# Compressive Shack-Hartmann Wavefront Sensor based on Deep Neural Networks

Peng Jia,1,2,3 Mingyang Ma,1 Dongmei Cai,1 Weihua Wang,1 Juanjuan Li1 and Can
Li4
1College of Physics and Optoelectronics, Taiyuan University of Technology,
Taiyuan, 030024, China
2Key Laboratory of Advanced Transducers and Intelligent Control Systems,
Ministry of Education and Shanxi Province,
Taiyuan University of Technology, Taiyuan, 030024, China
3Department of Physics, Durham University, South Road, Durham DH1 3LE, UK
4College of Information and Computer Science, Taiyuan University of
Technology, Taiyuan, 030024, China E-mail<EMAIL_ADDRESS>(JP)E-mail:
<EMAIL_ADDRESS>(MM)E-mail: dm<EMAIL_ADDRESS>(DM)

(Accepted XXX. Received YYY; in original form ZZZ)

###### Abstract

The Shack-Hartmann wavefront sensor is widely used to measure aberrations
induced by atmospheric turbulence in adaptive optics systems. However if there
exists strong atmospheric turbulence or the brightness of guide stars is low,
the accuracy of wavefront measurements will be affected. In this paper, we
propose a compressive Shack-Hartmann wavefront sensing method. Instead of
reconstructing wavefronts with slope measurements of all sub-apertures, our
method reconstructs wavefronts with slope measurements of sub-apertures which
have spot images with high signal to noise ratio. Besides, we further propose
to use a deep neural network to accelerate wavefront reconstruction speed.
During the training stage of the deep neural network, we propose to add a
drop-out layer to simulate the compressive sensing process, which could
increase development speed of our method. After training, the compressive
Shack-Hartmann wavefront sensing method can reconstruct wavefronts in high
spatial resolution with slope measurements from only a small amount of sub-
apertures. We integrate the straightforward compressive Shack-Hartmann
wavefront sensing method with image deconvolution algorithm to develop a high-
order image restoration method. We use images restored by the high-order image
restoration method to test the performance of our the compressive Shack-
Hartmann wavefront sensing method. The results show that our method can
improve the accuracy of wavefront measurements and is suitable for real-time
applications.

###### keywords:

techniques: image processing – instrumentation: adaptive optics – techniques:
high angular resolution

††pubyear: 2015††pagerange: Compressive Shack-Hartmann Wavefront Sensor based
on Deep Neural Networks–Compressive Shack-Hartmann Wavefront Sensor based on
Deep Neural Networks

##  1 Introduction

The Shack-Hartmann wavefront sensor (SH-WFS) is a wavefront sensing device
that normally consists of a lenslet array and an opto-electronic sensor. In
real applications, the SH-WFS firstly divides the wavefront into several sub-
apertures and light in each sub-apertures will be focused by a lenslet.
Wavefront slope measurements in each sub-aperture will then be evaluated
according to displacements of spots formed by each lenslet. Finally, wavefront
difference maps could be obtained through integration of wavefront slope
measurements (Platt & Shack, 2001).

The principle of the SH-WFS indicates us that as the number of sub-apertures
increases, the SH-WFS can measure wavefront difference maps with higher
spatial resolution. Meanwhile, because the atmospheric turbulence is highly
variable in the temporal domain, the SH-WFS needs to work in very high frame-
rate. With more sub-apertures and higher frame-rates, there would be less
photons that could be received by each sub-aperture in each frame. Considering
the brightness of guide stars in adaptive optics systems is low, there is
obvious a trade-off between the highest speed and highest spatial resolution
that a SH-WFS can measure. In real applications, if the atmospheric turbulence
is strong or the reference target is dim, slope measurements calculated from
spot images with low signal to noise ratio would lead to unreliable wavefront
measurements (Thomas et al., 2006). To solve this problem, many new methods
are proposed (Gilles & Ellerbroek, 2008; Schreiber et al., 2009; Basden et
al., 2015; Townson et al., 2015; Anugu et al., 2018; Adam, 2019).

Because wavefront difference maps are normally continuous surfaces, they can
be reconstructed with undersampled signals represented by different basis
functions. This concept, known as compressive sensing (Donoho, 2006), has
already been applied in holography (Clemente et al., 2013) and single pixel
camera (Duarte et al., 2008). In these applications, original signals can be
reconstructed with only a small number of undersampled measurements.

For SH-WFS, Polans et al. (2014) propose to represent wavefront difference
maps with Zernike polynomials (Hosseini & Michailovich, 2009) and have shown
that the compressive sensing could increase the speed of wavefront
measurements. Because the atmospheric turbulence is random media that
satisfies a particular statistical property (Roggemann et al., 1996), new
sparse basis functions, such as the golden section sparse basis function
(Juanjuan et al., 2018), are further proposed to reduce reconstruction error.
Compressive SH-WFS methods mentioned above could reconstruct wavefront
difference maps with only tens of percentage of all sub-apertures. Because
slope measurements introduced by spot images with low signal to noise ratio
(SNR) is the main error of SH-WFS, the compressive wavefront sensing method
indicates us a possibility that we could choose spot images with high SNR to
better reconstruct wavefront difference maps. With only a small amount of
slope measurements, it would be possible to increase the speed and the spatial
resolution for a SH-WFS at the same time.

The basis functions and the corresponding reconstruction algorithms are two
important factors for a compressive SH-WFS. Normally it would cost scientists
a long time to design adequate basis functions and corresponding
reconstruction algorithms. In essence, design of basis functions and
reconstruction algorithms is the procedure of finding a complex function that
could map undersampled slope measurements to original wavefront difference
maps. In recent years, deep neural networks (DNN) have become a successful
problem solver in representation of complex functions (Liu et al., 2017). For
wavefront sensing problems, Nishizaki et al. (2019); Andersen et al. (2019)
develop DNNs to learn mapping functions between wavefront represented by
Zernike coefficients and images blurred by the wavefront. DuBose et al. (2020)
propose to train a DNN that can map slope measurements and intensity
measurements to original wavefront difference maps, which can increase the
detection ability of SH-WFS for strong atmospheric turbulence. In this paper,
we take advantage of the complex function representation ability of DNNs and
propose to design a compressive SH-WFS method.

The compressive Shack-Hartmann wavefront sensing method (CS-WFS) proposed in
this paper uses a DNN to learn mapping functions between sparsely sampled
slope measurements and original wavefront difference maps. In real
applications, spot images with high signal to noise ratio will be selected to
calculate slope measurements and then the trained DNN could reconstruct
original wavefront difference maps from these measurements. To accelerate
algorithm development, we introduce a drop-out layer to simulate the
compressive sensing process during the training stage. This paper is organized
as follows. In Section 2, we will introduce the CS-WFS. In Section 3, we will
integrate the CS-WFS with the deconvolution from wavefront sensing algorithm
(Primot et al., 1990) to develop high-order image deconvolution method. We use
deconvolved images to test the effectiveness of our method. In Section 4, we
will make our conclusions and anticipate our future works.

##  2 Compressive Shack-Hartmann Wavefront Sensing Method Based on deep neural
networks

The classical SH-WFS approximates continuous wavefronts with discrete slope
measurements from equally distributed sub-apertures and reconstruct wavefronts
from slope measurements. The classical wavefront reconstruction algorithm is
the Southwell algorithm. However when the wavefront error is large or the
number of photons obtained by each sub-aperture is small, SNR of spot images
would be too low to obtain reliable slope measurements (Adam, 2019). In this
case, reconstructing wavefront difference maps with slope measurements
calculated from spot images of high SNR would be a possible way to assure
accuracy of SH-WFS. Because atmospheric turbulence induced aberrations are
stochastic, sub-apertures that have spot-images with high SNR would distribute
randomly and unequally. Therefore we need a compressive SH-WFS method to
reconstruct wavefront difference maps.

The concept of the compressive SH-WFS was firstly proposed by Polans et al.
(2014) with the name of SPARZER. The SPARZER is capable to reconstruct
original wavefront difference maps represented by Zernike polynomials with
only $5\%$ of randomly sub-sampled SH-WFS measurements. However, since
wavefront difference maps are represented by Zernike polynomials in the
SPARZER, the spatial resolution of wavefront measurement is limited by the
maximal orders of Zernike polynomials. Besides, the SPARZER requires around 30
iterations to reconstruct the wavefront, which limits its application in
astronomy observations that require real-time and high-order wavefront
measurements.

Meanwhile, DNNs can be trained straightforwardly in an end-to-end way and
could be directly used for wavefront sensing without numerical iterations. In
recent years, DNNs have been used in slope estimation, wavefront
reconstruction and wavefront estimation from images (Mello et al., 2014; Li &
Li, 2018; Hu et al., 2019; Xu et al., 2020; Hu et al., 2020). In this paper,
we will build a straightforward compressive SH-WFS method (CS-WFS) based on
DNN compressive sensing concepts (Lu et al., 2018a; Wu et al., 2019). We will
discuss details of the CS-WFS in Section 2.1.

###  2.1 The Structure of the CS-WFS Method

The physical process of the CS-WFS in real applications is shown in figure 1.
Comparing with classical SH-WFS, the CS-WFS has an additional SNR based sub-
aperture selection step. This step is used to guarantee that slope
measurements are reliable for further wavefront reconstruction. Because there
are many different slope calculation methods (Nicolle et al., 2004a; Gilles &
Ellerbroek, 2008), the SNR should be defined accordingly. In this paper, we
select the classical centre-of-gravity algorithm and use SNR defined in
equation 1 as selection criterion.

$SNR=\frac{signal-noise}{\sigma_{noise}}$ (1)

The $SNR$ is calculated by each sub-apertures. $signal$ is the summation of
grey scale values of pixels which are larger than a predefined threshold (one
standard deviation larger than the mean grey scale value for this paper).
$noise$ represents the average grey scale of the background and
$\sigma_{noise}$ is the standard deviation of the background.

Figure 1: The physical process of the CS-WFS method. The first step is the
same as that of SH-WFS, which will sample the wavefront $W_{org}$ with
lenslets to form complete spot images. Then the CS-WFS will select spot images
with enough SNR as sparse spot images to obtain effective slope measurements.
At last these effective images will be sent to the deep neural network in CS-
WFS to reconstruct wavefront maps $W_{dir}$.

Steps of the CS-WFS are shown in algorithm 1. It includes four main steps:
1\. Calculation of SNR from spot images. In this step, we will calculate SNR
of spot images for each sub-apertures.
2\. Selection of sub-apertures. After calculation of SNR of spot images of all
sub-apertures, we could either select slope measurements that are larger than
a predefined threshold $SNR_{t}$ (for relatively weak atmospheric turbulence)
or slope measurements from spot images with $N_{s}$ largest SNR (for strong
atmospheric turbulence) for wavefront reconstruction.
3\. Calculation of slope measurements. Spot images from selected sub-apertures
will be used to calculate the centre of spot images. Then we can obtain slope
measurements from the centre of spot images.
4\. Reconstruction of wavefront difference maps. With DNNs in CS-WFS, we can
reconstruct wavefront difference maps from these slope measurements.

Algorithm 1 Main steps of the CS-WFS method

1:

2:$SNR_{t}$ Threshold of SNR of spot images

3:$N_{s}$ Minimal number of sub-apertures

4:

5:$F_{DNN}(SLOPE)$ DNN that can map sparse slope measurements $SLOPE$ to
wavefront difference maps

6:

7:Evaluating SNR of spot images in each sub-apertures as $L_{SNR}$

8:Sorting $L_{SNR}$ and calculating number of sub-apertures $N_{SNR}$ that
have spot images with SNR larger than $SNR_{t}$

9:

10:if $N_{SNR}\leq N_{s}$ then

11: Selecting sub-apertures with the biggest $N_{s}$ SNR to calculate slope
measurements ($SLOPE$) from spot images

12:

13:else

14: Selecting sub-apertures with the biggest $N_{SNR}$ SNR to calculate slope
measurements ($SLOPE$) from spot images

15:end if

16:Reconstructing wavefront path difference maps from $SLOPE$ with
$F_{DNN}(SLOPE)$

The flowchart of algorithms of the CS-WFS method is shown in figure 2. The
main part of the CS-WFS is the DNN that could reconstruct wavefront difference
maps from sparsely sampled slope measurements. To make results obtained by
DNNs comprehensible, we use two sequentially connected DNNs to reconstruct
wavefront difference maps. The first DNN is a deep convolutional neural
network (DCNN), which is mainly used to estimate complete slope measurements
from sparsely sampled slope measurements. The second DNN is a U-Net, which is
mainly used to reconstruct wavefront difference maps from complete slope
measurements.

Figure 2: The flow chart of the CS-WFS. The sparse spot images will be used to
calculate sparse slope measurements along x and y directions. Then these
sparse slope measurements will be used to calculate complete slope
measurements with the DCNN and then the complete slope measurements will be
used to reconstruct wavefronts by the U-Net.

The structure of the DCNN in the CS-WFS is shown in figure 3. The DCNN
includes 8 convolutional layers to estimate complete slope measurements from
sparsely sampled slope measurements. In front of the first layer is a drop-out
layer (Hinton et al., 2012). The drop-out layer will randomly set a percentage
of weights as zero to prevent over-fitting of a neural network. In this paper
we use it to simulate the compressive sensing process. During the training
stage, we set the drop-out rate to a predefined value. When the complete slope
measurements are input into the DCNN network, the predefined proportional
(drop-out rate) slope measurements value are set to zero through the drop-out
layer to obtain the sparse slope measurements. Then the DCNN will be able to
learn the function that could estimate complete slope measurements from
sparsely sampled slope measurements.

Figure 3: The structure of the DCNN that are used to estimate complete slope
measurements from sparse slope measurements. It includes 8 convolutional
layers and a drop-out layer in front of them is used to simulate the
compressive sensing process during training.

The structure of the U-Net is shown in figure 4. It is used to reconstruct
wavefront difference maps from complete slope measurements. The U-Net is
modified from the structure proposed in DuBose et al. (2020). In this paper,
the U-Net merges complete slope measurements along x and y directions to
reconstruct wavefront difference maps.

Figure 4: The structure of the U-Net for the wavefront reconstruction.
Complete slope measurements of x and y directions will be transferred into two
separate branches and then outputs of these two branches will be merged
together to estimate wavefront difference maps.

###  2.2 Training of the CS-WFS Method

We use a high-fidelity atmospheric turbulence phase screen model (Jia et al.,
2015a, b) and an adaptive optics system simulation platform (Basden et al.,
2018) to generate atmospheric turbulence phase screens and corresponding spot
images to train and test the CS-WFS. The Fried parameter ($r_{0}$) is set to
0.06, 0.10, 0.14, 0.18 meter and the diameter of the telescope is set to 1
metre. In this paper, the SH-WFS has $32\times 32$ sub-apertures and we assume
there are no photo-electronics noise in the SH-WFS. With this configuration,
we could obtain noise free spot images. Besides, we resize atmospheric
turbulence phase screens to $32\times 32$ pixels as wavefront difference maps
that should be obtained by our CS-WFS method.

Complete slope measurements are calculated by noise free spot images through
the centre-of-gravity algorithm. The U-Net is trained by complete slope
measurements and their corresponding resized phase screens. Meanwhile, we use
complete slope measurements as the training set of the DCNN. During the
training stage, we set the drop-out rate as 0.9 in the drop-out layer of the
DCNN, which means only $10\%$ slope measurements can transmit to following
layers (compressive rate 0.9). The drop-out rate refers to percentage of
signals that are directly set to zero and the compressive rate refers to
percentage of slope measurements that are directly set as zero. We find that
if the DCNN is trained with drop-out rate of 0.9 (compressive rate 0.9), the
CS-WFS can reconstruct all wavefronts that have smaller or larger compressive
rates. Therefore we set the compressive rate as 0.9 during training stage.
After training, compressive slope measurements will be sent into the DCNN and
the drop-out rate will be set to zero (all slope measurements will be
transmitted to next layers - compressive rate equals to 0.0). Then the DCNN
could reconstruct wavefronts from these compressive slope measurements.

The DCNN and the U-Net can be linked sequentially and trained together or
separately. We find it is more efficient to train the DCNN and the U-Net
together. During training stage, the initial learning rate of the DCNN is
0.001 and that of the U-Net is 0.0001. The batch size is 100 and we use 130
epochs to train them. The wavefront slope and wavefront phase for training are
processed by linear normalization. The loss function of the CS-WFS is defined
in equation 2, where $err_{C}$ is the loss function of the DCNN and $err_{G}$
is the loss function of the U-Net. For both of these two DNNs, we use the mean
square error (MSE) as the loss function and Adam algorithm to optimize these
two DNNs (Kingma & Ba, 2014).After training, the CS-WFS can restore wavefronts
from slope measurements with 21 ms in a computer with Nvidia Tesla K40c GPU
under double float numerical accuracy. The speed can be further increased with
DNNs implemented in FPGAs with numerical accuracy of int8 or int16.

$err_{Plus}=err_{C}/200+err_{G}.$ (2)

###  2.3 Performance of the CS-WFS Method in Measurement of Wavefronts

In this section, we generate 200 atmospheric turbulence phase screens for
different blur levels ($r_{0}$ of 6 cm, 12 cm and 18 cm) to test the
performance of the CS-WFS. The input of the network for testing is the
wavefront slope that has not been normalized. We use the root mean square
error ($RMSE$) of residual wavefronts defined below as the evaluation
function.

$RMSE=\sqrt{\frac{1}{N}\sum_{i=1}^{n}(W_{org}-W_{dir})^{2}}.$ (3)

where $W_{org}$ and $W_{dir}$ are original and reconstructed wavefronts, N is
the total number of pixels that are used to represent the wavefront. The RMSE
is defined with unit of radians. In this paper, we assume the reconstructed
wavefront has the same dimension as that of the slope measurements. Therefore
$N$ equals to $32\times 32$ (1024) pixels.

We firstly test the robustness of the CS-WFS under different compressive
rates. We train a CS-WFS with a compressive rate of 0.9 (compressive rate of
0.9: $10\%$ sub-apertures are used in applications) and use slope measurements
with compressive rate from 0.0 to 0.9 to test performance of the CS-WFS. The
results are shown in figure 5. We can find that the CS-WFS is a robust
wavefront measure method: although we use a predefined compressive rate to
train the CS-WFS, the trained CS-WFS can reconstruct wavefront with different
compressive rates. In this figure, we can also find that the classical method
has good performance when all sub-apertures are used for reconstruction. When
the number of effective sub-apertures reduces, the performance of the
classical wavefront reconstruction algorithm will drop down.

Figure 5: Comparison of the RMSE value of residual wavefront obtained by the
classical method and the compressive sensing method. net stands for the
compressive wavefront reconstruction method. trad stands for the classical
compressive wavefront reconstruction method. The number at the end of net or
trad stands for the Fried parameter in centimetre. Compressive rate stands for
percentage of slope measurements that are removed from all measurements.

Then we test the CS-WFS algorithms with noisy wavefront slope measurements. To
better control different noise levels in measurements from SH-WFS, we generate
noisy slope measurements through adding random Poisson noise in noise free
slope measurements. We will adjust magnitude of noise and compare noisy
wavefront slopes with original slopes. When the difference between noisy
slopes and original slopes is larger than $10\%$ of PV values of original
wavefront slopes, we will set that as "noisy wavefront slopes" or "error
slope" or "incorrect slope". The noise level is set as fraction of error slope
measurements of all slope measurements in one frame.

Due to the lack of original slope measurements in real applications, the
method to determine whether the slope measurements are error measurements is
hard to use. Therefore error slopes will be selected according to the SNR of
spot images. If the SNR of spot images is used as selection criterion, three
different situations would occur: all correct slope measurements are selected,
fewer correct slope measurements are selected and some incorrect slope
measurements are selected. We test the performance of the CS-WFS method under
these three different situations. As shown in figure 6, there are different
percentage of sub-apertures which have incorrect slope measurements. We set
the drop-out rate as 0.9 during the training stage. In real applications, we
choose percentage of sub-apertures that are removed from observation data from
$60\%$ to $120\%$ of the number of sub-apertures that have incorrect slope
measurements. We can find that the CS-WFS is robust and can reconstruct
wavefronts with more or less effective slope measurements. Besides, there is a
weak relation that as the number of sub-apertures with incorrect slope
measurements increases, it would be better to remove more sub-apertures.

(a) diameter = 1m , $r_{0}=6cm$

(b) diameter = 1m , $r_{0}=12cm$

(c) diameter = 1m , $r_{0}=18cm$

Figure 6: RMSE of Residual wavefront reconstructed from different number of
selected sub-apertures for different noise level (percentage of sub-apertures
with incorrect slope measurements). (a) stands for Fried parameter of 6 cm.
(b) stands for Fried parameter of 12 cm and (c) stands for Fried parameter of
18 cm. The part of RMSE less than Marechal Criterion is considered to have
reached the diffraction limit.

##  3 Application of the CS-WFS in Deconvolution from Wavefront Sensing

Deconvolution from wavefront sensing (DWFS) is a powerful low-cost high-
resolution imaging technique (Primot et al., 1990; Welsh & Roggemann, 1995;
Mugnier et al., 2001; Jefferies & Hart, 2011). During observations, the DWFS
firstly records wavefront measurements and short-exposure images
simultaneously. Then the DWFS deconvolves short-exposure images with the point
spread function (PSF) estimated from wavefront measurements. Because the DWFS
uses light from the observed target for wavefront sensing and imaging, its
observation ability for dim targets are limited. In this part, we take
advantage of the higher spatial resolution and better reconstruction accuracy
of the CS-WFS method and integrate the DWFS and the CS-WFS as a new image
restoration method.

The flowchart of the new image restoration method is shown in figure 7.
Blurred images used in this section are generated through convolution of short
exposure PSFs and high resolution images. The short exposure PSFs are also
generated by Monte Carlo simulation platform (Basden et al., 2018) with a high
fidelity atmospheric turbulence phase screen model (Jia et al., 2015a, b). We
set the Fried parameter ($r_{0}$) as 6 cm, 12 cm and 18 cm. Meanwhile, we set
the diameter of telescope as 1 m. Original wavefronts are stored as $W_{org}$.
We set the number of sub-apertures in SH-WFS as $32\times 32$. Through Monte
Carlo simulation, we could obtain spot images from wavefronts. With classical
wavefront reconstruction algorithm, we obtain wavefront optical difference
maps from these spot images as $W_{dir}$. Then we can obtain the PSF (TO-PSF)
from $W_{dir}$ through far field diffraction. Meanwhile, we can also obtain
the PSF (NO-PSF) from complete slope measurements with the CS-WFS method. PSFs
mentioned above are obtained by noise free complete slope measurements. TN-
PSFs are obtained by classical wavefront reconstruction method from complete
noisy slope measurements. NS-PSFs are obtained by CS-WFS method from
compressive noisy slope measurements.

Figure 7: The flowchart of the CS-WFS and the DWFS in this paper. Original
wavefronts are used to generate blurred images. Four PSFs are estimated from
original wavefronts or wavefront measurements. They are PSF estimated by the
classical wavefront reconstruction algorithm from noise free complete slope
measurements (TO-PSF), PSF estimated by the CS-WFS algorithm from noise free
complete slope measurements (NO-PSF), PSF from noisy complete slope
measurements (TN-PSF) and PSF from noisy compressive slope measurements (NS-
PSF).

According to noise levels, we select slope measurements with different
compressive rates and obtain NS-PSFs and NO-PSFs. All these PSFs are used as
prior condition for deconvolution algorithm. In this paper, we use the maximum
likelihood deconvolution algorithm from MATLAB to deconvolve blurred images
and use estimated PSFs as prior PSF. We calculate SSIM (Structural Similarity
defined in MATLAB) between deconvolved images and original images to evaluate
the quality of deconvolved images. The results are shown in table 1. We can
find that the CS-WFS is robust to different compressive rates (NNR is larger
than TNR).

Table 1: SSIM of restored images with different percentages of slope measurements that are incorrect. Fried parameter stands for Fried parameter in centimetre. Compressive rate stands for fraction of slope measurements that are removed for wavefront reconstruction. Blurred Image stands for SSIM of blurred images. Restored Image stands for SSIM of restored images with corresponding blurred PSFs as prior information. TOR and NOR stand for SSIM of images restored with PSF obtained from the classical method and the compressive wavefront reconstruction method with wavefront measurements of no noise. TNR and NNR stand for SSIM of images restored with the classical and the compressive wavefront reconstruction algorithms with noisy wavefront measurements. We find that in the case of no noise, the classical algorithm has better restoration performance than the network method. Meanwhile, when the noise level increases, the network method has better restoration performance than the classical algorithm. And as the noise level increases, the network method is more robust than the classical algorithm. Fried Parameter | Compressive Rate | Blurred Image | Restored Image | TOR | NOR | TNR | NNR
---|---|---|---|---|---|---|---
| 0.1 |  |  |  |  | 0.2910±0.0075 | 0.3037±0.0105
| 0.3 |  |  |  |  | 0.2868±0.0066 | 0.3056±0.0091
6 cm | 0.5 | 0.1952±0.0021 | 0.3094±0.0039 | 0.3076±0.0069 | 0.3019±0.0114 | 0.2841±0.0069 | 0.3055±0.0081
| 0.7 |  |  |  |  | 0.2809±0.0078 | 0.3053±0.0080
| 0.9 |  |  |  |  | 0.2756±0.0073 | 0.3073±0.0091
| 0.1 |  |  |  |  | 0.4352±0.0072 | 0.4531±0.0095
| 0.3 |  |  |  |  | 0.4258±0.0074 | 0.4531±0.0097
12 cm | 0.5 | 0.2493±0.0035 | 0.4360±0.0070 | 0.4610±0.0074 | 0.4515±0.0100 | 0.4191±0.0077 | 0.4494±0.0111
| 0.7 |  |  |  |  | 0.4138±0.0075 | 0.4479±0.0114
| 0.9 |  |  |  |  | 0.4050±0.0096 | 0.4509±0.0111
| 0.1 |  |  |  |  | 0.5090±0.0139 | 0.5248±0.0160
| 0.3 |  |  |  |  | 0.5032±0.0138 | 0.5211±0.0175
18 cm | 0.5 | 0.2750±0.0071 | 0.5065±0.0155 | 0.5368±0.0153 | 0.5242±0.0160 | 0.4971±0.0123 | 0.5162±0.0191
| 0.7 |  |  |  |  | 0.4863±0.0115 | 0.5130±0.0202
| 0.9 |  |  |  |  | 0.4731±0.0119 | 0.5153±0.0204

Blurred images and restored images are shown in figure 8. img stands for the
original image. Blurred image is one frame of images blurred by the
atmospheric turbulence phase screen. TOR stands for restored images with PSFs
estimated from original wavefront with the classical method. NOR stands for
restored images with PSFs estimated from original images with the neural
network. TNR stands for restored images from noisy wavefront with the
classical method and NNR stands for restored images from noisy wavefront with
the neural network. We can find that even with small amount of slope
measurements, our CS-WFS can still obtain wavefront that can effectively
restore images. Because the uncertainty brought by the slope measurements
selection step would reduce the performance of the DWFS, we also estimate the
SSIM of NGC 4192 for different slope selection situations as defined in
Section 2.3. The results are shown in table 2. The results are similar to
wavefront reconstruction results. A trained CS-WFS with DWFS can effectively
restore images with sparsely sampled wavefronts.

(a) The original image.

(b) Image blurred by atmospheric turbulence with Fried parameter of 6 cm.

(c) Images restored from original wavefront with the classsical method.

(d) Images restored from original wavefront with the neural network.

(e) Images restored from noisy wavefront with the classsical method.

(f) Images restored from noisy wavefront with the neural network.

Figure 8: (a) is the original image. (b) is the image blurred by random atmospheric turbulence with $r_{0}$ of 6 cm. (c) is the restored image with the PSF estimated from wavefronts obtained from classical method. (d) is the restored image with the PSF estimated from wavefronts obtained from the neural network. (e) is the restored image with the PSF estimated from noisy wavefronts from classical method. (f) is the restored image with the PSF estimated from noisy wavefronts from the neural network. Table 2: SSIM of restored images with different percentages of selected measurements for wavefront measurements with different noise level. Fried parameter stands for Fried parameter in centimetre. Selection rate stands for percentage of removed slope measurements (comparing with noisy slope measurements) that are used for wavefront reconstruction. Noise level with different number stands for wavefront measurements with different percentage of noisy slope measurements. Fried Parameter | Selection Rate | Noise level 0 | Noise level 0.1 | Noise level 0.3 | Noise level 0.5 | Noise level 0.7 | Noise level 0.9
---|---|---|---|---|---|---|---
| $60\%$ |  | 0.3030±0.0108 | 0.3048±0.0101 | 0.3054±0.0086 | 0.3058±0.0081 | 0.3054±0.0082
| $80\%$ |  | 0.3034±0.0107 | 0.3053±0.0096 | 0.3056±0.0084 | 0.3051±0.0078 | 0.3059±0.0082
6 cm | $100\%$ | 0.3019±0.0114 | 0.3037±0.0105 | 0.3056±0.0091 | 0.3055±0.0081 | 0.3053±0.0080 | 0.3073±0.0091
| $120\%$ |  | 0.3040±0.0104 | 0.3058±0.0085 | 0.3050±0.0085 | 0.3063±0.0085 |
| $60\%$ |  | 0.4525±0.0096 | 0.4535±0.0096 | 0.4530±0.0097 | 0.4509±0.0103 | 0.4493±0.0105
| $80\%$ |  | 0.4528±0.0096 | 0.4532±0.0098 | 0.4510±0.0108 | 0.4488±0.0109 | 0.4479±0.0109
12 cm | $100\%$ | 0.4515±0.0100 | 0.4531±0.0095 | 0.4531±0.0097 | 0.4494±0.0111 | 0.4479±0.0114 | 0.4509±0.0111
| $120\%$ |  | 0.4533±0.0096 | 0.4521±0.0099 | 0.4485±0.0111 | 0.4489±0.0120 |
| $60\%$ |  | 0.5249±0.0158 | 0.5237±0.0167 | 0.5221±0.0172 | 0.5177±0.0185 | 0.5140±0.0202
| $80\%$ |  | 0.5248±0.0160 | 0.5227±0.0172 | 0.5194±0.0182 | 0.5138±0.0191 | 0.5134±0.0203
18 cm | $100\%$ | 0.5242±0.0160 | 0.5248±0.0160 | 0.5211±0.0175 | 0.5162±0.0191 | 0.5130±0.0202 | 0.5153±0.0204
| $120\%$ |  | 0.5248±0.0161 | 0.5194±0.0184 | 0.5141±0.0197 | 0.5131±0.0197 |

##  4 Conclusions and future work

In this paper, we propose a compressive Shack-Hartmann wavefront sensing
method (CS-WFS). Our method uses a drop-out layer to simulate the compressive
sensing process during training. After training, the CS-WFS can reconstruct
wavefront with only several to tens of percentage of sub-apertures. We
integrate the CS-WFS with the DWFS to test the performance of our method. The
results show that our method is robust and can fast estimate wavefronts
effectively.
The CS-WFS provide an effective way to measure high order wavefronts, which
would further increase performance of high-order adaptive optics systems
(Fusco et al., 2006; Roberts et al., 2012; Shumko et al., 2014; Rao et al.,
2015) or PSF reconstruction algorithms (Martin et al., 2016; Beltramo-Martin
et al., 2019; Fétick et al., 2019; Fusco et al., 2020). Besides, the
compressive Shack-Hartmann wavefront sensing algorithm can reduce the
requirement of brightness of guide stars in an adaptive optics system. Wide
field adaptive optics system, such as ground layer adaptive optics system
could then use natural guide stars as wavefront references for observations
(Rigaut, 2002; Nicolle et al., 2004b; Travouillon et al., 2004; Chun et al.,
2016; Carbillet et al., 2017; Jia et al., 2018; Lu et al., 2018b). In our
future works, we will integrate the CS-WFS with the deep learning based PSF
model (Jia et al., 2020) to develop high-order PSF reconstruction method and
we will also further investigate applications of the compressive wavefront
sensing in wide field adaptive optics systems.

## Acknowledgements

Peng Jia would like to thank Professor Dingqiang Su from Nanjing University
for guidance of the principle of Shack-Hartmann wavefront sensor. Peng Jia
would like to thank Professor Xiangqun Cui from Nanjing Institute of
Astronomical Optics & Technology, China Academy of Sciences for guidance on
Shack-Hartmann wavefront sensor in ground layer adaptive optics systems for
large telescopes. Peng Jia would also like to thank Professor Yong Zhang from
Nanjing Institute of Astronomical Optics & Technology, China Academy of
Sciences for introducing of the Shack-Hartmann wavefront sensor in the Large
Sky Area Multi-Object Fibre Spectroscopic Telescope. All authors are grateful
for kindly suggestions provided by the reviewer. Data resources are supported
by China National Astronomical Data Center (NADC) and Chinese Virtual
Observatory (China-VO). This work is supported by Astronomical Big Data Joint
Research Center, co-founded by National Astronomical Observatories, Chinese
Academy of Sciences and Alibaba Cloud.

This work is supported by National Natural Science Foundation of China (NSFC)
(11503018, 61805173), the Joint Research Fund in Astronomy (U1631133) under
cooperative agreement between the NSFC and Chinese Academy of Sciences (CAS).
Authors acknowledge the French National Research Agency (ANR) to support this
work through the ANR APPLY (grant ANR-19-CE31-0011) coordinated by B. Neichel.
This work is also supported by Shanxi Province Science Foundation for Youths
(201901D211081), Research and Development Program of Shanxi (201903D121161),
Research Project Supported by Shanxi Scholarship Council of China
(HGKY2019039), the Scientific and Technological Innovation Programs of Higher
Education Institutions in Shanxi (2019L0225).

## Data Availability

The code used in this paper is written in Python programming language (Python
Software Foundation) with the package Pytorch and the complete version of our
code can be downloaded from PaperData Repository Powered by China-VO
(https://nadc.china-vo.org/article/20200722160959?id=101045) with a DOI number
of 10.12149/101045.

## References

  * Adam (2019) Adam P., 2019, Optics letters, 44, 2602
  * Andersen et al. (2019) Andersen T., Owner-Petersen M., Enmark A., 2019, Optics letters, 44, 4618
  * Anugu et al. (2018) Anugu N., Garcia P. J., Correia C. M., 2018, Monthly Notices of the Royal Astronomical Society, 476, 300
  * Basden et al. (2015) Basden A., Morris T., Gratadour D., Gendron E., 2015, Monthly Notices of the Royal Astronomical Society, 449, 3537
  * Basden et al. (2018) Basden A. G., Bharmal N. A., Jenkins D., Morris T. J., Osborn J., Peng J., Staykov L., 2018, SoftwareX, 7, 63
  * Beltramo-Martin et al. (2019) Beltramo-Martin O., Correia C., Ragland S., Jolissaint L., Neichel B., Fusco T., Wizinowich P., 2019, Monthly Notices of the Royal Astronomical Society, 487, 5450
  * Carbillet et al. (2017) Carbillet M., Aristidi E., Giordano C., Vernin J., 2017, Monthly Notices of the Royal Astronomical Society, 471, 3043
  * Chun et al. (2016) Chun M. R., et al., 2016, in Adaptive Optics Systems V. p. 990902
  * Clemente et al. (2013) Clemente P., Durán V., Tajahuerce E., Andrés P., Climent V., Lancis J., 2013, Optics letters, 38, 2524
  * Donoho (2006) Donoho D. L., 2006, IEEE Transactions on information theory, 52, 1289
  * DuBose et al. (2020) DuBose T. B., Gardner D. F., Watnik A. T., 2020, Optics Letters, 45, 1699
  * Duarte et al. (2008) Duarte M. F., Davenport M. A., Takhar D., Laska J. N., Sun T., Kelly K. F., Baraniuk R. G., 2008, IEEE signal processing magazine, 25, 83
  * Fétick et al. (2019) Fétick R., et al., 2019, Astronomy & Astrophysics, 628, A99
  * Fusco et al. (2006) Fusco T., et al., 2006, Optics Express, 14, 7515
  * Fusco et al. (2020) Fusco T., et al., 2020, Astronomy & Astrophysics, 635, A208
  * Gilles & Ellerbroek (2008) Gilles L., Ellerbroek B., 2008, Optics letters, 33, 1159
  * Hinton et al. (2012) Hinton G. E., Srivastava N., Krizhevsky A., Sutskever I., Salakhutdinov R., 2012, arXiv: Neural and Evolutionary Computing
  * Hosseini & Michailovich (2009) Hosseini M. S., Michailovich O. V., 2009, in 2009 17th European Signal Processing Conference. pp 115–119
  * Hu et al. (2019) Hu L., Hu S., Gong W., Si K., 2019, Optics Express, 27, 33504
  * Hu et al. (2020) Hu L., Hu S., Gong W., Si K., 2020, Optics Letters, 45, 3741
  * Jefferies & Hart (2011) Jefferies S. M., Hart M., 2011, Optics express, 19, 1975
  * Jia et al. (2015a) Jia P., Cai D., Wang D., Basden A., 2015a, Monthly Notices of the Royal Astronomical Society, 447, 3467
  * Jia et al. (2015b) Jia P., Cai D., Wang D., Basden A., 2015b, Monthly Notices of the Royal Astronomical Society, 450, 38
  * Jia et al. (2018) Jia P., Basden A., Osborn J., 2018, Monthly Notices of the Royal Astronomical Society, 479, 829
  * Jia et al. (2020) Jia P., Wu X., Yi H., Cai B., Cai D., 2020, The Astronomical Journal, 159, 183
  * Juanjuan et al. (2018) Juanjuan L., Dongmei C., Peng J., Can L., 2018, Opto-Electronic Engineering, 45, 170616
  * Kingma & Ba (2014) Kingma D. P., Ba J., 2014, arXiv preprint arXiv:1412.6980
  * Li & Li (2018) Li Z., Li X., 2018, Optics Express, 26, 31675
  * Liu et al. (2017) Liu W., Wang Z., Liu X., Zeng N., Liu Y., Alsaadi F. E., 2017, Neurocomputing, 234, 11
  * Lu et al. (2018a) Lu X., Dong W., Wang P., Shi G., Xie X., 2018a, arXiv preprint arXiv:1801.10342
  * Lu et al. (2018b) Lu J. R., et al., 2018b, in Adaptive Optics Systems VI. p. 107030N
  * Martin et al. (2016) Martin O., et al., 2016, in Adaptive Optics Systems V. p. 99091Q
  * Mello et al. (2014) Mello A., Kanaan A., Guzman D., Guesalaga A., 2014, Monthly Notices of the Royal Astronomical Society, 440, 2781
  * Mugnier et al. (2001) Mugnier L. M., Robert C., Conan J.-M., Michau V., Salem S., 2001, JOSA A, 18, 862
  * Nicolle et al. (2004a) Nicolle M., Fusco T., Rousset G., Michau V., 2004a, Optics letters, 29, 2743
  * Nicolle et al. (2004b) Nicolle M., Fusco T., Michau V., Rousset G., Blanc A., Beuzit J.-L., 2004b, in Advancements in Adaptive Optics. pp 858–869
  * Nishizaki et al. (2019) Nishizaki Y., Valdivia M., Horisaki R., Kitaguchi K., Saito M., Tanida J., Vera E., 2019, Optics express, 27, 240
  * Platt & Shack (2001) Platt B. C., Shack R., 2001, Journal of Refractive Surgery, 17
  * Polans et al. (2014) Polans J., Mcnabb R. P., Izatt J. A., Farsiu S., 2014, Optics Letters, 39, 1189
  * Primot et al. (1990) Primot J., Rousset G., Fontanella J., 1990, JOSA A, 7, 1598
  * Rao et al. (2015) Rao C., et al., 2015, Chinese Optics Letters, 13, 120101
  * Rigaut (2002) Rigaut F., 2002, in European Southern Observatory Conference and Workshop Proceedings. p. 11
  * Roberts et al. (2012) Roberts J. E., et al., 2012, in Adaptive Optics Systems III. p. 84470Y
  * Roggemann et al. (1996) Roggemann M. C., Welsh B. M., Hunt B. R., 1996, Imaging through turbulence. CRC press
  * Schreiber et al. (2009) Schreiber L., Foppiani I., Robert C., Diolaiti E., Conan J.-M., Lombini M., 2009, Monthly Notices of the Royal Astronomical Society, 396, 1513
  * Shumko et al. (2014) Shumko S., et al., 2014, in Adaptive Optics Systems IV. p. 914835
  * Thomas et al. (2006) Thomas S., Fusco T., Tokovinin A., Nicolle M., Michau V., Rousset G., 2006, Monthly Notices of the Royal Astronomical Society, 371, 323
  * Townson et al. (2015) Townson M., Kellerer A., Saunter C., 2015, Monthly Notices of the Royal Astronomical Society, 452, 4022
  * Travouillon et al. (2004) Travouillon T., Lawrence J. S., Jolissaint L., 2004, in Advancements in Adaptive Optics. pp 934–942
  * Welsh & Roggemann (1995) Welsh B. M., Roggemann M. C., 1995, Applied optics, 34, 2111
  * Wu et al. (2019) Wu Y., Rosca M., Lillicrap T., 2019, arXiv preprint arXiv:1905.06723
  * Xu et al. (2020) Xu Z., Wang S., Zhao M., Zhao W., Dong L., He X., Yang P., Xu B., 2020, Applied Optics, 59, 4768


# Learning to Understand Child-directed and Adult-directed Speech

Lieke Gelderloos
Tilburg University
<EMAIL_ADDRESS>
Grzegorz Chrupała
Tilburg University
<EMAIL_ADDRESS>
Afra Alishahi
Tilburg University
<EMAIL_ADDRESS>
###### Abstract

Speech directed to children differs from adult-directed speech in linguistic
aspects such as repetition, word choice, and sentence length, as well as in
aspects of the speech signal itself, such as prosodic and phonemic variation.
Human language acquisition research indicates that child-directed speech helps
language learners. This study explores the effect of child-directed speech
when learning to extract semantic information from speech directly. We compare
the task performance of models trained on adult-directed speech (ADS) and
child-directed speech (CDS). We find indications that CDS helps in the initial
stages of learning, but eventually, models trained on ADS reach comparable
task performance, and generalize better. The results suggest that this is at
least partially due to linguistic rather than acoustic properties of the two
registers, as we see the same pattern when looking at models trained on
acoustically comparable synthetic speech.

##  1 Introduction

Speech directed to children (CDS) differs from adult-directed speech (ADS) in
many aspects. Linguistic differences include the number of words per
utterance, with utterances in CDS being considerably shorter than utterances
in ADS, and repetition, which is more common in child-directed speech. There
are also paralinguistic, acoustic factors that characterize child-directed
speech: people speaking to children typically use a higher pitch and
exaggerated intonation.

It has been argued that the properties of CDS help perception or
comprehension. Kuhl et al. (1997) propose that CDS is optimized for
learnability. Optimal learnability may, but does not necessarily align with
optimization for perception or comprehension. Although speech with lower
variability may be easiest to learn to understand, higher variability may
provide more learning opportunities, leading to more complete language
knowledge.

In this paper, we explore how learning to extract meaning from speech differs
when learning from CDS and ADS. We discuss task performance on the training
register as well as generalization across registers. To tease apart the effect
of acoustic and linguistic differences, we also report on models trained on
synthesized speech, in which linguistic differences between the registers are
retained, but the acoustic properties are similar.

##  2 Related work

###  2.1 Child directed speech and learnability

The characteristics of child-directed speech are a major topic of study in
language acquisition research. For a comprehensive overview, see Soderstrom
(2007) and Clark (2009, Ch. 2, p. 32-41). With regards to acoustics, CDS is
reported to have exaggerated intonation and a slower speech rate (Fernald et
al., 1989). Kuhl et al. (1997) show that CDS contains more ‘extreme’
realizations of vowels. McMurray et al. (2013) show that these increased means
are accompanied by increased variance, and argue that any learning advantage
of CDS due to extreme vowel realizations is counteracted by increased
variance. However, it has also been argued that increased variance may be
beneficial to learning in the long run, as it gives the learner a more
complete set of examples for a category, which helps generalization. Guevara-
Rukoz et al. (2018) show that word forms in child-directed speech are
acoustically more diverse. At the utterance level, child-directed language
consists of shorter sentences and simpler syntax (Newport et al., 1977;
Fernald et al., 1989), and words more often appear in isolation (Ratner and
Rooney, 2001).

Studies on home recordings show that the availability of CDS input accounts
for differences in vocabulary growth between learners, whereas overheard
speech is unrelated (Hoff, 2003; Weisleder and Fernald, 2013). This does not
necessarily mean that it is easier to learn from CDS. Psycholinguistic
research has shown that infants across the world show a CDS preference, paying
more attention to it than to ADS (ManyBabies Consortium, 2020). Learning
advantages of CDS in children may therefore simply be because they grant it
more attention, rather than to properties of CDS that are advantageous for
learning. Computational models, however, have no choice in where they allocate
attention. Any learning advantages we find of either ADS or CDS in
computational studies must be due to properties that make speech in that
register more learnable to the model.

There has been some computational work comparing learning from ADS and CDS at
the level of word learning and phonetic learning. Studies on segmentability
use algorithms that learn to identify word units, with some studies reporting
higher segmentability for CDS (Batchelder, 2002; Daland and Pierrehumbert,
2011), while Cristia et al. (2019) report mixed results. Kirchhoff and
Schimmel (2005) train HMM-based speech recognition systems on CDS and ADS, and
test on matched and crossed test sets. They find that both ADS and CDS trained
systems perform best on the matching test set, but CDS trained systems perform
better on ADS than systems trained on ADS peform on CDS. They show that this
is likely caused by phonetic classes have larger overlaps in CDS.

To the authors’ knowledge, the current work is the first to computationally
explore learnability differences between ADS and CDS considering the process
of speech comprehension as a whole: from audio to semantic information.

###  2.2 Speech recognition with non-linguistic supervision

In recent years, several studies have worked on machine learning tasks in
which models directly extract semantic information from speech, without
feedback on the word, character, or phoneme level. Most prominently, work on
‘weakly supervised’ speech recognition includes work in which accompanying
visual information is used as a proxy for semantic information. By grounding
speech in visual information accompanying it, models can learn to extract
visually relevant semantic information from speech, without needing symbolic
annotation (Harwath et al., 2016; Harwath and Glass, 2017; Chrupała et al.,
2017; Merkx et al., 2019).

The topic is of interest for automatic speech recognition, as it provides
potential ways of training speech recognition without the need for vast
amounts of annotation. The utilization of non-linguistic information as
supervision is particularly useful for low-resource languages. For the purpose
of this study, however, we are interested in this set of problems because of
the parallel to human language acquisition. A language learning child does not
receive explicit feedback on the words or phonemes it perceives. Rather, they
learn to infer these structural properties of language, with at their disposal
only the speech signal itself and its weak and messy links to the outer world.

##  3 Task

The task is to match speech to a semantic representation of the language it
contains, intuitively ‘grounding’ it to the semantic context. The design of
this task is inspired by work in visual grounding. However, the availability
of CDS data accompanied by visual data is very limited. Instead of visual
representation, we use semantic sentence embeddings of the transcriptions.
Rather than training our model to imagine the visual context accompanying an
utterance, as in visual grounding, we train it to imagine the semantic
content. Note that since the semantic embeddings are based on the
transcriptions of the sentences themselves, they have a much closer relation
to the sentences than visual context representations would have.

The semantic sentence representations were obtained using SBERT, a BERT-based
architecture that yields sentence embeddings, which was finetuned on the STS
benchmark of SemEval (Reimers and Gurevych, 2019). This particular encoding
was chosen because it harnesses the semantic strength of BERT (Devlin et al.,
2019) in an encoding of the sentence as a whole. Speech is converted Mel-
frequency cepstrum coefficients.

##  4 Data

###  4.1 Natural speech: NewmanRatner corpus

Since we are interested in the effect of learning from child- versus adult
directed speech, we select data that differs in register, but is otherwise as
comparable as possible. The NewmanRatner corpus contains annotated recordings
of caregivers in conversation with their children and with experimenters
Newman et al. (2016). This dataset is suitable to our set-up, as it contains a
reasonable amount of transcribed CDS and ADS by the same speakers, which is
rare; and it is in English, for which pretrained state-of-the-art language
models such as (S)BERT (Devlin et al., 2019; Reimers and Gurevych, 2019) are
readily available.

Child-directed speech in the NewmanRatner corpus takes place in free play
between caregiver and child, whereas adult-directed speech is uttered in the
context of an interview. Stretches of speech have been transcribed containing
one or more utterances. We selected only utterances by caregivers and excluded
segments with multiple speakers. As the CDS portion of the corpus is larger
than the ADS portion, we randomly selected 21,465 CDS segments, matching the
number of ADS segments by caregivers. Validation and test sets of 1,000
segments were held out, while the remaining 19,465 segments were used for
training.

Table 1 lists some characteristic statistics of the CDS and ADS samples that
were used. The ADS sample contains a larger vocabulary than the CDS sample. On
average, ADS segments contain more than twice as many words, although they are
only 88 milliseconds longer on average. Therefore, the number of words per
second is twice as high in ADS as it is in CDS.

Dataset | CDS | ADS
---|---|---
Vocabulary size | 3,170 | 5,665
Total nr. of words | 97,118 | 203,084
Type/token ratio | .033 | .028
Words per utterance | 4.52 | 9.46
Utterance length in seconds | 3.37 | 3.46
Words per second | 1.34 | 2.74
Table 1: Descriptive statistics of the data

###  4.2 Synthetic speech

To tease apart effects of the acoustic properties of speech and properties of
the language itself, we repeat the experiment using synthesized version of the
ADS and CDS corpora. For this variant, we feed the transcriptions to the
Google text2speech API, using the 6 available US English WaveNet voices (van
den Oord et al., 2016). Note that the synthetic speech is much cleaner than
the natural speech, which was recorded using a microphone attached to clothing
of the caregiver, and contains a lot of silence, noise, and fluctuations in
volume of the speech.

Since synthetic speech for ADS and CDS is generated using the same pipeline,
the acoustic properties of these samples are comparable, but linguistic
differences between them are retained. Differences remain in the vocabulary
size, number of words per utterance and type token ratio, but the number of
words per second is now comparable. This means the length of utterances is
much larger for synthetic ADS sentences, since the average ADS sentence
contains approximately twice as many words as the average CDS sentence.

##  5 Model

The model and training set-up is based on Merkx et al. (2019). This model is
suited to our task, as it allows to learn to extract semantic information from
speech by grounding it in another modality, without requiring the speech to be
segmented. The speech encoder comprises a convolutional filter over the speech
input, feeding into a stack of 4 bidirectional-GRU layers followed by an
attention operator. The difference in our set-up is the use of SBERT sentence
embeddings instead of visual feature vectors. Using a margin loss, the model
is trained to make the cosine distance between true pairs of speech segments
and SBERT embeddings smaller than that between random counterparts. We train
for 50 epochs and following Merkx et al. (2019) we use a cyclic learning rate
schedule.111Code is available through Github:
https://github.com/lgelderloos/cds_ads

##  6 Results

###  6.1 Performance

Model trained on CDS
---
Testset | Med.r. | R@1 | R@5 | R@10
CDS | 4.67 | .28 | .52 | .61
ADS | 52.50 | .08 | .19 | .26
Combined | 30.67 | .15 | .30 | .37
Model trained on ADS
Testset | Med.r. | R@1 | R@5 | R@10
CDS | 37.83 | .10 | .24 | .33
ADS | 5.00 | .29 | .51 | .61
Combined | 20.83 | .17 | .32 | .40

Table 2: Test performance of models trained on natural speech

Model trained on synthetic CDS
---
Testset | Med.r. | R@1 | R@5 | R@10
CDS | 1.00 | .82 | .96 | .99
ADS | 1.00 | .59 | .79 | .86
Combined | 1.00 | .68 | .85 | .90
Model trained on synthetic ADS
Testset | Med.r. | R@1 | R@5 | R@10
CDS | 1.00 | .70 | .89 | .95
ADS | 1.00 | .84 | .94 | .97
Combined | 1.00 | .74 | .89 | .93

Table 3: Test performance of models trained on synthetic speech

Trained models are evaluated by ranking all SBERT embeddings in the test set
by cosine distance to speech encodings. Reported metrics are recall@1,
recall@5, and recall@10, which are the proportion of cases in which the
correct SBERT embedding is among the top 1, 5, or 10 most similar ones; and
the median rank of the correct SBERT embedding. Test results are reported for
the training epoch for which recall@1 is highest on validation data. We have
trained 3 differently randomly initialized runs for all four datasets, and
report the average scores on the test split of the dataset the model was
trained on, as well as its CDS or ADS counterpart, and a combined test set,
which is simply the union of the two.

As can be observed in table 3, on the combined test set, models trained on
adult directed speech slightly outperform models trained on child-directed
speech. However, models in the two registers perform very similarly when we
test them on the test set in the same register, with ADS having higher
recall@1, but CDS scoring better on the other metrics. When we test ADS models
on CDS, performance is lower than that of models that have been trained on
CDS. However, the drop on ADS between models trained on ADS and models trained
on CDS is even larger. The better performance on the combined test set, then,
seems to come from ADS models generalizing better to CDS than the other way
around.

General performance of all models trained and tested on synthetic speech,
which is much cleaner than the natural speech and more similar across
registers, is much higher than performance on natural speech (see table 3).
However, the same pattern can be observed: on the combined test set, ADS
models perform better than CDS models. When tested on the register they were
trained on, the models perform similarly, but models trained on ADS perform
better when tested on CDS than the other way around.

To summarize, models trained on ADS and CDS reach comparable scores when
evaluated on the same register they are trained on. However, training on ADS
leads to knowledge that generalizes better than training on CDS does. This
pattern holds even when training and evaluating on synthetic speech, when the
two registers are acoustically similar.

###  6.2 Learning trajectories

Figure 1: Validation performance in early training on natural speech

Figure 2: Validation performance in early training on synthetic speech

Learnability is not just about eventual attainment: it is also about the
process of learning itself. Although ADS and CDS models eventually perform
similarly, this is not necessarily the case during the training process.
Figures 2 and 2 show the trajectory of recall performance on the validation
set after the first 10 epochs of training. During these early stages of
learning, the models trained on ADS (dotted lines) are outperformed by those
trained on CDS (solid lines). This pattern is more pronounced in the models
trained on synthetic speech, but also present for models trained on natural
speech. After five epochs of training, average recall@1 is 0.12 for CDS models
and 0.09 for ADS models. For models trained on synthetic speech, average
recall@1 on validation data is 0.51 for ADS models and 0.59 for CDS models. In
later stages of training, models trained on ADS outperform CDS models on
validation data. At epoch 40, close to the optimally performing epoch for most
models, average recall@1 is 0.31 for ADS models and 0.28 for CDS models, and
0.86 and 0.81 for the synthetic counterparts, respectively.

Although models trained on adult-directed speech eventually catch up with
models trained on child-directed speech, CDS models learn more quickly at the
start.

##  7 Discussion

We find indications that learning to extract meaning from speech is initially
faster when learning from child-directed speech, but learning from adult-
directed speech eventually leads to similar task performance on the training
register, and better generalization to the other register. The effect is
present both in models trained on natural speech and in models trained on
synthetic speech, suggesting that it is at least partly due to differences in
the language itself, rather than acoustic properties of the speech register.

Our finding that models trained on ADS generalize better to CDS than the other
way around contrasts with the findings of Kirchhoff and Schimmel (2005). Our
results are in contrast to the idea that CDS is optimized for leading to the
most valuable knowledge, as it is the models trained on ADS that lead to
better generalization. Our finding that learning is initially faster for CDS
is more in line with the idea of learnability as ‘easy to learn’.

The better generalization of models trained on ADS may be due to ADS having
higher lexical and semantic variability, reflected in the larger vocabulary
and higher number of words per utterance. Since there is simply more to learn,
learning to perform the task is more difficult on ADS, but it leads to more
valuable knowledge. It is also possible that SBERT is better suited to encode
the semantic content of ADS, as ADS uterrances are likely to be more similar
to the sentences SBERT was trained on than CDS utterances are.

We must be prudent in drawing conclusions from the apparent effects we see in
this study, as the results on different datasets cannot be interpreted as
being on the same scale. Although all metrics are based on a rank of the same
number of competitors, the distribution of similarities and differences
between the semantic representations of these competitors may differ across
datasets. The combined test set scores are more directly comparable, but
ideally, we would like to compare the generalization of both models on an
independent test set.

In future work, we intend to curate a test set with data from separate
sources, which can serve as a benchmark for the models we study. We intend to
explore how a curriculum of CDS followed by ADS affects learning trajectories
and outcomes. We also intend to use tools for interpreting the knowledge
encoded in neural networks (such as diagnostic classifiers and
representational similarity analysis) to investigate the emergent
representation of linguistic units such as phonemes and words.

## References

  * Batchelder (2002) Eleanor O. Batchelder. 2002.  Bootstrapping the lexicon: A computational model of infant speech segmentation.  _Cognition_ , 83(2):167–206.
  * Chrupała et al. (2017) Grzegorz Chrupała, Lieke Gelderloos, and Afra Alishahi. 2017.  Representations of language in a model of visually grounded speech signal.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 613–622.
  * Clark (2009) Eve V. Clark. 2009.  _First language acquisition_ , 2nd edition.  Cambridge University Press, Cambridge.
  * Cristia et al. (2019) Alejandrina Cristia, Emmanuel Dupoux, Nan Bernstein Ratner, and Melanie Soderstrom. 2019.  Segmentability differences between child-directed and adult-directed speech: A systematic test with an ecologically valid corpus.  _Open Mind_ , 3:13–22.
  * Daland and Pierrehumbert (2011) Robert Daland and Janet B. Pierrehumbert. 2011.  Learning diphone-based segmentation.  _Cognitive science_ , 35(1):119–155.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186.
  * Fernald et al. (1989) Anne Fernald, Traute Taeschner, Judy Dunn, Mechthild Papousek, Bénédicte de Boysson-Bardies, and Ikuko Fukui. 1989.  A cross-language study of prosodic modifications in mothers’ and fathers’ speech to preverbal infants.  _Journal of child language_ , 16(3):477–501.
  * Guevara-Rukoz et al. (2018) Adriana Guevara-Rukoz, Alejandrina Cristia, Bogdan Ludusan, Roland Thiollière, Andrew Martin, Reiko Mazuka, and Emmanuel Dupoux. 2018.  Are words easier to learn from infant- than adult-directed speech? a quantitative corpus-based investigation.  _Cognitive Science_ , 42(5):1586–1617.
  * Harwath and Glass (2017) David Harwath and James Glass. 2017.  Learning word-like units from joint audio-visual analysis.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 506–517.
  * Harwath et al. (2016) David Harwath, Antonio Torralba, and James Glass. 2016.  Unsupervised learning of spoken language with visual context.  In _Advances in Neural Information Processing Systems 29_ , pages 1858–1866.
  * Hoff (2003) Erika Hoff. 2003.  The specificity of environmental influence: Socioeconomic status affects early vocabulary development via maternal speech.  _Child Development_ , 74(5):1368–1378.
  * Kirchhoff and Schimmel (2005) Katrin Kirchhoff and Steven Schimmel. 2005.  Statistical properties of infant-directed versus adult-directed speech: Insights from speech recognition.  _The Journal of the Acoustical Society of America_ , 117(4):2238–2246.
  * Kuhl et al. (1997) Patricia K. Kuhl, Jean E. Andruski, Inna A. Chistovich, Ludmilla A. Chistovich, Elena V. Kozhevnikova, Viktoria L. Ryskina, Elvira I. Stolyarova, Ulla Sundberg, and Francisco Lacerda. 1997.  Cross-language analysis of phonetic units in language addressed to infants.  _Science_ , 277(5326):684–686.
  * ManyBabies Consortium (2020) The ManyBabies Consortium. 2020.  Quantifying sources of variability in infancy research using the infant-directed-speech preference.  _Advances in Methods and Practices in Psychological Science_ , 3(1):24–52.
  * McMurray et al. (2013) Bob McMurray, Kristine A. Kovack-Lesh, Dresden Goodwin, and William McEchron. 2013\.  Infant directed speech and the development of speech perception: Enhancing development or an unintended consequence? _Cognition_ , 129(2):362 – 378.
  * Merkx et al. (2019) Danny Merkx, Stefan L. Frank, and Mirjam Ernestus. 2019.  Language Learning Using Speech to Image Retrieval.  In _Proceedings of Interspeech 2019_ , pages 1841–1845.
  * Newman et al. (2016) Rochelle S. Newman, Meredith L. Rowe, and Nan Bernstein Ratner. 2016.  Input and uptake at 7 months predicts toddler vocabulary: the role of child-directed speech and infant processing skills in language development.  _Journal of Child Language_ , 43(5):1158–1173.
  * Newport et al. (1977) Elissa Newport, Henry Gleitman, and Lila Gleitman. 1977.  Mother, I’d rather do it myself: Some effects and non‐effects of maternal speech style.  In Catherine E. Snow and Charles A. Ferguson, editors, _Talking to children: Language input and acquisition_ , pages 109–149. Cambridge University Press, Cambridge.
  * van den Oord et al. (2016) Aäron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alexander Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. 2016\.  WaveNet: A generative model for raw audio.  arXiv:1609.03499.
  * Ratner and Rooney (2001) Nan Bernstein Ratner and Becky Rooney. 2001.  How accessible is the lexicon in Motherese?  In Jürgen Weissenborn and Barbara Höhle, editors, _Approaches to Bootstrapping: Phonological, lexical, syntactic and neurophysiological aspects of early language acquisition_ , volume 23 of _Language Acquisition and Language Disorders_ , pages 71–78. John Benjamins Publishing Company.
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019.  Sentence-BERT: Sentence embeddings using Siamese BERT-networks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3982–3992.
  * Soderstrom (2007) Melanie Soderstrom. 2007.  Beyond babytalk: Re-evaluating the nature and content of speech input to preverbal infants.  _Developmental Review_ , 27(4):501–532.
  * Weisleder and Fernald (2013) Adriana Weisleder and Anne Fernald. 2013.  Talking to children matters: Early language experience strengthens processing and builds vocabulary.  _Psychological Science_ , 24(11):2143–2152.


# Dither Signal Design for PAPR Reduction in OFDM-IM over a Rayleigh Fading
Channel

Kee-Hoon Kim The author is with the School of Electronic and Electrical
Engineering and IITC, Hankyong National University, Anseong 17579, South Korea
(e-mail<EMAIL_ADDRESS>
###### Abstract

Orthogonal frequency division multiplexing with index modulation (OFDM-IM) is
a novel scheme where the information bits are conveyed through the subcarrier
activation pattern (SAP) and the symbols on the active subcarriers.
Unfortunately, OFDM-IM inherits the high peak-to-average power ratio (PAPR)
problem from the classical OFDM. The OFDM-IM signal with high PAPR induces in-
band distortion and out-of-band radiation when it passes through high power
amplifier (HPA). There are attempts to reduce PAPR by adding dither signals in
the idle subcarriers, where the dither signals can have various amplitude
constraints according to the characteristic of the corresponding OFDM-IM
subblock. But, there is no result for the specific amplitude constraint for
the dither signals over a Rayleigh fading channel. In this letter, based on
pairwise error probability (PEP) analysis, a specific constraint for the
dither signals is derived over a Rayleigh fading channel.

###### Index Terms:

Index modulation (IM), orthogonal frequency division multiplexing (OFDM),
peak-to-average power ratio (PAPR)

##  I Introduction

Orthogonal frequency division multiplexing with index modulation (OFDM-IM) [1]
is an emerging technique which is the application of the spatial modulation
(SM) [2] principle to the subcarriers in an OFDM system. In OFDM-IM, the
subcarriers are partitioned into many subblocks and, in each subblock, the
subcarriers have two states, active or idle. Then, OFDM-IM conveys information
through not only modulated symbols but also the indices of the active
subcarriers. For the same spectral efficiency, OFDM-IM is shown to have a
superior bit error rate (BER) performance when compared to the classical OFDM
[1]. Also, OFDM-IM systems have a better energy efficiency compared to the
classical OFDM [3].

The authors in [4] pointed out that OFDM-IM inherits the high peak-to-average
power ratio (PAPR) problem from the classical OFDM. It is known that the high
PAPR induces in-band distortion and out-of-band radiation in consideration of
nonlinear high power amplifier (HPA). Numerous PAPR reduction schemes have
been researched for decades such as selected mapping (SLM), partial transmit
sequence (PTS), and tone injection (TI) [5]. To solve the high PAPR problem in
OFDM-IM, we may borrow the PAPR reduction schemes designed for the classical
OFDM. However, those methods are not efficient because they do not consider
the unique characteristic of OFDM-IM structure. Specifically, they do not
exploit the idle subcarriers in OFDM-IM [6].

Therefore, the authors in [6] proposed a PAPR reduction method using the idle
subcarriers in OFDM-IM. In specific, the scheme in [6] introduces dither
signals in the idle subcarriers for reducing PAPR of OFDM-IM signals. This is
the first PAPR reduction method exploiting the special structure of OFDM-IM.
This methodology is quite reasonable because the dither signals in the idle
subcarriers do not considerably affect the error performance in high signal-
to-noise ratio (SNR) region. This is because the symbol error event with
diversity order of one dominates the system performance in the high SNR region
and the dither signals cannot affect this error event. To suppress the harmful
effect of the dither signals, they also proposed the equivalent amplitude
constraint for the dither signals.

Besides, the previous work in [7] considered the fact that the amplitude
characteristics of the subblocks are distinct when quadrature amplitude
modulation (QAM) is employed in the active subcarriers. Therefore, a variable
amplitude constraint for dither signals is proposed. As a result, the
constraint in [7] gives the dither signals more freedom in average while
maintaining good demodulation performance. However, in [7], the amplitude
constraint is derived under the assumption of an additive white Gaussian noise
(AWGN) channel without fading. Also, the low complexity power based detection
algorithm is considered in the derivation, where the power based detection is
less preferred in the recent literature because of its degraded performance.

In this letter, based on rigorous pairwise error probability (PEP) analysis, a
new amplitude constraint for the dither signals is proposed over a Rayleigh
fading channel, which is theoretically meaningful. By using the proposed
amplitude constraint, dither signals can be well designed over a fading
channel.

##  II System Model

###  II-A OFDM-IM

Consider an OFDM-IM system with $N$ subcarriers and total $m$ information
bits. These $m$ information bits are divided into $g$ subblocks, where each
subblock conveys $p$ bits, i.e., $m=pg$. Also, the $p$ bits in each subblock
are mapped to one subblock of length $n$ in the frequency domain, where
$n=N/g$. The specific mapping procedure is that only $k$ out of $n$
subcarriers in each subblock are activated, where the subcarrier activation
pattern (SAP) is determined by the first $p_{1}$ bits of the $p$ bits. Then,
the $M$-ary modulated symbols on the $k$ active subcarriers are determined by
the remaining $p_{2}=k\log_{2}M$ bits of the $p$ bits, i.e., $p=p_{1}+p_{2}$.
Of course, the idle subcarriers have zero values in the frequency domain [1].

We denote the set of the indices of the $k$ active subcarriers in the
$\beta$-th OFDM-IM subblock, $\beta=0,1,\cdots,g-1$, as

$I^{\beta}=\\{i_{0}^{\beta},i_{1}^{\beta},\cdots,i_{k-1}^{\beta}\\}$ (1)

with $i_{\gamma}^{\beta}\in\\{0,1,\cdots,n-1\\}$ for $\gamma=0,1,\cdots,k-1$.
Also, we denote the set of $k$ modulated symbols as

$S^{\beta}=\\{S_{0}^{\beta},S_{1}^{\beta},\cdots,S_{k-1}^{\beta}\\}$ (2)

where $S_{\gamma}^{\beta}\in\mathcal{S}$ and $\mathcal{S}$ is the $M$-ary
signal constellation.

By considering $I^{\beta}$ and $S^{\beta}$, the $\beta$-th OFDM-IM subblock in
frequency domain can be generated as

$X^{\beta}=[X_{0}^{\beta}~{}X_{1}^{\beta}~{}\cdots~{}X_{n-1}^{\beta}]^{T}$ (3)

where $X_{i}^{\beta}$ is the $i$-th element of $X^{\beta}$ and
$i=0,1,\cdots,n-1$. After all of the $g$ subblocks
$\\{X^{\beta}\\}_{\beta=0}^{g-1}$ are generated, they are concatenated into
the symbol sequence $\mathbf{X}$ of length $N$ in frequency domain. To achieve
the frequency diversity gain as much as possible, concatenation in an
interleaved pattern is generally employed [8]. By the interleaved pattern, the
elements in an OFDM-IM subblock can experience independent fading channels.

Then, to obtain the OFDM-IM signal sequence $\mathbf{x}$ in time domain, the
symbol sequence $\mathbf{X}$ in frequency domain is processed by the inverse
discrete Fourier transform (IDFT) as

$\mathbf{x}=\mbox{IDFT}(\mathbf{X}).$ (4)

For transmission, cyclic prefix (CP) insertion and digital-to-analog (D/A)
conversion are sequentially performed and then the PAPR of the resultant
continuous-time OFDM-IM signal ${x}(t)$ is defined by

$\mbox{PAPR}({x}(t))=\frac{\max_{t}|{x}(t)|^{2}}{\mathbb{E}[|{x}(t)|^{2}]}.$
(5)

Practically, to capture PAPR of the continuous-time OFDM-IM signal, four-times
oversampling of $\mathbf{x}$ is used.

The receiver firstly detects the SAP, called index demodulation in this
letter, and demodulates the $M$-ary modulated symbols in the active
subcarriers. There have been several detection algorithms for OFDM-IM systems
[1, 9]. For low computational complexity, a power based detection algorithm
can be employed, but it degrades the error performance of OFDM-IM. In [10,
11], it is known that the optimal maximum likelihood (ML) detection can be
implemented with low complexity and thus it is preferred to employ the ML
detection.

##  III PAPR Reduction Using Dither Signals

###  III-A PAPR Reduction Using Dither Signals of an Equivalent Amplitude
Constraint [6]

In OFDM-IM, there are two types of error events, an index demodulation error
event and a symbol error event. The former is the error event when the SAP is
incorrectly detected and the latter is the error event when the modulated
symbols in the active subcarriers are incorrectly detected though the SAP is
correctly detected. As described in [1], the symbol error event has the
diversity order of one and the index demodulation error event has the
diversity order of two. By virtue of the frequency diversity gain, the index
demodulation error event occurs less frequently than the symbol error event in
high SNR region. That is, the symbol error event dominates the overall error
performance of the OFDM-IM system in high SNR region. Therefore, in [6] there
is an attempt to reduce PAPR by inserting the dither signals in the idle
subcarriers. Clearly, this dither signal does not affect the symbol error
event and only degrades the index demodulation error performance. Also, the
authors in [6] introduces an equivalent amplitude constraint for the dither
signals in the idle subcarriers.

Specifically, the dither signal added in the $\beta$-th OFDM-IM subblock is

$D^{\beta}=[D_{0}^{\beta}~{}D_{1}^{\beta}~{}\cdots~{}D_{n-1}^{\beta}]^{T}$ (6)

and the constraint in [6] with a hyperparameter $R$ is

$D_{i}^{\beta}=\left\\{\begin{array}[]{ll}|D_{i}^{\beta}|<R,&\hbox{$i\in(I^{\beta})^{c}$}\\\
0,&\hbox{$i\in I^{\beta}$}\end{array}\right.$ (7)

where $(I^{\beta})^{c}$ is the complement set of $I^{\beta}$, i.e., indices of
the idle subcarriers. Then, $X^{\beta}+D^{\beta}$ becomes a new $\beta$-th
OFDM-IM subblock with dither signals. Under the constraint in (7), the values
of $D^{\beta}$ are determined in order to reduce PAPR of $x(t)$. In [6],
convex programming is used, but iterative clipping and filtering with trimming
the dither signals to satisfy the constraint in (7) without phase change can
be alternatively used for low complexity.

###  III-B PAPR Reduction with Dither Signals of a Variable Amplitude
Constraint [7]

Meanwhile, in [7], a variable amplitude constraint for dither signals is
proposed. This is motivated from the fact that the OFDM-IM subblocks have
different robustness against channel noise if higher modulation than 16-QAM is
employed. Using this, the amplitude constraint of dither signals can be varied
for subblocks. Let us briefly review the work in [7].

First, for the $\beta$-th OFDM-IM subblock $X^{\beta}$, we define $A^{\beta}$
as

$A^{\beta}=\min(|S_{0}^{\beta}|,|S_{1}^{\beta}|,\cdots,|S_{k-1}^{\beta}|).$
(8)

For ease of understanding, we assume that 16-QAM is employed with the signal
constellation $\\{\pm 1\pm j1,\pm 1\pm j3,\pm 3\pm j1,\pm 3\pm j3\\}$ from now
on. (Clearly, the scheme in [7] can be easily described for higher modulations
such as 64-QAM or 256-QAM.) Then, $A^{\beta}$ can be one of
$\\{\sqrt{2},\sqrt{10},\sqrt{18}\\}$.

Second, assuming an AWGN channel with noise power $N_{0}$, the PEP for the
$\beta$-th subblock is

$P(X\rightarrow\hat{X})=Q\left(\frac{\|X-\hat{X}\|}{\sqrt{2N_{0}}}\right)$ (9)

where, without loss of generality, we omit $\beta$ for simplicity and
$\hat{X}=[\hat{X}_{0}~{}\hat{X}_{1}~{}\cdots~{}\hat{X}_{n-1}]^{T}$. Also,
$Q(\cdot)$ means the Q-function, the tail distribution function of the
standard normal distribution. It is clear that the PEP in (9) depends on the
Euclidean distance between $X$ and $\hat{X}$, $\|X-\hat{X}\|$.

Third, we focus on the index demodulation error event because the dither
signals do not affect the symbol error event as we mentioned. Then, the
fundamental index demodulation error in the $\beta$-th subblock is the case
when the $u$-th subcarrier is detected as active and the $v$-th subcarrier is
detected as idle when actually the opposite is true. The shortest Euclidean
distance inducing this fundamental index demodulation error is

$\min\sqrt{|\hat{X}_{u}|^{2}+|X_{v}|^{2}}=\sqrt{2+(A^{\beta})^{2}}$ (10)

where the index demodulation error performance of the $\beta$-th subblock is
dominated by this metric. That is, as $A_{\beta}$ becomes larger, the
robustness of the $\beta$-th OFDM-IM subblock against the index demodulation
error increases. Therefore, dither signals with large amplitude may be
inserted if $A_{\beta}$ is large.

Using this fact, in [7], the dither signals can have different amplitudes
according to $A^{\beta}$ as

$D_{i}^{\beta}=\left\\{\begin{array}[]{ll}|D_{i}^{\beta}|<R_{0},&\hbox{$i\in(I^{\beta})^{c}$
and $A^{\beta}=\sqrt{2}$}\\\
|D_{i}^{\beta}|<R_{1},&\hbox{$i\in(I^{\beta})^{c}$ and
$A^{\beta}=\sqrt{10}$}\\\ |D_{i}^{\beta}|<R_{2},&\hbox{$i\in(I^{\beta})^{c}$
and $A^{\beta}=\sqrt{18}$}\\\ 0,&\hbox{$i\in I_{\beta}$}.\end{array}\right.$
(11)

Also, the constraint proposed in [7] is

$\sqrt{2}-R_{0}=\sqrt{10}-R_{1}=\sqrt{18}-R_{2}$ (12)

where they assume the AWGN channels and using the power based detection
algorithm at the receiver.

One example of the possible candidates is $R_{0}=0.2$, $R_{1}\simeq 1.9$, and
$R_{2}\simeq 3$, which gives the dither signals more freedom than the
equivalent amplitude constraint with $R=0.2$ in (7). However, the constraint
in (12) is inappropriate for frequency selective fading channels because the
high values of $R_{1}$ and $R_{2}$ destroy the diversity.

##  IV Proposed Dither Signals Design over a Rayleigh Fading Channel

In this section, we propose an amplitude constraint of the dither signals with
consideration of a Rayleigh fading channel. First, we denote the channel
frequency response (CFR) of the $i$-th element in the $\beta$-th OFDM-IM
subblock as $H_{i}$ (with omission of $\beta$) and the CFR matrix is denoted
as $H=\mathrm{diag}(H_{0},H_{1},\cdots,H_{n-1})$. Note that
$H_{0},H_{1},\cdots,H_{n-1}$ are approximately independent because we use
concatenation in an interleaved pattern. (This is a valid assumption unless we
use channels with less than $n$ taps in time domain.) For a given matrix $H$,
the conditional PEP of the $\beta$-th OFDM-IM subblock with the dither signal
$D$ is

$\displaystyle P(X+D\rightarrow\hat{X}|H)$ (13)
$\displaystyle=P(\|Y-H\hat{X}\|^{2}<\|Y-H{X}\|^{2}|H)$ (14)
$\displaystyle=P(\|H(X+D)+Z-H\hat{X}\|^{2}<\|HD+Z\|^{2}|H)$ (15)
$\displaystyle=P(2\cdot\mathrm{Re}\\{(HD+Z)^{H}H(X-\hat{X})\\}<-\|H(X-\hat{X})\|^{2}|H)$
(16)

where $Y=H(X+D)+Z$ is the $\beta$-th received OFDM-IM subblock and $Z$ is AWGN
with $Z_{i}\sim\mathcal{CN}(0,N_{0})$.

Since the dither signals cannot affect the symbol error events, let us
consider the fundamental index demodulation error case that the $u$-th
subcarrier is detected as active and the $v$-th subcarrier is detected as idle
when actually the opposite is true. Then, (16) becomes

$\displaystyle P(X+D\rightarrow\hat{X}|H)$ (17)
$\displaystyle=P(\mathrm{Re}\\{-|H_{u}|^{2}D_{u}^{*}\hat{X}_{u}-H_{u}Z_{u}^{*}\hat{X}_{u}+H_{v}Z_{v}^{*}X_{v}\\}$
(18)
$\displaystyle~{}~{}~{}~{}~{}~{}<-\frac{1}{2}(|H_{u}\hat{X}_{u}|^{2}+|H_{v}X_{v}|^{2})|H)$
(19)

where $D_{u}$ is the dither signal in the $u$-th idle subcarrier in $X$. If we
consider the dither signal $D_{u}$ satisfying

$|\mathrm{Re}\\{D_{u}\\}|+|\mathrm{Im}\\{D_{u}\\}|=\sqrt{2}R$ (20)

then the left hand side (LHS) term in (19) becomes

$-R|H_{u}|^{2}|\hat{X}_{u}|+\mathrm{Re}\\{-H_{u}Z_{u}^{*}\hat{X}_{u}+H_{v}Z_{v}^{*}X_{v}\\}$
(21)

under the assumption of
$|\mathrm{Re}\\{\hat{X}_{u}\\}|=|\mathrm{Im}\\{\hat{X}_{u}\\}|$ and
$\hat{X}_{u}$ and $D_{u}$ lie on the same quadrant. (This assumption is valid
because we will consider the weakness case inducing the index demodulation
error.) Clearly, (21) is Gaussian distributed as

$\mathcal{N}(-R|H_{u}|^{2}|\hat{X}_{u}|,\frac{N_{0}}{2}(|H_{u}\hat{X}_{u}|^{2}+|H_{v}X_{v}|^{2})).$
(22)

Then, (19) becomes

$\displaystyle P(X+D\rightarrow\hat{X}|H)$ (23)
$\displaystyle=Q\left(\frac{|H_{u}\hat{X}_{u}|^{2}+|H_{v}X_{v}|^{2}-2R|H_{u}|^{2}|\hat{X}_{u}|}{\sqrt{2N_{0}}\sqrt{|H_{u}\hat{X}_{u}|^{2}+|H_{v}X_{v}|^{2}}}\right)$
(24)
$\displaystyle=Q\left(\frac{1}{\sqrt{2N_{0}}}\left(\sqrt{|H_{u}\hat{X}_{u}|^{2}+|H_{v}X_{v}|^{2}}-\frac{2R|H_{u}||H_{u}\hat{X}_{u}|}{\sqrt{|H_{u}\hat{X}_{u}|^{2}+|H_{v}X_{v}|^{2}}}\right)\right)$
(25)

where the term
$\sqrt{|H_{u}\hat{X}_{u}|^{2}+|H_{v}X_{v}|^{2}}-\frac{2R|H_{u}||H_{u}\hat{X}_{u}|}{\sqrt{|H_{u}\hat{X}_{u}|^{2}+|H_{v}X_{v}|^{2}}}$
in (25) can be depicted as the red line in Fig. 1.

Figure 1: A geometrical representation of
$\sqrt{|H_{u}\hat{X}_{u}|^{2}+|H_{v}X_{v}|^{2}}-\frac{2R|H_{u}||H_{u}\hat{X}_{u}|}{\sqrt{|H_{u}\hat{X}_{u}|^{2}+|H_{v}X_{v}|^{2}}}$
by the led line. Note that
$\frac{|H_{u}\hat{X}_{u}|}{\sqrt{|H_{u}\hat{X}_{u}|^{2}+|H_{v}X_{v}|^{2}}}$ is
$\cos\theta$.

As seen in Fig. 1, the length of the red line can be alternatively
approximated as the length of the green line. This is approximation is
reasonable because $|H_{u}\hat{X}_{u}|$ and $|H_{v}X_{v}|$ are similar based
on the fact that $\mathbb{E}[|H_{u}\hat{X}_{u}|]=\mathbb{E}[|H_{v}X_{v}|]$ and
the value of $R$ is relatively small compared to $|\hat{X}_{u}|$. The length
of the green line is $\sqrt{|H_{u}(|\hat{X}_{u}|-2R)|^{2}+|H_{v}X_{v}|^{2}}$
in Fig. 1 and thus (25) becomes

$P(X+D\rightarrow\hat{X}|H)\simeq
Q\left(\frac{\sqrt{|H_{u}(|\hat{X}_{u}|-2R)|^{2}+|H_{v}X_{v}|^{2}}}{\sqrt{2N_{0}}}\right).$
(26)

Meanwhile, in [12, 13], it is known that the unconditional PEP with
independent Rayleigh channel frequency responses $H_{i}\sim\mathcal{CN}(0,1)$
in high SNR region is expressed as

$\displaystyle P(X\rightarrow\hat{X})$ (27) $\displaystyle=\int
P(X\rightarrow\hat{X}|H)p(H)dH$ (28) $\displaystyle=\int
Q\left(\frac{\|H(X-\hat{X})\|}{\sqrt{2N_{0}}}\right)p(H)dH$ (29)
$\displaystyle\simeq\frac{(4N_{0})^{\Gamma_{X,\hat{X}}}}{2\Pi_{i\in\mathcal{G}_{X,\hat{X}}}\eta_{i}}$
(30)

where $p(H)$ is the probability distribution function (pdf) of $H$, $\eta_{i}$
is the $i$-th element of $|X-\hat{X}|^{2}$,
$\mathcal{G}_{X,\hat{X}}=\\{i|\eta_{i}\neq 0\\}$, and
$\Gamma_{X,\hat{X}}=|\mathcal{G}_{X,\hat{X}}|$. By combining (26) and (30), we
have

$P(X+D\rightarrow\hat{X})\simeq\frac{(4N_{0})^{2}}{2(|\hat{X}_{u}|-2R)^{2}\cdot|X_{v}|^{2}}.$
(31)

That is, the unconditional PEP $P(X+D\rightarrow\hat{X})$ depends on the
metric as

$(|\hat{X}_{u}|-2R)\cdot|{X}_{v}|.$ (32)

Then, the weakness case inducing the fundamental index demodulation error in
the $\beta$-th subblock depends on the metric as

$\min(|\hat{X}_{u}|-2R)\cdot|X_{v}|=(\sqrt{2}-2R)\cdot A^{\beta}.$ (33)

Clearly, the index demodulation error performance of the $\beta$-th OFDM-IM
subblock is dominated by the value of (33).

As in the scheme in [7], the proposed scheme is valid if QAM modulation is
considered. If 16-QAM is employed with the signal constellation $\\{\pm 1\pm
j1,\pm 1\pm j3,\pm 3\pm j1,\pm 3\pm j3\\}$, the proposed variable amplitude
constraint is

$\displaystyle D_{i}^{\beta}$ (34)
$\displaystyle=\left\\{\begin{array}[]{ll}|\mathrm{Re}\\{D_{i}^{\beta}\\}|+|\mathrm{Im}\\{D_{i}^{\beta}\\}|<\sqrt{2}R_{0},&\hbox{$i\in(I^{\beta})^{c}$,
$A_{\beta}=\sqrt{2}$}\\\
|\mathrm{Re}\\{D_{i}^{\beta}\\}|+|\mathrm{Im}\\{D_{i}^{\beta}\\}|<\sqrt{2}R_{1},&\hbox{$i\in(I^{\beta})^{c}$,
$A_{\beta}=\sqrt{10}$}\\\
|\mathrm{Re}\\{D_{i}^{\beta}\\}|+|\mathrm{Im}\\{D_{i}^{\beta}\\}|<\sqrt{2}R_{2},&\hbox{$i\in(I^{\beta})^{c}$,
$A_{\beta}=\sqrt{18}$}\\\ 0,&\hbox{$i\in I_{\beta}$}\end{array}\right.$ (39)

with

$(\sqrt{2}-2R_{0})\cdot\sqrt{2}=(\sqrt{2}-2R_{1})\cdot\sqrt{10}=(\sqrt{2}-2R_{2})\cdot\sqrt{18}$
(40)

which leads to the same value of (33) for all subblocks. (Since the bottleneck
of the OFDM-IM error performance relates to the smallest value of (33) for all
$g$ subblocks, this strategy could be the optimal solution.) It is
straightforward to generate the constraints for other modulations.

Table I shows several examples of the values of $R_{0}$, $R_{1}$, and $R_{2}$
satisfying the proposed amplitude constraint in (40). We remark that the
values of $R_{1}$ and $R_{2}$ are suppressed because of frequency selective
fading channels, compared to the values from (12).

TABLE I: Examples of $R_{0}$, $R_{1}$, and $R_{2}$ values satisfying (40) $R_{0}$ | $R_{1}$ | $R_{2}$
---|---|---
0.1 | 0.435 | 0.504
0.2 | 0.480 | 0.538
0.3 | 0.525 | 0.571
0.4 | 0.569 | 0.604
0.5 | 0.614 | 0.638

##  V Simulation Results

For modulating the symbols in the active subcarriers, 16-QAM is used. Also, we
use $N=128$, $n=4$, and $k=2$. The OFDM-IM subblocks are concatenated in an
interleaved pattern and dither signals are generated by five times iterative
clipping and filtering with trimming for all schemes. To capture PAPR of the
continuous-time OFDM-IM signal, four-times oversampling is used for the
clipping and filtering. At the receiver, the optimal ML detection with low
computational complexity is employed.

Fig. 2 shows the BER performance over a Rayleigh fading channel with eight
channel taps. The SNR means the average energy per bit over $N_{0}$. For fair
comparison, both symbols in the active subcarriers and dither signals in the
idle subcarriers are considered when we calculate the average energy per bit.
The meaning of four labels in the legend is as follows; the original OFDM-IM
signal without clipping; the OFDM-IM signal under the proposed amplitude
constraint, $R_{0}=0.5$, $R_{1}=0.614$, and $R_{2}=0.638$; the OFDM-IM signal
under the equivalent amplitude constraint in [6], $R=0.5$; the OFDM-IM signal
under the constraint in [7], $R_{0}=0.2$, $R_{1}=1.9$, and $R_{2}=3$, which is
originally designed for AWGN channels. In comparison, we do not consider the
traditional PAPR reduction schemes such as SLM, PTS, and TI in [5] because the
PAPR reduction schemes exploiting the idle subcarriers do not exist.

As we expected, the proposed constraint and the equivalent amplitude
constraint in [6] have almost the same BER performance because the two
constraints induce the same PEP for the bottleneck of the OFDM-IM error
performance. In specific, they have the same smallest value of (33) for all
$g$ subblocks. The constraint originally designed for AWGN channels in [7]
shows the error floor over 25 dB SNR point because the large values of $R_{1}$
and $R_{2}$ destroy the diversity gain over a fading channel.

Figure 2: BER performance of the OFDM-IM signals with different amplitude
constraints. Figure 3: PAPR reduction performance of the OFDM-IM signals with
different amplitude constraints. Four-times oversampling is used.

Fig. 3 shows the PAPR reduction performance of the four cases. The ordinate is
the complementary cumulative distribution function (CCDF) of the PAPR. Using
the proposed amplitude constraint can reduce much PAPR than the scheme in [6]
using the equivalent amplitude constraint. This result mainly comes from the
fact that the proposed constraint has larger freedom of dither signals.
Although the PAPR reduction performance gap between the two schemes is small,
the specific constraint based on the rigorous analysis is theoretically
meaningful to design the dither signals in OFDM-IM systems.

##  VI Conclusion

In this letter, we proposed the variable amplitude constraint for the dither
signals for PAPR reduction in OFDM-IM systems over a Rayleigh fading channel,
which is based on the rigorous PEP analysis. By using the proposed variable
amplitude constraint, the PAPR reduction performance can be increased compared
to the conventional case using the equivalent amplitude constraint with almost
the same BER performance.

## References

  * [1] E. Başar, Ü. Aygölü, E. Panayırcı, and H. V. Poor, “Orthogonal frequency division multiplexing with index modulation,” _IEEE Trans. Signal Process._ , vol. 61, no. 22, pp. 5536–5549, 2013\.
  * [2] R. Mesleh, H. Haas, S. Sinanovic, C. W. Ahn, and S. Yun, “Spatial modulation,” _IEEE Trans. Veh. Technol._ , vol. 57, no. 4, p. 2228, 2008.
  * [3] L. Zhao, H. Zhao, K. Zheng, and Y. Yang, “A high energy efficient scheme with selecting sub-carriers modulation in OFDM system,” in _2012 IEEE International Conference on Communications (ICC)_. IEEE, 2012, pp. 5711–5715.
  * [4] N. Ishikawa, S. Sugiura, and L. Hanzo, “Subcarrier-index modulation aided OFDM-will it work?” _IEEE Access_ , vol. 4, pp. 2580–2593, 2016\.
  * [5] S. H. Han and J. H. Lee, “An overview of peak-to-average power ratio reduction techniques for multicarrier transmission,” _IEEE Wireless Commun._ , vol. 12, no. 2, pp. 56–65, 2005.
  * [6] J. Zheng and H. Lv, “Peak-to-average power ratio reduction in OFDM index modulation through convex programming,” _IEEE Commun. Lett._ , vol. 21, no. 7, pp. 1505–1508, 2017.
  * [7] K.-H. Kim, “PAPR reduction in OFDM-IM using multilevel dither signals,” _IEEE Commun. Lett._ , vol. 23, no. 2, pp. 258–261, 2019.
  * [8] Y. Xiao, S. Wang, L. Dan, X. Lei, P. Yang, and W. Xiang, “OFDM with interleaved subcarrier-index modulation,” _IEEE Commun. Lett._ , vol. 18, no. 8, pp. 1447–1450, 2014.
  * [9] A. I. Siddiq, “Low complexity OFDM-IM detector by encoding all possible subcarrier activation patterns,” _IEEE Commun. Lett._ , vol. 20, no. 3, pp. 446–449, 2016.
  * [10] B. Zheng, F. Chen, M. Wen, F. Ji, H. Yu, and Y. Liu, “Low-complexity ML detector and performance analysis for OFDM with in-phase/quadrature index modulation,” _IEEE Commun. Lett._ , vol. 19, no. 11, pp. 1893–1896, 2015.
  * [11] X. Zhang, H. Bie, Q. Ye, C. Lei, and X. Tang, “Dual-mode index modulation aided OFDM with constellation power allocation and low-complexity detector design,” _IEEE Access_ , vol. 5, pp. 23 871–23 880, 2017.
  * [12] M. K. Simon and M.-S. Alouini, _Digital communication over fading channels_. John Wiley & Sons, 2005, vol. 95.
  * [13] T. Van Luong and Y. Ko, “Spread OFDM-IM with precoding matrix and low-complexity detection designs,” _IEEE Trans. Veh. Technol._ , vol. 67, no. 12, pp. 11 619–11 626, 2018.