#  IR Motion Deblurring

Nisha Varghese  Mahesh Mohan M. R  A. N. Rajagopalan

<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Camera gimbal systems are important in various air or water borne systems for
applications such as navigation, target tracking, security and surveillance. A
higher steering rate (rotation angle per second) of gimbal is preferable for
real-time applications since a given field-of-view (FOV) can be revisited
within a short period of time. However, due to relative motion between the
gimbal and scene during the exposure time, the captured video frames can
suffer from motion blur. Since most of the post-capture applications require
blur-free images, motion deblurring in real-time is an important need. Even
though there exist blind deblurring methods which aim to retrieve latent
images from blurry inputs, they are constrained by very high- dimensional
optimization thus incurring large execution times. On the other hand, deep
learning methods for motion deblurring, though fast, do not generalize
satisfactorily to different domains (e.g., air, water, etc). In this work, we
address the problem of real-time motion deblurring in infrared (IR) images
captured by a gimbal-based system. We reveal how _a priori_ knowledge of the
blur-kernel can be used in conjunction with non-blind deblurring methods to
achieve real-time performance. Importantly, our mathematical model can be
leveraged to create large-scale datasets with realistic gimbal motion blur.
Such datasets which are a rarity can be a valuable asset for contemporary deep
learning methods. We show that, in comparison to the state-of-the-art
techniques in deblurring, our method is better suited for practical gimbal-
based imaging systems.

##  1 Introduction

Camera gimbal systems, used in navigation, target tracking, security and
surveillance, work by controlling the trajectory of a camera-mounted on a
gimbal in order to obtain an extended FOV. During exposure, due to the
relative motion between the scene and the gimbal, the captured images are
typically degraded by motion blur. A motion blurred frame is the result of
aggregation of different world-to-sensor projections of the scene, over the
exposure interval, onto the image sensor [1][10]. Motion blur degrades image
quality which can in turn lead to low probability of threat detection and
higher false alarm rate during tracking and surveillance. Since most of the
applications require blur-free images, motion deblurring is a very important
problem. Moreover, such systems warrant motion deblurring in real-time. Even
though there exist different effective algorithms to retrieve latent images
from blurry inputs, they are not suitable for real-time applications due to
their higher execution time.

Traditional methods for motion deblurring mainly comprise of two stages [1].
First, a motion blur model is designed to relate the sharp image to the
observed blurred image via camera motion parameters (i.e., blur kernel).
Second, on inverting this model, the camera motion parameters and
corresponding sharp photograph are estimated. The problem of deblurring is an
ill-posed one and sharp image estimation typically necessitates the use of
natural image priors [18][19]. Traditional deblurring methods typically
proceed by alternating minimization (AM) of camera motion and sharp image.
Further, AM proceeds in a scale-space manner to accommodate large blurs to
keep the optimization dimension low [1].

A number of traditional methods exist for blind motion deblurring. In [3], a
parameterized geometric model of the blurring process is proposed in which the
blurred image is represented using the transformed pixel value, due to
rotations, and the time it spends at each rotation angle. Even though this
method accommodates space variant blur kernels, a high computational cost
renders it ineffective for real-time applications. There are situations in
which the blur kernel can be assumed to be space-invariant thus reducing the
complexity. The work [14] assumes a convolutional model and proposes a
sparsity prior to estimate the blur kernel. The latent image is then
reconstructed using standard Richardson-Lucy (RL) deconvolution [6][7]
algorithm. The method in [15] is aimed at handling blurred images in the
presence of outliers. It models the data fidelity term so that the outliers
have little effect on kernel estimation. A natural image prior known as
extreme channel prior is proposed in [16] by taking advantage of both dark and
bright channel priors for kernel estimation. Deblurring is done by alternating
minimization of camera motion and sharp image. The method of [17] introduces
an image prior based on a higher-order Markov random fields model termed as
super-Gaussian fields to address the problem of sparsity of the prior. A scale
normalization technique is introduced in [19] based on the $L^{p}$ norm. [18]
proposes a simplified sparsity prior of local patch-wise minimal pixels. Since
it only requires values for non-overlapping patches, it is much simpler than
the dark-channel prior. However, all these methods incur heavy processing
time.

Another class of methods use deep learning networks. They work by optimizing
the weights of a neural network, to emulate the non-linear relationship
between blurred-sharp image pair, using a training dataset. Both the effects
of camera motion and dynamic object motion can be handled by these networks
[2]. As these methods typically require only a single pass over the network
for deblurring (unlike the AM), their processing time is comparatively quite
less [22][23][24][25].

Typically, in surveillance applications, infrared (IR) imagery is most
suitable since it has a 24$\times$7 observation option during any atmospheric
or illumination conditions [8][9]. We consider a commonly used gimbal set-up
which extends the FOV via yaw-based gimbal motion. Our system consists of an
IR camera which is mounted on a gimbal set-up. The gimbal undergoes fast to
and fro yaw motion to surveil the extended-FOV. Further, the gimbal motion is
stabilized such that the motion disturbances in gimbal over other rotation
directions (i.e., pitch and roll) are negligible. The speed is selected such
that a given FOV is revisited as quickly as possible. The gimbal is equipped
with an inertial measurement unit (IMU), consisting of gyroscopes and
accelerometers. The IMU contains the information to calculate the angular
velocity of the device that it is mounted on.

In this work, we specifically address the problem of real-time motion
deblurring in IR images captured by a gimbal based system. It is important to
note that state-of-the-art deblurring methods are designed for an
unconstrained system such as random hand-held motion. Consequently, the number
of unknowns and hence the processing time for these traditional methods is
unacceptably high as they do not leverage the peculiarities of gimbal-based
systems where camera motion is usually constrained or is tractable. This
limits their applicability to the problem on hand. Deep learning networks, on
the other hand, need a realistic training dataset with a large number of
blurred and corresponding deblurred image-pairs. This requirement is often
difficult to meet practically, especially in the case of IR images. Further,
one pertinent problem is regarding generalization, i.e., a network trained
using a particular dataset struggles when confronted with unseen examples from
a different domain.

In contrast to the afore-mentioned methods, we propose an effective real-time
motion deblurring approach for IR images captured using gimbal-based systems.
We advocate _a priori_ estimation of the blur kernel followed by _non-blind_
deblurring for real-time performance. Our main contributions are as follows:

  1. 1.

This work provides a mathematical framework to encapsulate gimbal-based motion
blur via extensive evaluations with real gimbal data.

  2. 2.

We leverage our derived model to devise a cost-effective non-blind deblurring
solution that is almost real-time.

  3. 3.

We experimentally verify that our method outperforms competing methods by a
significant margin.

  4. 4.

Our model can be used to create large-scale motion blur datasets for gimbal-
based systems that can be harnessed by deep learning methods.

##  2 Our Approach

Blur due to camera rotations is typically associated with space-variant (SV)
blur kernels [3][4][5]. i.e., the nature of the kernel is different at
different locations in the image. Figure 1(a) shows plots of the point spread
function (PSF) distribution in four corners of an image for real camera motion
(using [4]). Even though SV methods [3][4] are effective for real camera
motion, their high computational cost renders them unsuitable for gimbal-based
systems. The SV methods optimize for high-dimensional camera motion, whereas
the gimbal motion is predominantly a 1D rotational motion. The blur kernel
corresponding to yaw-only motion is given in Fig. 1(b). Note that PSFs
produced by a general camera motion is quite different as compared to yaw-only
motion PSFs. Observe that the PSF is mostly space-invariant.

|
---|---
(a) | (b)
Figure 1: PSF distribution for (a) real camera motion, and (b) yaw-only
motion.

We will verify this interesting feature both theoretically and experimentally
in the subsequent sections.

For space-invariant blurring, we can model the blurred image $\mathbf{B}$ as a
clean image $\mathbf{L}$ convolved with a blur kernel $\mathbf{k}$ and
corrupted by noise $\mathbf{n}$ [26]:

$\mathbf{B}=\mathbf{L}*\mathbf{k}+\mathbf{n}$ (1)

where $*$ denotes the convolution operator. Note that the noise $\mathbf{n}$
captures the modelling errors, image noise and the effect of outliers, e.g.,
saturated pixels and impulse noise, which cannot be well-modeled by the linear
convolution model. Blind deblurring methods (mentioned in the previous
section) estimate both $\mathbf{L}$ and $\mathbf{n}$. Patch-wise minimal pixel
prior method [18] and normalized blind deconvolution [19] are some promising
blind deblurring methods proposed recently, but lack real-time performance.

The task of deblurring an image with a known blur kernel is known as _non-
blind_ deblurring. Since the problem is only to find out the clean image
$\mathbf{L}$ from the blurred image $\mathbf{B}$ and the blur kernel
$\mathbf{k}$, the processing time for non-blind deconvolution is comparatively
far less than any blind deblurring method. In this work, we explore the
exciting prospect of estimating the blur kernel a priori followed by non-blind
deblurring.

Numerous non-blind deconvolution approaches exist, varying greatly in their
speed and sophistication. Wiener filter [26] is a traditional non-blind
deconvolution method which finds the deblurred image from the knowledge of
blurred observation and kernel along with mitigation of additive noise. The
Richardson–Lucy (RL) algorithm [6][7], is an iterative procedure for
recovering an underlying image that has been blurred by a known PSF. The main
problem in any fast non-blind deconvolution method is the presence of ringing
artifacts in the deblurred results. The challenge addressed by RL algorithm is
in balancing the recovery of image details and suppression of ringing. There
are works which have proposed new image priors to model natural image
statistics so as to improve the quality of deblurred results, especially from
a ringing perspective. [11] employs a hyper-Laplacian prior and the
deconvolution is several orders of magnitude faster than existing techniques.
A patch-wise model is proposed in [12] that uses a Gaussian mixture prior.
[13] tackles the problem of outliers to improve the quality of the deblurred
results.

Among the many existing non-blind deblurring methods, we zero-in on three
methods, namely, Wiener filter (WF) [26], RL deconvolution [6][7] and
deblurring using hyper-Laplacian priors [11] due to their attractive execution
times. Our central idea is that if we know the PSF for a blurred frame, then
we can speed-up deblurring by a large factor. For example, if we can compute
accurately the kernel for each steering rate in a cost-effective manner, then
we can achieve considerable speed-up by employing one of the above three non-
blind deblurring methods. However there is a caveat: Non-blind deblurring
methods can yield good results only if the blur kernels are known accurately.
We next present two blur-kernel estimation methods for the problem on hand.

##  3 Blur kernel estimation using blur-sharp pair

The strength of this method lies in the fact that it does not warrant
knowledge of motion or camera parameters. Recollect that each pixel in a
motion blurred image has contributions from the neighbouring pixels. It is
well-known that temporal averaging of a number of consecutive sharp images,
captured from a camera, yields a blurred image that resembles a motion blurred
image. We make use of this fact to get the blurred images corresponding to
different amounts of motion blur. For this purpose, the camera is panned very
slowly (steering rate of 1 deg/sec is typical) so as to obtain blur-free
images of the scene. Blurred images corresponding to different steering rates
can then be obtained by suitably averaging a certain number of sharp frames.
The blurred images along with their sharp image counterpart can be used to
estimate the blur kernel corresponding to the motion blur content in the
blurred image by solving Eq. (2) using conjugate gradient.

$\mathbf{k}=\arg\min_{\mathbf{k}}||\mathbf{B}-\mathbf{L}*\mathbf{k}||_{2}\text{
\hskip 14.22636pts.t.\hskip 8.5359pt}\sum\mathbf{k}=1$ (2)

where $*$ denotes convolution operation, $\mathbf{B}$ is the blurred image and
$\mathbf{L}$ is the corresponding sharp image. This method works for any type
of blur kernel. We may take an initial guess for $\mathbf{k}$ that is
eventually refined to the original kernel after a pre-defined number of
iterations.

Figure 2 gives an example of blur-sharp pair used for PSF estimation. The
blurred image in Fig. 2(b) is formed synthetically by blurring the sharp image
(Fig. 2(a)) with a diagonal motion blur kernel of size 21 pixels at an angle
of 30$\degree$ (Fig. 2(d)). By solving Eq. (2), the estimated kernel from the
blur-sharp pair is given in Fig. 2(c). It can be seen that the estimated blur
kernel is quite close to that of Fig. 2(d).

|
---|---
(a) Sharp | (b) Blurred
|
(c) Estimated kernel | (d) Original kernel
Figure 2: Comparison of estimated kernel (c) (from blur sharp pair (a-b)), and
the original kernel (d). Note that the Blur kernel estimation method using
blur-sharp pair estimates PSFs similar to the ground truth.

As mentioned earlier, frames captured with a steering rate of 1 deg/sec
contain almost no motion blur as the motion is minimal. We treat these frames
as sharp frames. A blurred image can be formed by averaging $N$ images where
$N\in{1,2,3,\cdots,13}$. Thus, we can arrive at the average image as the
blurred image and the center frame as the sharp image. As an example, if
$N=5$, then we average 5 consecutive frames to get the blurred image
$\mathbf{B}$ and frame number 3 is treated as the sharp image $\mathbf{L}$. If
$N$ is an even number, frame number $\frac{N}{2}+1$ is taken as the sharp
image. We can then find kernel $\mathbf{k}$ from the blurred image
$\mathbf{B}$ and the sharp image $\mathbf{L}$ using Eq. (2). A uniform kernel
of a certain size can be taken as the initial estimate of $\mathbf{k}$. It is
better to take planar scenes to form the average image to get a consistent
kernel since depth differences in the scene can lead to errors in the estimate
of $\mathbf{k}$.

Thus we can get an estimate of the PSF (i.e., blur kernel) from blur-sharp
pairs for different values of $N$. Note that a large $N$ increases the blur
content in the averaged frame and hence the size of the blur kernel. The blur
kernel corresponding to different values of $N$ can be estimated to match the
blur in images corresponding to different steering rates (from 10 deg/sec to
60 deg/sec). The value of $N$ to be taken from steering 1 deg/sec to make the
blur equivalent to the blur in a given steering rate $s_{r}$ can be found out
either empirically by matching the deblurred results for different steering
rates or it can be derived if the exposure time $t_{exp}$ of the camera is
known. The rotation angle of a camera with steering rate $s_{r}$ and exposure
time $t_{exp}$ is given by

$\theta_{s_{r}}=t_{exp}\cdot s_{r}$ (3)

A steering rate of 1 deg/sec covers 1$\degree$ in 1 sec. If $f_{r}$ is the
frame rate (in frames/sec) then, the number of frames is given by

$N=f_{r}\cdot\theta_{s_{r}}$ (4)

##  4 Analytical modeling of Blur kernel

In this section, as an alternative, we wish to find the PSF mathematically for
the case of gimbal motion that pans a scene with an IR mounted camera with
known intrinsic camera parameters. Unlike the method discussed in Section 3,
this is an entirely analytical approach to find the PSF from camera motion and
camera intrinsics and does not need the blurred observations at all.

Consider a 2D impulse signal (a value 1 at the center of the image and 0
elsewhere). During exposure, due to relative motion between camera and the
scene, the impulse signal will be distributed across adjacent pixels in the
captured image. This forms the PSF for that motion model. To find the PSF
analytically, we first assume a maximum pixel spread $s$ (for each steering
rate), from the center of the image, due to the camera motion. We next find
the rotation angle of the camera corresponding to every pixel displacement
upto the maximum pixel displacement $s$. Then we find the transformation of
the center pixel value using homography equations for every rotation angle of
the camera. First, we will derive the equation for the rotation angle
corresponding to a pixel spread that occurs in the image due to yaw-rotation
induced motion blur.

###  4.1 Rotation angle $\theta$ corresponding to a pixel spread

Figure 3: Relation between rotation angle and the pixel spread.

Figure 3 illustrates the situation on hand. The camera is rotated by an angle
$\theta$ where $w$ and $h$ are the dimensions of the image, $f$ is the focal
length of the camera, $d_{m}$ is the maximum displacement from the center of
the image, and $r$ is the distance from the edge pixel point to camera center.

When the camera rotates by an angle $\theta$, an arc with length $r\theta$
will be formed on the rear of the sensor plane whose image is the pixel spread
on the image plane. Assuming a linear arc length $r\theta$, we can write the
arc length as the projection from the sensor plane (as seen in Fig. 3(b)).
Therefore,

$r\theta=p_{s}\cdot\cos\phi$ (5)

where $\phi$ is the angle between the arc and the sensor plane, and $p_{s}$ is
the corresponding pixel spread. From the figure, it can be seen that the angle
between the arc and the sensor plane is equal to the angle between $f$ and
$r$. Therefore $\cos\phi$ can be written as $\cos\phi=f/r$ and $r$ can be
written in terms of $f$ and $d_{m}$. Therefore Eq. (5) becomes

$\sqrt{f^{2}+d_{m}^{2}}\theta=p_{s}\cdot\left(\frac{f}{\sqrt{f^{2}+d_{m}^{2}}}\right)$
(6)

Hence, the rotation angle $\theta$ corresponding to the pixel spread can be
written as

$\theta=p_{s}\cdot\left(\frac{f}{f^{2}+d_{m}^{2}}\right)$ (7)

We next list the steps to find the blur kernel corresponding to the pan motion
of the camera given the intrinsic and motion parameters of the camera.

  1. 1.

The intrinsic camera matrix is given by,

$\mathbf{K}=\begin{bmatrix}f&0&w/2\\\ 0&f&h/2\\\ 0&0&1\end{bmatrix}$ (8)

If the focal length ($f$) of the camera is not known, then it can be found out
using FOV $\alpha$ and the frame dimensions $w$ and $h$ as,

$f=\frac{\sqrt{h^{2}+w^{2}}}{2\tan(\alpha/2)}$ (9)

  2. 2.

Find the rotation angles of the camera $\theta_{1}$ and $\theta_{max}$
corresponding to 1 pixel spread and maximum pixel spread $s$ respectively,
using Eq. (7).

  3. 3.

Since the rotation is only about the y axis, the rotation matrix $\mathbf{R}$
corresponding to the rotation angle $\theta$ can be written as

$\mathbf{R}=\mathbf{R}_{y}=\begin{bmatrix}\cos\theta&0&\sin\theta\\\ 0&1&0\\\
-\sin\theta&0&\cos\theta\end{bmatrix}$ (10)

Find the rotation matrices for the rotation angles from $-\theta_{max}\text{\
to}+\theta_{max}$ with step size $\theta_{1}$.

  4. 4.

For the pure rotation model, the homography $\mathbf{H}$ [3] can be written
as,

$\mathbf{H}=\mathbf{KRK}^{-1}$ (11)

Find all the homography matrices corresponding to every rotation obtained in
the previous step.

  5. 5.

Take the center pixel value $\mathbf{x}$ of the image and find its transformed
point $\mathbf{x^{\prime}}=\mathbf{H}\mathbf{x}$ for all $\mathbf{H}$ and mark
1 at every $\mathbf{x^{\prime}}$ if $\mathbf{x^{\prime}}$ is an integer. If
$\mathbf{x^{\prime}}$ is not an integer, then distribute 1 to the neighbouring
positions of $\mathbf{x^{\prime}}$ using bilinear interpolation.

  6. 6.

Normalise the PSF so that it sums to 1.

  7. 7.

This will give the PSF corresponding to the center position.

  8. 8.

If we are interested in obtaining PSFs corresponding to different positions of
the image, repeat steps 5 and 6 for pixel locations other than the center
position.

The above derivation can be extended in a straightforward manner to general
gimbal-trajectories.

###  4.2 Selection of maximum pixel spread

Note that in order to model blur kernel analytically, we assume that the
maximum pixel spread $s$ in the blur kernel is known for a particular steering
rate. It equals half the kernel length corresponding to that steering rate. A
higher steering rate will imply a higher value of $s$. If the exposure time is
known, then the maximum spread $s$ can be determined a priori as follows.

Let the exposure time of the camera be $t_{exp}$ and the steering rate be
$s_{r}$. The total angle of rotation $\phi^{\prime}$ during the exposure time
can be written as

$\phi^{\prime}=s_{r}\cdot t_{exp}$ (12)

Then by using Eqn. (7), we can find the maximum pixel spread $s$ from the
center as

$s=\left(\frac{\phi^{\prime}}{2}\right)\cdot\left(\frac{f^{2}+d_{m}^{2}}{f}\right)$
(13)

##  5 Experimental Results

Datasets: We present results on real IR images corresponding to seven
different steering rates and captured from an actual gimbal system that pans a
high-altitude scene. The steering rates are 1 deg/sec and 10 to 60 deg/sec
with a step-size of 10. Each frame is of size 558 $\times$ 481 pixels. The
video corresponding to steering rate of 1 deg/sec contains 3067 frames while
the other 6 videos contain on the average about 675 frames. We perform
deblurring on a machine with Intel Xeon e5-2630v4 processor $@$ 2.2 GHz.

In this section, we provide results of our method i.e., non-blind deblurring
with WF, RL and [11] using the blur kernel estimated i) from blur-sharp pair,
and ii) analytically. For comparisons, we examine different competing methods.
Among blind deblurring methods, we choose [19] as it is the best in its class.
We also use the PSF returned by [19] to perform non-blind deblurring with
Wiener filter, RL [6][7] and [11].

We begin by presenting qualitative comparisons for competing methods.
Deblurring results for steering rate 50 deg/sec and 60 deg/sec using [19] are
given in Fig. 4. While the results are somewhat satisfactory, the processing
time is about 10 minutes per image which is prohibitively high.

|  |
---|---|---
(a) Blurred - SR50 | (b) Kernel | (c) Deblurred output
|  |
(d) Blurred - SR60 | (e) Kernel | (f) Deblurred output
Figure 4: Deblurred results of blind deblurring method [19]. (a): steering rate 50 deg/sec, (d): steering rate 60 deg/sec. Figure 5: Comparisons of non-blind deblurring methods using the kernel obtained from [19]. (a): steering rate 50 deg/sec, (e): steering rate 60 deg/sec. |  |  |
---|---|---|---
(a) Blurred - SR50 |  (b) Krishnan et al. [11] | (c) RL | (d) Wiener
|  |  |
(e) Blurred - SR60 |  (f) Krishnan et al. [11] | (g) RL | (h) Wiener

Since blind deblurring methods cannot be implemented in real-time, we next use
three non blind deblurring methods (as explained in section 2) with a known
PSF. Since the blind method [19] gives good results, it stands to reason that
it is able to fairly model the underlying blur kernel associated with the
blurred images. Hence, we use the PSFs returned from [19] and perform non-
blind deblurring using Wiener filter, RL [6][7] and [11].

_Edge-tapering_ : Before applying any non-blind deblurring method, we perform
edge-tapering on the blurred images to remove boundary related ringing
artifacts that stem from high frequency drop-off during deblurring. Edge-
tapering is done using a PSF $\mathbf{k}_{et}$ (we use a Gaussian filter of
size 30 pixels and $\sigma=10$), which gives an output image $\mathbf{J}$
which is the weighted sum of the original image $\mathbf{I}$ and its blurred
version. The weighting array is determined by the autocorrelation function of
$\mathbf{k}_{et}$. The image $\mathbf{J}$ contains the original image portion,
$\mathbf{I}$, in its central region, and the blurred version of $\mathbf{I}$
(using $\mathbf{k}_{et}$) near the edges.

The results given in Fig. 5 are comparable to that of [19] but come at a
fraction of the computational cost incurred by [19].

###  5.1 Results using PSF from blur-sharp pair

We generate blurred images by averaging a set of sharp frames. In our dataset,
frames captured with a steering rate of 1 deg/sec can be treated as sharp
frames since the motion blur is minimal for this steering rate.

Using Eq. (4), we determine the number of images $N$ to be averaged for
different steering rates (frame rate $f_{r}=$ 30 frames/sec, exposure time
$t_{exp}=$ 5 ms). For steering rate 60 deg/sec: $N$ is found to be 9; for
steering rate 50 deg/sec: $N=$ 8; for steering rate 40 deg/sec: $N=$ 6; for
steering rate 30 deg/sec: $N=$ 5; for steering rate 20 deg/sec: $N=$ 4; and
for steering rate 10 deg/sec: $N=$ 3\. Examples of blur-sharp pairs formed
from steering rate 1 deg/sec frames and the estimated blur kernels are given
in Fig. 6. The PSFs thus estimated are next used in the non-blind deblurring
methods to deblur the real IR images captured at corresponding steering rates.
Note that the actual PSF in a real blurred image need not match the PSF
estimated from blur-sharp pairs due to presence of noise.

|  |
---|---|---
(a) Mean of 6 frames | (b) Center-4th frame | (c) Kernel
|  |
(d) Mean of 9 frames | (e) Center-5th frame | (f) Kernel
Figure 6: Blur-sharp pairs and the estimated kernel obtained using Eq. 2. (a): steering rate 40 deg/sec, (d): steering rate 60 deg/sec. Figure 7: Deblurred results with non-blind deblurring methods using the PSF estimated from blur-sharp pair. (a): steering rate 40 deg/sec, (e): steering rate 60 deg/sec. |  |  |
---|---|---|---
(a) Blurred - SR40 |  (b) Krishnan et al. [11] | (c) RL | (d) Wiener
|  |  |
(e) Blurred - SR60 |  (f) Krishnan et al. [11] | (g) RL | (h) Wiener

The deblurred results shown in Fig. 7 reveal good performance. Visually, we
observe that the deblurring quality of Wiener filter is better than other
methods. Krishnan et al. [11] has a few ringing artifacts.

###  5.2 Results using analytical PSFs

To use the proposed method discussed in Section 4, we need camera intrinsics.
For the camera that we have used in our experiments, the FOV
$(\alpha)=8\degree$ and the dimensions of the image are $w=558$ pixels and
$h=481$ pixels. Thus $f$ is determined using Eq. 9. Camera rotation angle
$\theta$ corresponding to one pixel spread upto the maximum pixel spread $s$
is found using Eq. (7). The maximum pixel spread for each steering rate is
selected using Eq. (13) with $t_{exp}$ = 5 ms.

Figure 8: Analytical PSFs obtained at different positions of the image. |  |  |
---|---|---|---
(a) Blurred - SR40 |  (b) Krishnan et al. [11] | (c) RL | (d) Wiener
|  |  |
(e) Blurred - SR60 |  (f) Krishnan et al. [11] | (g) RL | (h) Wiener
Figure 9: Deblurred results using analytical PSFs. (a): steering rate 40
deg/sec, (e): steering rate 60 deg/sec. Note that non-blind methods with
analytical PSFs obtained using our proposed method exhibit quite good
deblurring performance.

By following the procedure described in Section 4, the analytical PSFs
obtained at different positions of the image are given in Fig. 8. It shows
that the blur is mostly space-invariant with some small variations at the
edges of the kernels. We use the average of these 5 PSFs to deblur the real
images and the corresponding deblurred results are given in Fig. 9. It can be
observed that the deblurred results obtained using analytically modelled PSFs
are quite good.

In Fig. 10, we give comparison results under one roof for a blurred image
corresponding to steering rate of 60 deg/sec. It is amply evident that the
results of our method (shown in b and c) yield best performance consistently
across Wiener filter, RL as well as [11]. From the figure, it can be seen that
the deblurred quality of non-blind methods using the estimated kernels from
our approach is better than using the blur kernel returned from the blind
deblurring method of [19]. Estimation of kernel using [19] is not only
computationally expensive but also does not offer any advantage in getting
good deblurring quality. Our PSF estimation methods are less computationally
expensive and deliver good results. Visually, motion deblurring quality is
better by using Krishnan et al. [11] and Wiener filter.

###  5.3 Quantitative comparisons and Time complexity

The nomenclature that we follow for the deblurring methods is as follows.

  1. 1.

$k_{blind}$: PSF obtained from the blind method of [19].

  2. 2.

$k_{BSpair}$: PSF estimated using blur-sharp pair.

  3. 3.

$k_{anal}$: Analytical PSF derived using camera motion and intrinsic
parameters.

|  |  |
---|---|---|---
(A) Blurred | 1(a) | 2(a) | 3(a)
|  |  |
| 1(b) | 2(b) | 3(b)
|  |  |
| 1(c) | 2(c) | 3(c)
|  1\. Krishnan et al. [11] | 2\. RL | 3\. Wiener
Figure 10: Comparisons of non-blind deblurring methods (1-3 in each column)
for steering rate of 60 deg/sec using PSFs from three different methods (a:
from blind method [19], b: estimated using blur-sharp pair, c: analytical
PSF). Note that our method (b) and (c) deliver the best performance
consistently.

In order to quantitatively evaluate efficacy, from the steering rate of 1
deg/sec images, we synthetically generated 20 blurred images and added
additive white Gaussian noise (AWGN) with signal-to-noise ratio (SNR) values
of 32 dB, 35 dB and 38 dB. These images are then deblurred using the blind
deblurring method of [19] as well as the non-blind deblurring methods
($k_{blind}$, $k_{BSpair}$ and $k_{anal}$) for all three SNR values. The
average PSNR and SSIM values (higher is better) calculated over these 20
deblurred images are given in Table 1. Note that, in all the cases, our
proposed method outperforms [19] as well as the non-blind deblurring methods
with kernels derived from [19]. As the noise level increases, [19]
deteriorates considerably due to kernel estimate errors. In contrast, our
methods (especially $k_{anal}$) outperform the latter by a large margin in
noisy scenarios.

Table 1:  Quantitative evaluations using PSNR and SSIM: NBD - Non-blind Deblurring. |  PSNR(dB)$\uparrow$ | SSIM$\uparrow$
---|---|---
Noise | 38 dB | 35 dB | 32 dB | 38 dB | 35 dB | 32 dB
[19] | 25.3 | 25.2 | 24.1 | 0.63 | 0.52 | 0.37
NBD |  [11] |  RL |  WF |  [11] |  RL |  WF |  [11] |  RL |  WF |  [11] |  RL |  WF |  [11] |  RL |  WF |  [11] |  RL |  WF
$k_{blind}$ |  27.6 |  26.2 |  26.5 |  27.1 |  24.9 |  26.2 |  25.9 |  23.1 |  25.2 |  0.68 |  0.52 |  0.68 |  0.59 |  0.40 |  0.59 |  0.46 |  0.27 |  0.48
$k_{BSpair}$ |  31.3 |  27.8 |  27.5 |  29.7 |  25.3 |  27.1 |  27.3 |  22.5 |  25.9 |  0.76 |  0.54 |  0.70 |  0.66 |  0.38 |  0.61 |  0.50 |  0.26 |  0.49
$k_{anal}$ |  29.9 |  29.2 |  27.9 |  29.0 |  28.1 |  27.5 |  27.6 |  26.2 |  27.0 |  0.83 |  0.58 |  0.79 |  0.73 |  0.42 |  0.68 |  0.56 |  0.29 |  0.54
Table 2:  Quantitative evaluations using no-Reference image quality scores. Method |  NIQE$\downarrow$ |  PIQE$\downarrow$
---|---|---
Blind[19] | 9.8 | 47.5
Non-blind Methods |  [11] |  RL |  Wie- ner |  [11] |  RL |  Wie- ner
$k_{blind}$ | 8.9 | 9.7 | 10.5 | 40.8 | 43.4 | 43.9
$k_{BSpair}$ | 7.5 | 9.3 | 8.9 | 38.7 | 38.1 | 42.3
$k_{anal}$ | 5.9 | 5.56 | 7.9 | 33.9 | 25.5 | 39.7

For the real deblurred images, since ground truth is not available, we use no-
reference image quality scores such as naturalness image quality evaluator
(NIQE) [20] and perception based image quality evaluator (PIQE) [21] for
quantitative evaluation. These scores are inversely correlated to the quality
of image. The scores of different methods are given in Table 2. The score of
each method is its average score computed over 14 different deblurred images
from steering rates 60 deg/sec and 50 deg/sec. It can be observed that non-
blind deblurring methods again perform better than the state-of-the-art blind
deblurring method of [19]. The quality of deblurred image using PSF estimated
from our proposed method is better than using the PSF from the blind method.
RL deconvolution output has less ringing and less noise which is reflected as
lower scores (higher quality) in some cases.

On an Intel Xeon e5-2630v4 processor $@$ 2.2GHz, the blind deblurring method
[19] processes a single frame in about 10 minutes. With a known PSF, and by
using a single core of the processor, the processing time of the non-blind
deblurring methods, Krishnan et al. [11], RL and Wiener filter are 0.3 sec,
0.24 sec, and 0.04 sec, respectively. Even by using only a single core, the
Wiener filter method delivers real-time performance (0.04 sec/frame).

For further speed up, it is possible to use multiple cores to parallelise
deblurring of several frames. Using 24 parallel workers, the processing time
of different non-blind methods is given in Tables 3 and 4. Table 3 gives the
processing time for a real-time blurred video input (assuming 30 frames/sec)
while Table 4 gives the processing time for real-time visualization of
deblurred results of an already available blurred video (assuming 312 frames).
Using parallel processing, Wiener filter achieves excellent real-time
performance (0.014 sec/im-age) for a real-time video input. For the case of
312 frames, both the Wiener filter and Krishnan et al. [11] achieve real-time
performance.

In summary, PSF estimated using our proposed methods followed by Wiener
deconvolution is optimal both in terms of deblurring quality and real-time
processing requirement. The PSF for each steering rate can be computed in
advance using our proposed methods and can be stored in a Look Up Table (LUT).
For a given steering rate of the input video, the corresponding PSF can be
obtained from LUT and used in conjunction with the Wiener filter to get a
real-time deblurred video output.

_Additional Remarks_ : The ability to generate real blur-sharp pairs
corresponding to a given steering rate has significant implication for the
deep learning literature too. The availability of IR datasets for deblurring
is quite scarce which precludes the use of deep learning methods. Importantly
our approach can be used to generate a large number of blur-sharp pairs to
emulate realistic motion blur in gimbal-based systems which can then be used
to train deep networks.

Table 3: Processing time for a real-time video input. Method |  Execution time
---|---
|  per 30 frames |  per frame
Krishnan et al. [11] | 2.8 sec | 0.09 sec
RL deconv. | 2.25 sec | 0.075 sec
Wiener deconv. | 0.434 sec |  0.014 sec
Table 4: Processing time for real -time visualisation of an already available video. Method |  Execution time
---|---
|  per 312 frames |  per frame
Krishnan et al. [11] | 13.4 sec |  0.04 sec
RL deconv. | 21.6 sec | 0.069 sec
Wiener deconv. | 3.35 sec |  0.0107 sec

##  6 Conclusions

In this work, we dealt with the problem of real-time motion deblurring in
gimbal-based systems deployed for surveillance and security applications. Our
comprehensive studies on a real dataset captured from an actual gimbal-based
system revealed that existing methods fall significantly short of delivering
real-time deblurring performance. Towards this end, we model the motion blur
either using a low steering rate dataset or analytically using the knowledge
of camera motion and intrinsic parameters. In our method we use the estimated
kernels in conjunction with non-blind deblurring methods to deliver excellent
performance both in terms of deblurring quality and real-time processing. The
blur-sharp pairs thus generated for creation of realistic datasets can be
leveraged by deep learning networks too.

## References

  * [1] A. N. Rajagopalan, Rama Chellappa: Motion Deblurring: Algorithms and Systems, Cambridge University Press, USA (2014)
  * [2] Orest Kupyn, Volodymyr Budzan, Mykola Mykhailych, Dmytro Mishkin and Jiri Matas: DeblurGAN: Blind Motion Deblurring Using Conditional Adversarial Networks, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2018)
  * [3] Whyte, Oliver and Sivic, Josef and Zisserman, Andrew and Ponce, Jean: Non-uniform deblurring for shaken images, International journal of computer vision (IJCV) 98 (2), pp. 168-186 (2012)
  * [4] Köhler, Rolf and Hirsch, Michael and Mohler, Betty and Schölkopf, Bernhard and Harmeling, Stefan: Recording and playback of camera shake: Benchmarking blind deconvolution with a real-world database, Proceedings of the European Conference on Computer Vision (ECCV), pp. 27-40 (2012)
  * [5] A.N. Rajagopalan and S. Chaudhuri. A block-shift variant blur model for recovering depth from defocused images, Proceedings of the IEEE International Conference on Image Processing (ICIP), pp. 636–639 (1995)
  * [6] Richardson W.: Bayesian-based iterative method of image restoration. Journal of the Optical Society of America A 62, pp. 55–59 (1972)
  * [7] Lucy L.: Bayesian-based iterative method of image restoration, Journal of Astronomy 79, pp. 745–754 (1974)
  * [8] A. A. Sarawade and N. N. Charniya: Infrared Thermography and its Applications: A Review, 3rd International Conference on Communication and Electronics Systems (ICCES), pp. 280-285 (2018)
  * [9] Antoni Rogalski: Infrared detectors: status and trends, Journal of Progress in Quantum Electronics 27(2), pp. 59-210 (2003)
  * [10] Purna Chandra Rao, A.N. Rajagopalan and Guna Seetharaman: Harnessing motion blur to unveil splicing. IEEE Transactions on Information Forensics and Security, vol. 9, pp. 583-595 (2014)
  * [11] Krishnan, Dilip and Fergus, Rob: Fast Image Deconvolution using Hyper-Laplacian Priors, Proceedings of the Advances in Neural Information Processing Systems (NIPS), pp. 1033-1041 (2009)
  * [12] D. Zoran and Y. Weiss: From learning models of natural image patches to whole image restoration, Proceedings of International Conference on Computer Vision, pp. 479-486 (2011)
  * [13] S. Cho, Jue Wang and S. Lee: Handling outliers in non-blind image deconvolution, Proceedings of International Conference on Computer Vision, pp. 495-502 (2011)
  * [14] Fergus, Rob and Singh, Barun and Hertzmann, Aaron and Roweis, Sam T and Freeman, William T: Removing camera shake from a single photograph, ACM transactions on graphics (TOG) 25(3), pp. 787-794 (2006)
  * [15] Dong, Jiangxin and Pan, Jinshan and Su, Zhixun and Yang, Ming-Hsuan: Blind image deblurring with outlier handling, Proceedings of the IEEE International Conference on Computer Vision (ICCV), pp. 2478-2486 (2017)
  * [16] Yan, Yanyang and Ren, Wenqi and Guo, Yuanfang and Wang, Rui and Cao, Xiaochun: Image deblurring via extreme channels prior, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 4003-4011 (2017)
  * [17] Liu, Yuhang and Dong, Wenyong and Gong, Dong and Zhang, Lei and Shi, Qinfeng: Deblurring Natural Image Using Super-Gaussian Fields, Proceedings of the European Conference on Computer Vision (ECCV) (2018)
  * [18] F. Wen and R. Ying and Y. Liu and P. Liu and T. -K. Truong: A Simple Local Minimal Intensity Prior and An Improved Algorithm for Blind Image Deblurring, IEEE Transactions on Circuits and Systems for Video Technology (2020)
  * [19] Jin, Meiguang and Roth, Stefan and Favaro, Paolo: Normalized Blind Deconvolution, Proceedings of the European Conference on Computer Vision (ECCV) (2018)
  * [20] Mittal, A., R. Soundararajan, and A. C. Bovik: Making a Completely Blind Image Quality Analyzer, IEEE Signal Processing Letters 22(3), pp. 209–212 (2013)
  * [21] N. Venkatanath, D. Praneeth, Bh. M. Chandrasekhar, S. S. Channappayya, and S. S. Medasani: Blind Image Quality Evaluation Using Perception Based Features, Proceedings of the 21st National Conference on Communications (NCC), Piscataway, NJ: IEEE, (2015)
  * [22] Maitreya Suin, Kuldeep Purohit, A. N. Rajagopalan: Spatially-Attentive Patch-Hierarchical Network for Adaptive Motion Deblurring, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3606-3615 (2020)
  * [23] Hongguang Zhang, Yuchao Dai, Hongdong Li, Piotr Koniusz: Liu, H., Qin, J., Fu, Z. et al.: Fast simultaneous image super-resolution and motion deblurring with decoupled cooperative learning, Journal of Real-Time Image Processing 17, pp. 1787–1800 (2020)
  * [24] Kuldeep Purohit, Anshul B. Shah, and A.N. Rajagopalan, “Bringing alive blurred moments”, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1-10 (2019)
  * [25] Kuldeep Purohit and A. N. Rajagopalan, “Region-adaptive dense network for efficient motion deblurring”, AAAI conference on Artificial Intelligence, (2020)
  * [26] Rafael C. Gonzalez, Richard Eugene Woods : Digital image processing, Prentice Hall (2002)


# Encoding of lexical tone in self-supervised models of spoken language

Gaofei Shen1 Michaela Watkins2 Afra Alishahi1 Arianna Bisazza3 Grzegorz
Chrupała1
1 Tilburg University 2 University of Amsterdam 3 University of Groningen
{g.shen<EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS>

###### Abstract

Interpretability research has shown that self-supervised Spoken Language
Models (SLMs) encode a wide variety of features in human speech from the
acoustic, phonetic, phonological, syntactic and semantic levels, to speaker
characteristics. The bulk of prior research on representations of phonology
has focused on segmental features such as phonemes; the encoding of
suprasegmental phonology (such as tone and stress patterns) in SLMs is not yet
well understood. Tone is a suprasegmental feature that is present in more than
half of the world’s languages. This paper aims to analyze the tone encoding
capabilities of SLMs, using Mandarin and Vietnamese as case studies. We show
that SLMs encode lexical tone to a significant degree even when they are
trained on data from non-tonal languages. We further find that SLMs behave
similarly to native and non-native human participants in tone and consonant
perception studies, but they do not follow the same developmental trajectory.

Encoding of lexical tone in self-supervised models of spoken language

Gaofei Shen1 Michaela Watkins2 Afra Alishahi1 Arianna Bisazza3 Grzegorz
Chrupała1 1 Tilburg University 2 University of Amsterdam 3 University of
Groningen {g.shen<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>

##  1 Introduction

Explaining the inner workings of self-supervised models of written and spoken
language has been the focus of much recent work. Transformer-based (Vaswani et
al., 2017) written language models have been shown to encode many types of
linguistic information (Conneau et al., 2018; Hewitt and Manning, 2019). The
analysis of self-supervised Spoken Language Models (SLMs) is also gaining
traction: architectures such as wav2vec2 (Baevski et al., 2020) and HuBERT
(Hsu et al., 2021) have been shown to encode linguistic information at the
phonetic, phonological, syntactic and semantic levels of human speech without
labeled data (Abdullah et al., 2021; Ma et al., 2021; de Seyssel et al., 2022;
Bartelds et al., 2022; Martin et al., 2023; Shen et al., 2023; Pasad et al.,
2024).

The majority of research on representations of phonetic and phonological
information in SLMs focuses on the segmental level. Segmental refers to units
of speech that do not spread but remain localized. Phonemes (e.g. vowels and
consonants) are the smallest abstract units of sound that help to distinguish
one unit from another (e.g. pat vs bat). Suprasegmental, in contrast, refers
to features that are not necessarily limited to single units, but can spread
across multiple phonemes or phrases. Examples include tone, stress patterns,
and intonation, which can all entail syllable and phrase level changes Singh
and Fu (2016). The representation of suprasegmental information in SLMs is
important to study, as it is one of the main distinguishing features of speech
compared to text: spoken utterances use suprasegmental cues to convey
information that is sometimes not explicitly marked in a corresponding written
sentence.111Mandarin does not explicitly mark tone in writing, but Vietnamese
does. As a first step, in this work, we focus on lexical tone as a highly
constrained, relatively well-understood example of a suprasegmental feature.

We firstly examine to what extent SLMs trained on tonal and non-tonal
languages encode tone information in their internal representations. We find
that SLMs are capable of capturing tonal information, regardless of whether
they are trained on tonal or non-tonal languages.

Secondly, we investigate the impact of supervised fine-tuning on the automatic
speech recognition (ASR) task. We find that fine-tuning enhances tone
representations for models trained on tonal languages, but reduces them for
models trained on non-tonal languages.

Thirdly, we explore whether SLMs exhibit the same perceptual patterns as
native and non-native human listeners. We find that models show patterns
similar to humans in discrimination of Mandarin tones and consonants, but find
no evidence that they follow a similar developmental trajectory.

##  2 Tones

Estimates suggest that more than 60% of the world’s languages use some degree
of tonal contrast (Yip, 2002). The primary focus of this work is on lexical
tone, the process by which lexical items are distinguished from one another
primarily by pitch cues (Chen et al., 2022). We focus primarily on pitch cues:
while there are suggestions that articulatory cues alone are not sufficient
for tonal distinction in languages such as Vietnamese, with socio-linguistic
register also being important if present in the language in question Brunelle
(2009), this is still subject to further research in perception and production
studies to validate as the body of work remains small, particularly regarding
perception. We therefore state here that non-tonal phonemic units (e.g.
vowels, consonants) can be defined primarily by non-pitch articulatory cues,
such as vowel height, voicing, and duration. In contrast, tonal units make use
of pitch cues, with F0 (fundamental frequency) contour usually considered to
be the primary cue (Rhee et al., 2021). In ambiguous contexts, other pitch
cues can be used in combination with non-pitch cues such as amplitude, voice
quality (e.g. breathy vs creaky), and spectral tilt (Rhee et al., 2021).

The Tone Bearing Unit (TBU) is the segment containing tone; this is typically,
although not always, found on the syllable level. In the case of the primary
language investigated in this paper, Mandarin, syllables are morpheme based
(as in, Mandarin syllables are morphemes). Thus in Mandarin every morpheme
contains obligatory tone, and is in of itself the TBU (Jun and Kubozono,
2020).

Figure 1: F0 contours of the four Mandarin tones measured from pronunciations
recorded by one of the co-authors, a native speaker of Mandarin Chinese. The
four syllables are pronounced in isolation (notation: mā T1, má T2, mǎ T3, mà
T4).

We compare SLMs trained on non-tonal languages as well as three fully lexical
tonal languages: Mandarin, Cantonese and Vietnamese. The models are tested
primarily on Mandarin data. Mandarin demonstrates full tonality, with tone
found on each morpheme (Hyman, 2018), and has been widely studied for tone
perception as well as acquisition. Secondarily, we also test on data from
another lexical tone language, Vietnamese, to assess if our results
generalize.

Mandarin Chinese is typically described as containing four (lexical) tones and
one neutral tone that only occurs in unstressed syllables (Wu et al., 2020).
The tones are conventionally assigned the labels 1-4 (T1-4); Figure 1
illustrates the four Mandarin tones.

Since one morpheme (one character) corresponds to one tone in Mandarin
Chinese, we can use the Pinyin transcription to obtain our tone labels easily
(see Figure 1 for notations); for example: 今 天 天 气 很 好
Jīn tiān tiān qì hěn hǎo
T1 T1 T1 T4 T3 T3
‘The weather today is very good.’ The tone label corresponds to the tone of
the character when it is pronounced in isolation (base form). However,
Mandarin features tone sandhi, i.e. the tone assigned to individual morphemes
can change in pronunciation based on the tone of the adjacent morpheme (sandhi
form). One instance of tone sandhi rules in Mandarin is T3 sandhi (Chen,
2000): if two T3 (‘dipping’) tones occur next to one another, the first will
adjust to T2 (‘rising’) to avoid two consecutive T3 tones, as can be seen in
examples 2 and 2, after Chen (2000). Tone labels obtained from Pinyin
transcriptions only take the base form into account.

小 |
---|---
xiǎo |
T3 |
‘small’ |
小 | 狗 |
---|---|---
xiǎo | gǒu |
T3 | T3 | base form
T2 | T3 | sandhi form
‘small | dog’ |

The primary pitch cue that distinguishes the individual Mandarin tones from
each other is F0; however, secondary pitch cues are also present such as voice
quality and spectral tilt Belotel-Grenie and Grenie (1994); Huang (2020).

Vietnamese also has obligatory tones on every syllable, similar to Mandarin’s
morphemic TBU (Kirby, 2011). We adhere to the eight tone system described by
Kirby (2011) in our experiment setup.

Cantonese is a Sinitic language related to Mandarin, and also features lexical
tone, with six tonal distinctions (Zee, 1991) as opposed to Mandarin’s four.

##  3 Related work

The present paper builds both on works interpreting the inner workings of SLMs
and on experiments on perception of aspects of human speech.

###  3.1 Analyzing SLMs

The transformer architecture (Vaswani et al., 2017) has dominated the SLM
realm. Researchers have developed many methods to analyze the inner-working of
these models. Pasad et al. (2021) provide an overview on the variety of
linguistic featurs encoded by self-supervised SLMs. The models tend to follow
an autoencoder-like behavior with the middle layers showing the strongest
encoding of a variety of linguistic features.

More recently, research has focused on specific properties of the input audio
that is being encoded by the models. Martin et al. (2023) tested whether SLMs
can distinguish between voiced and voiceless consonants. Shen et al. (2023)
showed that self-supervised as well as visually-supervised SLMs are capable of
encoding syntactic properties to some extent. Some prior works in the field
have touched on the encoding of suprasegmental features in SSL speech models.
Bartelds et al. (2022) showed the hidden state activations of SLMs are capable
of capturing intonational and durational information on the phrase level,
indicating that they can encode non-segmental information to a significant
degree.

Many recent interpretability studies are inspired by psycholinguistics and
child language development research. With the rise of probing and other
interpretability methods, researchers replicated experimental paradigms in
psychology and linguistics to better understand the capabilities of models
compared to humans. For example, Wilcox et al. (2023) tested text language
models using psycholinguistic experimental paradigms, showing that they are
capable of learning syntactic dependencies with relatively little input data.

On the speech side, Lavechin et al. (2023) presented evidence that self-
supervised SLMs can develop limited language-specific perception. Cruz Blandón
et al. (2023) proposed comparing model behavior using checkpoints in the SLM
pre-training process with data in child language development. They showed that
computational language models can be a valuable resource in testing or
confirming linguistic theories in the language development field. The
methodology is mostly concerned with the overall learning of the language
model in the output stage. Our work contributes to the explanation of the
inner workings of SLMs.

###  3.2 Human perception experiments

In terms of tone perception, F0 is a clear primary cue (Ryant et al., 2014b;
Rhee et al., 2021; Chen et al., 2022), but other secondary pitch cues serve to
assist when speech is ambiguous and/or disrupted. Given that conversational
speech contains non-trivial speech recognition difficulties such as e.g. tone
sandhi and coarticulation, individual variation, and context omission Ryant et
al. (2014b), secondary cues play a role in the distinction of tones. An
example of this is voice quality, where for example lowering F0 (introducing
‘creaky’ voice) increased perceptual saliency for T3, whereas T1 and T4
accuracy decreased and T2 remained unaffected (Huang, 2020; Chai, 2019; Kuang,
2017). This emphasises the fact that F0 does not operate in isolation, but
that covariation between pitch and voice quality is inherent in Mandarin.
Spectral cues (e.g. amplitude differences, spectral tilt) have also been
suggested to be sufficient for adult speakers in tone production, while
children are thought to hyperarticulate the tonal differences in speech Rhee
et al. (2021).

Suprasegmental cues appear to be preferred in experiments that compare
segmental and suprasegmental cues against each other. Human infants are more
sensitive to suprasegmental cues, with even newborns showing the same
preference (Mehler et al., 1988; Nazzi et al., 1998). Several studies observe
that tonal sensitivity develops earlier than perception of vowels and
consonants (Xi et al., 2009; Yeung et al., 2013), with sensitivity to non-
native tonal distinctions remaining longer than perception of non-tonal non-
native phoneme categories Liu and Kager (2014); Shi et al. (2017).

Comparing vowels, consonants and tones, Singh et al. (2015) show that Mandarin
learning children’s sensitivity to consonants and vowels develop at a similar
rate and shows departure from tones. The effect of tone mispronunciation is
much larger than that of vowel or consonant mispronunciation for toddlers, but
the pattern is reversed in preschoolers (Singh et al., 2015).

###  3.3 Automatic classification of tones

Automatic tone classification in Mandarin traditionally uses F0 contour and
mel-frequency cepstral coefficients (MFCC) features. Advances in deep learning
brought improvements in performance of tone classification. Ryant et al.
(2014a) compare MFCC features and F0 contour as input to a neural tone
classifier. MFCC features, while not explicitly encoding the F0 contour
information, achieve an error rate of 15.56% for Tone 1-4 classification. The
combination of MFCC features and F0 contours extracted with different methods
did not see an improvement in the classifier’s performance, indicating that
the classifier was able to extract F0 contour from the MFCC features, or it
was able to predict Mandarin tones reliably without F0 contour information.
However, it is possible that the classifier was able to exploit associations
between specific phonemes strings and tone labels, and hence avoid learning to
detect tone based on pitch and voice quality cues.

After the introduction of self-supervised SLMs, Yuan et al. (2021) fine-tuned
an English pre-trained wav2vec2 model (Baevski et al., 2020) for Mandarin tone
classification and achieved a tone error rate of 6% on the same dataset as
(Ryant et al., 2014a). Clearly, SLMs can handle the task of classifying
Mandarin lexical tone with labeled fine-tuning. The aim of the present paper
is not to compete with the existing implementations of Mandarin tone
classifiers; rather we aim to uncover the tone encoding capabilities that
emerge without explicit supervision.

##  4 Methodology

We use a number of wav2vec2-based (Baevski et al., 2020) models pre-trained
and fine-tuned on datasets of different languages for our investigation. As
examples of tonal languages, we choose Mandarin, Vietnamese and Cantonese,
whereas English and French serve as non-tonal language examples. The models
trained in the languages above are then tested on test data from Mandarin and
Vietnamese.

To examine the encoding of tone, we train linear probing classifiers on the
hidden state activations extracted from the aforementioned models for every
morpheme in our testing datasets. 222We release the codebase for our
experiments at https://github.com/techsword/tone-encoding-in-speech-model

|  | Size (hours) |
---|---|---|---
Training language | Tonality | Pre-training | Fine-tuning | Speech type
English (Baevski et al., 2020) | Non-tonal | 960 | 960 | Read
French (Parcollet et al., 2023) | Non-tonal | 1,000 | - | Read
Mandarin (Lu and Chen, 2022) | Tonal | 1,000 | 178 | Read
Vietnamese (Nguyen, 2021) | Tonal | 13,000 | 250 | YouTube audio/Read
Cantonese (Huang and Mak, 2023) | Tonal | 2,800 | - | Spotaneous + Read
Table 1: Description of the datasets used in pre-training/fine-tuning models.

###  4.1 Datasets

##### Training data.

We examine SLMs that were trained on datasets of the following languages:

Mandarin pre-trained with AISHELL-2 (Du et al., 2018) and fine-tuned with
AISHELL-1 (Bu et al., 2017). English pre-trained and fine-tuned with
LibriSpeech (Panayotov et al., 2015). Vietnamese pre-trained with unlabelled
YouTube audio and fine-tuned with the VLSP dataset for ASR (Nguyen, 2021).
Cantonese pre-trained on a combined dataset of older Cantonese adult speech
and YouTube audio (Huang and Mak, 2023). French pre-trained on MLS French
(Pratap et al., 2020). Table 1 summarizes the characteristics of these
datasets.

##### Test data.

We primarily use the Mandarin Chinese THCHS-30 dataset (Wang and Zhang, 2015)
for testing models’ encoding of Mandarin tone. THCHS-30 consists of 30 hours
of Mandarin speech recorded in a laboratory environment. The dataset is
transcribed into both Chinese characters and Mandarin Pinyin. We also obtain
character-level forced alignment with the Charsiu aligner (Zhu et al., 2022).

To test the generalizability of our results, we also use the Vietnamese VIVOS
dataset (Luong and Vu, 2016), which consists of 15 hours of Vietnamese read
speech recorded in a laboratory environment. The dataset is transcribed into
Vietnamese orthography. We then convert the transcription into International
Phonetic Alphabet (IPA) with tone labels with vPhon (Kirby, 2008). We use the
Montreal Forced Aligner (McAuliffe et al., 2017) to obtain a syllable-level
forced alignment.

##### Pre-training data.

For the experiments on SLM’s learning trajectory and perceptual patterns (see
Section 5.3), we pre-train SLMs from scratch on the following datasets:

  * •

MAGICDATA (Magic Data Technology Co., 2019), containing 755 hours of read
Mandarin Chinese. The dataset was pre-split into a 712-hour training set and a
28-hour validation set.

  * •

LibriSpeech (Panayotov et al., 2015), see details in Table 1. We split a
subset of the LibriSpeech dataset into a 710-hour training set and a 29-hour
validation set.

###  4.2 Spoken Language Models

##### Architecture.

With the exception of the Cantonese model, all models investigated in this
paper are based on the base configuration of wav2vec2 (Baevski et al., 2020).
Wav2vec2-base consists of five convolutional feature encoder and twelve
transformer layers. The feature encoder processes the audio waveform input
into latent speech representations, and the transformer layers encode the
feature encoder output into contexual representations. The wav2vec2-base
models has 95M parameters. The Cantonese model uses the wav2vec2-conformer
architecture with 180M parameters.

##### Training objectives.

The fully self-supervised _pre-training_ objective in wav2vec2 consists in
discriminating between the matched and unmatched segment representations for a
masked portion of the latent speech representation. The ASR _fine-tuning_
objective consists in transcribing the audio input into output tokens in the
orthography of the target language and is realized by adding a linear layer on
top of a pre-trained wav2vec2 model.

##### Checkpoints.

For the experiments in Section 5.3 we pre-train two SLMs with the fairseq
toolkit (Ott et al., 2019) on LibriSpeech for English and MAGICDATA for
Mandarin; we train both models for 85,000 steps using 8 Nvidia A100-40GB GPU
with update frequency = 8 to simulate training with 64 GPUs. Each model
finished training in approximately 96 hours. We save checkpoints every 5,000
steps.

###  4.3 Probing classifiers

##### Preprocessing.

We follow previous work (Ryant et al., 2014a) in removing segments transcribed
with the neutral tone from the Mandarin tone classification task. Mandarin
neutral tones primarily appear in unstressed syllables (cf. Section 2) and
hence are more susceptible to variations.

##### Generating classifier input.

We extract the hidden state activations of models as a response to audio
samples in the test data. We average-pool the hidden state output
corresponding to the duration of individual syllables to obtain a vector using
forced alignment timestamps. The resulting 768-dimensional vectors are input
to the classifiers. To control for the influence of lexical cues on tone
detection, we construct an exclusive train-test-split such that phoneme
strings appearing in the test set do not appear in the training set. This
setup prevents the probing classifier from exploiting associations between
tones and phoneme sequences. We employ a randomized 80:20 train-test split
with the split sizes shown in Table 2.

Language | Split | Samples
---|---|---
Mandarin | Train | 223,851
Mandarin | Test | 45,772
Vietnamese | Train | 124,248
Vietnamese | Test | 29,629
Table 2: Train/test splits for the tone probing classifier, for the Mandarin
and Vietnamese data.

##### F0 and MFCC baselines.

We closely follow Ryant et al. (2014a) and use F0 contours and 40-dimensional
mel-frequency cepstral coefficients (MFCC) features as baselines. We use
Librosa (McFee et al., 2023) to extract the MFCC features and Praat (Jadoul et
al., 2018; Boersma and Weenink, 2021) to extract the F0 contours from the
audio samples. We then find the center frame for each word using the alignment
timestamps and concatenate all frames in a 21 frame window (10-1-10) for both
F0 and MFCC features. We end up with a 21-dimensional vector for F0 contours
and 840-dimensional vector for MFCC features as our baseline classifier
inputs.

##### Text baseline.

In addition to audio baselines, we also include a text-based transformer model
in our comparison. BERT (Devlin et al., 2019) serves as a reference point to
show how much information is encoded in the speech signal as opposed to what
can be guessed from pure text. We use a Chinese pre-trained
BERT333https://huggingface.co/bert-base-chinese that encodes Chinese
characters into vectors. We extract per-word hidden state outputs with a
resulting 768-dimensional vector.

##### Tone classifiers.

We use the syllable activation vectors as input to a Ridge linear classifier
that predicts the lexical tone of the input morpheme. We select the final
model via 5-fold cross-validation, and report the classification accuracy on
the test split. The regularization strength $\alpha$ was tuned for values
$\\{10^{n}~{}|~{}n\in\\{-4,-3,-2,-1,0,1,2\\}\\}$.

##### Consonant classifiers.

When comparing tone to consonant classification, we employ the same classifier
setup for consonant and replicate the perception experiment in Wang and Chen
(2020) in Section 5.3.1. Since we only investigate consonants that appear
solely in the onset position and the rest of the phonemes are not relevant to
our task, we use the same syllable vectors as above instead of obtaining a
phoneme vector with using phoneme level alignment. We construct exclusive
train-test-splits that contain unique rimes (nucleus + coda) of the syllables.
Specific details of the train/test split for this experiment can be found in
Table 3.

Split | Samples
---|---
Train | 92,413
Test | 15,688
Table 3: Train/test split for the consonant probing classifier, for the
Mandarin data.

##  5 Results

In this section, we present a series of experiments for analyzing the encoding
of tone in SLMs.

###  5.1 Tone encoding across languages

Figure 2: Classification accuracy of Mandarin lexical tones using layer-wise
representations from models pre-trained on tonal and non-tonal languages.

Figure 3: Classification accuracy of Vietnamese lexical tones with hidden-
state activations from models pre-trained on tonal and non-tonal languages.

Figure 3 shows the tone classification accuracy using the layer-wise
representations of all models pre-trained on non-tonal (left) versus tonal
(right) languages. We see that all layers of all models perform better than
the F0 and MFCC baselines, which themselves outperform the text-based BERT
baseline. The classification accuracy for tonal language models is overall
higher, and increases in the higher layers of the models. Models trained on
non-tonal languages also show substantial encoding of tone; but remarkably,
there is a substantial drop in classification accuracy in their final layers
while the corresponding decrease is much less pronounced in tonal language
models.

We repeat the tone classification experiment for Vietnamese tones. Results in
Figure 3 show the Cantonese model performs slightly better than the English
model, especially towards the later layers; the Mandarin model, however,
patterns similar to the English model. This is could be due to the fact that
Mandarin has fewer tonal contrasts than Vietnamese and Cantonese (cf. Section
2), or, more likely, that Vietnamese uses different acoustic cues such as
phonation type and voice quality in tonal perception than f0 contours or
height in Mandarin (Brunelle, 2009).

Studies on human participants show that speakers of other tonal languages can
perform better at identifying Mandarin lexical tones compared to non-tonal
language speakers (So and Best, 2010), but other literature suggest that
listeners are more sensitive to specific cues in tonal perception that are
present in their native languages (DiCanio, 2012; Schaefer and Darcy, 2014).
The SLMs we tested show the former pattern for Mandarin tone classification.
Regarding Vietnamese tones, the result is more equivocal suggesting that
Cantonese tone representations generalize to Vietnamese to some extent, while
Mandarin ones do not.

###  5.2 Impact of ASR fine-tuning

We examine how fine-tuning for ASR impacts the encoding of tone in SLMs. Since
tonal information is crucial for correctly transcribing tonal language input,
SLMs fine-tuned for tonal languages are expected to perform better at our tone
classification task. Figure 5 compares the tone classification accuracy of the
English and Mandarin models, pre-trained only (left) versus pre-trained and
then fine-tuned (right); Figure 5 shows the corresponding results for English
and Vietnamese.

We find that fine-tuning affects the encoding of tone for non-tonal vs tonal
language models in opposite ways: classification accuracy benefits from fine-
tuning for Mandarin, but is harmed by it for the English model. The same
pattern holds for English vs Vietnamese on the Vietnamese tone data in Figure
5.

These results likely reflect the fact that ASR fine-tuning encourages the SLM
to increase its specialization in identifying the language-specific
information needed to output the written form of the language. Tonal
information may not contribute much to this objective in non-tonal languages,
and thus fine-tuning would tend to remove it. In tonal-language ASR however,
tone information may be crucial to correctly transcribe the input audio, for
example, when disambiguating Mandarin syllables that consist of the same
segmental phonemes and only differ in tone, in order to output the correct
Chinese character.

Figure 4: Classification accuracy of Mandarin lexical tones using layer-wise
representations from models pre-trained and fine-tuned on Mandarin and
English.

Figure 5: Classification accuracy of Vietnamese lexical tones using layer-wise
representations from models pre-trained and fine-tuned on Vietnamese and
English.

###  5.3 Comparison to human perception

In this section we report the results motivated by tone and consonant
perception patterns in humans.

####  5.3.1 Learning trajectory

Children have a higher sensitivity to tone than consonant distinctions early
on. For children speaking a non-tonal language, this sensitivity towards tone
continues longer than sensitivity towards non-native segmental features, i.e.
consonants and vowels (Shi et al., 2017; Liu and Kager, 2014).

Here we aim to determine the corresponding learning trajectory in SLMs by
testing them during pre-training. Figure 6 shows the accuracy of classifying
Mandarin consonants and tones in the best performing layer of SLMs trained on
English and Mandarin as a function of the number of training steps.

Although we observe classification accuracy of the SLMs quickly surpasses the
F0 and MFCC baselines after 10,000 steps, we do not detect an obvious
difference in the overall pattern between the case of consonants and tones.
This suggest that SLMs do not follow the same differential trajectory as
children, at least as measured via our methodology.

Figure 6: Classification accuracy of Mandarin lexical tones versus consonants
for models pre-trained on English and Mandarin. Figure 7: Binary
classification accuracy for Mandarin tonal pairs, for English and Mandarin
models. Figure 8: Classification accuracy for Mandarin consonant groups, for
English and Mandarin models. The F0 baseline with its much lower
classification accuracy is omitted from this figure for clarity.

####  5.3.2 Tone and consonant contrasts

Non-native speakers can have difficulty distinguishing between T2-T3 and T1-T4
tone pairs in Mandarin (Hao, 2012). While native adult listeners have shown
near perfect tone identification accuracy (So and Best, 2010; Tsukada and
Kondo, 2019), there is literature documenting native Mandarin-learning
toddlers and also some adults having more difficulty in distinguishing the
tone pair T2-3 compared to other combinations of Mandarin lexical tone. Huang
and Johnson (2011) showed that the tone pair T2-3 is the most confusable for
both native speakers of American English and Mandarin while T1-4 also being
confusable for native Mandarin speakers. We investigate this pattern in pre-
trained SLMs via a dedicated probing experiment, using the final (85,000
steps) checkpoint of the pre-trained models in Section 5.3.1. As can be seen
in Figure 7, tone pairs T1-4 and T2-3 show the largest differences in the best
classification accuracy between the Mandarin and English models, which roughly
matches human perceptual pattern.

We complement the results on the development of tone contrast with a parallel
experiment on those Mandarin consonant contrasts which are challenging for
speakers of English. Each member of a contrasting group is perceived as the
same phoneme by English speakers due to perceptual assimilation (Wang and
Chen, 2020). Table 4 displays the resulting mapping to English phoneme
categories.

| Mandarin | English
---|---|---
Group | Pinyin | IPA | Alphabet | IPA
1 | sh, x |  s, C | sh |  S
2 | ch, zh, q |  t sh, t s, tCh | ch |  tS
3 | s, z, c |  s, ts, tsh | s |  s
Table 4: Perceptual mapping of Mandarin consonants onto English consonants
(Wang and Chen, 2020).

Figure 8 shows that accuracy for consonant groups 2 and 3 match closely for
the two models. Group 1 shows a discrepancy, possibly due to the potential
mapping of Mandarin x /C/ into two English consonants sh /S/ and z /z/, as
hypothesized by Wang and Chen (2020).

##  6 Conclusion

We analyze the tone encoding capabilities of spoken language models trained on
three tonal and two non-tonal languages, using classifier probes with data
from two tonal languages: Mandarin and Vietnamese. We find that SLMs trained
on either tonal or non-tonal languages encode tonal information in Mandarin
and Vietnamese to a significant degree.

We also find that fine-tuning for the speech recognition task enhances the
tone encoding capabilities of models trained on tonal languages but reduces
them for models trained on non-tonal languages. While we see evidence
suggesting that the learning trajectories of SLMs in pre-training do not
follow the same developmental trajectories found in human language
acquisition, we find that SLMs show patterns similar to that of human
listeners in tone and consonant perception experiments.

While this paper focused on investigating the encoding of lexical tones with
Mandarin and Vietnamese as case studies, Mandarin and Vietnamese are in no way
representative of all the diverse tonal languages that exist in the world. The
encoding of other suprasegmental features such as stress patterns and
intonation is equally important to study in future work. We hope that
explaining self-supervised models of spoken languages provides a unique
perspective for us to contribute to a better understanding of how languages
work. Given the rise of SLMs trained with multilingual datasets (Conneau et
al., 2021; Radford et al., 2022), it would be interesting to investigate if
the multilingual SLMs encode segmentals and suprasegmentals from different
languages more robustly. This paper serves as a starting point in the research
of the encoding of suprasegmental cues in spoken language models.

##  7 Limitations

We selected SLMs based on the wav2vec2 architecture in our experimental
design, but we acknowledge that the training data of the models selected is
quite varied in their size and quality (noisy vs clean speech) as described in
Section 4.1. This is partially due to the scarce availability of (high-
quality) speech data for underrepresented languages, especially the many tonal
languages of the world. Hence SLMs pre-trained on monolingual datasets of
these languages are also sparse. The Vietnamese wav2vec2 model (Nguyen, 2021)
was trained on a significantly larger amount of data (13k hours) than the
other models tested (around 1000 hours). It is possible that in addition to
the inclusion of tonal languages in training, the amount of training data also
played a role in increasing the tonal encoding capabilities of SLMs. However,
literature has shown that more training data does not always have a positive
impact on the models performance if the additional data is noisy (Parcollet et
al., 2023). At the same time, we note that the Cantonese model (Huang and Mak,
2023), in addition to being pre-trained on a larger dataset, is also different
in architecture. Additionally, our use of two read speech datasets as test
data does not fully reflect the linguistic diversity of different accents and
dialects in Mandarin and Vietnamese. Future work needs to go wider and deeper
in both model architecture and dataset diversity in order to uncover more
generalizable patterns in different languages.

##  8 Acknowledgements

This publication is part of the project InDeep: Interpreting Deep Learning
Models for Text and Sound (with project number NWA.1292.19.399) of the
National Research Agenda (NWA-ORC) program. Funding by the Dutch Research
Council (NWO) is gratefully acknowledged. We also thank SURF (www.surf.nl) for
the support in using the National Supercomputer Snellius.

## References

  * Abdullah et al. (2021) Badr M. Abdullah, Iuliia Zaitova, Tania Avgustinova, Bernd Möbius, and Dietrich Klakow. 2021.  How Familiar Does That Sound? Cross-Lingual Representational Similarity Analysis of Acoustic Word Embeddings.
  * Baevski et al. (2020) Alexei Baevski, Henry Zhou, Abdelrahman Mohamed, and Michael Auli. 2020.  Wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations.
  * Bartelds et al. (2022) Martijn Bartelds, Wietse de Vries, Faraz Sanal, Caitlin Richter, Mark Liberman, and Martijn Wieling. 2022.  Neural representations for modeling variation in speech.  _Journal of Phonetics_ , 92:101137.
  * Belotel-Grenie and Grenie (1994) Agnes Belotel-Grenie and Michel Grenie. 1994.  Phonation types analysis in standard chinese.  In _3rd International Conference on Spoken Language Processing, ICSLP 1994_ , pages 343–346. The International Society for Computers and Their Applications (ISCA).
  * Boersma and Weenink (2021) Paul Boersma and David Weenink. 2021.  Praat: Doing phonetics by computer [Computer program].
  * Brunelle (2009) Marc Brunelle. 2009.  Tone perception in northern and southern vietnamese.  _Journal of Phonetics_ , 37(1):79–96.
  * Bu et al. (2017) Hui Bu, Jiayu Du, Xingyu Na, Bengu Wu, and Hao Zheng. 2017.  AISHELL-1: An Open-Source Mandarin Speech Corpus and A Speech Recognition Baseline.
  * Chai (2019) Yuan Chai. 2019.  The source of creak in Mandarin utterances.  In _Proceedings of ICPhS 2019_.
  * Chen (2000) Matthew Y. Chen. 2000.  _Tone Sandhi: Patterns across Chinese Dialects_.  Cambridge Studies in Linguistics. Cambridge University Press, Cambridge.
  * Chen et al. (2022) Yue Chen, Yingming Gao, and Yi Xu. 2022.  Computational Modelling of Tone Perception Based on Direct Processing of f0 Contours.  _Brain Sciences_ , 12(3):337.
  * Conneau et al. (2021) Alexis Conneau, Alexei Baevski, Ronan Collobert, Abdelrahman Mohamed, and Michael Auli. 2021.  Unsupervised Cross-Lingual Representation Learning for Speech Recognition.  In _Interspeech 2021_ , pages 2426–2430. ISCA.
  * Conneau et al. (2018) Alexis Conneau, German Kruszewski, Guillaume Lample, Loïc Barrault, and Marco Baroni. 2018.  What you can cram into a single $&!#* vector: Probing sentence embeddings for linguistic properties.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 2126–2136, Melbourne, Australia. Association for Computational Linguistics.
  * Cruz Blandón et al. (2023) María Andrea Cruz Blandón, Alejandrina Cristia, and Okko Räsänen. 2023.  Introducing Meta-analysis in the Evaluation of Computational Models of Infant Language Development.  _Cognitive Science_ , 47(7):e13307.
  * de Seyssel et al. (2022) Maureen de Seyssel, Marvin Lavechin, Yossi Adi, Emmanuel Dupoux, and Guillaume Wisniewski. 2022.  Probing phoneme, language and speaker information in unsupervised speech representations.  In _Interspeech_.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.
  * DiCanio (2012) Christian T. DiCanio. 2012.  Cross-linguistic perception of Itunyoso Trique tone.  _Journal of Phonetics_ , 40(5):672–688.
  * Du et al. (2018) Jiayu Du, Xingyu Na, Xuechen Liu, and Hui Bu. 2018.  AISHELL-2: Transforming Mandarin ASR Research Into Industrial Scale.
  * Hao (2012) Yen-Chen Hao. 2012.  Second language acquisition of Mandarin Chinese tones by tonal and non-tonal language speakers.  _Journal of Phonetics_ , 40(2):269–279.
  * Hewitt and Manning (2019) John Hewitt and Christopher D. Manning. 2019.  A Structural Probe for Finding Syntax in Word Representations.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4129–4138, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Hsu et al. (2021) Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. 2021.  HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units.
  * Huang and Mak (2023) Ranzo Huang and Brian Mak. 2023.  Wav2vec 2.0 ASR for Cantonese-Speaking Older Adults in a Clinical Setting.  In _INTERSPEECH 2023_ , pages 4958–4962. ISCA.
  * Huang and Johnson (2011) Tsan Huang and Keith Johnson. 2011.  Language Specificity in Speech Perception: Perception of Mandarin Tones by Native and Nonnative Listeners.  _Phonetica_ , 67(4):243–267.
  * Huang (2020) Yaqian Huang. 2020.  Different attributes of creaky voice distinctly affect Mandarin tonal perception.  _The Journal of the Acoustical Society of America_ , 147(3):1441–1458.
  * Hyman (2018) Larry M. Hyman. 2018.  What tone teaches us about language.  _Language_ , 94(3):698–709.
  * Jadoul et al. (2018) Yannick Jadoul, Bill Thompson, and Bart de Boer. 2018.  Introducing Parselmouth: A Python interface to Praat.  _Journal of Phonetics_ , 71:1–15.
  * Jun and Kubozono (2020) Sun-Ah Jun and Haruo Kubozono. 2020.  Asian Pacific Rim.  In Carlos Gussenhoven and Aoju Chen, editors, _The Oxford Handbook of Language Prosody_ , Oxford Handbooks. Oxford University Press, Oxford, New York.
  * Kirby (2008) James Kirby. 2008.  vPhon: A Vietnamese phonetizer (version 2.1.1).
  * Kirby (2011) James P. Kirby. 2011.  Vietnamese (hanoi vietnamese).  _Journal of the international phonetic association_ , 41(3):381–392.
  * Kuang (2017) Jianjing Kuang. 2017.  Covariation between voice quality and pitch: Revisiting the case of Mandarin creaky voice.  _The Journal of the Acoustical Society of America_ , 142(3):1693–1706.
  * Lavechin et al. (2023) Marvin Lavechin, Maureen De Seyssel, Marianne Métais, Florian Metze, Abdelrahman Mohamed, Hervé Bredin, Emmanuel Dupoux, and Alejandrina Cristia. 2023.  Statistical learning models of early phonetic acquisition struggle with child-centered audio data.
  * Liu and Kager (2014) Liquan Liu and René Kager. 2014.  Perception of tones by infants learning a non-tone language.  _Cognition_ , 133(2):385–394.
  * Lu and Chen (2022) Ke-Han Lu and Kuan-Yu Chen. 2022.  A context-aware knowledge transferring strategy for CTC-based ASR.
  * Luong and Vu (2016) Hieu-Thi Luong and Hai-Quan Vu. 2016.  A non-expert Kaldi recipe for Vietnamese speech recognition system.  In _Proceedings of the Third International Workshop on Worldwide Language Service Infrastructure and Second Workshop on Open Infrastructures and Analysis Frameworks for Human Language Technologies (WLSI/OIAF4HLT2016)_ , pages 51–55, Osaka, Japan. The COLING 2016 Organizing Committee.
  * Ma et al. (2021) Danni Ma, Neville Ryant, and Mark Liberman. 2021.  Probing Acoustic Representations for Phonetic Properties.  In _ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 311–315.
  * Magic Data Technology Co. (2019) Ltd. Magic Data Technology Co. 2019.  MAGICDATA Mandarin Chinese Read Speech Corpus.
  * Martin et al. (2023) Kinan Martin, Jon Gauthier, Canaan Breiss, and Roger Levy. 2023.  Probing Self-supervised Speech Models for Phonetic and Phonemic Information: A Case Study in Aspiration.  In _INTERSPEECH 2023_ , pages 251–255. ISCA.
  * McAuliffe et al. (2017) Michael McAuliffe, Michaela Socolof, Sarah Mihuc, Michael Wagner, and Morgan Sonderegger. 2017.  Montreal Forced Aligner: Trainable Text-Speech Alignment Using Kaldi.  In _Interspeech 2017_ , pages 498–502. ISCA.
  * McFee et al. (2023) Brian McFee, Matt McVicar, Daniel Faronbi, Iran Roman, Matan Gover, Stefan Balke, Scott Seyfarth, Ayoub Malek, Colin Raffel, Vincent Lostanlen, Benjamin Van Niekirk, Dana Lee, Frank Cwitkowitz, Frank Zalkow, Oriol Nieto, Dan Ellis, Jack Mason, Kyungyun Lee, Bea Steers, Emily Halvachs, Carl Thomé, Fabian Robert-Stöter, Rachel Bittner, Ziyao Wei, Adam Weiss, Eric Battenberg, Keunwoo Choi, Ryuichi Yamamoto, CJ Carr, Alex Metsai, Stefan Sullivan, Pius Friesch, Asmitha Krishnakumar, Shunsuke Hidaka, Steve Kowalik, Fabian Keller, Dan Mazur, Alexandre Chabot-Leclerc, Curtis Hawthorne, Chandrashekhar Ramaprasad, Myungchul Keum, Juanita Gomez, Will Monroe, Viktor Andreevitch Morozov, Kian Eliasi, Nullmightybofo, Paul Biberstein, N. Dorukhan Sergin, Romain Hennequin, Rimvydas Naktinis, Beantowel, Taewoon Kim, Jon Petter Åsen, Joon Lim, Alex Malins, Darío Hereñú, Stef Van Der Struijk, Lorenz Nickel, Jackie Wu, Zhen Wang, Tim Gates, Matt Vollrath, Andy Sarroff, Xiao-Ming, Alastair Porter, Seth Kranzler, Voodoohop, Mattia Di Gangi, Helmi Jinoz, Connor Guerrero, Abduttayyeb Mazhar, Toddrme2178, Zvi Baratz, Anton Kostin, Xinlu Zhuang, Cash TingHin Lo, Pavel Campr, Eric Semeniuc, Monsij Biswal, Shayenne Moura, Paul Brossier, Hojin Lee, and Waldir Pimenta. 2023\.  Librosa/librosa: 0.10.1.  Zenodo.
  * Mehler et al. (1988) J. Mehler, P. Jusczyk, G. Lambertz, N. Halsted, J. Bertoncini, and C. Amiel-Tison. 1988.  A precursor of language acquisition in young infants.  _Cognition_ , 29(2):143–178.
  * Nazzi et al. (1998) T. Nazzi, J. Bertoncini, and J. Mehler. 1998.  Language discrimination by newborns: Toward an understanding of the role of rhythm.  _Journal of Experimental Psychology. Human Perception and Performance_ , 24(3):756–766.
  * Nguyen (2021) Thai Binh Nguyen. 2021.  Vietnamese end-to-end speech recognition using wav2vec 2.0.
  * Ott et al. (2019) Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019.  Fairseq: A Fast, Extensible Toolkit for Sequence Modeling.  pages 48–53, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Panayotov et al. (2015) Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. 2015.  Librispeech: An ASR corpus based on public domain audio books.  In _2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 5206–5210.
  * Parcollet et al. (2023) Titouan Parcollet, Ha Nguyen, Solene Evain, Marcely Zanon Boito, Adrien Pupier, Salima Mdhaffar, Hang Le, Sina Alisamir, Natalia Tomashenko, Marco Dinarelli, Shucong Zhang, Alexandre Allauzen, Maximin Coavoux, Yannick Esteve, Mickael Rouvier, Jerome Goulian, Benjamin Lecouteux, Francois Portet, Solange Rossato, Fabien Ringeval, Didier Schwab, and Laurent Besacier. 2023.  LeBenchmark 2.0: A Standardized, Replicable and Enhanced Framework for Self-supervised Representations of French Speech.
  * Pasad et al. (2024) Ankita Pasad, Chung-Ming Chien, Shane Settle, and Karen Livescu. 2024.  What Do Self-Supervised Speech Models Know About Words?
  * Pasad et al. (2021) Ankita Pasad, Ju-Chieh Chou, and Karen Livescu. 2021.  Layer-wise Analysis of a Self-supervised Speech Representation Model.
  * Pratap et al. (2020) Vineel Pratap, Qiantong Xu, Anuroop Sriram, Gabriel Synnaeve, and Ronan Collobert. 2020.  MLS: A Large-Scale Multilingual Dataset for Speech Research.  In _Interspeech 2020_ , pages 2757–2761.
  * Radford et al. (2022) Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2022.  Robust Speech Recognition via Large-Scale Weak Supervision.
  * Rhee et al. (2021) Nari Rhee, Aoju Chen, and Jianjing Kuang. 2021.  Going beyond F0: The acquisition of Mandarin tones.  _Journal of Child Language_ , 48(2):387–398.
  * Ryant et al. (2014a) Neville Ryant, Malcolm Slaney, Mark Liberman, Elizabeth Shriberg, and Jiahong Yuan. 2014a.  Highly Accurate Mandarin Tone Classification In The Absence of Pitch Information.  In _Speech Prosody 2014_ , pages 673–677. ISCA.
  * Ryant et al. (2014b) Neville Ryant, Jiahong Yuan, and Mark Liberman. 2014b.  Mandarin tone classification without pitch tracking.  In _2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 4868–4872, Florence, Italy. IEEE.
  * Schaefer and Darcy (2014) Vance Schaefer and Isabelle Darcy. 2014.  Lexical function of pitch in the first language shapes cross-linguistic perception of Thai tones.  _Laboratory Phonology_ , 5(4):489–522.
  * Shen et al. (2023) Gaofei Shen, Afra Alishahi, Arianna Bisazza, and Grzegorz Chrupała. 2023.  Wave to Syntax: Probing spoken language models for syntax.  In _INTERSPEECH 2023_ , pages 1259–1263.
  * Shi et al. (2017) Rushen Shi, Jun Gao, André Achim, and Aijun Li. 2017.  Perception and Representation of Lexical Tones in Native Mandarin-Learning Infants and Toddlers.  _Frontiers in Psychology_ , 8.
  * Singh and Fu (2016) Leher Singh and Charlene S. L. Fu. 2016.  A New View of Language Development: The Acquisition of Lexical Tone.  _Child Development_ , 87(3):834–854.
  * Singh et al. (2015) Leher Singh, Hwee Hwee Goh, and Thilanga D. Wewalaarachchi. 2015.  Spoken word recognition in early childhood: Comparative effects of vowel, consonant and lexical tone variation.  _Cognition_ , 142:1–11.
  * So and Best (2010) Connie K. So and Catherine T. Best. 2010.  Cross-language Perception of Non-native Tonal Contrasts: Effects of Native Phonological and Phonetic Influences.  _Language and speech_ , 53(Pt 2):273–293.
  * Tsukada and Kondo (2019) Kimiko Tsukada and Mariko Kondo. 2019.  The Perception of Mandarin Lexical Tones by Native Speakers of Burmese.  _Language and Speech_ , 62(4):625–640.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017.  Attention Is All You Need.
  * Wang and Zhang (2015) Dong Wang and Xuewei Zhang. 2015.  THCHS-30 : A Free Chinese Speech Corpus.
  * Wang and Chen (2020) Xinchun Wang and Jidong Chen. 2020.  The Acquisition of Mandarin Consonants by English Learners: The Relationship between Perception and Production.  _Languages_ , 5(2):20.
  * Wilcox et al. (2023) Ethan Gotlieb Wilcox, Richard Futrell, and Roger Levy. 2023.  Using Computational Models to Test Syntactic Learnability.  _Linguistic Inquiry_ , pages 1–44.
  * Wu et al. (2020) Yaru Wu, Martine Adda-Decker, and Lori Lamel. 2020.  Mandarin lexical tones: A corpus-based study of word length, syllable position and prosodic position on duration.  pages 1908–1912.
  * Xi et al. (2009) Jie Xi, Wei JIANG, Linjun Zhang, and Hua Shu. 2009.  Categorical Perception of VOT and Lexical Tones in Chinese and the Developmental Course.  _Acta Psychologica Sinica_ , 41:572–579.
  * Yeung et al. (2013) H. Henny Yeung, Ke Heng Chen, and Janet F. Werker. 2013.  When does native language input affect phonetic perception? The precocious case of lexical tone.  _Journal of memory and language_ , 68(2):123–139.
  * Yip (2002) Moira Yip. 2002.  _Tone_.  Cambridge Textbooks in Linguistics. Cambridge University Press, Cambridge.
  * Yuan et al. (2021) Jiahong Yuan, Neville Ryant, Xingyu Cai, Kenneth Church, and Mark Liberman. 2021\.  Automatic recognition of suprasegmentals in speech.
  * Zee (1991) Eric Zee. 1991.  Chinese (Hong Kong Cantonese).  _Journal of the International Phonetic Association_ , 21(1):46–48.
  * Zhu et al. (2022) Jian Zhu, Cong Zhang, and David Jurgens. 2022.  Phone-to-audio alignment without text: A Semi-supervised Approach.  _IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_.