# SpeechTokenizer: Unified Speech Tokenizer for
Speech Large Language Models

Xin Zhang , Dong Zhang∗, Shimin Li, Yaqian Zhou, Xipeng Qiu††footnotemark:
School of Computer Science, Fudan University
Shanghai Key Laboratory of Intelligent Information Processing, Fudan
University

<EMAIL_ADDRESS>
<EMAIL_ADDRESS>

https://0nutation.github.io/SpeechTokenizer.github.io/ Equal contribution,
order is randomCorresponding author

###### Abstract

Current speech large language models build upon discrete speech
representations, which can be categorized into semantic tokens and acoustic
tokens. However, existing speech tokens are not specifically designed for
speech language modeling. To assess the suitability of speech tokens for
building speech language models, we established the first benchmark,
SLMTokBench. Our results indicate that neither semantic nor acoustic tokens
are ideal for this purpose. Therefore, we propose SpeechTokenizer, a unified
speech tokenizer for speech large language models. SpeechTokenizer adopts the
Encoder-Decoder architecture with residual vector quantization (RVQ). Unifying
semantic and acoustic tokens, SpeechTokenizer disentangles different aspects
of speech information hierarchically across different RVQ layers. Furthermore,
We construct a Unified Speech Language Model (USLM) leveraging
SpeechTokenizer. Experiments show that SpeechTokenizer performs comparably to
EnCodec in speech reconstruction and demonstrates strong performance on the
SLMTokBench benchmark. Also, USLM outperforms VALL-E in zero-shot Text-to-
Speech tasks. Code and models are available at
https://github.com/ZhangXInFD/SpeechTokenizer/.

##  1 Introduction

Large language models (OpenAI, 2023; Touvron et al., 2023) have demonstrated
remarkable performance on various natural language processing tasks. This has
inspired numerous works to build speech language models (Borsos et al., 2022),
which have achieved significant breakthroughs across various speech processing
tasks (Wang et al., 2023; Zhang et al., 2023; Rubenstein et al., 2023; Dong et
al., 2023). A key commonality among these works is the utilization of discrete
speech representations. Current discrete speech representations can be
categorized into two types: semantic tokens and acoustic tokens (Borsos et
al., 2022). Semantic tokens are typically from self-supervised pre-trained
models with masked language modeling as training objective (Hsu et al., 2021;
Baevski et al., 2020; Chung et al., 2021). Derived through k-means clustering
on representations from a specific intermediate layer, semantic tokens are
depicted as sequences with one-dimensional structure. Acoustic tokens can be
extracted from neural audio codecs with reconstruction as training objective
(Zeghidour et al., 2021; Défossez et al., 2022). Utilizing residual vector
quantization (RVQ) with hierarchical quantizers for discretization, acoustic
tokens are represented as matrices consisting of two dimensions: timesteps and
quantizers.

Building upon two speech tokens, there exist three modeling approaches for
speech language models, as listed in Table 1: i) Semantic language models are
constructed using semantic tokens and employ an external unit vocoder for
speech synthesis. (Lakhotia et al., 2021; Zhang et al., 2023; Hassid et al.,
2023). While capturing semantically accurate content, their speech generation
results in poor quality and a loss of acoustic details. ii) Acoustic language
models are built on acoustic tokens. Taking VALL-E (Wang et al., 2023) as an
example, despite achieving impressive zero-shot text-to-speech (TTS)
capabilities, it still suffers from problems like inaccurate content, due to
the complex information within acoustic tokens. iii) Hierarchical speech
language models comprise semantic token language models and acoustic token
language models, which capture content information and acoustic details
respectively (Borsos et al., 2022; Rubenstein et al., 2023; Dong et al.,
2023). This structure shows promising results in both content and speech
quality, but the multi-stage modeling approach is more complex, leading to
several drawbacks such as error accumulation and slower processing speed.
Additionally, there is significant information redundancy between semantic
tokens and acoustic tokens, which introduces unnecessary modeling
complexities. An ideal speech language model should not only accurately model
content, but also generating diverse, high-quality speech, while maintaining
an architecture of elegant simplicity. Correspondingly, ideal speech tokens
should meet the following two key characteristics: i) Strong alignment with
text; ii) Effective preservation of speech information.

Figure 1: Illustration of information composition of different discrete speech
representations. Speech tokens are represented as colored circles and
different colors represent different information.

However, existing speech tokens are not explicitly designed for speech
language modeling, and there has been no exploration into their suitability
for building speech language models. To address this gap, we build Speech
Language Model Token Benchmark (SLMTokBench
111https://github.com/0nutation/SLMTokBench), to assess the suitability of
speech tokens for constructing speech language models. Our evaluation reveals
that semantic tokens exhibit a high alignment with text while losing some
information in speech, such as timbre. Acoustic tokens excel in preserving
speech information effectively but do not demonstrate a strong alignment with
text. With these observations, we aim to build a specialized speech tokens
designed for speech language models by unifying semantic and acoustic tokens.
Specifically, we can conduct information disentanglement in the RVQ structure
of acoustic tokens, enabling the first RVQ quantizer to generate tokens
containing content information, similar to semantic tokens, while the
subsequent quantizers complement the remaining paralinguistic information, as
illustrated in Figure 1.

| Accurate Content | High-quality Speech | Single Tokenzier
---|---|---|---
Semantic LM | $\surd$ | $\times$ | $\surd$
Acoustic LM | $\times$ | $\surd$ | $\surd$
Hierarchical LM | $\surd$ | $\surd$ | $\times$
USLM (ours) | $\surd$ | $\surd$ | $\surd$

Table 1: Comparision between different speech language models. Semantic LM
refers to semantic language models. Acoustic LM refers to acoustic language
models. Hierarchical LM refers to hierarchical speech language models. USLM
refers to our unified speech language model.

Speech representation disentanglement approaches mainly adopt a parallel
disentanglement strategy. This entails simultaneously feeding speech into
separate content encoders, speaker encoders, etc., in order to extract
different representations and attain information disentanglement (Qian et al.,
2020; Polyak et al., 2021). However, this approach heavily relies on prior
knowledge and introduces strong inductive biases, leading to increased
complexity and a potential oversight of certain speech information, such as
prosody. Similar to VQVC (Wu & Lee, 2020), we adopt residual structure to
achieve serial disentanglement of speech. For neural audio codec, the
cumulative representations from all RVQ layers encompass all dimensions of
speech information. By ensuring the first layer representations contain
content-related information, the subsequent residual layers will naturally
fill in the gaps with remaining details—specifically, modeling the
paralinguistic information.

With the above motivation, we propose SpeechTokenizer, a unified speech
tokenizer for speech large language models. SpeechTokenizer adopts the
Encoder-Decoder architecture with residual vector quantization. Unifying
semantic and acoustic tokens, SpeechTokenizer disentangles different aspects
of speech information hierarchically across different RVQ layers. By employing
a semantic teacher to guide the first RVQ quantizer, the first layer tokens
can effectively capture content information. With residual structure, the
subsequent quantizers complement the remaining paralinguistic information.
Building upon SpeechTokenizer, we build a Unified Speech Language Model
(USLM222https://github.com/0nutation/USLM) consisting of autoregressive and
non-autoregressive models. Experimental results show that SpeechTokenizer
performs comparably to EnCodec (Défossez et al., 2022) in speech
reconstruction and demonstrates strong performance on the SLMTokBench
benchmark. The USLM notably outperforms VALL-E (Wang et al., 2023) in zero-
shot Text-to-Speech (TTS) tasks.

Our contributions include the following:

  * •

We propose SpeechTokenizer, which is specially designed for speech large
language models and unifiy the semantic and acoustic tokens through
disentangling different aspects of speech information hierarchically.

  * •

We establish SLMTokBench, the first benckmark to assess the suitability of
speech tokens for constructing speech language models.

  * •

We construct a unified speech language model based on SpeechTokenizer, which
outperforms VALL-E on zero-shot TTS task.

  * •

We analyze the impact of information disentanglement and demonstrate
SpeechTokenizer’s potential for extending to unseen languages.

##  2 SLMTokBench: Speech Language Model Token Benchmark

To build powerful speech language models, discrete speech representations
should possess the following two key characteristics: i) Strong alignment with
text; ii) Effective preservation of speech information. Building on this
premise, we establish speech Language Model Token Benchmark (SLMTokBench) to
assess the suitability of speech tokens for constructing speech language
models. In this benchmark, we evaluate the alignment between speech tokens and
text by estimating their mutual information. We assess preservation of speech
information within speech tokens by evaluating the quality of resynthesized
speech. SLMTokBench is available on https://github.com/0nutation/SLMTokBench.

###  2.1 Text Alignment Evaluation

Mutual Information is a measure of how much the knowledge of one variable
reduces the uncertainty about another variable. It quantifies the level of
dependence or relationship between the variables. We evaluate the degree of
text alignment by estimating the mutual information between speech tokens and
text. For notation, $X$ denotes discrete speech representations; $Y$ denotes
text; $\mathcal{I}(X;Y)$ denotes the mutual information; test dataset is
denoted as $\mathcal{D}=\\{(x_{i},y_{i})\\}_{i=1}^{N}$ and $\theta$ denotes
the downstream model

A measure of mutual information between variable $X$ and $Y$ can be formulated
as:

$\displaystyle\mathcal{I}(X;Y)=\int_{X}\int_{Y}log\frac{P(X,Y)}{P(X)P(Y)}$ (1)

where $P(X)$ and $P(Y)$ are the marginal distributions of $X$ and $Y$
respectively, and $P(X,Y)$ denotes the joint distribution of X and Y.

The variational contrastive log-ratio upper bound (vCLUB) (Cheng et al., 2020)
of mutual information is defined by:

$\displaystyle\mathcal{I}(X;Y)=E_{p(X,Y)}[\log
q_{\theta}(Y|X)]-E_{p(X)p(Y)}[\log q_{\theta}(Y|X)]$ (2)

where $q_{\theta}(Y|X)$ is the variational distribution to approximate the
ground-truth probability $P(Y|X)$ and can be parameterized by the downstream
model $\theta$.

With test dataset $\mathcal{D}$, $\mathcal{I}(X;Y)$ has an unbiased estimation
as:

$\displaystyle\mathcal{\hat{I}}(X;Y)=\frac{1}{N^{2}}\sum_{i=1}^{N}\sum_{j=1}^{N}[\log
q_{\theta}(y_{i}|x_{i})-\log q_{\theta}(y_{j}|x_{i})]$ (3)

The downstream model is a vanilla 2-layer 1024-unit BLSTM optimized by CTC
loss on characters and it takes speech tokens as inputs. Specifically, for
each discrete representation, we first establish an embedding matrix, which
can be either randomly initialized or derived from the k-means centroid matrix
or vector quantization codebooks obtained during the discretization process.
We use the embedding matrix to embed the discrete representations and obtain
continuous representations, which are then fed into the downstream models. We
train the downstream model on LibriSpeech train-clean-100 subset and use dev-
clean subset for estimating mutual information. We also calculate the word
error rate (WER) on the test set. For downstream model training, we configure
the training setup with a batch size of 32, a learning rate of 1e-4, and a
total of 200k global steps.

###  2.2 Information Preservation Evaluation

To evaluate the preservation of speech information in discrete speech
representations, we convert speech tokens back to speech and evaluate
resynthesized speech by automatic metrics on content and timbre. We train a
unit-HiFIGAN (Polyak et al., 2021) on LibriSpeech dataset to convert HuBERT
units to waveform. Notably, to avoid interference from additional information,
we don’t supply any speaker information during training. For Encodec tokens,
we used the Encodec decoder to directly produce the waveform. Content
preservation is evaluated by computing the WER through transcribing the
resynthesized speech using the Whisper en-medium model (Radford et al., 2023).
Timbre preservation is evaluated by utilizing WavLM-TDNN (Chen et al., 2022)
to calculate speaker similarity between the synthesized and groundtruth
speech. We randomly sample 300 speech samples from LibriSpeech test set for
evaluation.

###  2.3 Comparing Semantic $\&$ Acoustic Tokens

We use HuBERT L9 units to representat semantic tokens and EnCodec codes to
represent acoustic tokens. As shown in Table 3, semantic tokens achieve high
mutual information with text but their resynthesized speech has low speaker
similarity. That means semantic tokens align well with the text and are
excellent for capturing the meaning and context of spoken words, but they can
lack the paralinguistic features of speech, such as timbre, or prosody.
Acoustic tokens achieve low WER and high speaker similarity for resynthesized
speech but have low mutual information with text. That shows acoustic tokens
do tend to capture the rich details of the speech signal, such as pitch,
timbre, but they often lack the semantic interpretability that would align
them closely with the corresponding text.

##  3 SpeechTokenizer

###  3.1 Model Structure

We represent a single-channel audio signal as a sequence
$x\in\mathcal{R}^{T}$, where $T=d\cdot f_{sr}$ denotes the length with
duration $d$ at sampling rate $f_{sr}$. As illustrated in Figure 2,
SpeechTokenizer model consists of four main components:

  * •

an encoder network $E$ that takes an audio as input and output a latent
representation $z$.

  * •

a residual vector quantizer $Q$ that compresses representation $z$ to $z_{q}$
from a set of finite codebooks.

  * •

a decoder network $G$ that produces the reconstructed audio signal $\hat{x}$
from the compressed representation $z_{q}$.

  * •

a semantic teacher model $S$ that provides semantic representation $s$ to
guide the residual vector quantization process during training.

SpeechTokenizer is trained end-to-end with reconstruction loss, together with
a perceptual loss in the form of discriminators and semantic distillation loss
as training objective. We adopt HuBERT (Hsu et al., 2021) as semantic teacher
model in this study.

Encoder & Decoder Architecture Inspired by SoundStream (Zeghidour et al.,
2021) and EnCodec (Défossez et al., 2022), our model employs a convolutional-
based encoder-decoder architecture. The encoder is constructed as a sequential
series of components: starting with a 1D convolutional layer featuring $C$
channels and a kernel size of 7, followed by a set of $B$ residual
conventional blocks. Each block is composed of two dilated convolutions with
dilation rate of $(1,1)$ and kernel size of $(3,1)$ and a skip-connection,
followed by a strided convolutional down-sampling layer, with a kernel size
$K$ of the twice the stride $R$. Whenever down-sampling, the number of
channels is doubled. Unlike in EnCodec that the convolution blocks are
followed by a two-layer LSTM, we use BiLSTM to augment the semantic modeling
ability. A final 1D convolution layer with a kernel size of 7 is used to set
the dimensionality of embeddings to $D$. We use $C=32,B=4$ and $(2,4,5,8)$ as
strides. We use ELU (Clevert et al., 2016) as a non-linear activation either
layer normalization (Ba et al., 2016) or weight normalization (Salimans &
Kingma, 2016).The decoder mirrors the encoder and uses transposed convolutions
and LSTM instead of stride convolutions and BiLSTM, with the strides in
reverse order as in the encoder. The decoder outputs the final audio signal.

Residual Vector Quantizer We use Residual Vector Quantizer (RVQ) to quantize
the encoder output. Vector quantization (VQ) involves selecting the code
closest to the input vector from a codebook of a given size as the
quantization representation. Residual vector quantizer cascades $N_{q}$ layers
of VQ to alleviate the conflict between codebook size and quantization
performance. The unquantized input vector is passed through the first
quantizer and quantization residuals are computed. These residuals are then
iteratively quantized by a sequence of $N_{q}-1$ additional quantizers. For
each residual step $i\in\\{1,\cdots,N_{q}\\}$, $z_{i}$ denotes the quantize-
before vector and $q_{i}$ denotes quantize-after vector and $c_{i}$ denotes
the corresponding code index.

We follow the same training procedure as EnCodec. The code selected for each
input is updated using an exponential moving average with a decay of 0.99, and
codes which have not been assigned any input vector for several batches are
replaced with input vectors randomly sampled within current batch. Straight-
through-estimator (Bengio et al., 2013) is used to compute the gradient of
encoder, e.g. as if the quantization step was the identity function during the
backward phase. Finally, a commitment loss, consisting of the MSE between the
input of the quantizer and its output, with gradient only computed with
respect to its input, is added to the overall training loss.

Discriminator We use the same dicriminators as HiFi-Codec Yang et al. (2023)
that consist of three discriminators: A multi-scale STFT-based (MS-STFT)
discriminator; a multi-period discriminator (MPD) and a multi-scale
discrinator (MSD). The MS-STFT discriminator utilizes networks with identical
structures that operate on multi-scaled complex-valued STFT, where the real
and imaginary parts are concatenated. For each sub-network, it is composed of
a 2D convolutional layer (using kernel size $3\times 8$ with 32 channels),
followed by 2D convolutions with increasing dilation rates in the time
dimension (1, 2 and 4), and a stride of 2 over the frequency axis. A final 2D
convolution with kernel size $3\times 3$ and stride $(1,1)$ provide the final
prediction. For MSD and MPD, we follow the same settings as in HiFiGAN (Kong
et al., 2020) but adjust the channel number to align the discriminator’s
parameters more closely with that of MS-STFT.

Figure 2: Illustration of SpeechTokenizer framework.

###  3.2 Semantic Distillation

To achieve a hierarchical modeling of diverse information across different RVQ
layers, we employ semantic guidance for the first quantizer, enabling it to
capture content information. Leveraging a residual structure enables the
subsequent quantizers to complement the remaining paralinguistic information.

We employ HuBERT (Hsu et al., 2021) as our semantic teacher, as HuBERT is
demonstrated to encompass substantial content information (Mohamed et al.,
2022). We introduce two types of distillation: continuous representation
distillation and pseudo-label prediction.

For continuous representation distillation, we employ the 9th layer HuBERT
representation or the average representation across all HuBERT layers as
semantic teachers. The training objective of this approach can be written as:

$\displaystyle\mathcal{L}_{distll}=\frac{1}{T}\sum_{t=1}^{T}\log{\sigma(\cos{(Aq_{1}^{t},s^{t})})},$
(4)

where $q_{1}^{t}$ and $s^{t}$ respectively denote the quantized output of the
first VQ layer and the semantic teacher representation at timestep t.
$\cos(\cdot)$ is cosine similarity. $T$ denotes the number of time steps and
$A$ is the projection matrix. $\sigma(\cdot)$ denotes sigmoid activation.

For pseudo-label prediction, we adopt HuBERT units as the target label. The
training objective is constructed as:

$\displaystyle\mathcal{L}_{distll}=-\frac{1}{T}\sum_{t=1}^{T}u^{t}\log(\text{Softmax}(Aq_{1}^{t})),$
(5)

where $q_{1}^{t}$ and $u^{t}$ respectively denote the quantized output of the
first VQ layer and the HuBERT unit at timestep t. $T$ denotes the number of
time steps and $A$ is the projection matrix.

###  3.3 Training Objective

Our training approach includes both a reconstruction task and a semantic
distillation task. In the reconstruction task, we employ a GAN objective,
optimizing a combination of a reconstruction term, a discriminative loss term,
and RVQ commitment loss. In the semantic distillation task, the training
objective involves a semantic distillation loss term.

Reconstruction Loss The reconstruction loss comprises a time and a frequency
domain loss. For time domain, we minimize the L1 distance between $x$ and
$\hat{x}$, i.e.

$\displaystyle\mathcal{L}_{t}=\lVert x-\hat{x}\rVert_{1}.$

For frequency domain, we linearly combine the L1 and L2 losses over the mel-
spectrogram using several time scales. Formally,

$\displaystyle\mathcal{L}_{f}=\sum_{i\in
e}\lVert\mathcal{S}_{i}(x)-\mathcal{S}_{i}(\hat{x})\rVert_{1}+\lVert\mathcal{S}_{i}(x)-\mathcal{S}_{i}(\hat{x})\rVert_{2},$

where $\mathcal{S}_{i}$ is a 64-bins mel-spectrogram using a normalized STFT
with window size of $2^{i}$ and hop length of $2^{i}/4,e=5,\cdots,11$ is the
set of scales.

Discriminative Loss The adversarial loss is used to promote perceptual quality
and it is defined as a hinge loss over the logits of the discriminator,
averaged over multiple discriminators and over time. Let $K$ denote the number
of discriminators, the adversarial loss for the discriminators
$\mathcal{L}_{D}$ and for the generator $\mathcal{L}_{g}$ is constructed as
follows:

$\displaystyle\mathcal{L}_{D}=\frac{1}{K}\sum_{k=1}^{K}max(1-D_{k}(x),0)+max(1+D_{k}(\hat{x}),0),$
$\displaystyle\mathcal{L}_{g}=\frac{1}{K}\sum_{k=1}^{K}max(1-D_{k}(\hat{x}),0).$

Additionally, a feature matching loss for the generator is computed as follow:

$\displaystyle\mathcal{L}_{feat}=\frac{1}{KL}\sum_{k=1}^{K}\sum_{l=1}^{L}\frac{\lVert
D_{k}^{l}(x)-D^{l}_{k}(\hat{x})\rVert_{1}}{mean(\lVert
D_{k}^{l}(x)\rVert_{1})},$

where the mean is computed over all dimensions and $L$ is the number of layers
in discriminators.

RVQ Commitment Loss We add a commitment loss $\mathcal{L}_{w}$ between the
pre-quantized value, and its quantized value, without gradient computed for
the quantized value. RVQ commitment loss is defined as:

$\displaystyle\mathcal{L}_{w}=\sum_{i=1}^{N_{q}}\lVert
z_{i}-z_{q_{i}}\rVert_{2}^{2}.$

Generally, the generator is trained to optimize the following loss:

$\mathcal{L}_{G}=\lambda_{t}\mathcal{L}_{t}+\lambda_{f}\mathcal{L}_{f}+\lambda_{g}\mathcal{L}_{g}+\lambda_{feat}\mathcal{L}_{feat}+\lambda_{w}\mathcal{L}_{w}+\lambda_{distill}\mathcal{L}_{distill},$

where $\lambda_{t},\lambda_{f},\lambda_{g},\lambda_{feat},\lambda_{w}$ and
$\lambda_{distill}$ are hyper-parameters used to balance each loss term.

###  3.4 Unified Speech Language Model

Figure 3: Illustration of unified speech language models. AR refers to
autoregressive and NAR refers to non-autoregressive. Speech tokens are
represented as colored circles and different colors represent different
information.

As shown in Figure 3, we can build a unified speech language model upon
SpeechTokenizer. Consisting of autoregressive and non-autoregressive models,
it can hierarchically model information in speech. The autoregressive (AR)
model captures the content information by modeling tokens from the first RVQ
quantizer. The non-autoregressive (NAR) model complements paralinguistic
information for the AR model by generating tokens from the subsequent
quantizers conditioned on the first-layer tokens. We validate the
effectiveness of unified speech language model on zero-shot TTS task. USLM is
available on https://github.com/0nutation/USLM.

The AR model is built upon the first-layer tokens $c_{1}$. Utilizing a
transformer decoder-only architecture $\theta_{AR}$, we approach this
conversion as a casual language modeling task with the phoneme sequence $u$
serving as the prompt for the AR model. The training objective can be
formulated as

$\displaystyle\mathcal{L}_{AR}=-\log\prod_{t=0}^{T}p(c_{1}^{t}|c_{1}^{<t},u;\theta_{AR}).$

The NAR model produces tokens $c_{2:8}$ from the subsequent quantizers. Its
architecture resembles that of the AR model, comprising eight distinct
acoustic embedding layers and output prediction layers. To control the
characteristics of the speaker’s voice, n acoustic prompt $\hat{C}$ is
employed for timbre guidance. The model is conditioned on phoneme sequence
$u$, acoustic prompts $\hat{C}$ and tokens from previous quantizers, leading
to the formulation of the training objective as follows

$\displaystyle\mathcal{L}_{NAR}=-\log\prod_{i=2}^{8}p(c_{i}|c_{<i},\hat{C},u;\theta_{NAR}).$

During inference, we convert text input to phoneme sequence and speech prompt
to speech tokens. They are concatenated to form the prompts for AR and NAR
models. Conditioned on that, the AR model generates first-level tokens, while
the NAR model iteratively produces tokens of subsequent levels. The tokens
generated by the AR and NAR models are then concatenated to construct the
speech token matrix. Finally, we use the SpeechTokenizer decoder to generate
the waveform conditioned on the complete token matrix.

##  4 Experiments

###  4.1 Experimental Setups

Datasets For SpeechTokenizer training, we use LibriSpeech (Panayotov et al.,
2015) dataset. We randomly crop a 3-second segment from the speech samples at
each training iteration. For zero-shot TTS, we train AR and NAR models on the
English subset of Multilingual LibriSpeech (Pratap et al., 2020) dataset,
which contains 44K hours of transcribed speech data derived from LibriVox
audiobooks. We select speech samples with durations ranging from 3 to 14
seconds for training data. The sampling rate is 16KHz for all speech data.

Model For SpeechTokenizer, we introduce the details about model structure in
section 3.1. For zero-shot TTS experiments, AR model and NAR model are both
12-layer Transformer decoders with 16 attention heads, an attention dimension
of 1024 and the FFN dimension of 4096.

Training For SpeechTokenizer, the model are trained on 2 A800 GPUS for 20
epochs with maximum learning rate of 4e-4 and batch size of 20 per GPU. For
Unified Speech Language Model, both AR and NAR models are trained on 8 A800
GPUS for 500k steps with maximum learning rate of 5e-4. The AR model is
trained with batch size of 7500 tokens per GPU, and the NAR model is trained
with batch size of 5000 tokens per GPU.

Baselines We adopt EnCodec (Défossez et al., 2022) as the baseline for
SpeechTokenizer and VALL-E (Wang et al., 2023) as the baseline system for
zero-shot TTS. We train VALL-E under the same dataset and experimental setups
as EnCodec.

###  4.2 Speech Reconstruction Evaluation

We randomly sample 300 speech samples from LibriSpeech test set for speech
reconstruction evaluation. We take into account both subjective and objective
evaluation metrics.

Objective Metrics We use ViSQOL metrics (Hines et al., 2012) to measure the
speech quality. Additionally, we evaluate content accuracy through Word Error
Rate (WER) by transcribing the speech utilizing the Whisper en-medium model
(Radford et al., 2023).

Subjective Metrics We adopt a crowd-sourced methodology inpspired by MUSHRA
protocol (Series, 2014), with a hidden reference but no lowerpass-filtered
anchor, for subjective evaluation. We instruct evaluators to rate the
perceptual quality of the given samples on a scale of 1 to 100.

###  4.3 Unified Speech Language Model Evaluation

We conduct zero-shot TTS evaluation on the VCTK dataset, which comprises 108
speakers. There is no speaker overlap between the training data and VCTK
dataset. For each speaker, we randomly selected a 3s utterance as the prompts
while the textual content of a different utterance is used as the input text.

Objective Metrics We evaluate the TTS systems with speaker similarity and WER.
We evaluate the speaker similarity between the generated speech and the prompt
speech. We calculate the similarity with the following steps: 1) we utilize
WavLM-TDNN to calculate the speaker embedding for the generated speech and the
prompt speech. 2) we calculate the cosine similarity between the normalized
embeddings. We employ Whisper medium model to transcribe the generated speech
and calculate the WER.

Subjective Metrics We determine the Mean Opinion Score (MOS) and Similarity
Mean Opinion Score (SMOS) through human evaluations. MOS reflects the
naturalness of speech, while SMOS assesses the degree of similarity to the
original speaker’s voice. We engaged 12 and 6 native speakers as contributors
for MOS and SMOS evaluations, respectively. MOS and SMOS both span from 1 to
5, with higher values signifying greater speech quality and voice similarity
respectively.

###  4.4 Main Results

Speech Reconstruction

| Objective | Subjective
---|---|---
Tokenizer |  WER$\downarrow$ |  VISQOL$\uparrow$ |  MUSHRA$\uparrow$
Groundtruth | 4.58 | - | 91.46
EnCodec | 5.11 | 4.37 | 79.86
SpeechTokenizer | 5.04 | 4.30 | 90.55

Table 2: Results of speech reconstruction

Table 2 summarizes the results of speech reconstruction experiments. The
SpeechTokenizer achienves lower WER than Encodec, demonstrating its superior
ability to preserve content. Additionally, SpeechTokenizer attains a
comparable VISQOL score but a higher MUSHRA score than EnCodec, which
indicates its stronger capability in generating high-quality speech. We put
samples of speech reconstruction in our demo page
333https://0nutation.github.io/SpeechTokenizer.github.io/.

|  |  | Text Alignment | Information Preservation
---|---|---|---|---
Tokenizer |  | Teacher |  MI$\uparrow$ |  WER${}^{\dagger}\downarrow$ |  WER${}^{\ast}\downarrow$ |  SIM$\uparrow$
Groundtruth |  |  | - | - | 4.58 | 1.0
HuBERT | KM500 | - | 31.2 | 9.88 | 16.26 | 0.77
EnCodec | RVQ-1 | - | 16.5 | 61.52 | 38.34 | 0.92
EnCodec | RVQ-1:8 | - | 23.6 | 30.91 | 5.11 | 0.98
Ablations |  |  |  |  |  |
SpeechTokenizer | RVQ-1 | HuBERT avg | 30.9 | 15.58 | 9.57 | 0.74
SpeechTokenizer | RVQ-1:8 | HuBERT avg | 29.7 | 16.03 | 5.04 | 0.97
SpeechTokenizer | RVQ-1 | HuBERT L9 | 32.9 | 12.68 | 14.17 | 0.73
SpeechTokenizer | RVQ-1:8 | HuBERT L9 | 31.6 | 13.12 | 5.31 | 0.97
SpeechTokenizer | RVQ-1 | HuBERT units | 24.2 | 34.13 | 20.02 | 0.72
SpeechTokenizer | RVQ-1:8 | HuBERT units | 25.1 | 30.71 | 5.84 | 0.95

Table 3: Results on SLMTokBench. MI and WER† refer to mutual information and
word error rate of the downstream model. WER∗ and SIM refer to word error rate
and speaker similarity of resynthesized speech respectively. RVQ-$n$ denotes
the tokens of the $n^{th}$ RVQ layer. RVQ-$n$:$m$ denotes the tokens from the
$n^{th}$ layer to the $m^{th}$ layer.

Performance on SLMTokBench Table 3 displays the performance of SpeechTokenizer
on SLMTokBench. Compared with EnCodec-RVQ-1, SpeechTokenizer-RVQ-1 achieves
higher mutual information between text and lower WER of downstream model. This
suggests that SpeechTokenizer exhibits a stronger alignment with textual
content. Meanwhile, the of resynthesized speech of SpeechTokenizer RVQ-1
tokens achieves lower WER and speaker similarity, indicating its capability to
retain more content-related information while disregarding timbre
characteristics, similar to semantic tokens. The resynthesized speech of
SpeechTokenizer RVQ-1:8 tokens demonstrates low WER and high speaker
similarity, illustrating SpeechTokenizer’s competence in preserving
comprehensive speech information, similar to acoustic tokens. Furthermore, the
speaker similarity of resynthesized speech of SpeechTokenizer RVQ-1 tokens is
notably low, whereas that of SpeechTokenizer RVQ-1:8 tokens is considerably
high. This observation implies that the tokens from subsequent layers
compensate for the timbre information that is discarded by the first layer
tokens.

Zero-shot TTS As shown in Table 4, our USLM demonstrates lower WER than
VALL-E. This result highlights that SpeechTokenizer can contribute to a more
precise modeling of content information. Additionally, the USLM demonstrates
superior speaker similarity, implying that a decoupled information structure
is more conducive to modeling speaker-related information. We put samples of
zero-shot TTS on our demo page
444https://0nutation.github.io/SpeechTokenizer.github.io/.

|  | Objective | Subjective
---|---|---|---
Model | Tokenizer |  WER$\downarrow$ |  SIM$\uparrow$ |  MOS$\uparrow$ |  SMOS$\uparrow$
Groundtruth |  | 1.9 | 0.93 | 4.5 | 3.96
VALL-E | EnCodec | 7.9 | 0.75 | 3.08 | 3.31
USLM | SpeechTokenizer | 6.5 | 0.84 | 3.63 | 3.45

Table 4: Results of zero-shot TTS

##  5 Analysis

###  5.1 Choices of Semantic Teachers

We compare performance of using different semantic teacher representations in
Table 3. When comparing the performance of HuBERT units and HuBERT L9
representations as semantic teachers on SLMTokBench, we find that HuBERT L9
representations perform better in both Text Alignment and Information
Preservation, regardless of whether it’s RVQ-1 or RVQ-1:8. We speculate that
the reason may be that the discrete HuBERT units lose some content information
compared to the continuous representations, thereby providing weaker semantic
guidance to SpeechTokenizer.

When comparing HuBERT L9 representations with HuBERT average representations,
we find that in terms of Text Alignment, the mutual information is higher when
HuBERT L9 representations serve as the teacher. This is because HuBERT average
representations contain some timbre information, while HuBERT L9 offers purer
content information. On the other hand, HuBERT average shows better
performance in Information Preservation, reflected in a lower WER. We
speculate that this is due to a certain level of task conflict between
semantic distillation and reconstruction, where the former aims to retain only
content information while the later aims to preserve various aspects of
speech. The presence of some timbre information in HuBERT average
representations could to some extent alleviate this task conflict.

###  5.2 Effectiveness of Serial Information Disentanglement

Source | Reference |  WER$\downarrow$ |  SIM$\uparrow$
---|---|---|---
Groundtruth | 0.4 | 0.93
RVQ-1 | RVQ-2 | 2.6 | 0.72
RVQ-1 | RVQ-2:4 | 11.7 | 0.80
RVQ-1 | RVQ-2:8 | 35.4 | 0.82

Table 5: Results of one-shot voice conversion. Source and Reference refers to
source token matrix and reference token matrix respectively.

To demonstrate that different speech information can be hierarchically modeled
in SpeechTokenizer, we conduct one-shot voice conversion (VC) experiment. This
experiment aims to convert speech from any source speaker to an arbitrary
target speaker using only a few seconds of reference speech from the target
speaker. To use SpeechTokenizer for one-shot VC, the first step is to
transform the source speech and reference speech into token matrices. By
concatenating the RVQ-1 tokens of source token matrix with RVQ-2:8 tokens of
the reference token matrix, and then passing this combined token matrix to the
decoder, we can achieve voice conversion. The lengths of the reference and
source tokens may not align perfectly. To address this, we use truncation or
circular padding to ensure they share the same temporal length, thereby
facilitating the concatenation process. We conduct experiments on VCTK
dataset. We randomly selected one speech sample from a speaker to serve as the
source speech. From the remaining 107 speakers, we individually selected one
speech sample of different content to act as the reference speech. We employed
two metrics for evaluation: WER and speaker similarity.

Table 5 reports the results of one-shot VC experiments. From the table, we can
see that as the number of layers for reference tokens increases, speaker
similarity also gradually increases. This suggests that more information from
the reference speaker is being transferred over, proving that speaker
information is embedded in tokens from the second to the last layers. When the
reference tokens are selected from the second to the fourth layers, we achieve
low WER and high speaker similarity, resulting in a satisfactory one-shot VC
performance. This indicates that the information disentanglement is
successful. We put samples of one-shot VC on our demo page
555https://0nutation.github.io/SpeechTokenizer.github.io/.

We also visualize quantized outputs from different layers in Figure 4.
Specifically, We randomly select five speakers from the VCTK dataset and pick
10 random speech samples per speaker. We extract quantized output of different
RVQ layers of SpeechTokenizer. The first layer output is denoted as RVQ-1
representations, while the sum of the outputs from the second layer to the
eighth layer is denoted as RVQ-2:8 representations. By performing mean pooling
along the temporal dimension, each representation is converted into a single
vector. These vectors are then visualized in a 2D space using t-SNE, with
speech samples from the same speaker represented in the same color. From the
plot, it can be observed that the RVQ-1 representations for different speakers
are scattered randomly without discernible pattern. In contrast, the RVQ-2:8
representations for the same speaker tend to cluster together, while being
distinct from those of other speakers. This suggests that speaker-specific
information is contained from the second layer up to the eighth layer.

Figure 4: Visualization of quantized output of different RVQ layers of
SpeechTokenizer.The first layer is denoted as RVQ-1, while the sum of the
second layer to the eighth layer is denoted as RVQ-2:8.

###  5.3 Codebook Analysis

Tokenizer |  |  PNMI$\uparrow$
---|---|---
HuBERT | KM500 | 0.43
EnCodec | RVQ-1 | 0.28
SpeechTokenizer | RVQ-1 | 0.71

Table 6: PNMI of different discrete speech representation.  Figure 5:
Visualization of the conditional probability $P(\text{phoneme}|\text{code})$
on TIMIT train set. The y-axis is the phoneme set and the x-axis is the
codewords of the first RVQ layer sorted by the most correlated phoneme.

We investigate whether the tokens learned by the first RVQ quantizer relate to
phonetic information. Utilizing SpeechTokenizer or EnCodec, we derive speech
tokens from the TIMIT training set and extract the RVQ-1 tokens, denoted as
$q_{1}$. We then compute the conditional probability $p(phoneme|q_{1})$ based
on the co-occurrence between phonemes and the codes. The alignments are
constructed by selecting the phoneme that occurs most frequently in the
receptive field for each $q_{1}$.

Figure 5 visualizes the conditional probability $p(\text{phoneme}|q_{1})$ for
both SpeechTokenizer and EnCodec. A darker color block indicates a higher
$p(\text{phoneme}|q_{1})$. A more distinct contrast between the diagonal color
band and its surrounding area signifies greater phoneme purity, which in turn
suggests a more accurate mapping between the code and its corresponding
phoneme. For SpeechTokenizer, it’s evident that in the codebook of RVQ-1
quantizer, many discrete codes seem to specialize in capturing specific
phonetic sounds, indicating RVQ-1 quantizer can obtain a good alignment
between codes and labeled phonemes. However, for EnCodec, this phenomenon is
not as obvious.

Additionally, Figure 5 also reveals that over 600 codes from the EnCodec RVQ-1
codebook have never been utilized, suggesting a suboptimal utilization rate of
the codebook when EnCodec encodes speech. A lower utilization rate of the
codebook implies that more RVQ layers are required to ensure the quality of
synthesized speech, consequently necessitating the generation of more codes
during the construction of a spoken generative language model, resulting in
greater space, time and computation power consumption.

We further evaluate the models using Phone-Normalized Mutual Information
(PNMI) (Hsu et al., 2021). As shown in Table 6, RVQ-1 tokens of
SpeechTokenizer achieve a superior PNMI score to that of HuBERT units and
significantly outperforms EnCodec-RVQ-1. This suggests that the semantic
distillation process in SpeechTokenizer is effective, thereby explaining its
enhanced text alignment performance.

###  5.4 Extension to Unseen Language

Since the paralinguistic information is considered to be language-agnostic, we
attempted to apply SpeechTokenizer directly to unseen languages. We choose
German and Chinese. For German, we select samples from the German subset of
Multilingual LibriSpeech dataset for testing. For Chinese, we select samples
from the Aishell-3 dataset (Shi et al., 2021) for testing. We resynthesize
speech from RVQ-1 and RVQ-1:8 tokens. Resynthesized speech are displayed in
our demo website 666https://0nutation.github.io/SpeechTokenizer.github.io/. We
also analysis the melspectrogram of German speech and English speech in
Appendix A.

Results show that for languages either closely or distantly related to
English, resynthesized speech from RVQ-1 tokens tend to lose timbre and
prosody information while maintaining clear content. The resynthesized speech
generated from RVQ-1:8 tokens is very close to the grountruth. That suggests
SpeechTokenizer can achieve hierarchical information disentanglement on unseen
language, even though SpeechTokenizer is trained solely on English data. We
believe that SpeechTokenizer may possess the ability to extract content from
speech while disregarding language-dependent features. This ability holds
promise for the development of a multilingual SpeechTokenizer.

##  6 Related Work

Discrete Speech Representations There are two popular speech discrete
representations: semantic tokens and acoustic tokens. Semantic tokens can be
extracted from self-supervised learning of speech representations (Hsu et al.,
2021; Chung et al., 2021) and encode high-level representations that correlate
with coarse, symbolic features while paralinguistic information such as
speaker identity and acoustic details are removed. Acoustic tokens can be
extracted from neural audio codec (Zeghidour et al., 2021; Défossez et al.,
2022; Yang et al., 2023) and provide high-fidelity reconstruction of the
acoustic details. But they can not decouple different information of speech.
SpeechTokenizer unifies the two types of tokens, enabling both high-quality
audio reconstruction and decomposition of different information of speech.

Spoken Generative Language Models Speech discrete representation based spoken
generative language models have demonstrated remarkable performance on various
speech processing tasks (Borsos et al., 2022; Wang et al., 2023; Kharitonov et
al., 2023; Zhang et al., 2023). AudioLM (Borsos et al., 2022) proposes to
model speech based on audio codecs together with semantic codes, which can
synthesize speech in a textlesss setting. VALL-E (Wang et al., 2023) leverages
neural codec models to represent speech in discrete tokens from eight
quantizers. VALL-E comprises of an autoregressive language model that converts
phoenmes to acoustic tokens from the first quantizer and an non-autoregressive
language model to generate codes of the other seven quantizers. However,
VALL-E suffers from problems that some words may be unclear, missed, or
duplicated in speech synthesis due to the information gap between acoustic
tokens and phoneme. To bridge the gap, SPEAR-TTS (Kharitonov et al., 2023)
uses semantic tokens as a bridge between text and acoustic tokens. It first
generates semantic tokens from text and then produces acoustic tokens from
semantic tokens. However, this multi-stage modeling approach is more complex
and can lead to problems like error accumulation and slow inference speed. The
first quantizer of SpeechTokenizer generates semantic tokens, while the
remaining seven quantizers produce acoustic tokens by modeling the
paralinguistic information lost in the semantic tokens. SpeechTokenizer-based
VALL-E combines the advantages of VALL-E and SPEAR-TTS, where the
autoregressive model can perform text-to-semantic tokens conversion, and the
non-autoregressive model can achieve semantic-to-acoustic tokens conversion.

Speech Representation Disentanglement Human speech can be roughly decomposed
into three components: content, timbre, and prosody (Liu et al., 2023).
Content represents the main information in the speech, which can be expressed
using text or phonemes. Timbre represents the speaker’s characteristics, while
prosody encompasses intonation, stress, and rhythm of speech, reflecting how
the speaker conveys the content information. Current Speech Representation
Disentanglement (SRD) methods mostly separate speaker information from content
information for voice conversion (Qian et al., 2019; Casanova et al., 2022).
These approaches adopt a parallel disentanglement strategy, where the speech
is fed into parallel content and speaker encoders to obtain different
representations (Qian et al., 2020). However, this strategy heavily relies on
prior knowledge and introduces strong inductive biases, making the modeling
process more complex and potentially overlooking certain speech information
like prosody. Differently, VQVC (Wu & Lee, 2020) models the content embedding
as a series of discrete codes and take the difference between quantize-before
and quantize-after vector as the speaker embedding. Similarly, SpeechTokenizer
utilizes a residual structure to perform serial decomposition of speech
information and models different information as discrete tokens.

##  7 Conclusion

In this study, we present SLMTokBench, which serves as a guide for building
speech language models, to assess the effects of various speech token kinds.
Meanwhile, we propose a unified audio tokenizer, SpeechTokenizer, to unify the
discretization of both types of speech tokens to overcome the issue of
employing several models to extract semantic and acoustic discrete tokens
separately. Furthermore, We developed a unified speech language model (USLM)
based on SpeechTokenizer, with better results regarding the generated speech’s
content accuracy and quality. The study of a unified speech tokenizer is an
essential part of the further development of speech language model in terms of
efficiency and quality. The SLMTokBench and SpeechTokenizer proposed are a
step forward in unifying speech tokens, and more work deserves to continue
under this unifying paradigm.

## References

  * Ba et al. (2016) Ba, J. L., Kiros, J. R., and Hinton, G. E.  Layer normalization, 2016.
  * Baevski et al. (2020) Baevski, A., Zhou, Y., Mohamed, A., and Auli, M.  wav2vec 2.0: A framework for self-supervised learning of speech representations.  _Advances in Neural Information Processing Systems_ , 33:12449–12460, 2020.
  * Bengio et al. (2013) Bengio, Y., Léonard, N., and Courville, A.  Estimating or propagating gradients through stochastic neurons for conditional computation.  _arXiv preprint arXiv:1308.3432_ , 2013.
  * Borsos et al. (2022) Borsos, Z., Marinier, R., Vincent, D., Kharitonov, E., Pietquin, O., Sharifi, M., Teboul, O., Grangier, D., Tagliasacchi, M., and Zeghidour, N.  Audiolm: a language modeling approach to audio generation, 2022.
  * Casanova et al. (2022) Casanova, E., Weber, J., Shulby, C. D., Junior, A. C., Gölge, E., and Ponti, M. A.  YourTTS: Towards zero-shot multi-speaker TTS and zero-shot voice conversion for everyone.  In Chaudhuri, K., Jegelka, S., Song, L., Szepesvari, C., Niu, G., and Sabato, S. (eds.), _Proceedings of the 39th International Conference on Machine Learning_ , volume 162 of _Proceedings of Machine Learning Research_ , pp. 2709–2720. PMLR, 17–23 Jul 2022.  URL https://proceedings.mlr.press/v162/casanova22a.html.
  * Chen et al. (2022) Chen, S., Wang, C., Chen, Z., Wu, Y., Liu, S., Chen, Z., Li, J., Kanda, N., Yoshioka, T., Xiao, X., Wu, J., Zhou, L., Ren, S., Qian, Y., Qian, Y., Wu, J., Zeng, M., Yu, X., and Wei, F.  WavLM: Large-scale self-supervised pre-training for full stack speech processing.  _IEEE Journal of Selected Topics in Signal Processing_ , 16(6):1505–1518, oct 2022.  doi: 10.1109/jstsp.2022.3188113.  URL https://doi.org/10.1109%2Fjstsp.2022.3188113.
  * Cheng et al. (2020) Cheng, P., Hao, W., Dai, S., Liu, J., Gan, Z., and Carin, L.  Club: A contrastive log-ratio upper bound of mutual information, 2020\.
  * Chung et al. (2021) Chung, Y.-A., Zhang, Y., Han, W., Chiu, C.-C., Qin, J., Pang, R., and Wu, Y.  W2v-bert: Combining contrastive learning and masked language modeling for self-supervised speech pre-training, 2021.
  * Clevert et al. (2016) Clevert, D.-A., Unterthiner, T., and Hochreiter, S.  Fast and accurate deep network learning by exponential linear units (elus), 2016.
  * Dong et al. (2023) Dong, Q., Huang, Z., Tian, Q., Xu, C., Ko, T., Zhao, Y., Feng, S., Li, T., Wang, K., Cheng, X., Yue, F., Bai, Y., Chen, X., Lu, L., Ma, Z., Wang, Y., Wang, M., and Wang, Y.  Polyvoice: Language models for speech to speech translation, 2023.
  * Défossez et al. (2022) Défossez, A., Copet, J., Synnaeve, G., and Adi, Y.  High fidelity neural audio compression, 2022.
  * Hassid et al. (2023) Hassid, M., Remez, T., Nguyen, T. A., Gat, I., Conneau, A., Kreuk, F., Copet, J., Defossez, A., Synnaeve, G., Dupoux, E., Schwartz, R., and Adi, Y.  Textually pretrained speech language models, 2023.
  * Hines et al. (2012) Hines, A., Skoglund, J., Kokaram, A., and Harte, N.  Visqol: The virtual speech quality objective listener.  In _IWAENC 2012; International Workshop on Acoustic Signal Enhancement_ , pp. 1–4, 2012.
  * Hsu et al. (2021) Hsu, W.-N., Bolte, B., Tsai, Y.-H. H., Lakhotia, K., Salakhutdinov, R., and Mohamed, A.  Hubert: Self-supervised speech representation learning by masked prediction of hidden units.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , 29:3451–3460, 2021.
  * Kharitonov et al. (2023) Kharitonov, E., Vincent, D., Borsos, Z., Marinier, R., Girgin, S., Pietquin, O., Sharifi, M., Tagliasacchi, M., and Zeghidour, N.  Speak, read and prompt: High-fidelity text-to-speech with minimal supervision, 2023.
  * Kong et al. (2020) Kong, J., Kim, J., and Bae, J.  Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis.  _Advances in Neural Information Processing Systems_ , 33:17022–17033, 2020.
  * Lakhotia et al. (2021) Lakhotia, K., Kharitonov, E., Hsu, W.-N., Adi, Y., Polyak, A., Bolte, B., Nguyen, T.-A., Copet, J., Baevski, A., Mohamed, A., et al.  On generative spoken language modeling from raw audio.  _Transactions of the Association for Computational Linguistics_ , 9:1336–1354, 2021.
  * Liu et al. (2023) Liu, H., Wang, T., Fu, R., Yi, J., Wen, Z., and Tao, J.  Unifyspeech: A unified framework for zero-shot text-to-speech and voice conversion, 2023.
  * Mohamed et al. (2022) Mohamed, A., yi Lee, H., Borgholt, L., Havtorn, J. D., Edin, J., Igel, C., Kirchhoff, K., Li, S.-W., Livescu, K., Maaloe, L., Sainath, T. N., and Watanabe, S.  Self-supervised speech representation learning: A review.  _IEEE Journal of Selected Topics in Signal Processing_ , 16(6):1179–1210, oct 2022.  doi: 10.1109/jstsp.2022.3207050.  URL https://doi.org/10.1109%2Fjstsp.2022.3207050.
  * OpenAI (2023) OpenAI.  Gpt-4 technical report, 2023.
  * Panayotov et al. (2015) Panayotov, V., Chen, G., Povey, D., and Khudanpur, S.  Librispeech: An asr corpus based on public domain audio books.  In _2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pp. 5206–5210, 2015.  doi: 10.1109/ICASSP.2015.7178964.
  * Polyak et al. (2021) Polyak, A., Adi, Y., Copet, J., Kharitonov, E., Lakhotia, K., Hsu, W.-N., Mohamed, A., and Dupoux, E.  Speech resynthesis from discrete disentangled self-supervised representations, 2021.
  * Pratap et al. (2020) Pratap, V., Xu, Q., Sriram, A., Synnaeve, G., and Collobert, R.  MLS: A large-scale multilingual dataset for speech research.  In _Interspeech 2020_. ISCA, oct 2020.  doi: 10.21437/interspeech.2020-2826.  URL https://doi.org/10.21437%2Finterspeech.2020-2826.
  * Qian et al. (2019) Qian, K., Zhang, Y., Chang, S., Yang, X., and Hasegawa-Johnson, M.  Autovc: Zero-shot voice style transfer with only autoencoder loss, 2019\.
  * Qian et al. (2020) Qian, K., Zhang, Y., Chang, S., Hasegawa-Johnson, M., and Cox, D.  Unsupervised speech decomposition via triple information bottleneck.  In _International Conference on Machine Learning_ , pp. 7836–7846. PMLR, 2020.
  * Radford et al. (2023) Radford, A., Kim, J. W., Xu, T., Brockman, G., McLeavey, C., and Sutskever, I.  Robust speech recognition via large-scale weak supervision.  In _International Conference on Machine Learning_ , pp. 28492–28518. PMLR, 2023.
  * Rubenstein et al. (2023) Rubenstein, P. K., Asawaroengchai, C., Nguyen, D. D., Bapna, A., Borsos, Z., de Chaumont Quitry, F., Chen, P., Badawy, D. E., Han, W., Kharitonov, E., Muckenhirn, H., Padfield, D., Qin, J., Rozenberg, D., Sainath, T., Schalkwyk, J., Sharifi, M., Ramanovich, M. T., Tagliasacchi, M., Tudor, A., Velimirović, M., Vincent, D., Yu, J., Wang, Y., Zayats, V., Zeghidour, N., Zhang, Y., Zhang, Z., Zilka, L., and Frank, C.  Audiopalm: A large language model that can speak and listen, 2023.
  * Salimans & Kingma (2016) Salimans, T. and Kingma, D. P.  Weight normalization: A simple reparameterization to accelerate training of deep neural networks.  _Advances in neural information processing systems_ , 29, 2016.
  * Series (2014) Series, B.  Method for the subjective assessment of intermediate quality level of audio systems.  _International Telecommunication Union Radiocommunication Assembly_ , 2014.
  * Shi et al. (2021) Shi, Y., Bu, H., Xu, X., Zhang, S., and Li, M.  Aishell-3: A multi-speaker mandarin tts corpus and the baselines, 2021\.
  * Touvron et al. (2023) Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al.  Llama: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_ , 2023.
  * Wang et al. (2023) Wang, C., Chen, S., Wu, Y., Zhang, Z., Zhou, L., Liu, S., Chen, Z., Liu, Y., Wang, H., Li, J., He, L., Zhao, S., and Wei, F.  Neural codec language models are zero-shot text to speech synthesizers, 2023.
  * Wu & Lee (2020) Wu, D.-Y. and Lee, H.-y.  One-shot voice conversion by vector quantization.  In _ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pp. 7734–7738. IEEE, 2020.
  * Yang et al. (2023) Yang, D., Liu, S., Huang, R., Tian, J., Weng, C., and Zou, Y.  Hifi-codec: Group-residual vector quantization for high fidelity audio codec, 2023.
  * Zeghidour et al. (2021) Zeghidour, N., Luebs, A., Omran, A., Skoglund, J., and Tagliasacchi, M.  Soundstream: An end-to-end neural audio codec, 2021.
  * Zhang et al. (2023) Zhang, D., Li, S., Zhang, X., Zhan, J., Wang, P., Zhou, Y., and Qiu, X.  Speechgpt: Empowering large language models with intrinsic cross-modal conversational abilities, 2023.

##  Appendix A Melspectorgram Analysis

We plot the melspectrogram of raw speech, resynthesized speech of EnCodec
RVQ-1 tokens, and resynthesized speech of SpeechTokenizer RVQ-1 tokens. From
the figure 6, it’s evident that the melspectrogram corresponding to EnCodec
RVQ-1 largely retains the stripes and shapes in the raw melspectrogram. In
contrast, the speech resynthesized from SpeechTokenizer RVQ-1 essentially
loses all of the horizontal stripes, which indicates that timbre and prosody
information has been diminished.

Figure 6: Melspectorgram of raw speech, resynthesized speech of
SpeechTokenizer and EnCodec RVQ-1 tokens.

We alse plot melspectrogram of raw German speech and resynthesized German
speech of SpeechTokenizer RVQ-1 tokens. As shown in the Figure 7, the same
patterns observed in English speech are also present in German speech.

Figure 7: Melspectorgram of German speech and resynthesized speech of
SpeechTokenizer RVQ-1 tokens.


11institutetext: Laser Physics Applications Section, Raja Ramanna Center for
Advanced Technology, Indore-452013, India.

# A tunable Doppler-free dichroic lock for laser frequency stabilization

Vivek Singh  V. B. Tiwari  S. R. Mishra and H. S. Rawat

(Received: date / Revised version: date)

###### Abstract

We propose and demonstrate a laser frequency stabilization scheme which
generates a dispersion-like tunable Doppler-free dichroic lock (TDFDL) signal.
This signal offers a wide tuning range for lock point (i.e. zero-crossing)
without compromising on the slope of the locking signal. The method involves
measurement of magnetically induced dichroism in an atomic vapour for a weak
probe laser beam in presence of a counter propagating strong pump laser beam.
A simple model is presented to explain the basic principles of this method to
generate the TDFDL signal. The spectral shift in the locking signal is
achieved by tuning the frequency of the pump beam. The TDFDL signal is shown
to be useful for locking the frequency of a cooling laser used for magneto-
optcal trap (MOT) for ${}^{87}Rb$ atoms.

††journal: my journal

##  1 Introduction

Laser frequency stabilization is an essential requirement for various
applications including atom coolingchu , high resolution spectroscopyCE ,
precision measurementsJE  etc. A setup for laser cooling and trapping of atoms
requires several lasers which are actively frequency stabilized and locked at
few line-widths detuned from the peak of atomic absorption. In the laser
cooling experiments, the active frequency stabilization is achieved by
generating a reference signal which is based on absorption profile of atom
around the resonance. The reference locking signal can be either Doppler-
broadened with wide tuning range or Doppler-free with comparatively much
steeper slope but with limited tuning range. The most commonly used technique
for frequency locking involves locking to the side of the Doppler free peak
generated from the saturated absorption spectroscopy (SAS)Demt ; Mac . The
locking of the laser frequency to the peak of an atomic reference signal such
as in frequency modulated spectroscopy (FMS)Mud  provides high signal to noise
ratio (SNR) but needs dithering of laser frequency and a phase sensitive
detection system. A suitable dispersion-like reference signal for peak locking
can be generated using polarizing spectroscopy (PS)Wie ; Pear ; Tiwari ; VBT ;
Kale  without frequency modulation and phase sensitive detection. Along with
these techniques, there are also methods for laser frequency stabilization
which employ PS with external magnetic fieldKrzemien ; Sho . However, the
techniques involving polarization spectroscopy are sensitive to the
surrounding stray magnetic fieldGawlik . An alternative technique which can
generate narrow dispersion like reference signal exploits circular dichroism
of an atomic vapour in the presence of a magnetic field. In this technique,
the difference of Zeeman shifted saturated absorption signals generate the
Doppler free dichroic lock (DFDL)signalShim ; Wasik ; Harris ; Dian ; Hongli ;
Miyabe ; Noh ; Pahwa ; Marin ; Petelski . This technique requires only low
magnetic field for its operation and is less sensitive to the surrounding
stray magnetic field. However, the tuning range of this technique is limited
to few natural linewidths of the transition. The tuning range can be increased
by electronically adding an offset voltage to the signal or by optically
changing the quarter wave plate axis relative to the polarisation beam
splitter axisWasik . The locking range can also be extended by either using
Doppler-broadened version of this technique known as dichroic atomic vapor
laser lock (DAVLL)Corwin ; Alfred ; Imanishi ; Beverini  signal or by locking
at positive as well as negative slopes of DFDL signal Wasik . Another method
involves increasing the magnetic field VB  to extend the locking range of the
DFDL signal. This frequency tuning is often achieved by compromising the slope
of the signal which is a crucial parameter for frequency locking. A recently
reported study on DFDL spectroscopy of D-2 line of ${}^{87}Rb$ includes
coherence for calculation of line-shape of real atomic system hrNoh .
Here, we present a technique which generates a narrow tunable DFDL (TDFDL)
signal without compromising with the slope of the signal. The frequency tuning
of this dispersion like TDFDL signal is achieved by varying the pump laser
frequency using AOMs. We have studied the dependence of this frequency locking
signal’s amplitude and the slope on various experimental parameters such as
the pump beam power and the magnetic field. The locking performance of an
extended cavity diode laser (ECDL) system is investigated by measuring change
in the number of cold atoms in a magneto-optical trap (MOT). A simple
theoretical model to explain the generation of TDFDL signal is also presented.

##  2 Theoretical analysis

We adopt a simple two-level system involving hyperfine energy levels to
explain the generation of TDFDL signal. A linearly polarized weak probe beam
is overlapped by a counter-propagating strong pump beam in an atomic vapor
cell placed in a weak magnetic field. The propagation of the probe beam is
along the axis of magnetic field. The linearly polarized light can be resolved
into two counter-rotating circular components $\sigma_{\pm}$ using quarter
wave plate and polarizing beam splitter. In the absence of magnetic field
(B=0), different $m_{F^{\prime}}$ states are degenerate and hence,
$\sigma_{+}$ and $\sigma_{-}$ transitions overlap. When a small magnetic field
(few Gauss) is applied, the degeneracy is lifted and medium becomes dichroic
due to the displacement of $\sigma_{+}$ and $\sigma_{-}$ transitions. As a
result, the pair of spectrally shifted saturated absorption spectroscopy (SAS)
signals on subtraction generate the Doppler free dispersion like signalVB .
For atoms moving with velocity $\vec{v}$, the Doppler shifted laser frequency
in the frame of moving atom becomes
$\omega^{{}^{\prime}}=\omega_{0}-\vec{k}.\vec{v}$. When the laser beam passes
through a Rb atomic sample with a Maxwell-Boltzmann velocity distribution,
only those atoms which fall within the homogeneous linewidth $\Gamma$ around
the center frequency $\omega_{0}$ of Rb atom at rest can significantly
contribute to the absorption.

Figure 1: Schematic diagram showing resonant interaction of $\sigma_{+}$ and
$\sigma_{-}$ polarized pump and the probe beams when a weak magnetic field is
applied to an atom having lower level $F=0$ and upper level $F^{\prime}=1$ is
in motion.

If the pump beam having frequency ($\omega_{c}$) and the probe beam having
frequency ($\omega_{p}$) are in counter propagating directions, the atoms
having velocity components of $v_{z}$ along direction of the probe beam see
different values of frequency detunings with the pump and the probe beams in
the presence of magnetic field. The $\sigma_{\pm}$ polarized pump and the
probe beams can interact resonantly with this group of atoms if,

$\displaystyle\omega_{p}(\sigma_{\pm})-kv_{z}=\omega_{0}\pm\Delta_{B}$ (1)
$\displaystyle\omega_{c}(\sigma_{\pm})+kv_{z}=\omega_{0}\pm\Delta_{B}$ (2)

Using equation (1) and (2), the relation between the probe beam detuning
$\Delta_{p}^{\pm}=\omega_{p}(\sigma_{\pm})-\omega_{0}$ and the pump beam
detuning $\Delta_{c}^{\pm}=\omega_{c}(\sigma_{\pm})-\omega_{0}$ for resonant
interaction is given by

$\displaystyle\Delta_{c}^{\pm}+\Delta_{p}^{\pm}=\pm 2\Delta_{B}$ (3)

Again using equation (1) and (2), the difference between the pump and the
probe beam detunings is given by,

$\displaystyle\Delta_{cp}=\Delta_{c}^{\pm}-\Delta_{p}^{\pm}=-2kv_{z}$ (4)

Using equation (3) and (4), resonance condition for $\sigma_{\pm}$ probe beam
in presence of counter propagating pump beam and applied magnetic field can be
written as

$\displaystyle\Delta_{p}^{\pm}=-\dfrac{\Delta_{cp}}{2}\pm\Delta_{B}$ (5)

Therefore, the probe absorption signal will depend upon the difference between
the pump and the probe beams detunings and the applied magnetic field.
In the case of simple model, $F=0\rightarrow F^{\prime}=1$, the absorption
cross-section for $\sigma_{\pm}$ probe beam for an atom with velocity
component $v_{z}$ is given asDemt

$\displaystyle\rho_{\pm}(\Delta_{p}^{\pm},v_{z})=\dfrac{\rho_{0}(\Gamma/2)^{2}}{(\Delta_{p}^{\pm}\mp\Delta_{B}-kv_{z})^{2}+(\Gamma/2)^{2}}$
(6)

where $\rho_{0}$ is maximum absorption cross section at
$\Delta_{p}^{\pm}=\pm\Delta_{B}+kv_{z}$,
$\Delta_{B}=\mu_{B}Bg_{F^{\prime}}m_{F^{\prime}}/\hbar$ (for $F=0\rightarrow
F^{\prime}=1$ case) is Zeeman splitting of the levels in magnetic field, $k$
is the wave vector of laser light.
Due to pump laser beam ($\omega_{c}$), population density in ground state
($F=0$ level) decreases within the velocity interval
$dv_{z}=\frac{\Gamma}{k}$, while population density in magnetically shifted
upper level ($F^{\prime}=\pm 1$ level) increases. Therefore, population in the
lower level after pump has caused excitation from lower level is given by

$\ \Delta N(\Delta_{c}^{\pm},v_{z})=N^{0}(v_{z})[1-\\\
-g_{1}\dfrac{(\Gamma/2)^{2}}{(\Delta_{c}^{+}-\Delta_{B}+kv_{z})^{2}+(\Gamma/2)^{2}}\\\
-g_{2}\dfrac{(\Gamma/2)^{2}}{(\Delta_{c}^{-}+\Delta_{B}+kv_{z})^{2}+(\Gamma/2)^{2}}]\
$ (7)

where $\Gamma$ is a natural line width for $F=0\rightarrow F^{{}^{\prime}}=1$
transition, $g_{1}$ and $g_{2}$ are coefficients representing the depth of
dips burnt in the distribution by the pump beam and depends on the pump beam
intensity used. For our calculations, we have assumed $g=g_{1}=g_{2}$. The
population in the ground state with velocity range $v_{z}$ and $v_{z}+dv_{z}$
from where probe makes absorption transition is given as

$\displaystyle N^{0}(v_{z})dv_{z}=Cexp{[-(v_{z}/v_{p})^{2}]}dv_{z}$ (8)

where $v_{p}$ is most probable velocity and $C$ is a constant depending on
number of atoms in gas, density etc..
The absorption coefficient for a weak probe laser beam in presence of the pump
beam is given by

$\displaystyle\alpha_{\pm}(\Delta_{p}^{\pm},\Delta_{c}^{\pm})=\int_{-\infty}^{\infty}\rho_{\pm}(\Delta_{p}^{\pm},v_{z})\Delta
N(\Delta_{c}^{\pm},v_{z})dv_{z}$ (9)

The integration in (9) can be performed over velocity and assuming the Doppler
width, $\Delta_{D}>>\Gamma$ (where $\Delta_{D}=kv_{p}$) to obtain
$\alpha_{\pm}$ values.
For low optical density of the sample, the TDFDL signal (represented by $DS$)
can be calculated to be

$\displaystyle
DS(\Delta_{p}^{\pm},\Delta_{c}^{\pm})=\alpha_{+}(\Delta_{p}^{+},\Delta_{c}^{+})-\alpha_{-}(\Delta_{p}^{-},\Delta_{c}^{-})$
(10)

Using equation (7), (8) and (9), the TDFDL signal is

$\ DS(\Delta_{p}^{\pm},\Delta_{c}^{\pm})=\\\
C[(D_{+}-D_{-})-g(D_{+}L_{+}-D_{-}L_{-})]\ $ (11)

where
$D_{\pm}=\dfrac{1}{\sqrt{\pi}}\exp{[-((\Delta_{p}^{\pm}\mp\Delta_{B})/\Delta_{D})^{2}]}$
$L_{\pm}=\dfrac{\Gamma^{2}}{(\Delta_{p}^{\pm}+\Delta_{c}^{\pm}\mp
2\Delta_{B})^{2}+(2\Gamma/2)^{2}}$
$L_{\pm}$ can be written in terms of $\Delta_{cp}$ using equation (4),
$L_{\pm}=\dfrac{(\Gamma/2)^{2}}{(\Delta_{p}^{\pm}+\dfrac{\Delta_{cp}}{2}\mp\Delta_{B})^{2}+(\Gamma/2)^{2}}$

The signal consists of two parts: the first part represents a Doppler
broadened dichroic signal. The second part appears due to presence of pump
beam and it is represented by two Doppler-free absorption coefficients of
$\sigma_{+}$ and $\sigma_{-}$ components.

Figure 2: Calculated TDFDL signals $DS(\Delta_{p}^{\pm},\Delta_{c}^{\pm})$
(using equation (11)) for the parameter $\Delta_{D}=50\Gamma$, $C=1$,
$g=0.04$, $m_{F}=0$, $g_{F^{\prime}}=2/3$, $m_{F^{\prime}}=1$,
$\Gamma/2\pi=12$ MHz, $B=4G$ and $\Delta_{cp}/2\pi$ = +80 MHz, 0 MHz and -80
MHz corresponds to short dashed line, solid line and dashed line respectively.

Fig. 2 shows the calculated TDFDL signal for different pump laser detuning. As
the pump laser frequency is blue detuned with respect to the probe beam, the
zero-crossing in dispersion like TDFDL signal will be shifted to red and vice
versa. Therefore, the zero crossing in TDFDL signal can be shifted to any
frequency depending upon the pump laser detuning. Further, Fig. 3 (a) and (b)
shows the calculated amplitude and the slope of the TDFDL signal (12 MHz red
detuned) with the axial magnetic field respectively. We find that amplitude
(difference of maximum and minimum values) of TDFDL signal initially increases
with the applied magnetic field and gets saturated after certain value of the
magnetic field. The amplitude of TDFDL signal reaches maximum value when SAS
for $\sigma_{+}$ and $\sigma_{-}$ probe beams is just separated by increasing
the magnetic field. Further, the slope of the TDFDL signal increases with
magnetic field and reaches maximum value at magnetic field of $\sim$ 4 Gauss
as shown in fig. 3(b). Slope starts to decrease with further increase of the
magnetic field due to increased separation between $\sigma_{+}$ and
$\sigma_{-}$ probe laser beam signals.

Figure 3: Calculated dependence of (a) amplitude and (b) Slope of TDFDL signal
on magnetic field. $\Delta_{D}=50\Gamma$, $\Gamma/2\pi=12$ MHz, $C=1$,
$g=0.04$, $m_{F}=0$, $g_{F^{\prime}}=2/3$, $m_{F^{\prime}}=1$ and
$\Delta_{cp}/2\pi=+24MHz$.

##  3 Experimental setup

The TDFDL experimental setup is shown in Fig. 4. The External cavity diode
laser (ECDL) (DL100, Toptica) system operating at 780 nm is used in our
experiments. A combination of half wave plate and polarizing beam splitter
(PBS) is used to suitably divide laser output beam for its use in TDFDL setup,
a reference SAS setup and a magneto-optical trap (MOT) setup.
In TDFDL setup, laser light is split in two parts namely the pump laser beam
and the probe laser beam using the combination of half wave plate and PBS. The
pump laser beam is passed through a setup of two acousto-optical modulators
(AOMs), AOM-1 and AOM-2 having frequency range of 60 to 100 MHz. The pump
laser beam is initially passed through AOM-1. The first order (+1) diffracted
beam of AOM-1 is retro-reflected and passed again through AOM-1 before going
to pass through the second AOM i.e. AOM-2 (Fig. 4). In AOM-2, The first order
(-1) is made retro-reflected. Thus, we have used both the AOMs in double pass
configuration to avoid deviation of the output beam. The output pump beam is
overlapped with the probe laser beam in counter propagating direction inside a
5 cm long Rb vapor cell placed inside the solenoid with crossing of $\sim$ 10
$mrad$. The probe laser beam is monitored using a $\lambda/4$ wave plate and a
PBS combination of elements and photo-diodes for detection of $\sigma_{+}$ and
$\sigma_{-}$ parts of polarizations (Fig. 4). The difference of two photo-
diode signals gives dispersion like signal. The solenoid generates the
longitudinal field of $\sim$ 32 G/A. At the Rb vapor cell, the probe and the
pump laser beam has a diameter of 3 mm. Neutral density filters are used to
vary the pump laser beam power.

Figure 4: Schematic diagram of TDFDL spectroscopy setup. ECDL: external cavity
diode laser, MOT: magneto-optical trap, PBS: polarizing beam splitter, BD:
beam dump, BS: beam splitter, AOM: acousto-optic modulator, DA: differential
amplifier, PD: photo-diode, SAS: saturated absorption spectroscopy, M: mirror,
L: lens, $\lambda/4$ : quarter wave plate, $\lambda/2$ : half wave plate. Two
detectors (PD-1 and PD-2) monitor the absorption of light which has driven
$\sigma_{+}$ and $\sigma_{-}$ transitions.

##  4 Results and discussions

###  4.1 Frequency tuning of TDFDL spectra using AOMs

The frequency tuning of the TDFDL signal is achieved by varying the frequency
of the pump laser beam with the help of two AOMs in double pass configuration
as shown in Fig. 4. The pump laser beam after double pass through AOM-1, gets
a frequency shift of $2\omega_{1}$, where $\omega_{1}$ is shift in first order
diffraction in AOM-1. When this first order diffracted beam passes through
AOM-2, the frequency shift is $-2\omega_{2}$, where $\omega_{2}$ is shift in
-1st order in AOM-2. Therefore, the frequency of the pump beam is shifted by
$\Delta_{AOM}$ ($\Delta_{AOM}=2(\omega_{1}-\omega_{2})=\Delta_{cp}$, relative
to the probe beam frequency. Therefore, as we observed, (Fig. 5(a)), there is
no spectral shift in the TDFDL signal when AOM-1 and AOM-2 operate at the same
frequency. This is because, the pump and the probe laser beam will be
interacting with zero velocity group resulting a TDFDL signal as shown in Fig.
5(a) (solid curve). Further, if AOM-1 and AOM-2 operate at frequency 100 MHz
and 60 MHz respectively then frequency shift of pump beam ($\Delta_{AOM}$) as
compared to the probe beam becomes +80 MHz. Therefore, TDFDL signal will be
red shifted (-40 MHz) as shown by short dashed curve in Fig. 5(a). In the same
way, TDFDL signal can be shifted to blue detuned (+40 MHz), as shown by dashed
curve in Fig. 5(a), if AOM-1 and AOM-2 operate at frequency of 60 MHz and 100
MHz respectively. Thus, observed detuning behaviour of TDFDL signal shown in
Fig. 5(a) is in good agreement with our theoretical results shown in Fig. 2.
We have also observed that the slope and the amplitude of the signal remains
nearly unchanged throughout the whole tuning range. Thus, the frequency
stability achieved using TDFDL reference signal is expected to be nearly
independent of the frequency shifting used. This is qualitatively different
from the SAS signal where the slope and the amplitude changes with detuning
from the resonance peak. It is shown in Fig. 5b that slope of TDFDL signal is
$\sim$ 43 mV/MHz as compared to SAS signal with slope of $\sim$ 6 mV/MHz at
-12 MHz detuning. Here, we note that the relative difference in the detuning
of the pump and the probe beams can also be introduced using two different
lasers as reported in ref.Kale . However, in that case the locking performance
of one laser may influence the frequency stabilization of the other laser. We
note that the tuning of laser frequency using a DFDL setup can also be
achieved by changing the frequency of final laser beam using AOMs. But this
will be an inefficient method due to loss of power through AOMs used in the
final beam.

Figure 5: (a) Observed spectral shift of TDFDL signal for $F=2\rightarrow
F^{\prime}=3$ transition of ${}^{87}Rb$ atom for $\Delta_{AOM}/2\pi$ = +80
MHz, 0 MHz and -80 MHz (short dashed curve, solid curve and dashed curve
respectively). (b) TDFDL signal at -12 MHz detuning with reference to SAS
signal. Slope of SAS signal is $\sim$ 6 mV/MHz at point A whereas the slope of
TDFDL signal is $\sim$ 43 mV/MHz at point B. pump power = 1.13 mW, Probe power
= 140 $\mu$ W, and magnetic field = 9.7 G for the data in figure (b).

###  4.2 Dependence of TDFDL spectra on magnetic field strength and pump beam
power

This TDFDL signal is utilized to lock the laser frequency at zero-crossing
point of the signal. The lock stability of laser is dependent on peak to peak
amplitude and slope of the TDFDL signal at the zero crossing position.
Therefore, dependence of peak to peak amplitude and slope of the signal on
magnetic field strength and the pump beam power is important for the cooling
transition $F=2\rightarrow F^{\prime}=3$ transition of ${}^{87}Rb$ atoms.

Figure 6: Dependence of (a) amplitude and (b) slope of the TDFDL signal on
magnetic field for $F=2\rightarrow F^{\prime}=3$ transition of ${}^{87}Rb$
atom. The data is recorded for probe laser beam power of 140 $\mu$W and for
different pump laser beam power. TDFDL signal is 12 MHz red detuned from
$F=2\rightarrow F^{\prime}=3$ transition of ${}^{87}Rb$.

Fig. 6 shows the effect of magnetic field on (a) the amplitude and (b) slope
of TDFDL signal for different values of the pump beam power. The TDFDL signal
is 12 MHz red detuned from $F=2\rightarrow F^{\prime}=3$ transition of
${}^{87}Rb$. The amplitude of the signal increases up to a field of 20 G and
then starts to decrease. The slope of the signal initially increases with
magnetic field and then achieves its maximum value of $\sim$ 43 mV/MHz for the
pump power of 1.13 mW and applied magnetic field of $\sim$ 9.7 Gauss. A
reduction in slope is observed for magnetic fields larger than $\sim$ 9.7 G as
the separation between $\sigma_{+}$ and $\sigma_{-}$ transitions for the probe
absorptions becomes more than a few natural widths. Therefore, it is evident
from the graph that optimised slope and a large amplitude of TDFDL signal is
obtained for the magnetic field of $\sim$ 9.7 G and the pump beam power of
1.13 mW. The slope and the amplitude reduces significantly due to power
broadening at the pump beam power greater than 1.13 mW. The behaviour of
calculated TDFDL signal’s amplitude and slope with magnetic field is similar
to the experimental results. However, the optimum value of magnetic field
($\sim$9.7 Gauss) for the experimental TDFDL signal for $F=2\rightarrow
F^{\prime}=3$ transition of ${}^{87}Rb$ is different from the value for the
calculated signal as shown in Fig. 3(b). This may be due to the reason that
our calculations are based on simple two level model whereas the real atomic
system consists of multiple sub-levels in ground as well as excited state.

###  4.3 Frequency stabilization of laser using TDFDL signal and its
application to Rb-MOT setup

The frequency stability of an ECDL laser system using TDFDL locking signal has
been studied. A standard PID (Toptica, Germany) controller is used to lock the
laser frequency. The TDFDL signal was fed to this PID controller to lock the
laser frequency. The desired frequency was selected by setting the lock
reference level equal to the signal level at that frequency. The laser system
remained frequency locked for several hours during the course of experiments.
The error signal after frequency locking is recorded on a calibrated photo-
diode.

Figure 7: The variation in maximum number of cold atoms in ${}^{87}Rb$-MOT
with detuning of cooling laser frequency using TDFDL locking method. The MOT
is pulsed loaded using Rb dispenser with dispenser current of 7 A for 5 second
duration.

Our proposed TDFDL technique has two advantages. First is robust locking due
to use of a dispersive like signal with large slope, and second is large
tunability without compromising the slope. The TDFDL signal has been used to
lock the cooling laser frequency in the magneto-optical trap (MOT) setup. We
can tune this cooling laser in locked condition itself, to precisely control
the number of cold atoms in the MOT. The ${}^{87}Rb$ MOT is prepared in an
octagonal chamber which is made of stainless steel with a residual pressure
$\sim$ $1.4\times 10^{-8}$ torr. The dispenser used in this experiment is a
SAES Rb metal dispenser. Our dispenser is located at $\sim$ 19 cm form the MOT
centre. Two feed-through pins on which dispenser is mounted are connected to
external current supply from outside the chamber. Six laser beams of $\sim$ 5
mW power in each beam is used to form the MOT. The beam waist of each cooling
laser beam is about $\sim$ 6 mm. The laser beam of an additional diode laser
is combined with that of the cooling and trapping laser for re-pumping of the
atoms. The quadrupole magnetic field gradient in axial direction was $\sim$ 10
G/cm. The MOT was operated in pulsed mode. In this, a current of $\sim$ 7 A is
passed through Rb dispenser for 5 second duration and cooling laser frequency
is changed using TDFDL setup. Fluorescence signal of pulsed loaded MOT is
recorded on a calibrated photo-diode to measure the number in the MOT. In Fig.
7, the peak value of the number of atoms is plotted as a function of cooling
laser detuning. The temperature of Rb dispenser is kept constant between each
pulsed operation by flowing 1.5 A current to the dispenser. We have maintained
$\sim$ 5 minute gap between each data. The precise control over laser
frequency tuning using the TDFDL signal allowed the cooling laser detuning to
be changed in steps of 1 MHz. We also note that the tuning was possible
without breaking the frequency lock over a wide range. The cold atom number
was found to be maximum ($\sim$ $6.0\times 10^{6}$) at - 12 MHz of cooling
laser detuning. The atom cloud temperature is typically $\sim$ 300 $\mu K$ for
the values of cooling beam intensity and detuning used VB . The result is in
good agreement with the earlier work Lind ; Arnold .

Figure 8: The measured variation in number of cold atoms in the MOT and
cooling laser frequency with time, before and after locking the laser
frequency using TDFDL signal. The data is recorded with probe laser beam power
of $\sim$ 140 $\mu$W and with pump laser beam power of $\sim$ 1.13 mW. The
slope of TDFDL signal is $\sim$ 43 mV/MHz. The TDFDL signal was shifted $\sim$
\- 12 MHz with respect to peak of the cooling transition of ${}^{87}Rb$. For
these measurements, a continuous MOT was operated by applying a dc current of
$\sim$ 2.7 A in the Rb dispenser.

Fig. 8 shows the fluctuation in cooling laser frequency and the number of cold
atoms in the MOT before and after locking the laser frequency. It is evident
from this figure that both the number of cold atoms and error signal (i.e.
frequency deviation) show the significant reduction in fluctuation after
locking the laser frequency. Fig. 8 shows that fluctuation in number of cold
atoms in the MOT gets reduced from $\sim$ $\pm$ 10 $\%$ (before locking) to
$\sim$ $\pm$ 4.5 $\%$ (after locking) whereas the fluctuation in laser
frequency gets reduced from $\sim$ $\pm$ 1 MHz (before locking) to $\sim$
$\pm$ 0.25 MHz (after locking). The laser is locked for more than 1 hour.

##  5 Conclusion

In conclusion, a new laser frequency stablization technique which we call as
tunable Doppler free dichroic lock (TDFDL) spectroscopy has been demonstrated.
This technique provides a sharp and widely tunable dispersion like signal
useful for frequency locking of a laser. The amplitude and the slope of the
signal depends on the pump beam power and the applied magnetic field. Using
the TDFDL signal, the tuning of zero-crossing point in the TDFDL locking
signal is achieved by varying the relative frequency difference between the
pump and probe beams. The TDFDL signal has been utilized for frequency locking
and tuning of a laser used for cooling of ${}^{87}Rb$ atoms in a magneto
optical trap.

##  6 Acknowledgements

We are grateful to Amit Chaudhary for his help during the experimental work.

## References

  * (1) S. Chu, Rev. Mod. Phys. 70, 685 (1998).
  * (2) C. E. Wieman and L. Hollberg, Rev. Sci. Instrum. 1, 62 (1991).
  * (3) J. E. M. Goldsmith, E. W. Weber, and T. W. Hänsch, Phys. Rev. Lett. 41, 1525 (1978).
  * (4) W. Demtroder, Laser Spectroscopy (Springer-Verlag, Berlin 1988) 3rd ed
  * (5) K. B. MacAdam, A. Steinbach and C. Wieman, Am. J. Phys. 60, 1098 (1992).
  * (6) L. Mudarikwa, K. Pahwa and J. Goldwin, J. Phys. B: At. Mol. Opt. Phys. 45, 065002 (2012).
  * (7) C. Wieman and W.T. Hansch, Phys. Rev. Lett. 36, 1170 (1976).
  * (8) C. P. Pearman, C. S. Adams, S. G. Cox, P. F. Griffin, D. A. Smith and I. G. Hughes, J. Phys. B: At. Mol. Opt. Phys. 35, 5141 (2002).
  * (9) V. B. Tiwari, S. Singh, S. R. Mishra, H. S. Rawat and S. C. Mehendale, Opt. Communn. 263, 249 (2006).
  * (10) V. B. Tiwari, S. Singh, S. R. Mishra, H. S. Rawat and S. C. Mehendale, Appl. Phys. B 83, 93 (2006).
  * (11) Y. B. Kale, V. B. Tiwari, S. Singh, S. R. Mishra and H. S. Rawat, Journal of Optical Scociety of America B 31, 2531 (2014).
  * (12) L. Krzemien, K. Brzozowski, A. Noga, M. Witkowski, J. Zachorowski, M. Zawada and W. Gawlik, Opt. Communn. 284, 1247 (2011).
  * (13) S. Okubo, K. Iwakuni and T. Hasegawa, Opt. Communn. 285, 4107 (2012).
  * (14) W. Gawlik, Acta Phys. Pol. A 66, 401 (1984).
  * (15) U. Shim, J. A. Kim and W. Jhe, J. K. P. S. 35, 222 (1999).
  * (16) G. Wasik, W. Gawlik, J. Zachorowski and W. Zawadzki, Appl. Phys.B 75, 613 (2002).
  * (17) M. L. Harris, S. L. Cornish, A. Tripathi and I. G. Hughes, J. Phys. B: At. Mol. Opt. Phys. 41, 085401 (2008).
  * (18) D. Q. Su, T. F. Meng, Z. H. Ji, J. P. Yuan, Y. T. Zhao, L. T. Xiao and S. T. Jia , Appl. Opt. 53, 7011 (2014).
  * (19) H. Liu, S. Yin, J. Qian, Z. Xu and Y. Wang, J. Phys. B: At. Mol. Opt. Phys. 46, 085005 (2013).
  * (20) M. Miyabe, M. Kato, M. Oba, I. Wakaida, K. Watanabe and K. Wendt, Japanese-J. Appl. Phys. 45, 4120 (2006).
  * (21) G. W. Choi and H. R. Noh, J. Phys. B: At. Mol. Opt. Phys. 48, 115008 (2015).
  * (22) K. Pahwa, L. Mudarikwa and J. Goldwin, Optics Express 20, 17456 (2012).
  * (23) M. Pichler and David C. Hall, Opt. Communn. 285, 50 (2012).
  * (24) T. Petelski, M. Fattori, G. Lamporesi, J. Stuhler and G. M. Tino, Eur. Phys. J. D 22, 279 (2003).
  * (25) K. L. Corwin, Z-T. Lu, C. F. Hand, R. J. Epstein and C. E. Wieman, Appl. Opt. 37, 3295 (1998).
  * (26) A. M. Sikking, I. G. Hughes, P. Tierney and S. L. Cornish, J. Phys. B: At. Mol. Opt. Phys. 40, 187 (2007).
  * (27) S. Imanishi, U. Tanaka and S. Urabe , Japanese-J. Appl. Phys. 44, 6767 (2005) .
  * (28) N. Beverini, E. Maccioni, P. Marsili, A. Ruffini and F. Sorrentino, Appl. Phys. B 73, 133 (2006).
  * (29) V. B. Tiwari, S. R. Mishra, H. S. Rawat, S. Singh, S. P. Ram and S. C. Mehendale, Pramana-J. Phys. 65, 403 (2005).
  * (30) G. W. Choi and H. R. Noh, Opt. Communn. 367, 312 (2016).
  * (31) K. Lindquist, M. Stephens and C. Wieman, Phys. Rev. A 46, 4082 (1992).
  * (32) J. P. McGilligan, P. F. Griffin, E. Riis and A. S. Arnold, Opt. Express 23, 8948 (2015).