Listing 15: The PINN architecture code for the Fisher-KPP equation. Phenomenon | Modelling | Remarks
---|---|---
unidimensional modelling | First order differential equation |  Simplistic approach allows to have a score using a low quality data
Diffusion Modelling | Heat equation |  Unique/ works well without data/Does not take into consideration the pigmentation
Modelling of viscous diffusion | Burgers equation | Works well without data
Front wave transport Modelling | Fisher KPP | Works well without data
M odelling with non linear terms | Kordweg de Vries | Works well without data
Turing pattern Modelling | Turing equation |  Numerical instability problem
Table 14: The table shows the different partial differential equations studied
in this paper.

##  5 Spatial distribution extraction by computer vision

The extraction of the spatial distribution of ulcerative colitis and Crohn’s
disease using computer vision can be very useful for our study. By using image
processing algorithms, it is possible to accurately map out the areas affected
by the two diseases. This can help doctors better understand the progression
of the disease and adjust treatment accordingly. In addition, it can also
allows for faster and more accurate evaluation of treatment response, which
can be particularly useful in the context of testing new medications. The
traditional detection methods based on the expertise of gastroenterologists
are time and resource intensive. As a result, early detection and treatment of
these diseases can reduce casualties and improve the patient’s quality of life
later on. With recent advances in Deep Learning, powerful approaches for both
detection and classification that can handle complex environments have been
developed. In this paper, we propose a deep learning based architecture for
object classification in the context of Crohn disease. The goal is to assist
the PINN framework developed above with input data for the PDEs initial
conditions and boundary conditions. The proposed solution combines deep
learning and tweaked transfer learning models for object classification and
detection with balanced data for each image class. It can operate in a more
complex environment and takes into consideration the state of the input. Its
aim is to automatically detect damages, locate them and classify the disease
type.

###  5.1 Transfer learning for image transformation

Transfer learning is a deep learning technique that allows the use of pre-
trained models to perform image transformation tasks. The pre-trained models
have already learned general features from a large dataset, making them
suitable for transfer learning to related tasks. In the context of spatial
distribution extraction, transfer learning can be used to improve the accuracy
of feature extraction for specific image classes. The residual network ResNet
[54] has been used for this transfer learning taks. Its architecture is
available in many different forms, each with a different number of layers, a
variant with 50 layers of neural networks is referred known as Resnet50. The
Resnet50 and Resnet34 architectures differ significantly in one key area. In
this case, the building block was changed into a bottleneck design due to
worries about the amount of time needed to train the layers. In order to
create the Resnet 50 architecture, each of the Resnet34’s 2-layer bottleneck
blocks was changed to a 3-layer bottleneck block. Comparing this model to the
34-layer ResNet model, the accuracy of this model is noticeably higher (as
shown in the figure 24).

On the other hand, the discrete gradient algorithm known as inpainting and the
clustering techniques can then be used to fill in missing data and group
similar data, respectively. Principal component analysis (PCA) can also be
applied to reduce the dimensionality of the extracted features, enabling more
efficient processing of the data. Ultimately, these image transformation
techniques enable accurate and efficient computer vision solutions for
analyzing complex biological images.

Figure 24: Resnet architecture variants.

The data used to test this approach is the Kvasir dataset [35] which is a
collection of annotated medical images of the gastrointestinal (GI) tract,
designed for the purpose of computer-aided disease detection. The dataset is
important for research in the medical domain of detection and retrieval,
especially for single- and multi-disease computer aided detection. It contains
images classified into three important anatomical landmarks and three
clinically significant findings, as well as two categories of images related
to endoscopic polyp removal. The sorting and annotation of the dataset are
done by medical doctors. The Kvasir dataset may improve medical practice and
refine health care systems globally since it includes sufficient numbers of
images to be used for different tasks such as machine learning, deep learning,
image retrieval, and transfer learning. The Kvasir dataset is collected using
endoscopic equipment at Vestre Viken Hospital Trust in Norway and is carefully
annotated by medical experts from VV and the Cancer Registry of Norway (CRN).
The CRN is responsible for the national cancer screening programmes with the
goal to prevent cancer death by discovering cancers or pre-cancerous lesions
as early as possible. The Kvasir dataset is containing 8 classes of 2000
images per class. As a first step we opted to work with binary classification
by feeding the model with both the normal images and the images having the
disease.

⬇

1import os

2import random

3import numpy as np

4import torch

5import torchvision

6import torchvision.transforms as transforms

7import matplotlib.pyplot as plt

8

9def train_val_split(dataset, val_split=0.2, shuffle=True, random_seed=None):

10 if shuffle:

11 if random_seed is not None:

12 random.seed(random_seed)

13 dataset_size = len(dataset)

14 indices = list(range(dataset_size))

15 random.shuffle(indices)

16 else:

17 indices = list(range(len(dataset)))

18

19 split = int(np.floor(val_split * len(dataset)))

20 train_indices, val_indices = indices[split:], indices[:split]

21

22 train_sampler = torch.utils.data.SubsetRandomSampler(train_indices)

23 val_sampler = torch.utils.data.SubsetRandomSampler(val_indices)

24

25 return train_sampler, val_sampler

26

27def get_dataloaders(data_dir, image_size=(512, 512), batch_size=16,
val_split=0.2, shuffle=True, random_seed=None):

28 data_transforms = {

29 ’train’: transforms.Compose([

30 transforms.Resize(image_size),

31 transforms.ToTensor()

32 ]),

33 ’val’: transforms.Compose([

34 transforms.Resize(image_size),

35 transforms.ToTensor()

36 ])

37 }

38

39 image_datasets = {

40 x: torchvision.datasets.ImageFolder(

41 os.path.join(data_dir, x),

42 transform=data_transforms[x]

43 )

44 for x in [’train’, ’val’]

45 }

46

47 train_sampler, val_sampler = train_val_split(image_datasets[’train’],
val_split, shuffle, random_seed)

48

49 dataloaders = {

50 ’train’: torch.utils.data.DataLoader(

51 image_datasets[’train’], batch_size=batch_size, sampler=train_sampler

52 ),

53 ’val’: torch.utils.data.DataLoader(

54 image_datasets[’val’], batch_size=batch_size, sampler=val_sampler

55 )

56 }

57

58 return dataloaders

59

60def visualize_random_images(dataloader, classes):

61 # Get a batch of training data

62 inputs, labels = next(iter(dataloader))

63

64 # Make a grid from batch

65 out = torchvision.utils.make_grid(inputs)

66

67 plt.imshow(out.permute(1, 2, 0))

68 plt.title([classes[label] for label in labels])

69 plt.show()

70

71dataloaders = get_dataloaders(data_dir)

72visualize_random_images(dataloaders[’train’],
dataloaders[’train’].dataset.classes)

73

74device = torch.device("cuda" if torch.cuda.is_available()

75 else "cpu")

76model = models.resnet50(pretrained=True)

77print(model)

78

79for param in model.parameters():

80 param.requires_grad = False

81

82model.fc = nn.Sequential(nn.Linear(2048, 1024),

83 nn.ReLU(),

84 nn.Dropout(0.2),

85 nn.Linear(1024, 2),

86 nn.LogSoftmax(dim=1))

87criterion = nn.NLLLoss()

88optimizer = optim.Adam(model.fc.parameters(), lr=0.001)

89model.to(device)

90

91epochs = 100

92steps = 0

93running_loss = 0

94print_every = 10

95train_losses, test_losses = [], []

96for epoch in range(epochs):

97 for inputs, labels in trainloader:

98 steps += 1

99 inputs, labels = inputs.to(device), labels.to(device)

100 optimizer.zero_grad()

101 logps = model.forward(inputs)

102 #print(logps, labels.view(-1, 1))

103 loss = criterion(logps, labels)

104 loss.backward()

105 optimizer.step()

106 running_loss += loss.item()

107

108 if steps % print_every == 0:

109 test_loss = 0

110 accuracy = 0

111 model.eval()

112 with torch.no_grad():

113 for inputs, labels in testloader:

114 inputs, labels = inputs.to(device),labels.to(device)

115 logps = model.forward(inputs)

116 batch_loss = criterion(logps, labels)

117 test_loss += batch_loss.item()

118

119 ps = torch.exp(logps)

120 top_p, top_class = ps.topk(1, dim=1)

121 equals = top_class == labels.view(*top_class.shape)

122 accuracy +=torch.mean(equals.type(torch.FloatTensor)).item()

123 train_losses.append(running_loss/len(trainloader))

124 test_losses.append(test_loss/len(testloader))

125 print(f"Epoch {epoch+1}/{epochs}.. "

126 f"Train loss: {running_loss/print_every:.3f}.. "

127 f"Test loss: {test_loss/len(testloader):.3f}.. "

128 f"Test accuracy: {accuracy/len(testloader):.3f}")

129 running_loss = 0

130 model.train()

131torch.save(model, ’sicknessmodel.pth’)

132

133plt.plot(train_losses, label=’Training loss’)

134plt.plot(test_losses, label=’Validation loss’)

135plt.legend(frameon=False)

136plt.show()

137

138from sklearn.metrics import f1_score

139f1_score(y_true, y_pred, average=None)

Listing 16: The classificastion code using a transfer learning process with
the resnet50 pretrained module. Figure 25: Resnet50 loss evolution in function
of the number of epochs. The loss function used is the cross-entropy loss.

###  5.2 Inpainting and clustering

One major issue our data had is the the endoscopic camera’s light in work. In
order to tackle this problem we tried firstly to inpaint the images. We have
considered the areas with light on them as having missing values, so the
inpainting idea, which is the task of reconstructing missing regions in an
image, comes into play. It is an important problem in computer vision and an
essential functionality in many imaging and graphics applications, e.g. object
removal, image restoration, manipulation, re-targeting, compositing, and
image-based rendering. The technique consisted of finding the white pixels in
the images and dilute the surrounding pixels using a certain threshold on the
pixels’ values by working with a mask between 221 and 255, in order to
conserve the maximum amount of information. After inpainting the images, we
noticed that they still contain some noise in them. So in order to tackle this
problem, we tried kmeans [55] for clustering. Clustering algorithms are
unsupervised algorithms, meaning they don’t use labelled data. They are used
to assign data points from a population to different groups where data points
belonging to the same group have similar traits. In our instance, clustering
the image enabled us to blend the several image segments together in order to
lessen the noise that persisted even after the inpainting.

⬇

1import cv2

2import numpy as np

3import matplotlib.pyplot as plt

4

5def inpaint_image(img_path, morph_size=(7,7)):

6 # Load the image

7 img = cv2.imread(img_path)

8 if img is None:

9 raise ValueError("Failed to read image from the given path")

10

11 # Convert the image to grayscale

12 gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)

13

14 # Threshold the grayscale image to create a mask

15 mask = cv2.threshold(gray, 220, 255, cv2.THRESH_BINARY)[1]

16

17 # Perform morphological closing on the mask

18 kernel = cv2.getStructuringElement(cv2.MORPH_ELLIPSE, morph_size)

19 mask = cv2.morphologyEx(mask, cv2.MORPH_CLOSE, kernel, iterations=1)

20

21 # Inpaint the image using the mask

22 result = cv2.inpaint(img, mask, 21, cv2.INPAINT_TELEA)

23

24 # Convert the result to RGB

25 image = cv2.cvtColor(result, cv2.COLOR_BGR2RGB)

26

27 return image

28

29# Specify the path to the input image

30img_path = ’./kvasirpytorch/kvasir-dataset/ulcerative-
colitis/049c2045-5259-47d8-8f9e-bbbecd81789f.jpg’

31

32# Inpaint the image

33image = inpaint_image(img_path)

34

35# Display the result

36plt.imshow(image)

37plt.axis(’off’)

Listing 17: Implementation of the morphological mask and the inpainting
transformation.

Figure 26: In order from top and from left to right, the figures present: the
original and real image extracted directly from the Kvasir database, the image
after application of a segmentation, after application of the algorithm of
inpainting and finally after applying segmentation.

###  5.3 Image classification with PCA

Image classification using principal component analysis (PCA) was performed in
this study to explore an unconventional approach to solving computer vision
problems. While deep learning is commonly used to address these challenges, we
opted to apply gradient boosting. Initially, we extracted the palettes of each
image to obtain a correspondence table of selected colors in the RGB color
space. We extracted the first five dominant color sets and transformed the
images into vectors, which were arranged in a dataframe. However, before
proceeding with the classification process, we applied PCA to reduce the
dimensionality of the data and ensure that our points were represented
optimally, thus increasing the chances of successful classification.

Figure 27: PCA plot of the points

To identify the optimal number of palettes for the images, we plotted the
accuracy of the model against the number of palettes used. Based on the
results, we selected the optimal number of palettes for further tests. We then
evaluated our gradient boosting method, specifically XGBoost [56], a
distributed tree boosting algorithm that utilizes the gradient boosting
framework to create efficient, flexible, and portable machine learning
algorithms. Overall, this study demonstrates the potential of using gradient
boosting and PCA to solve computer vision problems and offers new insights
into the classification of image data. This table shows the confusion matrix
of our results.

P/A | Positive | Negative
---|---|---
Positive | 66 | 10
Negative | 6 | 68

Figure 28: Results of the classification algorithm.

###  5.4 Computer vision solution perspectives

In this part, we are discussing how the previous subsections are going to
introduce the complete architecture of the solution. While the partial
differential equations gives us a solution depending on time, classification
models only provide the current state of the disease. The proposed solution
would be a tailored deep learning architecture having both the regression and
the classification tasks in order to get the full picture on the evolution of
the disease. The regression part of the solution consists of a convolutional
long short term memory (ConvLSTM). This network is provided a sequence of
images depending on time and its output is the next sequence of the disease
meaning the next state. With this architecture we get both the current state
and its evolution both on time and surface.

Figure 29: Computer vision proposed architecture

While the previous propositions might seem a bit shallow, we propose another
approach which is image segmentation. This could be treated also as a
regression problem. We apply an image segmentation model on our images that
would return the surface that has the disease on it. The information we get is
later treated as a tabular data and with sufficient data we could apply a
state of the art regression model in order to determine the evolution of the
surface. Keep in mind that the information provided by the segmentation model
should be treated carefully because of the amount of disturbance that could be
provided by the mask on the detected object.

##  6 Conclusion

This study was carried out with the ultimate goal of being producible and
usable by the scientific community in a challenging Tunisian context.
Digestive cancers are a significant public health concern due to their high
mortality rates. Besides, the mortality depends on the stage at the time of
diagnosis: high mortality for late stage detection vs survival rate for pre-
cancerous stage detection. Ulcerative Colitis and Crohn’s disease may indicate
a potential evolution towards cancer. However, the current diagnostic methods
for these diseases are complicated: requires expert gastroenterologists,
expensive equipments, and has a major impact on the quality of life of
patients (invasive) making early detection challenging. Which brings us to try
to answer several open questions: Can we improve on current methods, make it
less expensive and with less heavy intervention then improving the patient’s
life quality?

This paper aimed to demonstrate the potential of AI in providing a more cost-
effective and less invasive solution for detection by solving partial
differential equations that model the propagation of these diseases. Our
findings provide open-source data and codes, promoting transparency and
encouraging further research. Our preliminary study used unsupervised learning
to model ulcerative colitis and Crohn’s disease due to the lack of good
quality data. Further investigations may be necessary to better understand the
modeling of these diseases and the potential to combine computer vision
techniques and regression models. We have shown that is possible to solve the
Fisher-KPPk equation with low quality data and with a deep neural network. The
difficulty was noticed rather in comparison with the Turing non-linear and
coupled PDEs system. For that, an investigation is necessary to understand the
phenomenon of explosion badly modeled. Finally, we believe that this work was
an opportunity to bring together three communities: gastro-enterologists,
mathematicians, and AI specialists through our proposed experimental framework
15.

## References

  * [1] S. Cuomo, V.S. Di Cola, F. Giampaolo, et al.  Scientific machine learning through physics–informed neural networks: Where we are and what’s next.  J Sci Comput, 92:88, 2022.
  * [2] Kevrekidis I.G. Lu L. Perdikaris P. Karniadakis, G.E. et al.  Physics-informed machine learning.  Nature Reviews Physics, 3(6):422–440, 2021.
  * [3] Aditi Krishnapriyan, Amir Gholami, Shandian Zhe, et al.  Characterizing possible failure modes in physics-informed neural networks.  Advances in Neural Information Processing Systems, 34:26548–26560, 2021.
  * [4] Tchelepi H.A. Fuks O.  Limitations of physics informed machine learning for nonlinear two-phase transport in porous media.  Journal of Machine Learning for Modeling and Computing, 1(1):19–37, 2020.
  * [5] Grégoire Nadin, Eric Ogier-Denis, Ana I. Toledo, and Hatem Zaag.  A turing mechanism in order to explain the patchy nature of crohn’s disease.  Journal of Mathematical Biology, 83(2):12, 2021.
  * [6] Yann LeCun, Yoshua Bengio, and Geoffrey Hinton.  Deep learning.  Nature, 521(7553):436–444, 2015.
  * [7] Hao-nan Wang, Ning Liu, Yi-yun Zhang, Da-wei Feng, Feng Huang, Dong-sheng Li, and Yi-ming Zhang.  Deep reinforcement learning: a survey.  Frontiers of Information Technology & Electronic Engineering, 21(12):1726–1744, 2020.
  * [8] Jonas Degrave, Federico Felici, Jonas Buchli, et al.  Magnetic control of tokamak plasmas through deep reinforcement learning.  Nature, 602(7897):414–419, 2022.
  * [9] Suman Ravuri, Karel Lenc, Matthew Willson, et al.  Skilful precipitation nowcasting using deep generative models of radar.  Nature, 597(7878):672–677, 09 2021.
  * [10] L.C. Evans.  Partial Differential Equations, volume 19 of Graduate Studies in Mathematics.  AMS, 2nd edition, 2010.
  * [11] Jean-Pierre Demailly.  Analyse numérique et équations différentielles : nouvelle édition avec exercices corrigés.  Grenoble Sciences. EDP Sciences, 05 2016.
  * [12] Sören Bartels.  Numerical Methods for Nonlinear Partial Differential Equations.  Springer Series in Computational Mathematics. Springer Cham, 1 edition, 2015.
  * [13] Jan Blechschmidt and Oliver G. Ernst.  Three ways to solve partial differential equations with neural networks — a review.  Gamm, 44(2):e202100006, 06 2021.  "Special Issue: Scientific Machine Learning - Part II".
  * [14] Frank Merle and Hatem Zaag.  On degenerate blow-up profiles for the subcritical semilinear heat equation.  arXiv, 2021.
  * [15] Catherine Le Berre, Ashwin N Ananthakrishnan, Silvio Danese, et al.  Ulcerative colitis and crohn’s disease have similar burden and goals for treatment.  Clin Gastroenterol Hepatol, 18(1):14–23, Jan 2020.
  * [16] Johan Burisch.  Crohn’s disease and ulcerative colitis. occurrence, course and prognosis during the first year of disease in a european population-based inception cohort.  Danish medical journal, 61(1):B4778, Jan 2014.
  * [17] Doreen Busingye et al.  Prevalence of inflammatory bowel disease in the australian general practice population: A cross-sectional study.  PloS one, 16(5):e0252458, 2021.
  * [18] M. Mosli, S. Alawadhi, F. Hasan, et al.  Incidence, prevalence, and clinical epidemiology of inflammatory bowel disease in the arab world: A systematic review and meta-analysis.  Inflamm Intest Dis, 6(3):123–131, 2021.
  * [19] Sami Karoui and et al.  Frequence et facteurs predictifs de colectomie et de coloproctectomie au cours de la rectocolite hemorragique.  La tunisie Medicale, 87(02):115–119, 2009.
  * [20] Maxime Collard.  Les mathématiques au secours de la biologie.  Société Nationale Française de Colo-Proctologie (SNFCP), Nov 2021\.
  * [21] S Kraszewski, W Szczurek, J Szymczak, et al.  Machine learning prediction model for inflammatory bowel disease based on laboratory markers. working model in a discovery cohort study.  J Clin Med, 10(20):4745, Oct 16 2021.
  * [22] R. Makkar and S. Bo.  Colonoscopic perforation in inflammatory bowel disease.  Gastroenterol Hepatol (N Y), 9(9):573–83, 2013.
  * [23] Ana Isis Toledo Marrero.  Reaction-diffusion equations and applications to biological control of dengue and inflammation.  PhD thesis, Université Paris-Nord - Paris XIII, 2021.
  * [24] Safaa Al Ali.  Mathematical modelling of chronic inflammatory bowel diseases.  PhD thesis, Université Paris-Nord - Paris XIII, 2021.
  * [25] Gartner.  Gartner hype cycle for artificial intelligence, 2021.
  * [26] Gartner.  Gartner hype cycle for artificial intelligence, 2022.
  * [27] G. Cybenko.  Approximation by superpositions of a sigmoidal function.  Mathematics of Control, Signals and Systems, 2(4):303–314, 1989\.
  * [28] Kurt Hornik, Maxwell Stinchcombe, and Halbert White.  Multilayer feedforward networks are universal approximators.  Neural Networks, 2(5):359–366, 1989.
  * [29] Andrew R. Barron.  Approximation and estimation bounds for artificial neural networks.  Machine Learning, 14(1):115–133, January 1994.
  * [30] Shiyu Liang and R. Srikant.  Why deep neural networks for function approximation?  In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017.
  * [31] Lu Lu, Pengzhan Jin, Guofei Pang, et al.  Learning nonlinear operators via deeponet based on the universal approximation theorem of operators.  Nature Machine Intelligence, 3(3):218–229, 2021.
  * [32] Zhiping Mao, Lu Lu, Olaf Marxen, et al.  Deepm mnet for hypersonics: Predicting the coupled flow and finite-rate chemistry behind a normal shock using neural-network approximation of operators.  Journal of Computational Physics, 447:110698, 2021.
  * [33] Shengze Cai, Zhicheng Wang, Lu Lu, et al.  Deepm mnet: Inferring the electroconvection multiphysics fields based on operator approximation by neural networks.  Journal of Computational Physics, 436:110296, 2021.
  * [34] John H. Lagergren, John T. Nardini, Michael Lavigne G., Erica M. Rutter, and Kevin B. Flores.  Learning partial differential equations for biological transport models from noisy spatio-temporal data.  Proc. R. Soc. A., 476(20190800), 2020.
  * [35] Konstantin Pogorelov, Kristin Ranheim Randel, Carsten Griwodz, et al.  Kvasir: A multi-class image dataset for computer aided gastrointestinal disease detection.  In Proceedings of the 8th ACM on Multimedia Systems Conference, MMSys’17, pages 164–169, New York, NY, USA, 2017. ACM.
  * [36] J. Y. Lee, J. Jeong, E. M. Song, et al.  Real-time detection of colon polyps during colonoscopy using deep learning: systematic validation with four independent datasets.  Scientific Reports, 10:8379, 2020.
  * [37] Marco Chierici, Nicolae Puica, Matteo Pozzi, et al.  Automatically detecting crohn’s disease and ulcerative colitis from endoscopic imaging.  BMC Medical Informatics and Decision Making, 22(6):300, 2022.
  * [38] M. Raissi, P. Perdikaris, and G.E. Karniadakis.  Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations.  Journal of Computational Physics, 378:686–707, 2019.
  * [39] L. Bottou.  Online Algorithms and Stochastic Approximations.  Cambridge University Press, Cambridge, UK, 1998.
  * [40] Diederik P. Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
  * [41] Jorge Nocedal and Stephen J. Wright.  Numerical Optimization.  Springer Series in Operations Research and Financial Engineering. Springer New York, NY, 2 edition, July 2006.
  * [42] James Martens.  Deep learning via hessian-free optimization.  In International Conference on Machine Learning, 2010.
  * [43] Dong C. Liu and Jorge Nocedal.  On the limited memory bfgs method for large scale optimization.  Mathematical Programming, 45(1):503–528, August 1989.
  * [44] John Duchi, Elad Hazan, and Yoram Singer.  Adaptive subgradient methods for online learning and stochastic optimization.  Journal of Machine Learning Research, 12(61):2121–2159, 2011.
  * [45] Tony Lyons.  The 2-component dispersionless burgers equation arising in the modelling of blood flow.  Communications on Pure and Applied Analysis, 11(4):1563–1576, 2012\.
  * [46] Matylda Jabłońska, Robert Sitarz, and Andrzej Kraslawski.  Forecasting research trends using population dynamics model with burgers’ type interaction.  32:619–624, 2013.
  * [47] Mahesh Gajendran, Priyadarshini Loganathan, Guillermo Jimenez, et al.  A comprehensive review and update on ulcerative colitis,.  Disease-a-Month, 65(12):100851, 2019.  A Comprehensive Review and Update on Ulcerative Colitis.
  * [48] Chaohao Xiao, Xiaoqian Zhu, Fukang Yin, and Xiaoqun Cao.  Physics-informed neural network for solving coupled korteweg-de vries equations.  Journal of Physics: Conference Series, 2031:012056, 2021.
  * [49] Safaa Al-Ali, John Chaussard, Sébastien Li-Thiao-Té, et al.  Automatic bleeding and ulcer detection from limited quality annotations in ulcerative colitis.  Inflammatory Bowel Diseases, 28(Supplement 1):S19 S20, 01 2022.
  * [50] Safaa Al-Ali, Sébastien Li-Thiao-Té, John Chaussard, and Hatem Zaag.  Mathematical modeling of the spatial distribution of lesions in inflammatory bowel disease.  feb 2020.
  * [51] Cyrille Rossant.  IPython Interactive Computing and Visualization Cookbook, Second Edition.  Packt Publishing, 2018.
  * [52] R. A. FISHER.  The wave of advance of advantageous genes.  Annals of Eugenics, 7(4):355–369, 1937.
  * [53] A.N. Kolmogorov, I.G. Petrovsky, and N.S. Piskunov.  Investigation of the equation of diffusion combined with increasing of the substance and its application to a biology problem.  Bulletin of Moscow State University Series A: Mathematics and Mechanics, 1:1–25, 1937.
  * [54] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778, 2016.
  * [55] David Arthur and Sergei Vassilvitskii.  k-means++: the advantages of careful seeding.  In Proceedings of the 18th Annual ACM-SIAM Symposium on Discrete Algorithms (SODA ’07), pages 1027–1035. Society for Industrial and Applied Mathematics, 01 2007.
  * [56] Tianqi Chen and Carlos Guestrin.  Xgboost: A scalable tree boosting system.  In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD ’16), pages 785–794. ACM, 08 2016\.


[affiliation=1,2]PoonehMousavi [affiliation=3]JarodDuret
[affiliation=4]SalahZaiem [affiliation=1,2]LucaDella Libera
[affiliation=5,2]ArtemPloujnikov [affiliation=6,2,1]CemSubakan
[affiliation=1,2,5]MircoRavanelli

# How Should We Extract Discrete Audio Tokens from Self-Supervised
Models?††thanks: Accepted at Interspeech 2024

###### Abstract

Discrete audio tokens have recently gained attention for their potential to
bridge the gap between audio and language processing. Ideal audio tokens must
preserve content, paralinguistic elements, speaker identity, and many other
audio details. Current audio tokenization methods fall into two categories:
Semantic tokens, acquired through quantization of Self-Supervised Learning
(SSL) models, and Neural compression-based tokens (codecs). Although previous
studies have benchmarked codec models to identify optimal configurations, the
ideal setup for quantizing pretrained SSL models remains unclear.

This paper explores the optimal configuration of semantic tokens across
discriminative and generative tasks. We propose a scalable solution to train a
universal vocoder across multiple SSL layers. Furthermore, an attention
mechanism is employed to identify task-specific influential layers, enhancing
the adaptability and performance of semantic tokens in diverse audio
applications.

###### keywords:

discrete audio token, semantic token, representation learning, speech
processing.

##  1 Introduction

Learning effective, efficient, and robust representations is a core problem in
modern audio and speech processing systems [1]. Over the past few years,
continuous representations learned by large self-supervised models such as
Wav2Vec2 [2], WavLM [3], and HuBERT [4] have achieved unprecedented
performance. A recent research trend consists of learning discrete audio
representations instead of continuous ones, resulting in what is known as
audio tokens. These discrete tokens offer several potential advantages.
Firstly, they facilitate the development of audio language models (LMs) [5, 6,
7, 8, 9, 10] and the creation of multi-modal large language models [11], which
can emit audio, text, and visual tokens. Additionally, their compression
potential can contribute to efficient data transmission and storage. Discrete
tokens also enable us to address audio generation tasks such as speech
enhancement and synthesis using classification methods, instead of relying on
complex high-dimensional regression models.

Following the terminology from [5, 12], audio tokenization techniques can be
broadly categorized into Compression-based (codecs) tokens and Semantic
tokens. Compression-based tokens [13, 14, 15, 16] utilize encoder-decoder
architectures coupled with Residual Vector Quantization (RVQ) [13]. They are
explicitly trained to accurately reconstruct the original audio, making them
particularly suitable for audio generation tasks. Semantic tokens [17, 18,
19], on the other hand, are generated through clustering or quantization of
the layers of Self-Supervised Learning (SSL) models [2, 3, 4]. Often, this
involves selecting a layer from the pretrained SSL model and clustering its
representations, typically with the k-means algorithm. Semantic tokens
primarily capture coarse information such as phonetic, semantics, and
syntactic details. Since they are not explicitly trained to achieve accurate
waveform reconstruction, it is more natural to use them in discriminative
tasks like Automatic Speech Recognition (ASR). Recent research, however, has
shown that semantic tokens can be effective for generative tasks as well [20,
21]. Additionally, semantic tokens have been used in a hybrid tokenizer [12,
22]. This hybrid approach combines semantic and compression-based tokens,
separating content information in the initial layer while preserving
paralinguistic details in subsequent layers. A similar strategy has been
widely adopted in audio LLMs [5, 6, 7]. Nevertheless, the most effective
setting for extracting semantic tokens remains largely unclear. Recent studies
have primarily focused on ASR and Speech Translation [23, 24, 25], without
considering a broader range of discriminative and generative tasks.

This paper addresses this gap by evaluating the effects of different
heuristics required to derive semantic tokens for several discriminative and
generative tasks, such as speech recognition, speaker recognition, emotion
classification, speech enhancement, and text-to-speech. We investigate various
crucial aspects, including the impact of the number of clusters and the
selection of the intermediate layer of the SSL model to discretize. The latter
factor turned out to be crucial and task-dependent, as early layers capture
low-level information and higher layers encode content and semantic nuances.
Common strategies include using the middle layer [20, 17] or leveraging the
last layer [25]. Instead of relying on partial information only, we introduced
a novel technique based on an informed layer selection mechanism. We propose
to cluster all layers and inject their information into the acoustic models
using learnable attention weights. This approach significantly boosts
performance while also providing valuable insights into the importance of each
layer.

Since there is no built-in decoder in semantic tokens, a vocoder model for
converting the semantic tokens into audio must be trained [26, 27]. Training
such a vocoder is computationally demanding, making it highly impractical to
train a separate vocoder for each layer or combination of layers. To address
this challenge, we propose a novel scalable vocoder capable of operating with
various layer combinations at no additional cost. This is achieved through a
layer dropout training scheme, inspired by the bitrate scalability mechanism
used in SoundStream [13]. Interestingly, our results show that the scalable
vocoder outperforms all vocoders trained on every specific layer. Finally, for
a comprehensive comparison, we provide experimental evidence using both in-
domain and out-of-domain datasets for training k-means. For reproducibility
and to encourage further research, we release the code, built on the popular
SpeechBrain [28] toolkit, and pretrained models
publicly111github.com/speechbrain/benchmarks/tree/DASB.

##  2 Model Design

The proposed architecture, illustrated in Fig. 1, consists of four components
a) Tokenizer,  b) Informed Layer Selector,  c) Acoustic Model, and  d)
Scalable Vocoder  . The following subsections will describe each module.

###  2.1 Tokenizer

For quantization, we cluster five layers taken from two pretrained SSL models
using the k-means algorithm independently for each layer. We consider two
widely-used models: WavLM-large222huggingface.co/microsoft/wavlm-large and
HuBERT-large333huggingface.co/facebook/hubert-large-ll60k, both having 24
layers. We choose two layers from the lower part (3, 7) to capture fine-
grained information, the middle layer (12), and two layers from the higher
part (18, 23) for encoding content and meaning. This selection is based on
observation from prior research [3, 29] which studied the contribution
patterns of different layers across various tasks. As a result, this set of
discrete hierarchical tokens captures rich information from the original audio
signal. Each of the K clusters is assigned a unique index. Additionally, we
store the continuous coordinates of each centroid for studying the effect of
initializing input embeddings in downstream acoustic models (Sec. 4.4). The
outcome of this tokenization process is a tensor $\mathbf{d}$ of shape
$B\times T\times n_{l}$, where B represents the batch size, T is the sequence
length, and $n_{l}$ is the number of discretized layers.

Figure 1: The proposed method for audio token extraction from SSL models: (A)
k-means discretizes the continuous representations of each layer, (B) an
attention mechanism merges the discrete layer representations, (C) the mixed
representations train acoustic models for discriminative and generative tasks,
(D) our scalable vocoder generates waveforms (if needed).

###  2.2 Informed Layer Selector

As evident from the SSL literature [30, 29, 31], the choice of the layer
within the SSL model significantly influences the performance of the
downstream task of interest. This decision is equally critical for semantic
tokens. Unlike prior methods that rely on heuristic layer selection [20, 17,
25], we integrate the information from our hierarchical multi-layer audio
tokens with an attention mechanism. The attention mechanisms comprise a
straightforward multi-layer perceptron (MLP) fed by the embeddings of the
audio tokens from each layer. The MLP generates a score for each selected
layer, that is normalized by a softmax function as shown in the following
equations:

$z_{l,t}=f\big{(}emb(d_{l,t})\big{)}\\\ $ (1)
$a_{l,t}=\frac{exp(z_{l,t})}{\sum_{k=1}^{nl}exp(z_{k,t})},\quad
h_{t}=\sum_{l}a_{l,t}z_{l,t},$ (2)

where, $z_{l,t}$ represents the score assigned to layer $l$ at time $t$ by the
MLP function $f$. The variable $emb$ refers to the lookup table that assigns
embeddings to discrete tokens in $d_{l}$. The variable $a_{l,t}$ denotes the
attention assigned to layer $l$ at time $t$, and lastly $h_{t}$ is the
representation that is fed to the downstream MLP model. Note that we learn
different layer combinations at each time-step, making this mechanism
particularly effective.

This simple yet effective approach offers several advantages. Firstly, it
enhances flexibility by reducing reliance on heuristic layer selection. The
model can now dynamically capture information from different layers for each
task. Additionally, as shown in Sec.4, this mechanism yields performance
improvements when compared with models utilizing information from a single SSL
layer. Lastly, the informed layer selections enhance interpretability,
enabling us to analyze the learned weights and understand the relative
importance of each layer for each downstream task.

###  2.3 Acoustic Model

The mixed representations are fed to a neural model trained to address various
downstream tasks444We train the attention mechanism, embeddings, and the
acoustic models jointly.. While previous studies [23, 24, 25] have primarily
focused on a few discriminative tasks, we aim to provide evidence across a
diverse range of speech applications, considering both discriminative and
generative tasks. We consider ASR, speaker identification, and emotion
recognition as discriminative tasks. For generative tasks, we focus on text-
to-speech and speech enhancement. The details for each task are reported in
Sec. 3.

###  2.4 Scalable Vocoder

Although SSL models such as Wav2vec2, HuBERT, and WavLM are not designed for
accurate waveform reconstruction, we can potentially adapt them for generative
tasks by training a vocoder on top of their representations. The dominant
approach involves training a separate vocoder for each possible layer
combination. However, this approach is impractical and computationally
demanding since each downstream task may require a different set of layers. In
this work, we propose a universal and scalable vocoder capable of
accommodating various layer combinations. To train such a model, we modify
HiFi-GAN [16] to accept a variable number of multi-layer discrete tokens as
input. We introduce a layer dropout mechanism, similar to structured dropout
[32]. For each input example, we randomly sample $k$ layers from the range
$[1,n_{l}]$, as shown in the following equations:

$\mathbf{d}_{S}\sim\text{Sample}(\mathbf{d},k),\quad\mathbf{o}=V(\mathbf{d}_{S}),\\\
$ (3)

where `Sample($\cdot$)' randomly selects $k$ layers from the discrete
representations $\mathbf{d}$, and $V$ represents the vocoder function that
outputs the waveform $\mathbf{o}$. Layers are combined with an attention
mechanism that assigns weights to different layers and ensures that the
dimensionality of the embeddings remains consistent regardless of the number
of layers. The model is trained to decode audio by considering all possible
combinations of layers. During inference, the desired combination of layers
can be selected. In addition to its flexibility, this vocoder has demonstrated
superior performance compared to vocoders trained on single layers, as we will
show in Sec. 4.

##  3 Experiments

The tasks in our experiments are divided into two groups: Discriminative tasks
involving transcription and classification, and generative tasks producing
audio. For the downstream architecture choices and training procedures, we
follow the best-performing approaches for classic continuous self-supervised
representations [30]. We employ 1000 centroids across all tasks, except for
ASR and emotion recognition, where we adopt 2000 centroids based on insights
from prior research on ASR with discrete representations [25]. The effect of
this selection is probed in Sec. 4.3.

(a)

(b)  (c)

Figure 2: Performance of the Scalable Vocoder (SV) at different layers
compared to a Single-Layer Vocoder (SLV). Vocoders and tokenizers are trained
using the LJSpeech dataset with 1000 and 2000 centroids.

###  3.1 Discriminative Tasks

Automatic Speech Recognition (ASR): We consider two CTC-based speech
recognition tasks. The first one is English ASR using Librispeech train-
clean-100 for training and test-clean, test-other for testing. The second one
uses French data coming from the CommonVoice (CV) 16.1 Corpus [33]. We select
$100$ hours for training, keeping the original validation and test sets. We
use two layers of BiLSTM as a downstream head. The evaluation metric is the
Word Error Rate (WER).

Speaker Identification (SID): We employ an ECAPA-TDNN model [34] to determine
the speaker identity of each utterance. The widely used VoxCeleb1 [35] is
adopted, and the evaluation metric is accuracy (ACC).

Emotion Recognition (ER): We use ECAPA-TDNN for emotion recognition [36] on
the IEMOCAP dataset. The task consists of predicting one of the four
considered classes: happy, sad, angry, and neutral. The evaluation metric is
accuracy (ACC).

###  3.2 Generative Tasks

Speech Enhancement (SE): We utilize a non-autoregressive transformer encoder
[37], which consists of 6 layers, 4 attention heads, a model dimension of 256,
and a feed-forward layer dimension of 2048. Input tokens are extracted from
the noisy signal, and target tokens from the clean one. Training is conducted
end-to-end using cross-entropy loss. Noisy samples are generated by mixing
clean samples from LJSpeech [38] with noise from WHAM! [39]. The signal-to-
noise ratios (SNRs) are uniformly distributed between 0 and 5 dB. Due to the
misalignment of the vocoder's output with the target at the sample level,
metrics like Si-SNR can be degraded. Therefore, we use the deep noise
suppression mean opinion score (DNSMOS) [40] for the speech quality metric,
following a previous study [20]. Intelligibility is evaluated through the
differential word error rate (dWER) [41], which measures the WER between the
transcribed enhanced signal and the transcribed target signal. Transcriptions
are obtained using the small version of Whisper [42].

Text-to-Speech (TTS): We train an end-to-end autoregressive Transformer[37]
with 6 layers in the encoder, 12 layers in the decoder, 4 attention heads, a
model dimension of 512, and a feed-forward layer in 2048. To facilitate
convergence, we employ guided attention [43]. The model takes text embeddings
as its input and generates the audio tokens for each considered layer. We
utilize a shared transformer decoder, where each tokenizer head has its own
learned embedding, and there is a distinct final linear layer for each token.
We train all models on the LJSpeech dataset [38]. For assessing speech
quality, we use UTMOS [44] to estimate human quality ratings. To evaluate
fidelity to the text, we assess generated samples using the WER computed with
the small version of Whisper [42].

##  4 Results

###  4.1 Scalable Vocoder

Figure 3: Attention analysis across various tasks and layers of the discrete
WavLM model with in-domain tokenizers. Table 1: Assessing the impact of the
number of clusters and embedding initialization on discrete WavLM-Large across
different tasks.

Setting | ASR (EN) | ASR (FR) | SID | ER | SE | TTS
---|---|---|---|---|---|---
WER $\downarrow$ |  WER $\downarrow$ |  ACC $\uparrow$ |  ACC $\uparrow$ | DNSMOS$\uparrow$ | dWER$\downarrow$ | UTMOS$\uparrow$ |  WER $\downarrow$
Effect of Number Of Clusters
1000 | 7.15 | 34.61 | 79.0 | 61.8 | 3.93 | 6.75 | 3.65 | 5.76
2000 | 6.96 | 32.94 | 79.5 | 67.2 | 3.93 | 6.58 | 3.55 | 5.62
Effect of Embedding Initialization
Random | 6.96 | 32.94 | 81.0 | 67.2 | 3.93 | 6.75 | 3.65 | 5.76
PreTrained & finetune | 8.93 | 35.81 | 77.5 | 63.9 | 3.93 | 6.82 | 3.64 | 6.62
PreTrained & freeze | 9.26 | 35.12 | 73.1 | 67.0 | 3.93 | 6.98 | 3.66 | 6.42

Table 2: Out-of-domain and in-domain performance of discrete HuBERT and WavLM
models across the downstream tasks.

SSL Model | Tokenizer | ASR (EN) | ASR (FR) | SID | ER | SE | TTS | Vocoder
---|---|---|---|---|---|---|---|---
WER $\downarrow$ |  WER $\downarrow$ |  ACC $\uparrow$ |  ACC $\uparrow$ | DNSMOS$\uparrow$ | dWER$\downarrow$ | UTMOS$\uparrow$ |  WER $\downarrow$ | UTMOS$\uparrow$ | dWER$\downarrow$
HuBERT Large[4] | In-Domain | 7.89 | 38.29 | 67.2 | 64.5 | 3.98 | 17.64 | 3.61 | 6.46 | 3.50 | 4.49
Out-Of-Domain | N/A | 39.50 | 67.8 | 61.7 | 3.95 | 15.92 | 3.54 | 5.45 | 3.48 | 2.92
WavLM Large[3] | In-Domain | 6.96 | 32.94 | 81.0 | 67.2 | 3.93 | 6.75 | 3.65 | 5.76 | 3.49 | 2.98
Out-Of-Domain | N/A | 36.25 | 79.0 | 61.9 | 3.96 | 6.49 | 3.61 | 5.73 | 3.68 | 2.95

Our results cover findings from two distinct setups: 1) a scalable vocoder
trained across five layers, and 2) a vocoder trained on a single layer. In
both setups, the tokenizers and the vocoders are trained with LJSpeech (in-
domain condition). In the first scenario, models are trained with HuBERT
discrete tokens and WavLM discrete tokens, each with the number of clusters
set to 1000. To further explore the influence of k-means cluster size on
speech quality, we introduce an additional model with the number of clusters
set to 2000. In the second setup, we focus on models trained specifically on a
single layer (3 or 23) using HuBERT discrete tokens and in-domain tokenizer.
This experiment aims to compare the performance of the scalable vocoder
against the vocoder trained on a single layer.

The results are summarized in Fig. 2. WavLM combined with an in-domain
tokenizer achieves higher UTMOS and lower dWER scores across all setups. About
the impact of the number of clusters, our experiment shows that setting $k$ to
2000 degrades the quality of synthesized speech. Finally, both models trained
on a single layer are outperformed on both evaluation metrics by the one
trained on five layers, confirming the benefits of the scalable approach.
Lastly, we explore an out-of-domain scenario where the tokenizers are trained
on LibriSpeech and the vocoders are on LJSpeech. As shown in Table 2 (last
column), we do not observe any significant performance degradation when using
the scalable vocoder in an out-of-domain condition.

###  4.2 Layer Analysis

Fig. 3 depicts the average weights assigned to different layers in the WavLM
model across various downstream tasks on the test dataset. In both TTS and the
scalable vocoder, lower levels get greater importance as they prioritize
effective reconstruction. Conversely, for ASR, the upper layers become more
crucial in capturing the semantic aspects of spoken utterances. In the case of
ER and SID, the third layer receives the highest weight. Our findings align
with the observed pattern in continuous representations [45]. For SE, all
layers are equally weighted, indicating the necessity of all hierarchical
levels to achieve optimal audio quality while preserving the semantic content
of the input.

###  4.3 Effect of Number of Clusters

We train k-means models with both 1000 and 2000 centroids and examine the
impact of the number of clusters across different tasks, as illustrated in
Table 1. In Generative tasks, TTS and SE, no significant differences are
observed between models trained with 1000 and 2000 clusters. However, for ASR
in both English and French, as well as ER, models with a higher number of
clusters outperform those with fewer clusters. In the case of SID, the model
trained with 1000 clusters exhibits comparable accuracy to the model with 2000
centroids. As expected, the ideal number of clusters is task-dependent. For
multi-modal LLMs where a single set of tokens is desired to solve multiple
tasks, we recommend a cluster count between 1000 and 2000.

###  4.4 Effect of Embedding Initialization

We study various configurations for initializing the embedding layers of audio
tokens (Table 1). Three options are considered:  1) Random initialization of
the embedding layers,  2) Initialization of the embedding layer with the
corresponding centroid's embedding, while freezing the layer, and  3)
Initialization of the embedding layer with the corresponding centroid's
embedding, without freezing the layer.  Across all tasks, there is no
advantage observed in initializing the embedding with pretrained centroid
embeddings, and random initialization consistently outperforms it in all
scenarios. However, discriminative tasks show greater benefits from random
initialization, while generative tasks exhibit comparable performance across
all three settings. This observation eliminates the need for having the same
embedding size as the SSL models, allowing the choice of a smaller and more
efficient embeddings.

###  4.5 Out-of-Distribution Generalization

To evaluate the robustness of discrete representations under distribution
shifts, we train tokenizers on both in-domain and out-of-domain datasets
(Table 2). In discriminative tasks, k-means models are trained using the same
dataset employed for training acoustic models. In the out-of-domain scenarios,
k-means models are trained on train-clean-100, train-clean-360, and train-
other-500. For generative tasks, k-means models are trained on LJSpeech for
in-domain evaluation and LibriSpeech-960h for OOD evaluation, while both the
acoustic model and vocoder are trained on LJSpeech. For all discriminative
tasks, the in-domain tokenizer outperforms its OOD counterpart. Interestingly,
in all generative tasks, training the model using the OOD tokenizer does not
adversely affect performance and, in some instances, even improves the
results. We speculate that this trend may arise because generative tasks
primarily depend on tokens capturing low-level information, which tends to be
more ``universal" and transferable across different domains.

##  5 Conclusions

Discrete semantic tokens, derived from the quantization of SSL models, play an
important role, providing ``pseudo-text" valuable for training text-free
speech language models and multimodal LLMs. We explore the optimal
configuration of semantic tokens across discriminative and generative tasks.
We introduce a novel technique involving an informed layer selection
mechanism, utilizing learnable attention weights to integrate information from
different SSL layers. This approach significantly enhances the performance and
interpretability of the model. Furthermore, we propose a scalable solution for
training a universal vocoder across multiple SSL layers, demonstrating its
superiority over vocoders trained on specific layers. As future work, we plan
to explore more diverse tasks and quantization methods, and the development of
a multi-speaker vocoder.

## References

  * [1] Y. LeCun, Y. Bengio, and G. Hinton, ``Deep learning,'' _nature_ , vol. 521, no. 7553, p. 436, 2015.
  * [2] A. Baevski, Y. Zhou, A. Mohamed, and M. Auli, ``wav2vec 2.0: A framework for self-supervised learning of speech representations,'' _Proc. of NeurIPS_ , 2020.
  * [3] S. Chen _et al._ , ``Wavlm: Large-scale self-supervised pre-training for full stack speech processing,'' _IEEE Journal of Selected Topics in Signal Processing_ , vol. 16, no. 6, pp. 1505–1518, 2022.
  * [4] W.-N. Hsu, B. Bolte, Y.-H. H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed, ``Hubert: Self-supervised speech representation learning by masked prediction of hidden units,'' _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 29, pp. 3451–3460, 2021.
  * [5] Z. Borsos _et al._ , ``Audiolm: a language modeling approach to audio generation,'' _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 31, pp. 2523–2533, 2023.
  * [6] P. K. Rubenstein _et al._ , ``Audiopalm: A large language model that can speak and listen,'' _arXiv preprint arXiv:2306.12925_ , 2023.
  * [7] Q. Chen _et al._ , ``Lauragpt: Listen, attend, understand, and regenerate audio with gpt,'' _arXiv preprint arXiv:2310.04673_ , 2023.
  * [8] T. Wang _et al._ , ``Viola: Unified codec language models for speech recognition, synthesis, and translation,'' _arXiv preprint arXiv:2305.16107_ , 2023.
  * [9] C. Wang _et al._ , ``Neural codec language models are zero-shot text to speech synthesizers,'' _arXiv preprint arXiv:2301.02111_ , 2023.
  * [10] X. Wang _et al._ , ``Speechx: Neural codec language model as a versatile speech transformer,'' _arXiv preprint arXiv:2308.06873_ , 2023.
  * [11] Gemini-Team, ``Gemini: A family of highly capable multimodal models,'' 2023.
  * [12] X. Zhang, D. Zhang, S. Li, Y. Zhou, and X. Qiu, ``Speechtokenizer: Unified speech tokenizer for speech large language models,'' _arXiv preprint arXiv:2308.16692_ , 2023.
  * [13] N. Zeghidour, A. Luebs, A. Omran, J. Skoglund, and M. Tagliasacchi, ``Soundstream: An end-to-end neural audio codec,'' _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 30, pp. 495–507, 2021.
  * [14] A. Défossez, J. Copet, G. Synnaeve, and Y. Adi, ``High fidelity neural audio compression,'' _arXiv preprint arXiv:2210.13438_ , 2022.
  * [15] R. Kumar, P. Seetharaman, A. Luebs, I. Kumar, and K. Kumar, ``High-fidelity audio compression with improved rvqgan,'' _Proc. of NeurIPS_ , 2023.
  * [16] D. Yang, S. Liu, R. Huang, J. Tian, C. Weng, and Y. Zou, ``Hifi-codec: Group-residual vector quantization for high fidelity audio codec,'' _arXiv preprint arXiv:2305.02765_ , 2023.
  * [17] A. Polyak, Y. Adi, J. Copet, E. Kharitonov, K. Lakhotia, W.-N. Hsu, A. Mohamed, and E. Dupoux, ``Speech Resynthesis from Discrete Disentangled Self-Supervised Representations,'' in _Proc. Interspeech_ , 2021.
  * [18] D. Wells, H. Tang, and K. Richmond, ``Phonetic analysis of self-supervised representations of english speech,'' in _Proc. of Interspeech_ , 2022.
  * [19] Y.-A. Chung, Y. Zhang, W. Han, C.-C. Chiu, J. Qin, R. Pang, and Y. Wu, ``W2v-bert: Combining contrastive learning and masked language modeling for self-supervised speech pre-training,'' in _Proc. of ASRU_ , 2021.
  * [20] Z. Wang, X. Zhu, Z. Zhang, Y. Lv, N. Jiang, G. Zhao, and L. Xie, ``SELM: Speech Enhancement Using Discrete Tokens and Language Model,'' _arXiv preprint arXiv:2312.09747_ , 2023.
  * [21] Y. Yang, F. Shen, C. Du, Z. Ma, K. Yu, D. Povey, and X. Chen, ``Towards universal speech discrete tokens: A case study for asr and tts,'' in _Proc. of ICASSP_ , 2024.
  * [22] Z. Du, S. Zhang, K. Hu, and S. Zheng, ``Funcodec: A fundamental, reproducible and integrable open-source toolkit for neural speech codec,'' _arXiv preprint arXiv:2309.07405_ , 2023.
  * [23] X. Chang _et al._ , ``Exploring speech recognition, translation, and understanding with discrete speech units: A comparative study,'' _arXiv preprint arXiv:2309.15800_ , 2023.
  * [24] D. Zhang, R. Ye, T. Ko, M. Wang, and Y. Zhou, ``DUB: Discrete unit back-translation for speech translation,'' in _Proc. of ACL_ , 2023.
  * [25] X. Chang, B. Yan, Y. Fujita, T. Maekaku, and S. Watanabe, ``Exploration of efficient end-to-end asr using discretized input from self-supervised learning,'' _arXiv preprint arXiv:2305.18108_ , 2023.
  * [26] J. Kong, J. Kim, and J. Bae, ``Hifi-gan: generative adversarial networks for efficient and high fidelity speech synthesis,'' in _Proc. of NeurIPS_ , 2020.
  * [27] R. Prenger, R. Valle, and B. Catanzaro, ``Waveglow: A flow-based generative network for speech synthesis,'' _arXiv preprint arXiv:1811.0000_ , 2018.
  * [28] T. P. Mirco Ravanelli _et al._ , ``SpeechBrain: A general-purpose speech toolkit,'' _arXiv preprint arXiv:2106.04624_ , 2021.
  * [29] S.-w. Yang _et al._ , ``Superb: Speech processing universal performance benchmark,'' _arXiv preprint arXiv:2105.01051_ , 2021.
  * [30] S. Zaiem, Y. Kemiche, T. Parcollet, S. Essid, and M. Ravanelli, ``Speech self-supervised representation benchmarking: Are we doing it right?'' in _Proc. of Interspeech_ , 2023.
  * [31] S. Zaiem, R. Algayres, T. Parcollet, E. Slim, and M. Ravanelli, ``Fine-tuning strategies for faster inference using speech self-supervised models: A comparative study,'' in _Proc. of ICASSP_ , 2023.
  * [32] N. Srivastava _et al._ , ``Dropout: a simple way to prevent neural networks from overfitting,'' _The journal of machine learning research_ , vol. 15, no. 1, pp. 1929–1958, 2014.
  * [33] R. Ardila, M. Branson, K. Davis, M. Kohler, J. Meyer, M. Henretty, R. Morais, L. Saunders, F. Tyers, and G. Weber, ``Common voice: A massively-multilingual speech corpus,'' in _Proc. of LREC_ , 2020.
  * [34] B. Desplanques, J. Thienpondt, and K. Demuynck, ``Ecapa-tdnn: Emphasized channel attention, propagation and aggregation in tdnn based speaker verification,'' in _Proc. of Interspeech_ , 2020.
  * [35] A. Nagrani, J. S. Chung, and A. Zisserman, ``Voxceleb: a large-scale speaker identification dataset,'' 2017.
  * [36] C. Busso, M. Bulut, C.-C. Lee, A. Kazemzadeh, E. Mower, S. Kim, J. N. Chang, S. Lee, and S. S. Narayanan, ``IEMOCAP: Interactive emotional dyadic motion capture database,'' _Language resources and evaluation_ , vol. 42, pp. 335–359, 2008.
  * [37] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, ``Attention is all you need,'' in _Proc. of NIPS_ , 2017.
  * [38] K. Ito, ``The LJ speech dataset,'' https://keithito.com/LJ-Speech-Dataset/, 2017.
  * [39] G. Wichern _et al._ , ``WHAM!: Extending speech separation to noisy environments,'' in _Proc. of Interspeech_ , 2019.
  * [40] C. K. Reddy, V. Gopal, and R. Cutler, ``DNSMOS P.835: A non-intrusive perceptual objective speech quality metric to evaluate noise suppressors,'' in _Proc. of ICASSP_ , 2022.
  * [41] Z.-Q. Wang _et al._ , ``Sequential multi-frame neural beamforming for speech separation and enhancement,'' in _Proc. of SLT_ , 2021.
  * [42] A. Radford _et al._ , ``Robust speech recognition via large-scale weak supervision,'' _arXiv preprint arXiv:2212.04356_ , 2022.
  * [43] H. Tachibana, K. Uenoyama, and S. Aihara, ``Efficiently trainable text-to-speech system based on deep convolutional networks with guided attention,'' in _Proc. of ICASSP_ , 2018.
  * [44] S. Takaaki _et al._ , ``UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022,'' _Proc. of Interspeech_ , 2022.
  * [45] S. Zaiem, Y. Kemiche, and T. Parcollet, ``Speech Self-Supervised Representations Benchmarking: a Case for Larger Probing Heads,'' 7 2023.