# Iterative pseudo-forced alignment by acoustic CTC loss
for self-supervised ASR domain adaptation

###### Abstract

High-quality data labeling from specific domains is costly and human time-
consuming. In this work, we propose a self-supervised domain adaptation
method, based upon an iterative pseudo-forced alignment algorithm. The
produced alignments are employed to customize an end-to-end Automatic Speech
Recognition (ASR) and iteratively refined. The algorithm is fed with frame-
wise character posteriors produced by a seed ASR, trained with out-of-domain
data, and optimized throughout a Connectionist Temporal Classification (CTC)
loss. The alignments are computed iteratively upon a corpus of broadcast TV.
The process is repeated by reducing the quantity of text to be aligned or
expanding the alignment window until finding the best possible audio-text
alignment. The starting timestamps, or temporal anchors, are produced uniquely
based on the confidence score of the last aligned utterance. This score is
computed with the paths of the CTC-alignment matrix. With this methodology, no
human-revised text references are required. Alignments from long audio files
with low-quality transcriptions, like TV captions, are filtered out by
confidence score and ready for further ASR adaptation. The obtained results,
on both the Spanish RTVE2022 and CommonVoice databases, underpin the
feasibility of using CTC-based systems to perform: highly accurate audio-text
alignments, domain adaptation and semi-supervised training of end-to-end ASR.

Index Terms: Forced-alignment, Iterative, Automatic Speech Recognition,
Connectionist Temporal Classification, Speech Segmentation, Closed-Captions

##  1 Introduction

The quality and quantity of data required to train ASR systems are
fundamental. Even more in end-to-end ASR systems, which require more data than
hybrid Deep Neural Network (DNN) - Hidden Markov Models (HMM) to learn
acoustic representations [1]. When it comes to medium and low-resource
languages, the maximum exploitation of data is sought, given that manual
annotations are costly and time-consuming. A common technique is to retrieve
audio-to-text alignments from available audio data that has text references
from the internet or TV broadcast (e.g. conferences, subtitles). Some
alignment methods only work under strict conditions as human-revised
transcriptions [2]. Nevertheless, in the major part of the data available, the
transcription does not exactly match the spoken content, depending on the
method that has been used to generate the text (e.g. respeaking for
subtitling). Therefore, other methods have explored alignment retrieval with
low-quality text references [3, 4, 5, 1]. Some of these systems use a post-
filtering process based on a confidence score to overcome mistaken references.
Additionally, there is work with unaligned speech and text but requires word-
level segmentation [6].

More recently, the use of non-labeled data has been explored. Either in semi-
supervised approaches, with pseudo-labeling of unlabeled audio [7]. Or with
self-supervised approaches, learning speech representations that are
subsequently used in a concrete task (supervised), requiring less quantity of
labeled data to achieve state-of-the-art results [8, 9, 10].

In this work, we train and adapt to the TV broadcast domain an ASR in the
Spanish language. Firstly, we fine-tune, only acoustically by a CTC loss, a
model that was formerly trained with non-labeled data from several languages.
Then, we use the trained ASR to perform utterance-level alignments on a corpus
comprised of Spanish TV broadcast data. For the alignments, we developed an
anchor-based algorithm that generates pseudo-forced alignments with an
iterative approach. The aligned data is used to continue training the model.
Hence, we propose a framework that consists of a multi-step training process
that iteratively improves the former ASR performance and adapts it, step by
step, to the specific broadcast TV acoustics and language domain.

##  2 Related work

###  2.1 Spanish Speech Recognition

Recent tools and challenges have promoted the improvement of Spanish ASR. To
begin with, Hugging Face’s Robust Speech Recognition Challenge sought
improvement in ASR systems in more than 50 languages. The participants used a
fine-tuned version of Wav2Vec2-XLS-R (300M, 1B, or 2B of parameters) with the
CommonVoice database, and decoding based on an n-gram language model (LM).
Additionally, any pre and post-processing system, such as denoising mechanisms
or spelling correctors, was allowed. The winner in the Spanish language
achieved a Word Error Rate (WER) of 8.82% in the test set of CommonVoice v6.0,
reducing it to 6.27% with an LM [11]. Moreover, the Speech-To-Text Albayzin
Evaluation consists of automatic transcription for Spanish TV shows. The
generated transcripts are compared with human revised transcriptions. In the
previous editions, the Machine Learning and Language Processing (MLLP)
research group achieved the best results, in 2021 a 16.06% WER in the test
split of RTVE2020DB [12].

###  2.2 Audio-to-Text alignments

Audio-to-text alignments enrich transcriptions with temporal references to the
spoken content. They are required in fields such as closed-captioning [13],
audiobooks or database generation for speech technologies [14, 2].
Traditionally, alignments have been performed with HMM-based models and the
Viterbi algorithm [15, 3, 2, 14]. Frameworks as sphinx [16] and MAUS [17] are
based on these technologies. This approach presents many inconveniences [15,
1, 13], such as the assumption that the transcription contains the exactly
spoken utterance, problems with out-of-vocabulary words, problems with long
audio tracks as the Viterbi algorithm fails to scale well with speech length;
and Viterbi losses may lead to a complete misaligned file due to incapacity to
recover.

Recently, in [1], a segmentation mechanism using CTC-based end-to-end networks
was proposed. It shows better results for utterance-alignment in German,
compared with different tools as MAUS (based in HTK [18]) or Gentle (based in
Kaldi [19]). Our work is based on this alignment mechanism so it will be
described in detail in section 3.1.

##  3 Methodology

###  3.1 CTC-alignment algorithm

The algorithm proposed in [1] requires an end-to-end ASR trained with already
aligned data and optimized with a CTC loss. Having a $T$-length acoustic
observation $\bm{x}_{t}$, associated with a text reference of length $M-2$. A
special token is added at the beginning of the sequence and the blank token
added at the end of the sequence. Let’s consider this $M$-length sequence of
characters to be aligned, which indexes are represented by $j\in[1,M]$, and a
discrete sequence of acoustic frames $a=\\{a_{1},a_{2},...,a_{T}\\}$ within
the interval $t\in[1,T]$. The alignment task is defined as the assigning of
the corresponding character index $j$ to each temporal element $a_{t}$ within
the $a$ sequence. The alignment works as follows:

  1. 1.

The network generates frame-wise character posteriors $P(c|\bm{x}_{t})$ that
are used to calculate the maximum joint probabilities ($k_{t,j}$). A trellis
diagram is drawn by taking the most probable of two possible transitions: the
next character in reference text or the blank symbol ($\varnothing$). The
transition cost for the first character in text reference is set to zero,
allowing arbitrary start of the sequence in any frame.

  2. 2.

Character-wise alignments ($a_{t}$) are calculated with backtracking. The last
character alignment, which is the start of the algorithm, is selected by
taking the most probable temporal position of the character. For the rest of
the alignments, the most probable transitions determine the alignments.

Going deeper, the joint probabilities are defined as follows:

$k_{t,j}=\begin{cases}\begin{split}\max(k_{t-1,j}\cdot
P(\varnothing|\bm{x}_{t}),\\\ k_{t-1,j-1}\cdot
P(c_{j}|\bm{x}_{t}))\end{split}&\text{if }t>0\wedge j>0\\\ 0,&\text{if
}t=0\wedge j>0\\\ 1,&\text{if }t=j=0\\\ \end{cases}$ (1)

Considering that the most probable temporal position of character $M$ is
$\psi=\arg\max_{t}{k_{t,M}}$, alignments are obtained with:

$a_{t}=\begin{cases}M-1,&\text{if }\psi\geq\arg\max_{t}{k_{t,M-1}}\\\
a_{t+1},&\begin{split}\text{if }k_{t,a_{t+1}-1}\cdot
P(c_{a_{t+1}-1}|\bm{x}_{t+1})\\\ <k_{t,a_{t+1}}\cdot
P(\varnothing|\bm{x}_{t+1})\end{split}\\\ a_{t+1}-1,&\text{else }\\\
\end{cases}$ (2)

The algorithm produces a posterior probability ($\rho_{t}$) for every aligned
frame, selecting the maximum probability between the aligned char or the blank
token. So, we capture the spikes in the temporal $\rho_{t}$ sequence.
Moreover, a fragment score is computed by splitting the $T$-length aligned
utterance in segments of length $L$. Thus, $m_{l}$ is the fragment score where
the index of the fragment is represented by $l\in[1,T/L]$, and is obtained by:

$m_{l}=\frac{1}{L}\sum_{t=lL}^{(l+1)L}\rho_{t}.$ (3)

Finally, the utterance alignment confidence score $s_{seg}$ is given by
selecting the worst fragment score, this inflicts a penalty in mismatches
between ASR results and text references.

$s_{seg}=\min_{l}m_{l},$ (4)

###  3.2 Iterative pseudo-forced alignment

The alignment method presented in section 3.1 cannot be used directly in our
case for several reasons. As we pretend to align long audio files using a
transformer-based ASR, we found the necessity to process chunks instead of the
whole audio. In addition, we noticed that even human-revised transcripts
present differences with the spoken content. Thus, we propose an iterative
chunk processing, where several combinations of audio and text are aligned
until finding the best possible match. Our proposal is implemented as follows:

  1. 1.

Pre-process audio with a Voice Activity Detector (VAD), removing only non-
speech segments longer than 30 seconds. Typically opening sequences of shows
and sections.

  2. 2.

Split the reference text in utterances with a maximum length of 24 words, as
this is the maximum word sequence in CommonVoice.

  3. 3.

Calculate initial time references based on total text length and total speech
time. We assign an audio duration proportional to text length. This assumes
constant speech velocity but is only used to have initial time references.

  4. 4.

Read audio and utterances from the last temporal anchor point. In the first
step, the previous anchor point is defined by the first voice event in the
audio file. As the algorithm progresses, new anchor points will be defined by
the accepted alignments.

Figure 1: Simplified alignment algorithm. $th$ represents the alignment
threshold, $W$ stands for the audio window length, $N$ is the number of
utterances to align, and $max\\_w$ stands for the maximum allowed window until
recalculate time references. Iterations whenever a good alignment has been
found are not considered.

  5. 5.

Perform iterative alignments with a fixed quantity of audio and a variable
amount of text. First, get frame-wise character posteriors for the acoustic
observation with the ASR. Then, perform alignments while the number of
utterances is bigger or equal to one. The confidence score of the last aligned
utterance is the unique criterion used to accept or reject the alignments. We
use a fixed threshold, $s_{seg}=0.13$ (-2.0 in log space), to select anchor
points. If the last aligned utterance provides a $s_{seg}<threshold$, we
reject alignments. Therefore, we remove an utterance from the text reference
and try again. In contrast, if $s_{seg}>=threshold$ we can accept the
alignments, so we store them. Nevertheless, we continue iterations by removing
utterances from text, looking for a better confident score. This process is
repeated until finding a bad alignment or until there is no improvement. That
is to say, after finding a good alignment if we find an even better alignment,
then the alignment with the best confidence score is stored. If we exhaust the
number of utterances without any acceptable alignment, we go back to the
previous step to read more audio and text.

In addition, we do not allow short utterances to be anchors. Concretely,
utterances with lengths inferior to or equal to $L=30$ frames (0.6 seconds)
cannot be anchors. We modify the obtained $s_{seg}$ by decreasing its value.
This is done because of the inability of the algorithm to provide a bad
alignment confident score for short utterances. A simplified version of the
algorithm is presented in Figure 1. Furthermore, as our unique criterion to
accept the alignments of a set of utterances is the confidence score of the
last aligned utterance, the algorithm allows bad alignments between anchors,
thus, we call it pseudo-forced. This is especially useful for low-quality
transcriptions. Figure 2 shows an example of six iterations of the algorithm.

Figure 2: Iterations of pseudo-forced alignment algorithm. Same number
represents the same audio window. Similarly, different letters represent a
different set of utterances under alignment. In iterations 2b and 4a, the
algorithm accepts alignments with $s_{seg}<threshold$ between anchors. $R$ is
the resultant alignment, with 7 utterances with $s_{seg}$ above the alignment
threshold and 3 utterances under the alignment threshold.

##  4 Experimental setup

###  4.1 Databases

The proposal of this work is assessed with CommonVoice [20] as out-of-domain
database and RTVE2022 [21] as target domain database. Spanish CommonVoice
dataset comprises more than 200 hours of reading speech, which is utterance-
level aligned and validated by volunteers [20]. Table 1 presents the amount of
data of the two versions used in this work.

Table 1: Number of hours and sentences for the different splits of the Spanish CommonVoice datasets. version | train | dev | test
---|---|---|---
| hours | N | hours | N | hours | N
6.1 | 236.9 | 161.8k | 25.2 | 15.1k | 25.9 | 15.1k
7.0 | 291.2 | 196.0k | 25.8 | 15.3k | 26.4 | 15.3k

RTVE2022DB combines the databases used in previous evaluations: 2018 and 2020
with released data in 2022 [21]. RTVE2018DB is a collection of shows from
public Spanish Television (RTVE) during the years 2015 to 2018. It has 569
hours of unaligned audio, partitioned into 4 different subsets: train, dev1,
dev2, and test. The train split consists of 460 hours of audio with closed
captions from TV shows. The dev1, dev2, and test splits contain 57, 15, and 41
hours of human-revised transcripts, respectively. Whereas RTVE2020DB is a
collection of shows from RTVE during the years 2019 to 2020. It includes a
test split with 55 hours of human transcribed audio.

###  4.2 End-to-end ASR

A purely end-to-end acoustic model has been used. Based on the large XLS-R
architecture, which consist of a convolutional feature encoder, followed by a
transformer with 24 blocks with an inner dimension of 4096 and a model
dimension of 1024. Concretely, we used the XLSR-53 model, which was pre-
trained with 56k hours of audio in 53 languages [9], and then fine-tuned in
the Spanish CommonVoice dataset (version 6.1). We added two linear layers
randomly initialized to the original Wav2Vec2 architecture. The resultant
model counts for more than 300M trainable parameters. Finally, the
transcription is obtained using simple Greedy decoding from the frame-wise
character posteriors that the model produces.

Two versions of the model were built. Firstly, a model that classifies 87
characters without punctuation, as a result of retrieving any unique char from
the CommonVoice database. Secondly, as the Spanish language has fewer
characters, we removed other languages’ utterances, characters, e.g., from the
Cyrillic alphabet. Characters coming from different alphabets used in proper
nouns are also discarded. As a result, the model classifies among 38
characters, including unaccented letters between a-z, the accented vowels á,
é, í, ó, ú, and the dieresis on the vowel u(ü). The unique difference between
both models is the number of output neurons.

###  4.3 Baseline

The 87-char model was trained during 150 epochs with CommonVoice 6.1. While
the 38-char model was trained during 15 epochs with CommonVoice 7.0. For the
second model, we used the trained Wav2Vec2 weights of the first model. In the
test split of the CommonVoice 7.0, the 87-char model achieved 8.8% WER, while
the 38-char model achieved 8.17% WER. These checkpoints are our baseline
experiments. Only the 38-char model was adapted to the TV domain, so,
hereinafter it will be simply referred as model or ASR.

###  4.4 Training setup

Based on the CTC recipe from the SpeechBrain toolkit [22], utterance audio
length is limited from 4 to 10 seconds. Similarly, signals are ordered by
length, using shorter ones in the first batches of the epoch, and longer clips
at the end. Regarding data augmentation, the unique technique used is
SpecAugment [23]. Furthermore, the model is optimized minimizing only a CTC
loss and the learning rates (LR) are updated using the New Bob scheduler.
Additionally, we manually restarted the LR at some points in training.
Finally, the best checkpoint in terms of WER is stored. The ASR has been
trained using a batch size of 3, setting the starting LRs for the linear
layers and Wav2Vec2 of 1.0 and $10^{-5}$, respectively.

###  4.5 Evaluation

Aiming to evaluate the usefulness of our iterative pseudo-forced alignment
method, we partially aligned the RTVE2018 corpus. Afterward, we adapted an
out-of-domain baseline ASR by fine-tuning it through the aligned data. From
this baseline model, we continued training it by combining out-of-domain data
with the aligned RTVE domain data. Note that the development split comprises
uniquely RTVE2018. Training and development splits of RTVE2018 have been
separated by show, avoiding speaker repetition. Concretely, the development
split comprises four of the total seventeen shows available.

Performance is evaluated with WER. For CommonVoice, it is computed with the
SpeechBrain toolkit. For RTVE2020, transcription is generated by processing
the audio file with a sliding window without overlap, and WER is computed with
the sclite tool of the NIST Speech Recognition Scoring Toolkit.

##  5 Results

###  5.1 Filtering methods and data recovery

We evaluated three different methods to filter alignments. First, discarding
utterances with $s_{seg}<-1.0$ (in log space). Secondly, using the Chebyshev’s
inequality for filtering out the 15% worst aligned utterances. Lastly,
normalizing the confidence score by considering the utterance length and then
discard alignments with $s_{seg}<-1.5$ (in log space). We heuristically
selected a reference length in seconds ($U=8.0$) that we consider enough to
detect mismatches between reference text and ASR. Likewise, $S$ is the aligned
segment length in seconds. Thus, the normalized segment score is calculated as
follows:

$s^{\prime}_{seg}=\frac{s_{seg}\cdot S}{U}$ (5)

The best results are obtained while using the first method commented above, so
we used only that filtering method. In Table 2 the quantity of aligned and
filtered data is presented. The first pass is performed using the baseline
ASR. For the second pass, the model is already trained with domain data (23
hours) during 15 epochs. Similarly, Figure 3 depicts $s_{seg}$ distribution
between the two different pass alignments, showing the improvement as the
model is adapted.

Table 2: Data recovery, in hours, from RTVE2018DB. Split | Hours | 1st pass | 2nd pass
---|---|---|---
|  | Aligned | Filtered | Aligned | Filtered
train | 460 | 0 | 0 | 17.5 | 14.7
dev1 | 55 | 45 | 18 | 53.6 | 29.9
dev2 | 15 | 8.5 | 5 | 14.3 | 9.1
test | 39 | 0 | 0 | 33.1 | 20.9
Figure 3: Alignment segment score ($s_{seg}$) of dev1 split from RTVE2018
across alignments. In the second pass, the confidence increases notoriously.
In $s_{seg}=-4.0$, a small peak is generated due to the penalty inflicted in
short utterances.

###  5.2 Domain adaptation

The domain adaptation process is presented in Figure 4 and final results are
presented in Table 3. Aligned data definitely contributes to the process. Best
results are obtained while using the biggest amount of domain data. In
RTVE2020 test, we reduced WER from 46.98% to 30.37%. Similarly, in CommonVoice
7.0 we reduced from 8.17% to 7.83%. This suggests that the use of more data,
even if it is from a different domain, contributes to ASR performance in both
used databases. To conclude, it is reasonable to think that there is still a
margin for improvement by using more complex decoding, using an LM, or
aligning more data, as only the 20% of RTVE2018 has been aligned.

Figure 4: WER across training epochs. In epoch 35, LR was restarted to 1.0 for the linear layers and $10^{-5}$ for Wav2Vec2. W stands for the length in seconds of the segmentation window used to get transcriptions. Table 3: Baseline and final WER %. DA stands for domain adaptation. test benchmark | 87-chars | 38-chars | DA
---|---|---|---
RTVE2020DB | 47.43 | 46.98 | 30.37
CommonVoice 6.1 | 8.51 | 8.13 | 7.80
CommonVoice 7.0 | 8.80 | 8.17 | 7.83

##  6 Conclusions

We propose an iterative pseudo-forced alignment algorithm to produce audio-to-
text alignments using the CTC paths. This allowed us to perform self-
supervised domain adaptation on ASR by aligning a Spanish speech corpora based
on broadcast TV. Our model is a purely acoustic end-to-end model that uses
simple Greedy decoding. Without any LM, we improved ASR performance over
generic and domain data. Our model achieved a 35.4% of relative WER reduction
from the baseline in TV domain data. Similarly, we achieved a 4.2% relative
WER reduction in out-of-domain data. This shows the potential of the alignment
method, even with low-quality text references.

## References

  * [1] L. Kürzinger, D. Winkelbauer, L. Li, T. Watzel, and G. Rigoll, “Ctc-segmentation of large corpora for german end-to-end speech recognition,” in _International Conference on Speech and Computer_. Springer, 2020, pp. 267–278.
  * [2] X. Anguera, J. Luque, and C. Gracia, “Audio-to-text alignment for speech recognition with very limited resources,” in _Fifteenth Annual Conference of the International Speech Communication Association_ , 2014.
  * [3] T. Hazen, “Automatic alignment and error correction of human generated transcripts for long speech recordings,” vol. 4, 01 2006.
  * [4] M. Kocour, “Automatic speech recognition system continually improving based on subtitled speech data,” Master’s thesis, Brno University of Technology, Faculty of Information Technology, 2019. [Online]. Available: https://www.fit.vut.cz/study/thesis/22041/
  * [5] M. Kocour, G. Cámbara, J. Luque, D. Bonet, M. Farrús, M. Karafiát, K. Veselý, and J. H. Ĉernocký, “Bcn2brno: Asr system fusion for albayzin 2020 speech to text challenge,” 2021.
  * [6] Y. Chen, C. Shen, S. Huang, and H. Lee, “Towards unsupervised automatic speech recognition trained by unaligned speech and text only,” _CoRR_ , vol. abs/1803.10952, 2018. [Online]. Available: http://arxiv.org/abs/1803.10952
  * [7] G. Synnaeve, Q. Xu, J. Kahn, T. Likhomanenko, E. Grave, V. Pratap, A. Sriram, V. Liptchinsky, and R. Collobert, “End-to-end asr: from supervised to semi-supervised learning with modern architectures,” _arXiv preprint arXiv:1911.08460_ , 2019.
  * [8] A. Baevski, H. Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech representations,” _CoRR_ , vol. abs/2006.11477, 2020. [Online]. Available: https://arxiv.org/abs/2006.11477
  * [9] A. Conneau, A. Baevski, R. Collobert, A. Mohamed, and M. Auli, “Unsupervised cross-lingual representation learning for speech recognition,” _arXiv preprint arXiv:2006.13979_ , 2020.
  * [10] A. Babu, C. Wang, A. Tjandra, K. Lakhotia, Q. Xu, N. Goyal, K. Singh, P. von Platen, Y. Saraf, J. Pino, A. Baevski, A. Conneau, and M. Auli, “Xls-r: Self-supervised cross-lingual speech representation learning at scale,” 2021\. [Online]. Available: https://arxiv.org/abs/2111.09296
  * [11] J. Grosman, “Fine-tuned XLS-R 1B model for speech recognition in Spanish,” https://huggingface.co/jonatasgrosman/wav2vec2-xls-r-1b-spanish, 2022.
  * [12] P. Baquero-Arnal, J. Jorge, A. Giménez, J. Iranzo-Sánchez, A. Pérez, G. V. Garcés Díaz-Munío, J. A. Silvestre-Cerdà, J. Civera, A. Sanchis, and A. Juan, “Mllp-vrain spanish asr systems for the albayzín-rtve 2020 speech-to-text challenge: Extension,” _Applied Sciences_ , vol. 12, no. 2, 2022. [Online]. Available: https://www.mdpi.com/2076-3417/12/2/804
  * [13] G. Bordel, M. Penagarikano, L. J. Rodríguez-Fuentes, A. Álvarez, and A. Varona, “Probabilistic kernels for improved text-to-speech alignment in long audio tracks,” _IEEE Signal Processing Letters_ , vol. 23, no. 1, pp. 126–129, 2016.
  * [14] V. Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Librispeech: An asr corpus based on public domain audio books,” in _2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2015, pp. 5206–5210.
  * [15] P. Moreno, C. Joerg, J.-M. Thong, and O. Glickman, “A recursive algorithm for the forced alignment of very long audio segments.” 01 1998.
  * [16] P. Lamere, P. Kwok, E. Gouvea, B. Raj, R. Singh, W. Walker, M. Warmuth, and P. Wolf, “The cmu sphinx-4 speech recognition system,” in _Ieee intl. conf. on acoustics, speech and signal processing (icassp 2003), hong kong_ , vol. 1, 2003, pp. 2–5.
  * [17] F. Schiel, “Automatic phonetic transcription of non-prompted speech,” 1999.
  * [18] S. J. Young and S. Young, “The htk hidden markov model toolkit: Design and philosophy,” 1993.
  * [19] D. Povey, A. Ghoshal, G. Boulianne, L. Burget, O. Glembek, N. Goel, M. Hannemann, P. Motlicek, Y. Qian, P. Schwarz _et al._ , “The kaldi speech recognition toolkit,” in _IEEE 2011 workshop on automatic speech recognition and understanding_ , no. CONF. IEEE Signal Processing Society, 2011.
  * [20] R. Ardila, M. Branson, K. Davis, M. Henretty, M. Kohler, J. Meyer, R. Morais, L. Saunders, F. M. Tyers, and G. Weber, “Common voice: A massively-multilingual speech corpus,” _arXiv preprint arXiv:1912.06670_ , 2019.
  * [21] E. Lleida, A. Ortega, A. Miguel, V. Bazán, C. Pérez, M. Gómez, and A. de Prada, “Rtve 2018, 2020 and 2022 database description.” [Online]. Available: http://catedrartve.unizar.es/reto2022/RTVE2022DB.pdf
  * [22] M. Ravanelli, T. Parcollet, P. Plantinga, A. Rouhe, S. Cornell, L. Lugosch, C. Subakan, N. Dawalatabad, A. Heba, J. Zhong, J.-C. Chou, S.-L. Yeh, S.-W. Fu, C.-F. Liao, E. Rastorgueva, F. Grondin, W. Aris, H. Na, Y. Gao, R. D. Mori, and Y. Bengio, “SpeechBrain: A general-purpose speech toolkit,” 2021, arXiv:2106.04624.
  * [23] D. S. Park, W. Chan, Y. Zhang, C.-C. Chiu, B. Zoph, E. D. Cubuk, and Q. V. Le, “Specaugment: A simple data augmentation method for automatic speech recognition,” _arXiv preprint arXiv:1904.08779_ , 2019.


# Towards ML-based diagnostics of focused laser pulse

Y. R. Rodimkov<EMAIL_ADDRESS>Department of Mathematical Software and
Supercomputing Technologies, Lobachevsky University, 603950 Nizhni Novgorod,
Russia  V. D. Volokitin<EMAIL_ADDRESS>Department of Mathematical Software
and Supercomputing Technologies, Lobachevsky University, 603950 Nizhni
Novgorod, Russia  Mathematical Center, Lobachevsky University, 603950 Nizhni
Novgorod, Russia  I. B. Meyerov<EMAIL_ADDRESS>Department of Mathematical
Software and Supercomputing Technologies, Lobachevsky University, 603950
Nizhni Novgorod, Russia  Mathematical Center, Lobachevsky University, 603950
Nizhni Novgorod, Russia  E. S. Efimenko<EMAIL_ADDRESS>Institute
of Applied Physics of the Russian Academy of Sciences, 603950 Nizhni Novgorod,
Russia

###### Abstract

Currently, machine learning (ML) methods are widely used to process the
results of physical experiments. In some cases, due to the limited amount of
experimental data, ML-models can be pre-trained on synthetic data simulated
based on the analytical theory and then fine-tuned using experimental data. A
limitation of this approach is the presence of the latent parameters of the
analytical model, which values are difficult or impossible to estimate.
Setting these parameters incorrectly may induce a dataset shift even when
applied to synthetic data. To overcome this problem, we train the ML-model on
a dataset with randomly varied latent parameters of the analythical model to
force the ML-model to concentrate on more general patterns that depend weakly
on the latent parameters. We applied this approach to the problem of tight
focusing of a laser pulse with the complex structure of the wavefront. We
observed good accuracy of reconstructing of the tilt parameters when training
and testing the ML-model on datasets generated for different values of the
latent parameters. This confirms that the ML-model was able to select relevant
information without over-fitting for specific features inherent in certain
values of the latent parameters. We believe that this approach will enrich
possible applications of ML-methods to an experimental diagnostics of laser
pulses.

tight focusing, deep learning, convolutional neural network, dataset shift

##  I Introduction

With the progress in optics technologies in the last decades, laser facilities
can now achieve unprecedented intensities. Femtosecond laser pulses focused to
spot size of the order of a few micrometers can now routinely achieve
intensities above $10^{22}$ W/cm2 (Yanovsky2008, ; Pirozhkov2017, ;
Tiwari2019, ), with a record intensity even exceeding $10^{23}$ W/cm2 Yoon2021
. These ultra-relativistic laser pulses allow for multiple applications such
as particle acceleration in plasmas Higginson2018 ; Gonsalves2019  and
production of $\gamma$-rays Sarri2014 , or exploring quanto-electrodynamic
effects in laser-plasma interaction in the high-field regime DiPiazza2012 .

Such intensities are reached by the OPCPA technique, which successively
stretches and compresses the pulse for further amplification (Strickland85, ).
Any imperfection in this process in the laser chain can thus introduce spatio-
temporal couplings (STC), i.e. correlations between the longitudinal and
transverse intensity profiles, which ultimately reduces the intensity at focus
(Kim2013, ; Bourassin2013, ). STC tends to become stronger for more powerful
laser systems – corresponding to an increasing waste of the pumping energy,
which becomes greatly harmful for applications.

Characterization of the laser pulse is thus crucial both for optimizing the
laser profile in experiments and for its accurate numerical description
without overestimation of the peak intensity – leading to better agreement
with the experimental results and thus ultimately to a deeper understanding of
the underlying physics.

Spatio-temporal characterization of the ultra-intense laser pulses is however
not straightforward. Direct detection is precluded due to the high frequency
of the pulse, and a combination of spectroscopy and interferometry has to be
used. 3D characterization is thus a trade-off between cost, time of
acquisition and resolution of the data. Recently, experimental techniques
leading to a complete spatio-temporal characterization of ultra-intense laser
pulses have been developed, such as TERMITES Pariente2016  or INSIGHT
Borot2018 , which have been applied to top-class PW-laser facilities
Jeandet2019 . Such measurements are however obtained thought extensive
manipulations, and effort are also pursued for simplifying this process
through the combination of experimental measurements and machine learning.
First attempts in this sense were realized in the 90’s, in which neural
network Krumbugel1996  and genetic algorithm Nicholson1999  were used with
frequency-resolved optical gating (FROG) for reconstructing the pulse phase.
More recently, a neural network trained on simulated data retrieved the pulse
phase even in the presence of high noise, and lowered the required knowledge
about the relation between the pulse and its measurement Zahavy2018 . Other
works have also proposed to reconstruct the pulse phase with neural network
from dispersion scan traces Kleinert2019 , 2D intensity patterns Ziv2020  or
with a multimode fiber Xiong2020 .

Training data for machine learning (ML) models can be collected experimentally
or through simulation. Quite often the amount of experimental data is not
sufficient for training the ML-model from scratch. In this case the training
data can be numerically simulated on the basis of some analytical theory, and
then fine-tuned on a real experimental data. This approach, although seems
very natural, requires good correspondence between the analytical theory and
experiment. The later can become very problematic, because the analytical
model may have hidden or latent parameters which affect the results, but can
not be directly measured or estimated in the experiment. The number of such
latent parameters for ML-models can be large. Setting parameters incorrectly
during data collection can cause a ML-model to break when applied to real data
due to dataset shift.

In this article, we demonstrate an approach to reconstructing the physical
parameters of a tightly focused laser pulse based on the energy flux
distribution in the case of the latent parameters uncertainty. In order to
make the laser pulse model closer to experimental conditions we impose a
spectral-dependent tilt on its wavefront, and the properties of this tilt act
as the latent parameters of the model. The resulting analytical model is
rather simplified representation of the real experimental conditions,
nevertheless it can help to find better approaches for the problem. This
parameterized model with randomly varied parameters was used for synthetic
data generation. We use this data to train the ML-model to solve the inverse
problem of reconstructing pulse parameters. In order to study the influence of
the choice of the latent parameters on accuracy of the ML-model we study the
generalization ability of this model by training and testing on different
subsets of the latent parameters values. By doing this we demonstrate
tolerance of the ML-model to wrong choice of the latent parameters during
training, that can be useful for the application of ML-methods to real
experimental data.

The article is structured as follows. In Section II the analytical model used
for data generation is described. In Section III we present the methodology of
our work, methods and metrics used in this paper, the solution of the inverse
problem and the analysis of the generalization ability of the ML-model. In
Section IV the results are discussed and some considerations related to
further development of the approach are given. Finally, in Section V the
conclusions are formulated.

##  II Generative model

The data was generated by simulation of the propagation of a tightly focused
laser pulse initialized in the far field zone to the focal plane. The modeling
was performed by means of the Hi-chi module (Panova2020, ), which uses a
spectral method for solving Maxwell’s equations. The cumulative energy flux
was calculated in the focal plane to be used for the ML-model training.

For simplicity the pulse propagates along the $x$-axis and the pulse wavefront
is chosen to be spherical, so any field component $u$ in the far field zone
can be set as

$u(x,y,z)=u_{l}(r(x,y,z))u_{t}(\alpha(x,y,z)),$ (1)

where $r=|\mathbf{r}|=|\mathbf{R}-\mathbf{R}_{0}|$ is the distance from the
center of the spherical wavefront $\mathbf{R}_{0}$ to the point $\mathbf{R}$
and $\alpha=\arcsin\sqrt{y^{2}+z^{2}}/r$ is the angle between the $x$-axis and
vector $\mathbf{r}$. The electromagnetic fields are defined as
$\mathbf{E}(\mathbf{R})=u(\mathbf{R})\mathbf{s}_{0}$ and
$\mathbf{B}(\mathbf{R})=u(\mathbf{R})\mathbf{s}_{1}$ with the normalized
vectors $\mathbf{s}_{0}=\mathbf{e_{y}}\times\mathbf{e_{y}}\times\mathbf{r}$
and $\mathbf{s}_{1}=\mathbf{e_{y}}\times\mathbf{r}$.

The longitudinal $u_{l}$ and transverse $u_{t}$ profiles are defined as
follows

$u_{\rm{l}}(x)=\sin\left(\frac{2\pi
x}{\lambda_{0}}+\varphi\right)\exp\left(-\frac{x^{2}}{2L^{2}}\right)\Pi\left(x,-4L,4L\right),$
(2)
$u_{\rm{t}}(\alpha)=\Pi\left(\alpha,0,\theta-\frac{\varepsilon}{2}\right)+\cos^{2}\left(\frac{\pi(\alpha-\theta+\varepsilon/2)}{2\varepsilon}\right)\Pi\left(\alpha,\theta-\frac{\varepsilon}{2},\theta+\frac{\varepsilon}{2}\right),$
(3)

where $\lambda_{0}$ is the central wavelength, $L$ is the length of the pulse
and $\Pi$ is the rectangular function, with $\Pi(x,x_{min},x_{max})=1$ if $x$
is in $[x_{min},x_{max}]$ and 0 otherwise. In this case $u_{\rm{t}}$
corresponds to a transverse flattop profile with no intensity outside of the
opening angle $\theta$ with the edge smoothing angle $\varepsilon$. The
opening angle $\theta$ is connected to the F-number of the focusing optics by
$\theta=\arctan(1/2\mathrm{F})$. An additional phase $\varphi$ is used to
model the imperfections of the wavefront that occur in experimental
conditions.

The proposed pulse definition does not account for complex structure of
wavefront inherent to real experimental conditions. In order to model complex
wavefront structure in a simplified manner we introduce three spectral
components by applying mask functions $f_{i}(k)$ (see Fig. 1) to the spectra
of the longitudinal profile $u_{\rm{l},i}(k)=u_{\rm{l}}(k)f_{i}(k)$, where
$u_{\rm{l}}(k)=\int_{-\infty}^{\infty}u_{\rm{l}}(x)e^{-ikx}dx$ is the spatial
Fourier spectra of the longitudinal profile $u_{\rm{l}}(x)$ and $k$ is the
wavenumber. Three spectral components are separated by the two wavenumbers
$k_{1}$ and $k_{2}$ such as $k_{1}>k_{0}>k_{2}$ with $k_{i}=2\pi
c/\lambda_{i}$, where $c$ is the speed of light. These two boundary
wavenumbers $k_{1,2}$ are varied in the range
$[k_{i,\rm{min}},k_{i,\rm{max}}$] to parameterize the laser pulse definition.
For convenience, later in the text we operate with corresponding wavelengths
$\lambda_{1,2}$ as boundary wavelengths and treat them as the latent
parameters of the model.

For each of these three spectral component we add a linear tilt by adding the
additional phase $\varphi_{i}$ in the longitudinal profile $u_{i}(x)$ in the
Eq. 2

$\varphi_{i}=a_{i}y^{\prime}_{i},~{}~{}y^{\prime}_{i}=y\cos\theta_{i}-z\sin\theta_{i},$
(4)

where $a_{i}$ is a constant characterizing the amplitude of the tilt and
$\theta_{i}$ is an angle characterizing the direction of the tilt in the
transverse plane. Further in this paper $a_{i}$ and $\theta_{i}$ are referred
to as the tilt amplitude and the tilt angle, respectively.

$k_{2}$$k_{0}$$k_{1}$$k_{2,min}$$k_{2,max}$$k_{1,min}$$k_{1,max}$$1$$0$$f_{2}(k)$$f_{0}(k)$$f_{1}(k)$
Figure 1:  Mask functions $f_{i}(k)$ used to create three spectral components
with different tilt properties.

After applying the mask function we calculate longitudinal profile
$u_{\rm{l},i}$ for each spectral component by performing inverse Fourier
transform
$u_{\rm{l},i}(x)=1/2\pi\int_{-\infty}^{\infty}u_{\rm{l}}(x)e^{ikx}dk$ and
calculate the corresponding spectral component initial distribution $u_{i}$
according to Eq. 1

$u_{i}(x,y,z)=u_{\rm{l},i}(r)~{}u_{\rm{t}}(\alpha).$ (5)

After initialization in the far field zone each component is propagated to the
focal plane using the spectral solver. Using the linearity of Maxwell’s
equations, we can then get the total field at the focal position as the sum of
fields of all spectral components as

$u_{\rm{tot}}(\mathbf{R})=\sum_{i}A_{i}u_{i}(\mathbf{R}),$ (6)

with $A_{i}$ defining the relative spectral component amplitude compared with
the central component corresponding to the mask function $f_{0}$ (i.e.
$A_{0}=1$). These relative amplitude parameters $A_{1,2}$ act as another pair
of latent parameters of our laser pulse model.

Finally, we calculate the cumulative energy flux in the focal plane

$I(y,z)=\frac{1}{4\pi}\int_{-\infty}^{\infty}\mathbf{E}_{\mathrm{tot}}(x,y,z)\times\mathbf{B}_{\mathrm{tot}}(x,y,z)\cdot\mathbf{e_{x}}dx.$
(7)

Summarizing, in our simplified model the energy flux depends on following
parameters: the boundary wavelengths $\lambda_{1,2}$, the relative amplitudes
$A_{1,2}$, the tilt amplitudes $a_{i}$ and the tilt angles $\theta_{i}$ with
$i\in[0,1,2]$. The last six parameters $a_{i}$ and $\theta_{i}$ were predicted
by the ML-model, while $\lambda_{1,2}$ and $A_{1,2}$ acted as the latent
parameters of the pulse model. Data was generated using central wavelength
$\lambda_{0}=800~{}$nm, F-number of the focusing optics F=1.5, the pulse is
initialized at $65\lambda_{0}$ from the spherical wavefront center. Other
parameters were set in the following ranges:
$\lambda_{1}\in[600\mathrm{nm},750\mathrm{nm}]$,
$\lambda_{2}\in[850\mathrm{nm},1000\mathrm{nm}]$, $A_{1,2}\in[0.5,2]$,
$a_{0,1,2}\in[0,0.6]$ and $\theta_{0,1,2}\in[0,2\pi]$.

##  III Results

###  III.1 Methodology

The problem under consideration is the inverse problem of predicting of six
parameters (the tilt angle and the tilt amplitude for three spectral
components) based on the cumulative energy flux in the focal plane. The
analytical model which describes the experiment has latent parameters, which
can not be directly reconstructed from the experiment. In our rather
simplified model such parameters are the boundary wavelengths $\lambda_{1,2}$
and the relative amplitudes of the spectral component with the lowest and
highest wavelengths, respectively, $A_{1,2}$. Determination of these
parameters in an experiment, in which the ML-model will be used after training
on synthetic data, is a difficult task. Thus, methodologically, the work is
divided into two stages. At the first stage, the inverse problem of
reconstructing the tilt parameters $a_{i}$ and $\theta_{i}$ is solved. In
order to do this a suitable architecture of the ML-model is chosen, and the
best way for reconstructing the parameters is searched for. At the second
stage, after the inverse problem is solved, the generalization ability of the
proposed ML-model is analyzed. To do this, the ML-model is trained on a subset
of latent parameters values, and then tested on the entire available set of
latent parameter values, i.e. the test set includes values of latent
parameters that were not present in the training set. Such experiments will
help to answer the main question of interest of this paper, whether the
trained ML-model is able to generalize the extracted information to unknown
values of latent parameters or not. The data and scripts required to reproduce
the numerical results may be downloaded from https://github.com/hi-
chi/Machine-Learning (the relevant examples are located in the ”Focus ML”
folder).

###  III.2 Methods and metrics

The cumulative energy flux in the focal plane can be treated as a single-
channel image, see Fig. 2. Neural networks are one of the most versatile
methods for analyzing data with the same type of continuous features. For
image analysis, the best results at the moment are shown by convolutional
neural networks (LeCun1998, ; Krizhevsky2012, ) and transformers (Vaswani2017,
; Dosovitskiy2020, ). It is known that transformers are difficult to train,
they are very demanding on the amount of data, and also require pre-training.
At the moment, in computer vision problems, there are not a large number of
successful applications of this method without model pre-training (Caron2021,
; Touvron2021, ). Therefore, convolutional neural networks were chosen for the
ML-based analysis of the flux.

After several experiments a convolutional neural net architecture was chosen
that consists of two consecutive blocks. Each block included two convolutional
layers and a max pooling layer, with the number of convolutions 16 and 32 in
each block, respectively. These two blocks were followed by four fully
connected layers with the number of neurons 1024, 512, 128, 32, and finally
there was a layer that predicts 6 tilt parameters. ReLU was used as an
activation function on each convolutional and fully connected layer, except
for the last one. For training, the root-mean-square error and the Adam
optimizer with default parameters in the Keras framework were used. The model
was trained for 30 epochs.

To assess the quality of ML-models in our study we computed the Mean Absolute
Percentage Error (MAPE) and the coefficient of determination ($R^{2}$):

$MAPE=\frac{100}{n}\sum\limits_{i=1}^{n}\frac{|\widehat{y_{i}}-y_{i}|}{\max(\widehat{y})},$
(8)
$R^{2}=1-\sum\limits_{i=1}^{n}\frac{(\widehat{y_{i}}-y_{i})^{2}}{(\overline{y}-y_{i})^{2}},$
(9)

where $\widehat{y_{i}}$ and $y_{i}$ are the true and the predicted value of
the $i$-th object, respectively, and $\overline{y}$ is the value of the
predicted parameter averaged over the training set.

For the reconstructed parameters a linear transformation to the range from -1
to 1 was used. Each energy flux profile was normalized in the range from zero
to one. In all experiments Keras and Tensorflow frameworks were used to train
neural networks.

###  III.3 The solution of the inverse problem

Our goal was to reconstruct the tilt amplitude $a_{i}$ and the tilt angles
$\theta_{i}$ for three spectral components of a laser pulse based on the
profile of the cumulative energy flux in the focal plane. The study of the
data and experiments showed that in order to achieve good accuracy the
reconstructed parameters should be carefully chosen. First, the tilt angles
$\theta_{i}$ have an obvious periodicity with a period of $2\pi$, and
therefore require a special treatment. There are different approaches to this
problem, for example, using a special loss function that takes into account
the periodicity of angles, or reconstructing trigonometric functions of angles
instead of angles themselves. Second, when the tilt amplitude $a_{i}$ is
small, the dependence of the flux on the corresponding tilt angle $\theta_{i}$
becomes weak, i.e. specific angle values are indistinguishable for the ML-
model, which leads to a strong increase in the angles-related error. This is
caused by the fact that the tilt angle and amplitude are not used separately
in computation of the additional phase $\varphi_{i}$ (see Eq. 4), but as the
product of the tilt amplitude and the trigonometric functions of the tilt
angle. Experiments showed that instead of reconstructing six original
parameters $a_{i}$ and $\theta_{i}$ it makes sense to reconstruct six derived
parameters that are the combination of the original ones
$p_{i}=a_{i}\sin\theta_{i}$, $q_{i}=a_{i}\cos\theta_{i}$. This transformation
can be interpreted as a transition from polar coordinates to Cartesian
coordinates. It should be noted that the proposed transformation is one-to-
one, and the original parameters $a_{i}$ and $\theta_{i}$ can be reconstructed
from derived parameters, but the accuracy of angle reconstructing for the
close to zero tilt amplitude can still be poor.

The ML-model architecture and training parameters are described in Section
3.2. For the ML-model training an experimental dataset of 50000 samples was
numerically generated. The experimental results showed that the parameters
$p_{i}$, $q_{i}$ are reconstructed with an accuracy close to ideal (see Table
1). It should be noted that the parameters $p_{1}$ and $q_{1}$ are
reconstructed worse than other parameters, although the accuracy is still
high. This effect is systematic and was observed in all experiments.
Apparently, this result is due to the fact that these parameters correspond to
the spectral component with the shortest wavelength, information about which
is worse extracted from the cumulative energy flux. The accuracy of
reconstructing of the parameters related to the central spectral component
$p_{0}$ and $q_{0}$ is the best, which can be explained by the fact that both
its amplitude and the central wavelength are fixed. Thus, the use of a
convolutional network makes it possible to solve the inverse problem with a
very high accuracy.

Table 1: Average values and standard deviations of the coefficient of determination (R2) and the mean error (MAPE) for reconstructing of derived parameters $p_{i}$ and $q_{i}$ by trained convolutional network. Metrics were averaged over 10 runs with different splits into training and test samples in a 90/10 ratio. Metrics | $p_{0}$ | $p_{1}$ | $p_{2}$ | $q_{0}$ | $q_{1}$ | $q_{2}$
---|---|---|---|---|---|---
$R^{2}$ | $0.991\pm 0.001$ | $0.948\pm 0.006$ | $0.981\pm 0.003$ | $0.991\pm 0.002$ | $0.951\pm 0.006$ | $0.982\pm 0.003$
MAPE, % | $2.71\pm 0.21$ | $5.70\pm 0.42$ | $3.77\pm 0.32$ | $2.74\pm 0.24$ | $5.66\pm 0.45$ | $3.78\pm 0.33$

###  III.4 Analysis of the generalization ability of the ML-model

####  III.4.1 The main idea.

In the previous section, we demonstrated the high accuracy of solving the
inverse problem using the convolutional neural network trained on model data.
At the same time, the application of ML-models trained on synthetic data to
the real physical experiment can be problematic, because the wrong choice of
values of the latent parameters of the analytical model during training may
significantly affect the accuracy of the ML-model on experimental data, which
is explained by a change in the data distribution. In this section we address
the ability of the ML-model to cope with such negative effects using synthetic
test dataset with other values of the latent parameters.

For a better understanding of the importance of a correct choice of the latent
parameters values, it makes sense to visually evaluate the influence of the
choice of boundary wavelengths $\lambda_{1,2}$ and relative amplitudes
$A_{1,2}$ on the cumulative energy flux profile in the considered analytical
model (see Fig. 2). Local changes in the energy distribution can be
noticeable, for example the distribution at the bottom in Fig. 2 (a) is
different from the case with the same parameters except $\lambda_{1,2}$ in
Fig. 2 (b). Similar differences can be seen if compare Fig. 2 (c) and (d)
where all parameters are the same except relative amplitudes $A_{1,2}$. These
differences show the data variability, but what is more important, is that the
general shape of the energy flux distribution in the profile is preserved,
which indicates that a significant part of the relevant information is
contained in the tilt angles and amplitudes.

This fact demonstrates the possibility of information generalization for
different values of latent parameters, although the good result is not
guaranteed. It is known that ML-models can overemphasize irrelevant
information. For example, in adversarial attacks, when a change in the data is
selected in a special way to change the prediction of the network
(Goodfellow2014, ; Yuan2019, ). There are also examples when irrelevant
features were changed in the data, which led to a dataset shift problem and a
decrease in the accuracy of the model (Wang2018, ; Wilson2020, ). In this
regard, it seems important to evaluate the influence of the choice of latent
parameters on the accuracy of the previously considered ML-model, as well as
the ability of the ML-model to extract relevant information even in the case
of previously unseen values of the latent parameters.

Based on this, the methodology for further experiments is following: the
previously considered convolutional network is trained from scratch on a
subset of the data with a reduced set of latent parameter values, and then it
is tested on a test dataset that includes all values of latent parameters,
including those not present in the training dataset. By doing this we try to
quantify the effect of wrong choice of latent parameters values during
training on the final accuracy of the ML-model. The experiments are divided
into two parts. In the first part, the boundary wavelengths $\lambda_{1,2}$
act as the studied latent parameters, and in the second part, the same is done
for the relative amplitudes $A_{1,2}$.

(a)

(b)

(c)

(d)

Figure 2: Examples of the cumulative energy flux distribution for different
values of latent parameters. (a, b) $\lambda_{1}$ and $\lambda_{2}$ are
different, $A_{1,2}$ are fixed; (c,d) $A_{1}$ and $A_{2}$ are different,
$\lambda_{1,2}$ are fixed. Parameters $a_{i}$ and $\theta_{i}$ are fixed for
(a), (b) and (c), (d).

####  III.4.2 Generalization for different wavelengths.

In the first experiment, the ML-model was trained on a newly generated dataset
with 50000 samples with the values of the boundary wavelengths
$\lambda_{1}\in[650~{}\mathrm{nm},700~{}\mathrm{nm}]$ and
$\lambda_{2}\in[900~{}\mathrm{nm},950~{}\mathrm{nm}]$. The ML-model was tested
on a dataset with 50000 samples from Section 3.3 with
$\lambda_{1}\in[600~{}\mathrm{nm},650~{}\mathrm{nm},700~{}\mathrm{nm},750~{}\mathrm{nm}]$
and
$\lambda_{2}\in[850~{}\mathrm{nm},900~{}\mathrm{nm},950~{}\mathrm{nm},1000~{}\mathrm{nm}]$.
All other parameters were randomly changed. The model architecture and
training parameters are the same as described in Section 3.2. The results of
the experiments are shown in Fig. 3.

It can be seen that the accuracy of determining the parameters on data with
the boundary wavelengths that are not included in the training set slightly
decreases. The ML-model shows the worst result when the values of the boundary
wavelengths become close to each other. In this case, the difference between
the spectral components from different spectral ranges decreases, which
complicates the reconstructing of parameter values.

Figure 3: The dependence of the coefficient of determination $R^{2}$ of all
parameters on the boundary wavelengths $\lambda_{1,2}$. The coefficient was
averaged over 10 runs with random initialization of neural network weights.
The black square represents the training data subset. The coefficient of
determination is shown by gradation of grey: the lower the coefficient, the
darker the color.

####  III.4.3 Generalization for different amplitudes.

In the second experiment, the ML-model was trained on a newly generated
dataset with 50000 samples with the values of the relative amplitudes
$A_{1,2}\in[0.95,1.1,1.25,1.4]$. The ML-model was tested on a dataset with
50000 samples from Section 3.3 with
$A_{1,2}\in[0.5,0.65,0.8,0.95,1.1,1.25,1.4,1.55,1.7,1.85,2]$. All other
parameters were randomly changed. The model architecture and training
parameters are the same as those used in Section 3.2. The results of the
experiments are shown in Fig. 4.

It can be seen that with an increase in the difference between the values of
the relative amplitudes used during training and testing (when approaching the
boundaries of the square in Fig. 4 (a)), the accuracy decreases. At the same
time, the accuracy in most cases remains acceptable, which indicates the
ability of the network to generalize the information received without over-
fitting for specific values of latent parameters.

An error for different parameters $p_{i},q_{i}$ contributes to the overall
accuracy differently depending on values of latent parameters $A_{1,2}$. If we
analyze in more detail the accuracy of reconstructing the parameters $p_{1}$,
we can see that the accuracy falls sharply if the value of the parameter
$A_{1}$ decreases (see Fig. 4 (b)). In this case, the relative contribution of
this spectral component to the energy flux becomes smaller, which complicates
the determination of its parameters. For parameters $p_{2}$ and $q_{2}$ the
situation is similar: when the value of the parameter $A_{2}$ takes on minimum
values, the accuracy of reconstructing the parameters decreases. In the case
of parameters $p_{0}$ and $q_{0}$, the accuracy deteriorates when the values
of the parameters $A_{1}$ or $A_{2}$ are close to the maximum values. Thus, it
can be summarized that in the case of a change in the relative amplitudes of
various spectral components, the key factor is the distinguishability of the
corresponding spectral component against the background of others. In other
words, the relative energy contribution of the spectral component to the
cumulative energy flux should be noticeable.

(a)

(b)

Figure 4:  The dependence of the coefficient of determination $R^{2}$ of (a)
all parameters, (b) $p_{1}$ parameter on the relative amplitudes $A_{1,2}$.
The coefficient was averaged over 10 runs with random initialization of neural
network weights. The black square represents the training data subset. The
coefficient of determination is shown by gradation of grey: the lower the
coefficient, the darker the color.

##  IV Discussion

This article is devoted not only to solving the problem of reconstructing the
tilt parameters of a laser pulse, but to a greater extent to studying the
generalization ability of the ML-model. The results of the experiment, when
the ML-model is trained on some values of latent parameters, and tested on
other values, can be treated differently. From the point of view of latent
parameters, the available information is extrapolated by the ML-model beyond
the set of parameters values used for training. Generally speaking, the
possibility of solving such type of problems with high accuracy is not
guaranteed, because the data distribution is different on training and testing
datasets, which generally leads to the dataset shift problem. Fortunately, in
the problem under consideration, it is the values of the reconstructed tilt
parameters that mainly determine the energy flux distribution, while the
latent parameters lead only to its modification. This fact explains the good
generalization ability of the model, which makes it possible to optimistically
assess the possibility of its application in the case when the values of
latent parameters are unknown and, as an ultimate case, to real experimental
data.

In the context of applying the trained ML-model to real experimental data, it
should be noted that in this work a simplified pulse model was used, when the
entire spectrum of the laser pulse was divided only into three spectral
components. In the case of adapting the model to a real physical experiment,
when the wavefront has a complex structure, such a partition must be done into
a larger number of spectral components. In this regard, it is important to
note that in our numerical experiments it was observed a systematic effect of
deterioration in the accuracy of reconstructing the parameters of spectral
components with a small wavelength or relative amplitude. In addition, the
effect of reducing the accuracy was observed in the case when the boundary
wavelengths become close, i.e. one of the spectral ranges becomes very narrow.
In all these cases, the influence of such spectral components on the energy
flux decreases, which makes it difficult to reconstruct the corresponding tilt
parameters. In addition, cross-influence of spectral components with close
wavelengths on each other is possible, when the values of tilt parameters from
one spectral component can be erroneously attributed by the ML-model to
another. The probability of this kind of errors increases sharply with a
decrease of the spectral component width, because the relative energy
contribution to the cumulative energy flux will also decrease. This means that
in the case of partitioning into a large number of spectral components, the
characteristics of which may be close, the accuracy of simple ML-models may
deteriorate significantly, which will require the use of more complex ML-
models or approaches.

##  V Conclusion

In this paper an ML-based approach for determining the parameters of the
spectral-dependent tilt of a focused laser pulse based on the cumulative
energy flux in the focal plane was proposed. A simplified formulation of the
problem was given with the division of the spectrum of a laser pulse into
three spectral components with an individual setting of the tilt parameters
for each of them. It was shown that it is possible to obtain good accuracy of
tilt parameters reconstructing using the convolutional neural network. In
order to study the generalization ability of the proposed ML-model the
methodology based on the study of the influence of the latent parameters
choice on the accuracy of the ML-model was used. For this purpose separate
datasets with different subsets of the latent parameters values were used
during training and testing. Two experiments were carried out in which the
boundary wavelengths and relative amplitudes acted as such latent parameters.
In both experiments, it was shown that the network has good generalization
ability and can reconstruct the tilt parameters with quite good accuracy even
for the latent parameter values not included in the training set. Significant
degradation of accuracy was observed only in cases where the relative energy
contribution of the spectral component to the energy flux became small. These
results indicate that the proposed approach is promising even in the presence
of uncertainty in the choice of the latent parameter values that are typical
for application of analytical models to a real experiment.

###### Acknowledgements.

This research was funded by the Ministry of Science and Higher Education of
the Russian Federation, agreement number 075-15-2020-808. The authors would
like to thank Julien Ferri for useful discussions and other contributions. The
authors acknowledge the use of computational resources provided by the
Lobachevsky University and Joint Supercomputer Center of the Russian Academy
of Sciences.

## References

  * (1) V. Yanovsky, V. Chvykov, G. Kalinchenko, P. Rousseau, T. Planchon, T. Matsuoka, A. Maksimchuk, J. Nees, G. Cheriaux, G. Mourou, and K. Krushelnick, Ultra-high intensity-300-TW laser at 0.1 Hz repetition rate., Optics express, 16 (3), 2109-2114 (2008).
  * (2) A. S. Pirozhkov, Y. Fukuda, M. Nishiuchi, H. Kiriyama, A. Sagisaka, K. Ogura, M. Mori, M. Kishimoto, H. Sakaki, N. P. Dover, K. Kondo, N. Nakanii, K. Huang, M. Kanasaki, K. Kondo, and M. Kando, Approaching the diffraction-limited, bandwidth-limited Petawatt, Optics express, 25 (17), 20486–20501 (2017).
  * (3) G. Tiwari, E. Gaul, M. Martinez, G. Dyer, J. Gordon, M. Spinks, T. Toncian, B. Bowers, X. Jiao, R. Kupfer, L. Lisi, E. McCary, R. Roycroft, A. Yandow, G. D. Glenn, M. Donovan, T. Ditmire, and B. M. Hegelich, Beam distortion effects upon focusing an ultrashort petawatt laser pulse to greater than 1022 W/cm2, 44 (11), 2764–2767 (2019).
  * (4) J. W. Yoon, Y. G. Kim, I. W. Choi, J. H. Sung, H. W. Lee, S. K. Lee, and C. H. Nam, Realization of laser intensity over 1023 W/cm2, Optica, 8 (5), 630-635 (2021).
  * (5) A. Higginson, R. J. Gray, M. King, R. J. Dance, S. D. R. Williamson, N. M. H. Butler, R. Wilson, R. Capdessus, C. Armstrong, J. S. Green, S. J. Hawkes, P. Martin, W. Q. Wei, S. R. Mirfayzi, X. H. Yuan, S. Kar, M. Borghesi, R. J. Clarke, D. Neely, and P. McKenna, Near-100 MeV protons via a laser-driven transparency-enhanced hybrid acceleration scheme, Nature communications, 9 (1), 1–9 (2018).
  * (6) A. J. Gonsalves, K. Nakamura, J. Daniels, C. Benedetti, C. Pieronek, T. C. H. de Raadt, S. Steinke, J. H. Bin, S. S. Bulanov, J. van Tilborg, C. G. R. Geddes, C. B. Schroeder, Cs. Tóth, E. Esarey, K. Swanson, L. Fan-Chiang, G. Bagdasarov, N. Bobrova, V. Gasilov, G. Korn, P. Sasorov, and W. P. Leemans, laser guiding and electron beam acceleration to 8 GeV in a laser-heated capillary discharge waveguide, Physical review letters, 122 (8), 084801 (2019).
  * (7) G. Sarri, D. J. Corvan, W. Schumaker, J. Cole, A. Di Piazza, H. Ahmed, C. Harvey, C. H. Keitel, K. Krushelnick, S. P. D. Mangles, Z. Najmudin, D. Symes, A. G. R. Thomas, M. Yeung, Z. Zhao, and M. Zepf, Ultrahigh brilliance multi-MeV $\gamma$-ray beams from nonlinear relativistic Thomson scattering, Physical review letters, 113 (22), 224801 (2014).
  * (8) A. Di Piazza, C. Müller, K. Z. Hatsagortsyan, and C. H. Keitel, Extremely high-intensity laser interactions with fundamental quantum systems, Reviews of Modern Physics, 84 (3), 1177 (2012).
  * (9) D. Strickland, and G. Mourou, Compression of amplified chirped optical pulses, Optics communications, 55 (6), 447–449 (1985).
  * (10) K. T. Kim, C. Zhang, A. D. Shiner, S. E. Kirkwood, E. Frumker, G. Gariepy, A. Naumov, D. M. Villeneuve, and P. B. Corkum, Manipulation of quantum paths for space-time characterization of attosecond pulses, Nature Physics, 9 (3), 159–163 (2013).
  * (11) C. Bourassin-Bouchet, M. M. Mang, F. Delmotte, P. Chavel, and S. de Rossi, How to focus an attosecond pulse, Optics express, 21 (2), 2506-2520 (2013).
  * (12) G. Pariente, V. Gallet, A. Borot, O. Gobert, and F. Quéré, Space-time characterization of ultra-intense femtosecond laser beams, Nature Photonics, 26 (20), 26444–26461 (2018).
  * (13) A. Borot, and F. Quéré, Spatio-spectral metrology at focus of ultrashort lasers: a phase-retrieval approach, Optics express, 26 (20), 26444–26461 (2018).
  * (14) A. Jeandet, A. Borot, K. Nakamura, S. W. Jolly, A. J. Gonsalves, C. Toth, H. S. Mao, W. P. Leemans, and F. Quéré, Spatio-temporal structure of a petawatt femtosecond laser beam, Journal of Physics: Photonics, 1 (3), 035001 (2019).
  * (15) M. A. Krumbügel, C. L. Ladera, K. W. DeLong, D. N. Fittinghoff, J. N. Sweetser, and R. Trebino, Direct ultrashort-pulse intensity and phase retrieval by frequency-resolved optical gating and a computational neural network, Optics letters, 21 (2), 143–145 (1996).
  * (16) J. W. Nicholson, F. G. Omenetto, D. J. Funk, and A. J. Taylor, Evolving FROGS: phase retrieval from frequency-resolved optical gating measurements by use of genetic algorithms, Optics letters, 24 (7), 490–492 (1999).
  * (17) T. Zahavy, A. Dikopoltsev, D. Moss, G. I. Haham, O. Cohen, S. Mannor, and M. Segev, Deep learning reconstruction of ultrashort pulses, Optica, 5 (5), 666–673 (2018).
  * (18) S. Kleinert, A. Tajalli, T. Nagy, and U. Morgner, Rapid phase retrieval of ultrashort pulses from dispersion scan traces using deep neural networks, Optics letters, 44 (4), 979–982 (2019).
  * (19) R. Ziv, A. Dikopoltsev, T. Zahavy, I. Rubinstein, P. Sidorenko, O. Cohen, and M. Segev, Deep learning reconstruction of ultrashort pulses from 2D spatial intensity patterns recorded by an all-in-line system in a single-shot, Optics express, 28 (5), 7528–7538 (2018).
  * (20) W. Xiong, B. Redding, S. Gertler, Y. Bromberg, H. D. Tagare, and H. Cao, Deep learning of ultrafast pulses with a multimode fiber, APL Photonics, 5 (9), 096106 (2020).
  * (21) E. Panova, V. Volokitin, E. Efimenko, J. Ferri, T. Blackburn, M. Marklund, A. Muschet, A. Gonzalez, P. Fischer, L. Veisz, I. Meyerov, and A. Gonoskov, Optimized Computation of Tight Focusing of Short Pulses Using Mapping to Periodic Space, Applied Sciences, 11 (3), 956 (2021).
  * (22) Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, Gradient-based learning applied to document recognition, Proceedings of the IEEE, 86 (11), 2278–2324 (1998).
  * (23) A. Krizhevsky, I. Sutskever and G. E. Hinton, Imagenet classification with deep convolutional neural networks, Advances in neural information processing systems, 25, (2012).
  * (24) A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser and I. Polosukhin, Attention is all you need, Advances in neural information processing systems, 30, (2017).
  * (25) A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, N. Houlsby, An image is worth 16x16 words: Transformers for image recognition at scale, arXiv preprint arXiv:2010.11929, (2020).
  * (26) M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin, Emerging properties in self-supervised vision transformers, Proceedings of the IEEE/CVF International Conference on Computer Vision, 9650–9660 (2021).
  * (27) H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, and H. Jégou, Training data-efficient image transformers & distillation through attention, International Conference on Machine Learning. - PMLR, 10347–10357 (2021).
  * (28) I. J. Goodfellow, J. Shlens and C. Szegedy, Explaining and harnessing adversarial examples, arXiv preprint arXiv:1412.6572, (2014).
  * (29) X. Yuan, P. He, Q. Zhu, and X. Li, Adversarial examples: Attacks and defenses for deep learning. IEEE transactions on neural networks and learning systems, 30(9), 2805–2824 (2019).
  * (30) M. Wang and W. Deng, Deep visual domain adaptation: A survey, Neurocomputing, 312, 135–153 (2018).
  * (31) G. Wilson, and D. J. Cook, A survey of unsupervised deep domain adaptation, ACM Transactions on Intelligent Systems and Technology (TIST), 11(5) 1–46 (2020).