Differential elastic nucleus-nucleus scattering in complete Glauber theory

Yu.M. Shabelski and A.G. Shuvaev

Petersburg Nuclear Physics Institute, Kurchatov National Research Center

Gatchina, St. Petersburg 188300, Russia

E-mail<EMAIL_ADDRESS>
E-mail<EMAIL_ADDRESS>
###### Abstract

The differential elastic cross sections of 12C – 12C and 11Li – 12C nuclei are
calculated in the complete Glauber theory. The role of the possible
correlations connected to the shell effects in 11Li nucleus is considered.

##  1 Introduction

High energy nuclear interactions are usually treated in the Glauber theory [1,
2, 3]. It is highly efficient in describing the hadron-nucleus scattering
where all the relevant diagrams can be summed up in the closed form. The
nucleus-nucleus scattering becomes rapidly more complicated for $A>4$ and
additional simplified approximations are usually made to get the result
analytically. An example of such calculations for the differential elastic and
reaction cross sections for 11Li – 12C scattering can be found in Refs. [4,
5].

In this paper the analytical calculations of the differential elastic cross
sections for 12C – 12C and 11Li – 12C scattering have been carried out in the
complete Glauber theory in the generating function approach Ref. [6]. The
results have been obtained for several commonly used forms of the nuclear
density, which allows one to get an insight to the possible nuclear
correlations, in particular, to those related to the shell structure of the
halo nucleus 11Li.

It is worth pointing out that it is the differential cross section rather than
the integrated reaction one that is more sensitive to the nuclear density
distribution, especially to its far distance parts.

##  2 Differential elastic cross sections

The amplitude of the elastic scattering of the incident nucleus $A$ on the
fixed target nucleus $B$ is given in the Glauber theory by the expression [7,
8]

$F_{AB}^{el}(q)\,=\,\frac{ik}{2\pi}\int
d^{\,2}b\,e^{iqb}\bigl{[}1\,-\,S_{AB}(b)\bigr{]},$ (1)

where $q$ is the transferred momentum and $k$ is the mean nucleon momentum in
nucleus $A$. The two-dimensional impact momentum $b$ lies in the transverse
plain to the vector $k$. The function $S_{AB}(b)$ is evaluated through the sum
of the independent phase shifts $\chi_{NN}(b)$ for each nucleon-nucleon
scattering,

$S_{AB}(b)\,=\,\langle\,A,\,B\,|\left\\{\prod\limits_{i\,j}\bigl{[}1-\Gamma_{NN}(b+x_{i}-y_{j})\bigr{]}\right\\}|\,A,\,B\,\rangle,$
(2)

with

$\Gamma_{NN}(b)\,=\,1\,-\,e^{i\chi_{NN}(b)}\,=\,\frac{1}{2\pi ik}\int
d^{\,2}q\,e^{-iqb}f_{NN}^{el}(q).$ (3)

The symbol $\langle\,A,\,B\,|\,\cdots\,|\,A,\,B\,\rangle$ denotes an average
over the nucleons’ positions $x_{i}$ and $y_{j}$ in the transverse plain. Each
pair $\\{i,j\\}$ enters the product only once, meaning that each nucleon from
the projectile nucleus can scatter on each nucleon no more than once.

The elastic nucleon-nucleon amplitude, $f_{NN}$, is mainly imaginary at the
beam energy about 1 GeV per nucleon,
$\mathrm{Re}f_{NN}/\mathrm{Im}f_{NN}\lesssim 10^{-1}$. The standard
parametrization is

$f_{NN}^{el}(q)\,=\,ik\frac{\sigma_{NN}^{tot}}{4\pi}e^{-\frac{1}{2}\beta
q^{2}},$ (4)

where $\sigma_{NN}^{tot}$ is the total nucleon-nucleon cross section. The
slope $\beta$ is related to an effective interaction radius
$a=\sqrt{2\pi\beta}$.

Figure 1: Differential elastic cross section of 12C- 12C scattering at the
energy about 1 GeV per nucleon. The dash-dotted line shows the pure Coulomb
scattering, the solid line is for the sum of the Coulomb and strong
scatterings, Eq.(13).

An efficient way to deal with the function (2) is provided by the generating
function $Z(u,v)$,

$S_{AB}(b)\,=\,\frac{1}{Z(0,0)}\frac{\partial^{A}}{\partial
u^{A}}\frac{\partial^{B}}{\partial v^{B}}\,Z(u,v)\biggl{|}_{u=v=0},$ (5)

for which the closed expression have been obtained in Ref. [6]

$\displaystyle Z(u,v)\,$ $\displaystyle=$
$\displaystyle\,e^{W_{y}(u,v)},~{}~{}~{}~{}~{}~{}~{}~{}z_{y}\,=\,1-\frac{1}{2}\frac{\sigma_{NN}^{tot}}{a^{2}},$
(6) $\displaystyle W_{y}(u,v)\,$ $\displaystyle=$
$\displaystyle\,\frac{1}{a^{2}}\int
d^{\,2}x\,\ln\bigl{(}\\!\\!\sum\limits_{M\leq A,N\leq
B}\frac{z_{y}^{M\,N}}{M!N!}\bigl{[}a^{2}u\rho_{A}^{\bot}(x-b)\bigr{]}^{M}\bigl{[}a^{2}v\rho_{B}^{\bot}(x)\bigr{]}^{N}\bigr{)}.$
(7)

The transverse densities entering this formula are expressed through the three
dimensional nucleon distributions in the colliding nuclei,

$\rho_{A,B}^{\bot}(x_{\perp})\,=\,\int
dz\,\rho_{A,B}(z,x_{\perp}),~{}~{}~{}\int
d^{2}x_{\perp}\,\rho_{A,B}^{\bot}(x_{\perp})\,=\,1.$

The function $W_{y}(u,v)$ (5) goes as the series built of the overlaps,

$t_{m,n}(b)=\frac{1}{a^{2}}\int
d^{\,2}x\,\bigl{[}a^{2}\rho_{A}^{\bot}(x-b)\bigr{]}^{m}\,\bigl{[}a^{2}\rho_{B}^{\bot}(x)\bigr{]}^{n},$
(8)

with $m\leq A$ and $n\leq B$ For relatively light nuclei, say, $A,B\lesssim
20$, the generating function can be explicitly differentiating with respect
$u$, $v$ variables yielding the final $S_{AB}(b)$ function. For more heavy
nuclei the differentiating amounts with a rather good accuracy to sending
these variables to $A$ and $B$ values (see Ref. [6] for details).

The density distribution in a relatively light nuclei with the atomic weight
$A\lesssim 20$ is well parametrized by the harmonic oscillator function

$\rho_{A}(r)\,=\,\rho_{0}\bigl{[}1+\frac{1}{6}\,(A-4)\frac{r^{2}}{\lambda^{2}}\bigr{]}e^{-\frac{r^{2}}{\lambda^{2}}},$
(9)

with $\rho_{0}$ being the normalization, and the factor $\lambda$ being
adjusted to match the nuclear mean square radius,
$R_{\mathrm{rms}}=\sqrt{r_{A}^{2}}$, $r_{A}^{2}=\int
d^{3}r\,r^{2}\rho_{A}(r)$. The simple form of this parametrization enables one
to get analytically all the overlaps relevant to 12C –12C scattering. For more
complicated densities the overlaps can be calculated numerically.

There exist several parametrizations for 11Li nucleon density, which are more
complex due to the halo structure. The density is supposed to be the sum of
two separate parts for the central core and for the surrounding it neutron
halo [9, 10, 11, 12]. Below we use the particular parametrization of the form
[13]

$\rho(r)\,=\,N_{c}\rho_{c}(r)\,+\,N_{v}\rho_{v}(r),$ (10)

$N_{c}$ is the number of the nucleons in the core, and $N_{v}$ is the number
of the valence neutrons in the halo. The core density is taken in Gaussian
form,

$\rho_{c}(r)\,=\,\frac{1}{\pi^{\frac{3}{2}}a_{c}^{3}}\,e^{-\frac{r^{2}}{a_{c}^{2}}},~{}~{}~{}~{}a_{c}=\sqrt{2/3}\,R_{c},$

where $R_{c}$ is the core mean square radius. The halo is also parametrized in
the Gaussian forms, which are different for the various shell structures
supposed for it,

$\begin{array}[]{ccc}\rho_{v}^{G}(r)\,=\,\frac{1}{\pi^{\frac{3}{2}}a_{G}^{3}}e^{-\frac{r^{2}}{a_{G}^{2}}},{}{}{}&~{}\rho_{v}^{O}(r)\,=\,\frac{2}{3\pi^{\frac{3}{2}}a_{O}^{5}}\,r^{2}e^{-\frac{r^{2}}{a_{O}^{2}}},{}{}{}&~{}\rho_{v}^{2S}(r)\,=\,\frac{2}{3\pi^{\frac{3}{2}}a_{2S}^{3}}\left(\frac{r^{2}}{a_{2S}^{2}}-\frac{3}{2}\right)^{2}e^{-\frac{r^{2}}{a_{2S}^{2}}},\\\
a_{G}=\sqrt{2/3}\,R_{v},&a_{O}=\sqrt{2/5}\,R_{v},&a_{2S}=\sqrt{2/7}R_{v},\end{array}$
(11)

where $R_{v}$ is the halo mean square radius. Whilst more complicated, these
forms also admit the straightforward analytical evaluation of $t_{m,n}(b)$
functions.

Figure 2: Differential elastic cross section of 11Li–12C scattering at the
energy about 1 GeV per nucleon. The different lines are for the sum of the
strong and Coulomb cross sections for the various halo density
parametrizations (11). The dotted line is for $\rho_{v}^{G}$, the dashed line
is for $\rho_{v}^{O}$, the solid line is for $\rho_{v}^{2S}$. The dash-dotted
line stands for the pure Coulomb cross section.

It is also necessary to account for the Coulomb scattering. The Coulomb
amplitude reads

$f_{C}(q)\,=\,-\frac{1}{2\pi}M_{AB}\,Z_{A}Z_{B}e^{2}\int
d^{3}z\,e^{iqz}V_{c}(z),~{}~{}~{}V_{c}(z)\,=\,\int
d^{3}xd^{3}y\,\frac{\rho_{A}(x)\rho_{B}(y-z)}{|x-y|}$

where $q=p-p^{\prime}$ is the momentum transfer, $M_{AB}$ is reduced mass of
the A and B nuclei, $Z_{Z}$, $Z_{B}$ are their charge. Passing to the form
factors one gets

$f_{C}(q)\,=\,-2M_{AB}Z_{A}Z_{B}e^{2}\,\frac{\rho_{A}(q)\rho_{B}(q)}{q^{2}},~{}~{}~{}\rho_{A,B}(q)\,=\,\int
d^{3}x\,e^{iqx}\rho_{A,B}(x),$ (12)

the core form factor having been taken for the halo nucleus.

Figure 3: The enlarged fragment of Fig.2. The notations are the same as in
Fig.2.

The elastic differential cross section is evaluated through the scattering
amplitude $f$ as

$d\sigma\,=\,|\,f\,|^{2}2\pi\,\sin\theta
d\theta\,=\,-\frac{\pi}{k^{2}}\,dt,~{}~{}~{}dt=2k^{2}\sin\theta d\theta,$

where $\theta$ and $k$ are the scattering angle and the momentum conjugated to
the relative distance in the center of mass system,
$k=(M_{A}p_{B}-M_{B}p_{A})/(M_{A}+M_{B})$,
$t=(p-p^{\prime})^{2}=2k^{2}(1-\cos\theta)$. The amplitude $f$ is a sum of the
Coulomb and the short range strong ones, thus

$\frac{d\sigma}{dt}\,=\,\frac{\pi}{k^{2}}\bigl{|}f_{C}(q)\,+\,F_{AB}^{el}(q)\bigr{|}^{2}\,=\,\frac{1}{4\pi}\,\biggl{|}\,\frac{2\pi}{k}f_{C}(q)\,+\,i\phi(q)\,\biggr{|}^{2},$
$\phi(q)\,=\,\int d^{2}b\,e^{iqb}[1-S_{AB}(b)].$

Given the fact that in the one photon exchange approximation the Coulomb
amplitude is real we have

$\frac{d\sigma}{dt}\,=\,\frac{1}{4\pi}\,\biggl{[}\,\frac{4\pi^{2}}{k^{2}}f_{C}^{2}(q)\,+\,\phi^{2}(q)\,\biggr{]}.$
(13)

##  3 Numerical calculations

The parameters of NN amplitude (4) have been chosen to be
$\sigma_{NN}^{tot}=43$ mb, $\beta=0.2$ fm2, the NN total cross section being
taken as an average over $pp$ and $pn$ values.

We start from 12C – 12C scattering. The distribution (9) has been used with
the value of the mean square radius $R_{\mathrm{rms}}=2.49$ fm fitted from
Monte-Carlo simulation of a 12C – 12C collision in Ref. [14]. The results of
the calculations are presented in Fig.1 The figure exhibits 2 maxima and 2
minima between them at the transferred momentum interval $-t=0.001-0.15$GeV2
(the point $-t=0$ is excluded for the singular Coulomb contribution). As is
seen from the figure the values the cross section takes at the minima are due
to the Coulomb interaction only. This is a consequence of the fact that the
real part of the strong NN amplitude is neglected.

The Fig.2 presents the results for the scattering of the exotic halo nucleus
11Li on 12C target obtained with the three parametrizations (11) with
$R_{c}=2.50$ fm, $R_{v}=5.04$ fm, $N_{c}=9$, $N_{v}=2$ [13]. Since the core
part is common for all of them the Coulomb contributions are identical and the
difference should be attributed to various shell structures supposed for the
neutrons in the halo. To expose this difference more explicitly the enlarged
fragment of Fig.2 is presented in Fig.3.

##  4 Conclusion

We have carried out the calculations of the differential cross sections of 12C
– 12C and 11Li – 12C scattering in the complete Glauber theory with account
for the Coulomb contribution. The Coulomb scattering plays an important role
in the nucleus-nucleus collisions especially at the small transverse momenta.
In the case of 11Li – 12C scattering we have considered 3 parametrization for
the halo taken from Ref. [13]. The results show a notable difference between
the obtained cross sections.

## References

  * [1] R.J. Glauber, Cross-sections in deuterium at high-energies Phys. Rev. 100, 242 (1955).
  * [2] R. J. Glauber and G. Matthiae, High-energy scattering of protons by nuclei, Nucl. Phys. B 21, 135 (1970).
  * [3] W. Czyz and L. C. Maximon, High-energy, small angle elastic scattering of strongly interacting composite particles, Annals Phys. 52, 59 (1969).
  * [4] G. D. Alkhazov and V. V. Sarantsev, Sensitivity of Cross Sections for Elastic Nucleus-Nucleus Scattering to Halo Nucleus Density Distributions, Phys. At. Nucl. 75, 1544 (2012) arXiv:1107.0533.
  * [5] G. D. Alkhazov and V. V. Sarantsev, Sensitivity of reaction cross sections to halo nucleus density distributions, Phys. At. Nucl. 77, 912-916 (2014) arXiv:1312.0782.
  * [6] Y. M. Shabelski and A. G. Shuvaev, Generating function for nucleus-nucleus scattering amplitudes in Glauber theory, Phys. Rev. C 104, no.6, 064607 (2021) arXiv:2104.04943
  * [7] C. Pajares and A. V. Ramallo, Effects of the multiple scattering structure in the propagation of hadronic properties in nucleus-nucleus collisions, Phys. Rev. D 31, 2800 (1985).
  * [8] V. M. Braun and Y. M. Shabelski, Multiple Scattering Theory for Inelastic Processes, Int. J. Mod. Phys. A 3, 2417 (1988).
  * [9] G. D. Alkhazov, A. V. Dobrovolsky and A. A. Lobodenko, Matter density distributions and radii of light exotic nuclei from intermediate-energy proton elastic scattering and from interaction cross sections, Nucl. Phys. A 734, 361 (2004).
  * [10] S. Ilieva, F. Aksouh, G. D. Alkhazov, L. Chulkov, A. V. Dobrovolsky, P. Egelhof, H. Geissel, M. Gorska, A. Inglessi, R. Kanungo, et al. Nuclear-matter density distribution in the neutron-rich nuclei 12,14 Be from proton elastic scattering in inverse kinematics, Nucl. Phys. A 875, 8 (2012).
  * [11] A. A. Korsheninnikov, E. Y. Nikolskii, C. A. Bertulani, S. Fukuda, T. Kobayashi, E. A. Kuzmin, S. Momota, B. G. Novatskii, A. A. Ogloblin, A. Ozawa, et al. Scattering of radioactive nuclei 6 He and 3 H by protons: Effects of neutron skin and halo in 6 He, 8 He, and 11 Li, Nucl. Phys. A 617, 45 (1997).
  * [12] M. V. Zhukov, B. V. Danilin, D. V. Fedorov, J. M. Bang, I. J. Thompson and J. S. Vaagen, Bound state properties of Borromean Halo nuclei: He-6 and Li-11, Phys. Rept. 231, 151 (1993).
  * [13] M. A. M. Hassan, M. S. M. Nour El-Din, A. Ellithi, E. Ismail and H. Hosny, The effect of halo nuclear density on reaction cross-section for light ion collision, Int. J. Mod. Phys. E 24, 1550062 (2015).
  * [14] C. Merino, I. S. Novikov and Y. M. Shabelski, Nuclear Radii Calculations in Various Theoretical Approaches for Nucleus-Nucleus Interactions, Phys. Rev. C 80, 064616 (2009).


# Improving Protein-peptide Interface Predictions in the Low Data Regime

Justin Diamond & Markus Lill
Department of Pharmaceutical Science
Univiersity of Basel
Klingelbergstrasse 50, 4056
Basel, Switzerland
<EMAIL_ADDRESS>
###### Abstract

We propose a novel approach for predicting protein-peptide interactions using
a bi-modal transformer architecture that learns an inter-facial joint
distribution of residual contacts. The current data sets for crystallized
protein-peptide complexes are limited, making it difficult to accurately
predict interactions between proteins and peptides. To address this issue, we
propose augmenting the existing data from PepBDB with pseudo protein-peptide
complexes derived from the PDB. The augmented data set acts as a method to
transfer physics-based contextdependent intra-residue (within a domain)
interactions to the inter-residual (between) domains. We show that the
distributions of inter-facial residue-residue interactions share overlap with
inter residue-residue interactions, enough to increase predictive power of our
bi-modal transformer architecture. In addition, this dataaugmentation allows
us to leverage the vast amount of protein-only data available in the PDB to
train neural networks, in contrast to template-based modeling that acts as a
prior.

##  1 Introduction

Compared to the experimental protein or protein small molecule structures,
like those in PDB (Berman et al., 2000), there is significantly less
crystallized structures of protein-peptide complexes which makes it difficult
to build a model like AlphaFold in this setting. Given that amino acids within
and between domains share fundamental characteristics derive-able from Quantum
Mechanics, there should be a transformation that improves the predictive power
of protein-peptide interactions from purely intra-protein interactions. In
principle, we want to determine the feasibility of such transfer-ability
methods by making simple augmentation of intra-protein residues to look more
like protein-peptide interactions. We take the augmentation as the cutting of
a fake peptide sequence from a protein sequence to obtain pseudo protein-
peptide complexes derived from the PDB to mimic that of the PepBDB (Huang et
al., 2018).

This could generalize to more heterogeneous contexts such as multiple peptides
interactions or protein domain interfaces, making it more general and
applicable to a wider range of problems. By learning the coupling information
where covalent bonding patterns are muteable, our model can generalize to
different sequences and even different types of molecules made up of amino
acids, i.e. branching or cyclic peptides.

We frame our problem as classification via contact predictions of inter-facial
residues of a peptide - protein complex from sequence information alone, with
the goal of predicting whether two residues between a protein and peptide are
within some Angstrom distance.

##  2 Background

CASP (Critical Assessment of Techniques for Protein Structure Prediction)
conference was initiated in the early 90s to compare/develop the best
computational techniques for in-silico prediction of protein structure. In
2020s, they declared this problem generally solved with the advent of Deep
Learning techniques initially centered around template based models (Zhang et
al., 2008), then including RESNET architectures (Xu, 2017) from Computer
Vision and finalized via a variation of Large Language Models (LLM) seen in
DeepMind’s AlphaFold (Senior et al., 2020) and Facebook’s Evolutionary Scale
Models (Rives et al., 2019). The key take away was residue-residue co-
evolutionary couplings obtained from large Multiple Sequence Alignments (MSAs)
contain enough information to reconstruct 3D information of proteins with
these new computational techniques. There are generally two direct successor
problems: crystal protein-protein/peptide complex reconstruction and
macromolecule dynamics i.e. computing equilibrium thermodynamic state
variables. We look to advance the first.

Macromolecule 3D reconstruction from amino-acid sequences beyond proteins
remains a key goal. Structure reconstruction initially centered around
transforming predicted residue-residue contacts information into an energy
potential which is used to optimized a protein structure. DeepMind was the
first to do this in a end-to-end differentiable manner. Most recently,
diffusion models (Wu et al., 2022) have shown great reconstruction abilities.

We generalize the problem setting to obtaining inter residue-residue
information from intra residue-residue couplings.

##  3 Methods

###  3.1 Bi-Modal Transformer

The bi-modal transformer, similar to (Lu et al.,2016), takes in two inputs,
the protein sequence $X_{p}$ of length $N$ and the peptide sequence $X_{l}$ of
length $M$, both of which are tokenized into a sequence of amino acids. The
transformer uses the guided-attention mechanism to compute the attention
matrix between the protein and peptide sequences and at each layer, the
protein and peptide sequences are alternated such that at one layer the
protein sequence embedding is updated with respect to the peptide one, and the
next vice versa.

In the first layer, the attention matrix, used to update the protein sequence,
between the protein and peptide sequences is computed as follows:

$A_{p,l}=\text{softmax}(\frac{Q_{p}K_{l}^{T}}{\sqrt{d_{k}}})V_{p}$

Where $Q_{p}$, $K_{l}$, $V_{p}$ are the query, key, and value matrices for
updating the protein sequence, and $d_{k}$ is the dimension of key used to
scale the dot-product attention.

The hidden representation of the protein and peptide sequences are obtained by
concatenating the attention matrices:

$H_{p}=\text{concat}(A_{p,l})$

In the next layer, the attention matrix between peptide and protein is
computed:

$A_{l,p}=\text{softmax}(\frac{Q_{l}K_{p}^{T}}{\sqrt{d_{k}}})V_{l}$

Where $Q_{l}$, $K_{p}$, $V_{l}$ are the query, key, and value matrices for the
peptide sequence.

$H_{l}=\text{concat}(A_{l,p})$

And this process continues for a fixed number of layers.

Finally, the objective function is obtained by performing logistic regression
on the concatenated representation of the protein and peptide sequences:

$Y=\sigma(W[H_{p},H_{l}]+b)$

Where $W$ is the weight matrix, $b$ is the bias term, and $\sigma$ is the
sigmoid function.

The overall objective of the model is to maximize the likelihood of the
correct interaction between the protein and peptide sequences, given by:

$\mathcal{L}=\sum_{i=1}^{n}y_{i}\log
p(y_{i}|x_{i})+(1-y_{i})\log(1-p(y_{i}|x_{i}))$

where $n$ is the number of examples, $x_{i}$ is the pair of protein and
peptide sequences, and $y_{i}$ is the true label indicating the interaction
between the protein and peptide, which is defined as a 10 Angstrom cutoff
threshold.

###  3.2 Evaluation Metrics

To compare the true and learned distribution of contacts, we use the softmax
function on the sum of the output of the model over all peptide residues for
each protein residue to obtain a probability distribution over protein
residues indicating the propensity of this residue to be in contact with one
of the peptide residues. The softmax function is defined as:

$P_{i}=\frac{e^{\sum_{j=1}^{M}x_{i,j}}}{\sum_{k=1}^{N}e^{\sum_{j=1}^{M}x_{k,j}}}$

Where $P_{i}$ is the probability of protein residue $i$ interacting with any
peptide residue, $x_{i,j}$ is the output of the model for the interaction
between protein residue i and peptide residue j, N is the number of protein
residues and M is the number of peptide residues.

Once we have the true and predicted probability distributions, we can use the
Kullback-Leibler divergence (KL divergence) to evaluate the goodness of the
model. The KL divergence between two probability distributions $P$ and $Q$ is
defined as:

$D_{KL}(P||Q)=\sum_{i=1}^{N}P_{i}\log\frac{P_{i}}{Q_{i}}$

This measures the dissimilarity between the true and predicted probability
distributions. The lower the KL divergence, the better the model is at
reproducing the true distribution of contacts.

###  3.3 Creating Augmented Protein-peptide Complexes

We create the augmented protein-peptide dataset by obtaining protein
structures from the Protein Data Bank (PDB). For each structure, we calculate
the distance matrix between all pairs of amino acid residues:

$D_{i,j}=||R_{i}-R_{j}||$

Where $D_{i,j}$ is the distance between amino acid residues i and j, $R_{i}$
and $R_{j}$ are the coordinates of residues i and j respectively.

We then transform the distance matrix into a probability distribution over
protein residues being in contact with other residues by summing over one
dimension of the matrix and performing a softmax:

$P_{i}=\frac{e^{\sum_{j=1}^{N}D_{i,j}}}{\sum_{k=1}^{N}e^{\sum_{j=1}^{N}D_{k,j}}}$

Where $P_{i}$ is the probability of protein residue $i$, $D_{i,j}$ is the
distance between amino acid residues $i$ and $j$, N is the number of amino
acid residues.

We then sample indices from this distribution, where the indices indicate
where the protein will be split. We also sample a uniform distribution from
10-50 amino acids in length to describe the length of the segment to be cut
out from the protein. The protein is then concatenated together, with the
cutout segment becoming the pseudo-peptide sequence. From the full distance
matrix and the cutout segment, we can also calculate the pseudo-distance
matrix.

##  4 Results

To evaluate the performance of our proposed method, we performed a series of
experiments on a dataset of protein-peptide complexes. Our goal was to assess
the ability of our method to improve contact predictions between the protein
and peptide.

In our first experiment, we predict contacts between residues of proteins and
peptides determined at a 10 Angstrom distance threshold. We then apply
gaussian smoothing with convolutional kernels with all the filter weights set
to the value of 1, which has the effects of smoothing out the contact signals
as the predictions can be sensitive to small changes in residue positioning.

As can be seen in Figure 1, the augmented data at times seems to help improve
accuracy by moving around contact groupings, sharpening the predictions, and
refining the predictions. Although the individual protein-peptide contact
predictions are not always improved (there are cases where predictions are
worse), from the test loss described below, in general, the prediction
accuracy improves.

Figure 1: Predicted contacts (from left to right) of True, no augmented data,
20000 augmented examples, and 30000 augmented examples. The X axis is ligand
residue indices and the Y axis is protein residue indices. Brighter yellow
regions indicate higher probabilities of a residue-residue contact between the
peptide and protein.

Take for instance the far-right predictions in Figure 1. The second to the
left of the four shows a less resolved cluster of lower probability contacts
towards the top compared to the augmented model. This indicates that the
augmented models increase their confidence further from the true contact
labels.

In the middle four contact diagrams in contrast, the augmented models show
higher probabilities for some of the contact predictions which,
retrospectively, accurately determines correct binding regions. It also
suggests that adding a certain amount of extra augmented training data can
increase accuracy further. However, certain precautions are necessary to make
sure the model does not over train on the most frequent set of augmented
protein-peptide complexes. Adding more than 30000 augmented examples did not
noticeably improve the test losses.

In the second figure, we looked at the effects of data augmentation on test
and validation loss. Some complexes were similar to eachother, as measured by
TMscore (Zhang et al., 2005), and if two or more complexes are over a certain
threshold, then they are made sure to be in either training, validation, or
test but not any other. This shows why the validation loss is slightly higher
than the test loss. The augmented models outperform the baseline while the
baseline admits a quick downtrend in prediction accuracy due to the small size
of the training set, while the augmented models mediate this observation.

Figure 2:  Validation and test error for baseline (purple, orange), +20000
(red, blue), and +30000 (black, blue) augmented datasamples

In another experiment, we comparing the KL divergence of contact distributions
on the protein or peptide. As discussed above, to obtain distributions over
the sequences of residues we apply sum over the peptide (or protein )
sequences and apply softmax to obtain a probability distribution over protein
( or peptide) sequences which are compared via the Kullback-Leibler
divergence. This is shown in Table 1 in the appendix, which shows improvements
in binding site distribution predictions with the augmented data. Lower KL
divergence indicates that the augmented data could be useful to a degree, but
care must be taken to not weigh the augmented samples too heavily during
training. We also note that aggregating the contact probabilities to a
distribution over peptide residues may be useful in deciphering which residues
are most important for binding.

##  5 Discussion

###  5.1 Summary of the results

Our results show that by augmenting a bi-modal transformer network with pseudo
protein-peptide complexes derived from the PDB, we are able to improve
predictions of contacts between proteinpeptide complexes and their binding
site predictions slightly. This is an important step towards a better
understanding of the physics of amino-acid interactions, and especially in the
design of transfer learning methods capable of generalizing across
heterogeneous amino acid complexes.

###  5.2 Future Research

Future directions could include adding Multiple Sequence Alignments to the
model, which may help improve sequence embeddings by implicitly encoding
evolutionary couplings intra-sequence wise, with the goal of obtaining
evolutionary inter-residue couplings between sequences.

Another direction would be to reconstruct protein-peptide complexes with these
contact predictions used as constraints to be satisfied.

Lastly, it necessary to derive a similarity scheme between augmented examples
so that the models do not over train on the most common augmented residue-
residue contacts, which would ideally mean we need not weight the augmented
datasets loss by a decreasing factor.

##  6 Conclusion

Our approach of augmenting a bi-modal transformer network with pseudo protein-
peptide complexes represents a novel method for predicting protein-peptide
interactions with limited data access. It highlights the efficiencies of data-
augmentation of transfer-learning. In principle, this approach is a form of
data distillation, where knowledge is transferred from a conditional
distribution to generalized joint distribution and is particularly useful in
cases where the distribution of the data is unknown or difficult to obtain,
and maximum likelihood methods are limited by the lack of data.

##  7 References

[1] Senior, A. W., Evans, R., Jumper, J., Kirshner, D., Korvink, J., Sylaidi,
I., Kandel, A., Baran, R., Ziebart, M., Frolov, D., et al. (2020). Improved
protein structure prediction using potentials from deep learning. Nature,
586(7830), 705–710.

[2] Zhang, Y. (2008). I-Tasser: a unified platform for automated protein
structure and function prediction. Nat. Methods, 5(2), 145–151.

[3] Xu, M., Yu, L., Song, Y., Shi, C., Ermon, S., Tang, J. (2022). GeoDiff: A
Geometric Diffusion Model for Molecular Conformation Generation. In
International Conference on Learning Representations (pp. 1-15). Retrieved
from https://openreview.net/forumid=PzcvxEMzvQC

[4] Rives, A., Meier, J., Sercu, T., Goyal, S., Lin, Z., Liu, J., Guo, D.,
Ott, M., Zitnick, C. L., Ma, J., Fergus, R. (2019). Biological Structure and
Function Emerge from Scaling Unsupervised Learning to 250 Million Protein
Sequences. PNAS. doi:10.1101/622803

[5] Xu, J. (2017). Accurate De Novo Prediction of Protein Contact Map by
Ultra-Deep Learning Model. arXiv preprint arXiv:1703.06876.

[6] Lu, J., Yang, J., Batra, D., Parikh, D., Lee, D. D. (2016). Hierarchical
Question-Image CoAttention for Visual Question Answering. arXiv preprint
arXiv:1606.00061.

[7] Wen Z, He J, Tao H, Huang SY. PepBDB: a comprehensive structural database
of biological peptide-protein interactions. Bioinformatics. 2018 Jul 6. doi:
10.1093/bioinformatics/bty579.

[8] Berman, H. M., Westbrook, J., Feng, Z., Gilliland, G., Bhat, T. N.,
Weissig, H., Shindyalov, I. N., and Bourne, P. E. (2000). The Protein Data
Bank. Nucleic Acids Res. 28, 235-242. doi: 10.1093/nar/28.1.235.

[9] Wu, K. E., Yang, K. K., van den Berg, R., Zou, J. Y., Lu, A. X., Amini, A.
P., Microsoft, and Stanford. (2022, November 24). Protein structure generation
via folding diffusion. [Online]. Available: https://arxiv.org/abs/2211.10297

[10] Kolmogorov, A. (1998). On Tables of Random Numbers. Theoretical Computer
Science. 207(2), 387–395. doi: 10.1016/S0304-3975(98)00075-9. MR 1643414.

[11] Zhang, Y. and Skolnick, J. (2005). TM-align: a protein structure
alignment algorithm based on the TM-score. Nucleic Acids Res. 33(7),
2302-2309. doi: 10.1093/nar/gki524.

##  8 Appendix

Figure 3: More predicted examples.

Figure 4: More predicted examples.


#  Transcribing Educational Videos Using Whisper
A preliminary study on using AI for transcribing educational videos

Ashwin Rao
University of Helsinki

###### Abstract

Videos are increasingly being used for e-learning, and transcripts are vital
to enhance the learning experience. The costs and delays of generating
transcripts can be alleviated by automatic speech recognition (ASR) systems.
In this article, we quantify the transcripts generated by whisper for 25
educational videos and identify some open avenues of research when leveraging
ASR for transcribing educational videos.

##  1 Introduction

During the last decade, we have witnessed an increase in the volume of video
content that is disseminated over the Internet. The pandemic further
exacerbated this trend as people started to consume a wide category of videos
from their houses [1]. Along with lectures, we have also witnessed a rise in
the conferences and talks that are being recorded and uploaded online on
streaming sites. These videos augment the material taught in the classrooms
and are increasingly being leveraged for educational purposes [2].

Educational videos, like entertainment videos, are consumed in a combination
of personal devices such as laptops, tablets, smartphones, and studies. The
capabilities of the audio systems on these devices vary significantly, and a
given audio file may sound different on each of these devices [3]. Words in an
audio segment recorded by amateurs may sound clear and comprehensible on one
device, and the same audio segment may be unintelligible on another device.
Furthermore, the educational videos might include the voices of people from a
wide range of ethnicities, and the speakers might also not be native speakers
of the language in which they are speaking. Clearly, the audio quality of
educational videos is vital, and addressing acoustic issues can result in
drastic improvement in the quality of the material [4]. However, the video and
audio quality of educational videos might not be optimal for all devices
because they may not be professionally created, edited, and processed.

    WEBVTT
    Kind: captions
    Language: en

    00:00:00.040 --> 00:00:02.460
    The following content is
    provided under a Creative

    00:00:02.460 --> 00:00:03.870
    Commons license.

Figure 1: Example Closed Caption. The metadata (the file format and language)
is followed by the time stamps during which the text can be shown.

Audio transcripts and subcaptions help alleviate the issues in the audio
quality and enable the viewers to receive a correct interpretation of the
content. For instance, Gernsbacher has shown that captions are particularly
beneficial for persons watching videos in their non-native language [5].
Although generating transcripts has been non-trivial, recent advances in
speech-to-text generation have shown promising results in transcribing audio
content. In the context of videos, transcripts are different from subtitles:
transcripts typically refer to a textual copy of the words someone has said in
the video, while subtitles refer to the textual versions of the dialogues in
the video [6]. Subtitles can either be open or closed: open subtitles are
embedded in the video frames, while closed subtitles are stored separately and
can be overlayed over the video frames or can be displayed on a second screen.
A variant of closed subtitles is closed captions which contain an additional
description of the audio-video content being shown, such as the sound made by
animals, etc. At times, a transcript can also include additional description;
examples include laughter by students, audience clapping, etc. A key
difference between a transcript and the subtitles is that a transcript does
not contain the time stamp at which the words in the transcript were said.

In this article, we do a preliminary evaluation of the quality of transcripts
generated by whisper [7]. We focus on the speech-to-text translation, and not
on the time stamp at which the word was spoken. Although there is a wide range
of tools and models for generating transcripts, we focus our attention on
whisper. Our goal is to get an understanding of using whisper for academic
videos and identify open avenues of research in the area of leveraging ASR for
transcribing academic videos.

##  2 Methodology

### Tools used and data processing pipeline.

For our analysis, we first collect a set of 25 YouTube videos that have closed
captions that are not automatically generated; YouTube shows if the captions
are auto-generated or provided by the content creator. For each video, we use
yt-dlp to download the best audio files corresponding to the video and the
available captions (as transcripts). The downloaded captions are the baseline
for our evaluation. We do this because YouTube keeps multiple versions of the
same video, and dynamically adapts to the optimal audio/video quality
depending on the network connectivity. We then use whisper [7] to generate the
transcripts, and run it in our cluster powered by NVidia V100 GPUs [8]. The
generated transcripts are then compared with our baseline transcripts
downloaded from YouTube using jiwer. We summarize the tools used in Table 1.

Tool | Version | Usage
---|---|---
whisper | 20230314 | Speech to text conversion.
jiwer | 3.0.1 | Compare the text in two files.
yt-dlp | 2023.03.04 | Download audio files and transcripts.
opusinfo | 0.1.10 | Extract metadata from audio files.
Table 1: Software Tools

### Automatic Transcript Generation (Speech to Text).

In this article, we restrict ourselves to whisper [7]. Whisper offers multiple
models which can be used to process the transcribe the audio files, and in our
evaluation we restrict ourselves to the following five models (number of
parameters in parenthesis) of which large-v2 is a multi-lingual model: base.en
(74 M), tiny.en (39 M), small.en (244 M), medium.en (769 M), and large-v2
(1550 M). We acknowledge that there is a wide range of open-source tools and
models including Kaldi [9], Flashlight [10], and Paddlespeech [11]. We plan to
analyze the efficiency of these tools in our subsequent works.

### Metrics for evaluating transcript quality.

The Word Error Rate (WER) is a commonly used metric for comparing texts [12]
and it is computed as $WER=\dfrac{S+D+I}{N=H+S+D}$ where $H$ is the number of
hits (correct words), $S$ is the number of substitutions, $D$ is the number of
deletions, and $I$ is the number of insertions, and $N$ denotes the number of
words in the reference (baseline) against which the hypothesis (results of the
transcribing tool) is being evaluated. In contrast, the Match Error Rate (MER)
is the probability of an incorrect match [12], and is given by
$MER=\dfrac{S+D+I}{H+S+D+I}$. The Word Information Lost (WIL) is an
approximation for the Relative Information Lost (RIL) which is computed using
the hits, substitutions, insertions, and deletions [12]; the RIL measures the
statistical dependence between the reference and the hypothesis and is
calculated using the Shannon entropy. Our goal is not to compare the metrics,
and instead we rely on the WER, MER, and WIL to evaluate the performance of
the transcription. We use jiwer to compute the WER, MER, and WIL. It is known
that jiwer can end up computing a higher WER without normalizing the text [7],
and the WER depends on the normalization technique used. For this preliminary
analysis we avoid using any custom normalizations, and we plan to explore the
impact of normalization in a subsequent study.

Figure 2: Average Bitrate of the Audio Files.

### Dataset Description.

Of the 25 YouTube videos, 15 were from lectures on MIT OCW. The remaining 10
included 5 talks at Google, one talk at MIT OCW, and four Turing Award
lectures.111Availability: The details of these videos are available with our
code and datasets at: https://version.helsinki.fi/transcribe-educational-
videos/preliminary-study-dai2023/. In Figure 2, we present the playback
duration (size in seconds) of each of the videos and the average bitrate of
the audio file. The quality of the audio file is important because it can
affect the quality of the transcripts being generated, and we observe that the
audio files downloaded have an average bit rate of at least 92 kbps. Note that
the audio file was encoded in opus audio format which supports variable
bitrate and is optimized for speech [13]. We also observe that the audio files
were sampled at 48 kHz. Whisper internally converts the audio file to 16 kHz,
and we believe that the audio files in our dataset have a sufficiently higher
frequency from which audio segments can be sampled at 16 kHz.

##  3 Evaluation

In Figure 3, we present the time required to transcribe a video for a given
playback time (see Figure 3(a)), and also for a given word count in our
baseline transcripts (see Figure 3(b)). We observe that the time to transcribe
increases linearly with the playback duration and word count, and the larger
models require more time. We present these results to give a ballpark on what
to expect, and we are aware that these times are heavily biased to the audio
content, and the computational capabilities in our cluster.

(a) Time to transcribe a video for a given playback duration.

(b) Time to transcribe a video for a given word count.

Figure 3: Transcription Time. The transcription time, i.e., the time to
generate transcripts, increases linearly with the playback duration and word
count. The larger models require more time than their smaller counterparts.
Figure 4: Relative transcription time. If the playback time is 50 s and it
takes 10 s to generate the transcript then the fraction of playback time is
10/50 = 0.2, i.e., generating a transcript required 20% of the playback time.
(Range = min, max)

In Figure 4, we plot the fraction of the playback time that a given model took
to transcribe the video. We observe that even the large-v2 model was able to
complete the transcription process in less than 25% of the time required to
playback the video. For the videos in our dataset, and while running whisper
on our servers, we observe that the base, tiny, and small models took less
than 10% of the playback time to transcribe the video, and the larger models
took less than 25% of the playback time. A typical human transcriber would
require at least the playback time to listen to the whole audio. In Table 2,
we present a snippet of the transcripts generated using Whisper. In this
snippet, the speaker asks the audience member to repeat what they said because
of audio issues. We see that the original transcript marks the conversation as
inaudible while the whisper tries to guess what is said, and the results vary
with the model size. Clearly, this speed-up when using smaller models is
meaningless if the quality of the transcription is poor.

Reference: I’m —- [inaudible] student from Duke. I’m working on domain
specific architecture.
---
[dotted] base.en: I’m ——- —-, I’m a PhD student from Duke I’m working on
Thomas versus architecture There’s this thing called swallowing the
microphones, so you have to actually Say it again, what you do? So I’m —— —-,
a PhD student from Duke I’m working on Thomas versus architecture
[dotted] tiny.en: I’m —- — —-, I’m a PhD student from Duke, I’m working on So
I’m —- — —-, a PhD student from Duke, I’m working on Thomas’s certificate
architecture
[dotted] small.en: I’m — ——-. I’m a PhD student from Duke. I’m working on
There’s this thing called swallowing the microphone so you have to actually
yeah, so say it again what you do So I’m — ——- a PhD student from Duke. I’m
working on a domain specific architecture
[dotted] medium.en: I’m ——- —-. I’m a PhD student from Duke. I’m working on
domain-specific architecture. There’s this thing called swallowing the
microphones. You have to actually… Say it again. What did you do? I’m ——- —-,
a PhD student from Duke. I’m working on domain-specific architecture.
[dotted] large-v2: I’m —- — —- I’m a PhD student from Duke I’m working on
domain specific architecture there’s this thing called swallowing the
microphone so you have to actually yeah so say it again what you do so I’m —-
— —- a PhD student from Duke I’m working on domain specific architecture.
Table 2: Example transcript with high WER. The above transcripts are for a
segment at time offset 1h:02m:58s of the the following video
https://www.youtube.com/watch?v=3LVeEjsn8Ts#t=62m58s. Figure 5: Transcript
quality. The error bars represent the min and max across the files in the
dataset.

In Figure 5, we present the WER, MER, and WIL when using the various models.
Across all the metrics, we observe that the WER, MER, and WIL decreases as the
number of parameters in the models increases. An exception is for the large-v2
model. We believe that this is primarily due to the lack of using a normalizer
[7], and the audio segments that were marked inaudible in the original
transcripts. As shown in Table 2, whisper transcribes the conversation marked
inaudible by the human transcriber, and the volume of text generated (sans
punctuations) by the large-v2 model is larger than the other models thus
resulting in a higher error rate.

Figure 6: Fraction of Hits, Substitutions, Deletions, and Insertions. Error
bars represent the min and max across files in our dataset. The cutout zooms
into the Deletions and Insertions.

Along with the example provided in Table 2, we also observe a high WER, a high
WIL, and a high MER for other videos, as highlighted by the error bars in
Figure 5. To better understand this behavior, we present the fraction of hits,
substitutions, deletions, and insertions in Figure 6. Across all models, we
observe that the hits are above 80% for the majority of videos, and the
fraction of hits increases with the number of parameters. However, for some
videos, such as the one in Table 2, we observe a large number of
substitutions, insertions, and deletions.

One reason for the high error rates is that whisper does not provide inaudible
as output and tries to extract the text even from the audio which a human
transcriber might mark as inaudible. This is further exacerbated by not
leveraging the context. For instance, in the example shown in Table 2 the
conversation was about domain-specific architecture, and the question being
asked was on the same topic, and yet some of the models wrongly predicted the
outcome to be Thomas version architecture or Thomas’s certificate
architecture. These predictions are bullshit222We apologize for the use of
profanity, and we rely on the following quote by Harry Frankfurt [14] for
describing the term bullshit: “it is impossible for someone to lie unless he
thinks he knows the truth. Producing bullshit requires no such conviction.”
because they (and the underlying models) are indifferent to truth.
Furthermore, although only two substitutions are needed to replace thomas
certificate architecture to domain specific architecture, incorrect
predictions like these diminish the usefulness of the generated transcripts.
We believe that marking the audio segments as either inaudible or its
equivalent that indicates a low confidence in the transcription result would
be more beneficial in such scenarios. This is achievable by tweaking some
thresholds in whisper’s configurations, and we plan to explore their impact in
subsequent works.

##  4 Concluding Remarks and Avenues for Future Work

We performed a preliminary analysis on the transcription capabilities of
Whisper, however we cannot draw any strong conclusions: our dataset is heavily
biased to the videos picked by the author, and the results are only for the
models of one tool, whisper. However, we gained some insights such as the
importance of marking audio segments as inaudible, and how inaudible audio
segments affect the quality of transcripts generated by ASR systems.

Some avenues for future work in this area include: a) metrics that account for
the semantic information such as the importance of each word, and evaluate the
quality of transcripts in end-user studies; b) comparing the transcription
results from different models; c) evaluating transcription capabilities for
languages other than English, and also for non-native speakers for these
languages; d) quantifying the impact of multiple speakers from different
ethical backgrounds in the same video/audio; e) approaches to identify the
context of the lecture/talk, and leveraging it for better transcriptions; f)
quantifying the costs for generating transcripts for different accelerators,
and identifying effectiveness of accelerators for transcript generation on
end-user devices; and g) quantifying the quality of subtitles including the
timestamp of the words and descriptions of the sounds that are generated by
the ASR system.

### Acknowledgement.

The authors wish to thank the Finnish Computing Competence Infrastructure
(FCCI) for supporting this project with computational and data storage
resources

## References

  * [1] Anja Feldmann, Oliver Gasser, Franziska Lichtblau, Enric Pujol, Ingmar Poese, Christoph Dietzel, Daniel Wagner, Matthias Wichtlhuber, Juan Tapiador, Narseo Vallina-Rodriguez, Oliver Hohlfeld, and Georgios Smaragdakis.  The lockdown effect: Implications of the covid-19 pandemic on internet traffic.  In Proceedings of the ACM Internet Measurement Conference, IMC ’20, page 1–18, New York, NY, USA, 2020. Association for Computing Machinery.
  * [2] Daniel T Seaton, Sergiy Nesterko, Tommy Mullaney, Justin Reich, Andrew Ho, and Isaac Chuang.  Characterizing video use in the catalogue of mitx moocs.  Proceedings of the European MOOC Stakeholder Summit, pages 140–146, 2014.
  * [3] Why we all need subtitles now.  https://www.youtube.com/watch?v=VYJtb2YXae8.  accessed 2023-May-01.
  * [4] Craig H Richardson.  Improving audio quality in distance learning applications.  1998\.
  * [5] Morton Ann Gernsbacher.  Video captions benefit everyone.  Policy insights from the behavioral and brain sciences, 2(1):195–202, 2015.
  * [6] Subtitles - wikipedia.  https://en.wikipedia.org/wiki/Subtitles.  accessed 2023-May-01.
  * [7] Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever.  Robust speech recognition via large-scale weak supervision, 2022.
  * [8] https://wiki.helsinki.fi/display/it4sci/HPC+Environment+User+Guide.  accessed 2023-May-01.
  * [9] Daniel Povey, Arnab Ghoshal, Gilles Boulianne, Lukas Burget, Ondrej Glembek, Nagendra Goel, Mirko Hannemann, Petr Motlicek, Yanmin Qian, Petr Schwarz, et al.  The kaldi speech recognition toolkit.  In IEEE 2011 workshop on automatic speech recognition and understanding, number CONF. IEEE Signal Processing Society, 2011.
  * [10] Jacob Kahn, Vineel Pratap, Tatiana Likhomanenko, Qiantong Xu, Awni Hannun, Jeff Cai, Paden Tomasello, Ann Lee, Edouard Grave, Gilad Avidov, Benoit Steiner, Vitaliy Liptchinsky, Gabriel Synnaeve, and Ronan Collobert.  Flashlight: Enabling innovation in tools for machine learning, 2022.
  * [11] Hui Zhang, Tian Yuan, Junkun Chen, Xintong Li, Renjie Zheng, Yuxin Huang, Xiaojie Chen, Enlei Gong, Zeyu Chen, Xiaoguang Hu, Yanjun Ma Dianhai Yu, and Liang Huang.  Paddlespeech: An easy-to-use all-in-one speech toolkit.  In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Demonstrations. Association for Computational Linguistics, 2022\.
  * [12] Andrew Cameron Morris, Viktoria Maier, and Phil Green.  From WER and RIL to MER and WIL: improved evaluation measures for connected speech recognition.  In Eighth International Conference on Spoken Language Processing, 2004.
  * [13] Jean-Marc Valin, Koen Vos, and T Terriberry.  Rfc 6716: Definition of the opus audio codec, 2012.
  * [14] Harry G Frankfurt.  On bullshit.  Princeton University Press, 2005.

This work is licensed under a Creative Commons Attribution-ShareAlike 4.0
International License.