# RobustL2S: Speaker-Specific Lip-to-Speech Synthesis exploiting Self-
Supervised Representations

Neha Sahipjohn1  Neil Shah12  Vishal Tambrahalli1  Vineet Gandhi1  1 CVIT,
Kohli Centre for Intelligent Systems, IIIT Hyderabad, India 2 TCS Research
Pune  India

###### Abstract

Significant progress has been made in speaker-dependent Lip-to-Speech
synthesis, which aims to generate speech from silent videos of talking faces.
Current state-of-the-art approaches primarily employ non-autoregressive
sequence-to-sequence architectures to directly predict mel-spectrograms or
audio waveforms from lip representations. We hypothesize that the direct mel-
prediction hampers training/model efficiency due to the entanglement of speech
content with ambient information and speaker characteristics. To this end, we
propose RobustL2S, a modularized framework for Lip-to-Speech synthesis. First,
a non-autoregressive sequence-to-sequence model maps self-supervised visual
features to a representation of disentangled speech content. A vocoder then
converts the speech features into raw waveforms. Extensive evaluations confirm
the effectiveness of our setup, achieving state-of-the-art performance on the
unconstrained Lip2Wav dataset and the constrained GRID and TCD-TIMIT datasets.
Speech samples from RobustL2S can be found at https://neha-
sherin.github.io/RobustL2S/

##  1 Introduction

Understanding lip movements offers a distinct advantage in situations where
auditory cues are unavailable. It proves particularly valuable for individuals
with hearing impairments, speech disorders and aids in speech rehabilitation
by providing visual feedback [1]. The synthesis of accurate speech from lip
movements can assist in tasks such as movie dubbing [2], language learning,
forensic investigations [3], video conferencing in noisy conditions, voice
inpainting [4] or giving artificial voice to people who cannot produce
intelligible sound.

The problem of Lip-to-Speech synthesis is inherently ill-posed because a
sequence of lip movements can correspond to multiple possible speech
utterances [5]. Additional challenges arise from factors such as head pose
movements, non-verbal facial expressions, variations in capture quality, and
ambient noise, which further complicate the problem. Reliance on contextual
information, such as environment, place, topic, etc., can help alleviate the
Lipreading challenges [6, 7]; however, such information may not always be
available.

Most existing approaches constitute an encoder-decoder architecture; the
encoder maps the lip sequence to intermediate representations, which are then
directly decoded into mel-spectrograms. The major drawback of this approach is
that apart from speech content, the decoder is also forced to predict the
time-varying speaker and ambient noise characteristics present in the ground
truth Mel. We hypothesize that this dependence hurts the model’s performance
in terms of speech intelligibility, reducing its usability for various
downstream applications [8]. Our work addresses these limitations by taking a
modularized approach, exploiting the advances in Self-Supervised Learning
(SSL) in audio and audio-visual scenarios.

Figure 1: The proposed RobustL2S model utilizes lip encoder and speech encoder
to extract SSL representations from lip sequences and their corresponding
speech. A Seq2Seq model maps the lip representations to speech
representations, which are then decoded to synthesize speech.

Fig. 1 illustrates the proposed RobustL2S framework. In contrast to direct mel
prediction from lip features, we take a two-staged approach. The first step
extracts SSL representations of lip sequences and maps them to corresponding
speech SSL representations using a sequence-to-sequence (Seq2Seq) model. The
key idea is to use speech embeddings that disentangle the content from the
speaker and ambient information. The second stage maps the content-rich speech
embeddings to raw speech using a speaker-conditioned vocoder. The proposed
RobustL2S framework simplifies training and brings robustness to variations in
head-pose, ambient noise, and time-varying speaker characteristics, leading to
significant gains in speech intelligibility. To validate the efficacy of our
approach, we perform comprehensive experiments on GRID [9], TCD-TIMIT [10] and
Lip2Wav [5] datasets. The quantitative measures and MOS scores show that the
synthesized speech generated by our method accurately represents the intended
content and improves on the intelligibility/naturalness compared to current
state-of-the-art methods [11, 12] on all three datasets.

More formally, our work makes the following contributions: (1) We propose a
novel modularized framework for Lip-to-Speech synthesis exploiting self-
supervised embeddings for both lip and speech sequences (2) A Seq2Seq network
for cross-modal knowledge transfer to map lip SSL representations to speech
SSL representations; and (3) Thorough experimental results demonstrating that
RobustL2S is capable of synthesizing high-quality speech, achieving state-of-
the-art results in objective and subjective evaluation without requiring
additional data augmentation [11].

##  2 Related Work

###  2.1 Lip-to-Speech synthesis

####  2.1.1 Constrained Lip-to-Speech synthesis

Constrained lip-to-speech synthesis tackles speech generation from videos with
limited vocabulary and minimal head movement [9, 10]. Ephrat et al.[13]
introduced a CNN-based approach to predict Linear Predictive Coding features
from silent talking videos. They later enhanced their model to a two-tower
CNN-based encoder-decoder architecture [14], encoding raw frames and optical
flows separately. [15] propose a combination of an autoencoder for extracting
bottleneck features from audio spectrograms and a lipreading network
comprising CNN, LSTM, and fully connected layers, for visual feature
extraction. On the other hand, [16] utilize a stochastic modeling approach
employing a variational autoencoder. Other methods [17, 18, 19, 20] employ
GANs to synthesize speech from video frames. [21] train an attention-based
encoder-decoder model to reconstruct speech from silent facial movement
sequences without human annotations. [22] performs multi-modal supervision,
using text and audio, to complement the insufficient word representations to
reconstruct speech with correct contents from the input lip movements. [23]
achieves zero-shot Lip-to-Speech synthesis using variational autoencoder to
disentangle speaker and content information and a face identity encoder for
unseen speakers.

####  2.1.2 Unconstrained Lip-to-Speech synthesis

Unconstrained lip-to-speech synthesis focuses on generating speech from real-
world videos of individuals talking. This approach considers videos with an
extensive vocabulary and significant head movements. Seminal work by Prajwal
et al. [5] introduced a 3D-convolution plus autoregressive Seq2Seq model,
adapted from Tacotron2 [24] for single speaker Lip-to-Speech synthesis. Their
model generates mel-spectrograms based on the input video frames. In contrast,
[25] employed a New A Rev. architecture to expedite the inference process. It
uses 3D-convolution blocks, a transformer condition module, and a Glow decoder
module [26] for refining the mel-spectrograms. [27] expanded on [5] by
training a transformer model to learn a joint latent distribution for speech
generation. The VV-Memory architecture [28] combines audio and visual
information using a key-value memory structure to enable video-to-speech
reconstruction and speaker-independent speech retrieval. A recent work by [11]
proposed an end-to-end New A Rev. transformer for synthesizing speech directly
from unconstrained talking videos. Their approach involves a visual encoder,
an acoustic decoder, a linear layer, and a GAN-based vocoder operating on Mel-
spectrograms. LipSound2 [21] investigates cross-modal self-supervised pre-
training of an encoder-decoder architecture with a location-aware attention
mechanism to map face image sequences to mel-scale spectrograms. In contrast,
our proposed RobustL2S differs by using content-rich SSL representations [29]
and learning target speech representations from lip sequences.

###  2.2 Self-supervised representation for Lip-to-Speech synthesis

Traditional works on Lip-to-Speech synthesis encode lip or face sequences to
hidden states, followed by a decoder to generate Mel-spectrograms. An
independently trained vocoder transforms them into time-domain waves. However,
these Mel frames are highly correlated along both time and frequency axes
which may degrade the performance of entire Lip-to-Speech synthesis [30].
Moreover, these Mel-spectrograms have higher variance than that of quantized
speech SSL representations increasing the complexity of training a Seq2Seq
model. Due to its recent emergence, the utilization of SSL representation in
Lip-to-Speech synthesis remains limited.

In VCVTS [12], vector quantized contrastive predictive coding (VQCPC) units
are extracted from lip movements, and a speaker encoder, pitch predictor, and
decoder are used to infer Mel frames. A separate voice conversion model and
vocoder are employed for speaker representation learning and speech synthesis.

Revise [31] employs a combined audio-visual speech recognition module (P-AVSR
- initialized with AV-HuBERT - an audio-visual SSL model) and a modified text-
to-speech synthesis module (P-TTS) for generalized speech enhancement tasks.
P-AVSR predicts discrete units derived from a self-supervised speech model,
and P-TTS converts these units into speech using a modified HiFi-GAN[29]
trained on the LJSpeech[32] dataset. Our work closely relates with [31]
utilizing SSL representations for Lip-to-Speech synthesis. However, we utilize
disentangled AV-HuBERT and HuBERT [33] features and a decoupled training
procedure to train a Seq2Seq model. Additionally, the existing works have not
yet fully capitalized on SSL representations for speaker-specific Lip-to-
Speech generation, and our efforts aim to address this gap.

##  3 Method

###  3.1 Preliminaries

RobustL2S consists of three modules: an encoder that extracts lip and speech
representation from their corresponding sequences, a Seq2Seq model that maps
lip representations to speech representations, and a vocoder that synthesize
speech using the speech representations. We introduce four functions as
follows:

  * •

$f_{\text{l}}:L^{T\times W\times H}\mapsto L_{\text{ssl}}$, which maps the
input lip sequence to its corresponding SSL representation. Here, $T$
represents the number of time-steps (frames), and $H$ and $W$ correspond to
the spatial dimensions of the frames.

  * •

$f_{\text{s}}:X\mapsto S_{\text{ssl}}$, which maps the ground-truth raw speech
to its corresponding SSL representation.

  * •

$f_{\text{s2s}}:L_{\text{ssl}}\mapsto S_{\text{ssl}}$, which maps the lip
representation to its corresponding speech representation.

  * •

$f_{\text{voc}}:S_{\text{ssl}}\mapsto\widehat{X}$, which maps the speech
representation to the synthesized speech $\widehat{X}$.

###  3.2 Encoder

Although our framework is compatible with various off-the-shelf SSL models, we
specifically utilize AV-HuBERT [34] for $f_{\text{l}}$, our video encoder to
extract lip representations. We use HuBERT [33] for $f_{\text{s}}$, to extract
speech representation of target speech signal. The HuBERT and AV-HuBERT
models, are trained using a masked-prediction loss to predict cluster IDs,
which are learned using k-means clustering. They are initialized with MFCC
features derived from acoustic frames, and subsequently more complex features
derived from an audio or audio-visual encoder, depending on the specific model
being used. AV-HuBERT incorporates a modified ResNet [35, 36] as its frontend,
coupled with a transformer encoder. We opted for AV-HuBERT instead of Visual-
HuBERT for this task because AV-HuBERT exhibits superior performance in the
lip-reading task, as demonstrated by [34]. This improvement is observed when
the target cluster IDs are derived from both audio and visual modalities, as
opposed to a single modality, whether it be audio or visual. We also finetune
the pretrained AV-HuBERT using an attention-based Seq2Seq cross-entropy loss
as in [34].

###  3.3 Seq2Seq model

In recent years, Seq2Seq models have gained significant attention in the field
of cross-domain generation. The core concept of our approach is to align
representations from two different domains - visual and audio, that share a
common generating process. By recovering correspondences between these
domains, we facilitate the transfer of knowledge from one domain to the other.

Our Seq2Seq model, denoted as $f_{\text{s2s}}$, adopts a New A Rev.-based
encoder-decoder architecture to map lip representations to their corresponding
speech representations. The encoder and decoder consist of feed-forward
transformer blocks with self-attention [37], along with $1$-dimensional
convolutions inspired by Fastspeech2 [38]. A transposed convolution layer is
used at the encoder to match the rate of video and audio representations. The
encoder takes the lip representation $L_{\text{ssl}}$ and encodes it into a
sequence of fixed-dimensional vectors. The decoder generates predictions for
all representations of $S_{\text{ssl}}$ simultaneously. We train three
versions of the Seq2Seq model:

  * •

$f_{\text{s2s-units}}$: This encoder-decoder architecture utilizes Cross-
Entropy (CE) loss to train the model on the decoded speech units. The CE loss
measures the difference between the decoded units and the ground-truth speech
HuBERT units. The input to the architecture consists of cluster IDs from the
video encoder, and the decoder predicts the corresponding HuBERT cluster IDs
for the audio. The objective can be written as:

$\mathcal{L}_{\text{CE}}=-\sum_{i=1}^{N}S_{\text{ssl\\_units\\_i}}\log(\hat{S}_{\text{ssl\\_units\\_i}}),$
(1)

where $S_{\text{ssl\\_units}}$ are the ground-truth speech units,
$\hat{S}_{\text{ssl\\_units}}$ are the decoded speech units, and N are the
number of HuBERT units.

  * •

$f_{\text{s2s-features}}$: Here the model learns mapping from audio-visual
feature to corresponding speech feature vectors. This model utilizes L1 loss,
quantifying the difference between the decoded features and ground-truth
speech features. The objective can be written as:

$\mathcal{L}_{\text{L1}}=\frac{1}{T}\sum_{i=1}^{T}|S_{\text{ssl\\_features\\_i}}-\hat{S}_{\text{ssl\\_features\\_i}}|,$
(2)

where $S_{\text{ssl\\_features}}$ are the ground-truth speech features,
$\hat{S}_{\text{ssl\\_features}}$ are the decoded speech features, and $T$ is
the time-steps.

  * •

$f_{\text{s2s-features-ctc}}$: This architecture follows the same structure as
$f_{\text{s2s-features}}$, but also includes an additional fully connected
linear head to predict CTC tokens after the encoder layer. For given input lip
representation $L_{\text{ssl}}\in\mathbb{R}^{TxD}$ of length T and dimension
D, let $Enc_{\text{ssl}}$ be the output of encoder. The goal is to minimize
the negative log-likelihood by using
$P_{CTC}(S_{\text{ssl}}|Enc_{\text{ssl}})$ to train the model effectively
using the CTC approach and is defined as:

$\mathcal{L}_{CTC}:=-\log P_{CTC}(S_{\text{ssl}}|Enc_{\text{ssl}}).$ (3)

By weighted summing the L1 and CTC loss functions, the objective function can
be formulated as:

$\mathcal{L}_{\text{Tot}}=\alpha_{CTC}*\mathcal{L}_{\text{CTC}}+\alpha_{L1}*\mathcal{L}_{\text{L1}},$
(4)

where $\alpha_{\text{CTC}}\in\mathbb{R}$ and $\alpha_{\text{L1}}\in\mathbb{R}$
are the hyperparameter that balances the influence between two loss.

###  3.4 Speech Vocoder

We use a modified version of HiFiGAN-v2 [39] to synthesize speech. It has a
generator $G$ and a discriminator $D$. $G$ runs $S_{ssl}$ through transposed
convolutions for upsampling to recover the original sampling rate followed by
residual block with dilations to increase the receptive field to synthesize
the signal, $\widehat{X}:=G(S_{\text{ssl}})$.

The discriminator in our model has the task of distinguishing the synthesized
signal $\widehat{X}$ from the original signal $X$. It is evaluated using two
sets of discriminator networks. The multi-period discriminators operate on
equally spaced samples of the signals, focusing on capturing temporal patterns
and characteristics. On the other hand, the multi-scale discriminators analyze
the input signal at different scales, enabling the model to capture both fine-
grained details and global structure. The primary objective of the model is to
minimize the discrepancy, measured by $D(X,\widehat{X})$, between the original
signal and the synthesized signal. This optimization process applies to all
the parameters of the speech decoder, improving its overall performance and
fidelity.

##  4 Experiments

###  4.1 Datasets

####  4.1.1 Lip2Wav

The Lip2Wav dataset [5] is a large, person-specific, unconstrained dataset,
commonly used for learning Lip-to-Speech synthesis for individual speakers. It
consists of real-world lecture videos featuring $5$ different speakers. Each
speaker has approximately $20$ hours of video data, and the vocabulary size
exceeds $5000$ words for each speaker. We do experiments on all five speakers:
Chess Analysis (chess), Chemistry Lectures (chem), Hardware Security (hs),
Deep Learning (dl), and Ethical Hacking (eh).

####  4.1.2 GRID-4S

The GRID-4S is a subset of the GRID audio-visual dataset [9] specifically
designed for constrained Lip-to-Speech synthesis. This subset includes two
male speakers ($s1$, $s2$) and two female speakers ($s4$, $s29$), which are
frequently used in the literature [19, 5]. The videos in the dataset were
captured in an artificial environment. The vocabulary used in GRID-4S is
limited to only $51$ words. The sentences in the dataset follow a restricted
grammar, with each sentence containing $6$ to $10$ words.

####  4.1.3 TCD-TIMIT-3S

The TCD-TIMIT-3S is a subset of the TCD-TIMIT dataset, which comprises
recordings of $62$ speakers captured under studio conditions. Among these
speakers, three are trained lip-speakers. The primary objective of selecting
this subset was to enable comparison with previous studies [19, 5]. Our focus
was solely on the audio-visual data generated by these three lip-speakers.
Each lip-speaker delivers $375$ distinct sentences that exhibit phonetic
diversity. Additionally, there are two sentences that are spoken by all three
lip-speakers.

###  4.2 Implementation details

####  4.2.1 Data preparation

For the GRID-4S and TCD-TIMIT-3S datasets, we adhere to the convention of
randomly selecting $90\%$ data for training, $5\%$ for validation, and $5\%$
for testing, as established in previous works [40, 5, 17, 11]. For Lip2Wav, we
adopt the official data split [5]. For consistency, in line with previous
works, we evaluated RobustL2S on the Lip2Wav dataset using a speaker-dependent
setting [5, 14, 28]. This involved training the network separately with
individual speakers. However, for the GRID-4S and TCD-TIMIT-3S datasets, we
evaluated RobustL2S in a constrained (seen) speaker setting [40, 5, 17]. In
this case, we trained a single speaker model for each dataset. The video
sequences are resampled to a frame rate of $25$ frames per second (fps), while
the raw audio is sampled at $16kHz$. We utilize the SFD [41] face detector to
detect $68$ keypoints, allowing us to crop a mouth-centered region-of-interest
measuring $96\times 96$ pixels. In order to solely assess the advantages of
using SSL representation in our proposed setup, we opt not to employ any data
augmentation techniques to enhance the quality of synthesized speech. The
Lip2Wav dataset does not provide transcripts, so we rely on the Whisper small
model [42] to extract transcripts. These transcripts are then used for
finetuning the AV-HuBERT model.

####  4.2.2 SSL representation

We utilize the official fairseq repository implementation of the BASE models
AV-HuBERT [34] and HuBERT [33] for our experiments. We fine-tune the AV-HuBERT
pretrained model with an attention-based STS cross-entropy loss for visual
speech recognition [34]. To achieve this, a transformer decoder is added to
the pretrained model, which autoregressively decodes the AV-HuBERT features to
target character probabilities. The fine-tuned AV-HuBERT model extracts SSL
representations for lip sequences, while HuBERT is employed to extract
representations from speech signals. Both models provide $768$-dimensional
features. For $f_{\text{s2s-units}}$ model, following the approach in [34,
33], the lip features are clustered into $2000$ AV-HuBERT units, while the
speech features are clustered into $100$ HuBERT units, using k-means
clustering. For $f_{\text{s2s-features}}$ model, the output features from
HuBERT and AV-HuBERT models are used directly.

####  4.2.3 Seq2Seq model

Our model comprises a $6$-layer transformer encoder and decoder with a hidden
dimension of $512$ and $2$ attention heads. we set the batch size to $32$ and
the maximum number of steps to $20,000$. We employ the Adam optimizer with an
initial learning rate of $4.4$ x $10^{-2}$, along with an annealing rate of
$0.3$ and annealing steps at $[3000,4000,5000]$. The HuBERT model encodes
speech into features at a frame rate of $50Hz$, while the SSL unit from AV-
HuBERT is encoded at $25Hz$. To match these rates, we incorporate a
lightweight transposed convolution layer with a kernel size of $[4,3]$ and a
stride length of $[2,1]$. We set $\alpha_{\text{CTC}}$ and
$\alpha_{\text{L1}}$ mentioned in (4) to $0.001$ and $1$, respectively.

####  4.2.4 Speech Vocoder

We use the official implementation of the adapted
HiFiGAN-v2111https://github.com/facebookresearch/speech-resynthesis to
generate audio from speech SSL representations. This model employs encoding of
raw audio into a sequence of discrete tokens from a set of $100$ possible
HuBERT tokens, with a code hop size of $160$ raw audio samples. We set the
batch size to $16$, the learning rate to $2$x$10^{-4}$, the number of
embeddings to $100$, the embedding dimension to $128$, and the model input
dimension to $256$. Following the approach in [43], F0 is not used as a
feature in our training process. The aforementioned vocoder configuration is
effective for speech units. However, for our investigated feature-based
models, $f_{\text{s2s-features}}$ and $f_{\text{s2s-features-ctc}}$, we apply
a pre-trained k-means
222https://github.com/facebookresearch/fairseq/tree/main/examples/
textless_nlp/gslm/speech2unit clustering model for speech HuBERT units. During
the inference phase, the generated features undergo k-means clustering to
obtain discrete speech units, which are then

Table 1: Performance comparison: Seq2Seq model vs. evaluated variations vs. no Seq2Seq model employed on chemistry speaker of Lip2Wav dataset. Baseline (Ours) |  STOI $\uparrow$ |  ESTOI $\uparrow$
---|---|---
$f_{l}\text{(pre-trained)}$ \+ $f_{voc}$ | 0.447 | 0.22
$f_{l}\text{(finetuned)}$ \+ $f_{voc}$ | 0.50 | 0.27
$f_{l}\text{(finetuned)}$ \+ $f_{\text{s2s-units}}$ \+ $f_{voc}$ | 0.18 | 0.013
$f_{l}\text{(finetuned)}$ \+ $f_{\text{s2s-features}}$ \+ $f_{voc}$ | 0.583 | 0.397
$f_{l}\text{(finetuned)}$ \+ $f_{\text{s2s-features-ctc}}$ \+ $f_{voc}$ | 0.557 | 0.368

passed through the speech vocoder. We train the vocoders for the three
datasets separately. samples.

###  4.3 Evaluation metric

During our evaluation, we employ several metrics to assess the quality of the
synthesized speech. These include: Word Error Rate (WER), Short-Time Objective
Intelligibility (STOI) [44] and Extended Short-Time Objective Intelligibility
(ESTOI) [45]. Additionally, we conduct subjective evaluations using Mean
Opinion Score (MOS), where human evaluators rate the quality, intelligibility
and naturalness of the synthesized speech based on their subjective
perception.

##  5 Results

###  5.1 Need for Seq2Seq model

We tested our hypothesis of using a Seq2Seq model on the Lip2Wav dataset,
specifically for a chemistry speaker and report our findings in Table 1. Fine-
tuning the AV-HuBERT model using transcripts consistently improved objective
metrics by approximately $0.05$ units on both the metrics compared to the
pretrained version. Deploying the Seq2Seq model on the finetuned AV-HuBERT
features ($f_{l}\text{(finetuned)}$ \+ $f_{\text{s2s-features}}$ \+ $f_{voc}$)
resulted in an increase of approximately $0.08$ and $0.12$ units in STOI and
ESTOI metrics, respectively, compared to not using the Seq2Seq model
($f_{l}\text{(finetuned)}$ \+ $f_{voc}$). These results highlight the
effectiveness of our Seq2Seq approach using SSL representations for Lip-to-
Speech synthesis. The significant performance gap (approximately $0.39$ units
on both metrics) between the Seq2Seq model using SSL features and the model
using SSL units on both evaluated metrics approximates the amount of
information lost in speech reconstruction when audio-visual sequences are
represented as SSL units instead of SSL features. From now on, we will refer
to $f_{l}\text{(finetuned)}$ \+ $f_{\text{s2s-features}}$ \+ $f_{voc}$ as
RobustL2S. The inclusion of CTC loss in our $f_{l}\text{(finetuned)}$ \+
$f_{\text{s2s-features-ctc}}$ model resulted in a statistically insignificant
decrease of approximately $0.02$ units in STOI compared to the model without
CTC loss, $f_{l}\text{(finetuned)}$ \+ $f_{\text{s2s-features}}$. The decrease
may be due to the lack of ground-truth transcripts in the Lip2Wav dataset.
However, when evaluating our model using CTC loss on datasets (GRID-4S and
TCD-TIMIT-3S) with ground-truth transcripts, we observed a slight increase of
$0.02$ units. Nevertheless, our focus is on working with datasets in the wild
that generally lack ground-truth transcripts, so we proceed with experiments
excluding the CTC loss.

Table 2: Performance comparison in constrained-speaker setting on GRID-4S dataset Method |  STOI $\uparrow$ |  ESTOI $\uparrow$ |  WER $\downarrow$
---|---|---|---
Vid2speech [13] | 0.491 | 0.335 | 44.92 %
Lip2AudSpec [15] | 0.513 | 0.352 | 32.51 %
1D GAN-based [17] | 0.564 | 0.361 | 26.64 %
Vocoder-based [40] | 0.648 | 0.455 | 23.33 %
Ephrat et al. [14] | 0.659 | 0.376 | 27.83 %
Lip2Wav [5] | 0.731 | 0.535 | 14.08 %
VAE-based [16] | 0.724 | 0.540 | -
VCA-GAN [19] | 0.724 | 0.609 | 12.25 %
kim et al. [28, 46] | 0.738 | 0.579 | -
RobustL2S | 0.754 | 0.571 |  11.21 %
Table 3: Performance comparison in constrained-speaker setting on TCD-TIMIT-3S dataset Method |  STOI $\uparrow$ |  ESTOI $\uparrow$ |  WER $\downarrow$
---|---|---|---
Vid2speech [13] | 0.451 | 0.298 | 75.52 %
Lip2AudSpec [15] | 0.450 | 0.316 | 61.86 %
1D GAN-based [17] | 0.511 | 0.321 | 49.13 %
Ephrat et al. [14] | 0.487 | 0.310 | 53.52 %
Lip2Wav [5] | 0.558 | 0.365 | 31.26 %
VCA-GAN [19] | 0.584 | 0.401 | -
RobustL2S | 0.596 | 0.452 |  29.03 %

###  5.2 RobustL2S in Constrained settings

Table 2 and 3 summarizes the performance of our RobustL2S in the context of
Lip-to-Speech synthesis using constrained datasets: GRID-4S and TCD-TIMIT-3S.
We compare our results with existing Lip-to-Speech synthesis works, including
state-of-the-art approaches. We report the mean test scores on all four
speakers of the GRID-4S dataset and all three speakers of the TCD-TIMIT-3S
dataset, as documented in previous works. Remarkably, our RobustL2S approach
demonstrates significant improvements in terms of STOI and WER metrics when
compared to other approaches. This improvement is particularly noticeable on
the TCD-TIMIT-3S dataset, which contains a larger number of novel words that
were unseen during training. This observation highlights the ability of our
RobustL2S to accurately pronounce new words and effectively capture semantic
information from lip movements, resulting in the generation of more
intelligible speech.

Table 4: Performance comparison in speaker-dependent setting on Lip2Wav dataset Speaker | Method |  STOI $\uparrow$ |  ESTOI $\uparrow$
---|---|---|---
Chemistry Lectures (chem) | Ephrat et al. [5] | 0.165 | 0.087
GAN-based [47] | 0.192 | 0.132
Lip2Wav [5] | 0.416 | 0.284
Hong et al. [28] | 0.566 | 0.429
RobustL2S | 0.583 | 0.397
Chess Analysis (chess) | Ephrat et al. [5] | 0.184 | 0.098
GAN-based [47] | 0.195 | 0.104
Lip2Wav [5] | 0.418 | 0.290
Hong et al. [28] | 0.506 | 0.334
RobustL2S | 0.517 | 0.340
Deep Learning (dl) | Ephrat et al. [5] | 0.112 | 0.043
GAN-based [47] | 0.144 | 0.070
Lip2Wav [5] | 0.282 | 0.183
Hong et al. [28] | 0.576 | 0.402
RobustL2S | 0.627 | 0.419
Hardware Security (hs) | Ephrat et al. [5] | 0.192 | 0.064
GAN-based [47] | 0.251 | 0.110
Lip2Wav [5] | 0.446 | 0.311
Hong et al. [28] | 0.504 | 0.337
RobustL2S | 0.511 | 0.337
Ethical Hacking (eh) | Ephrat et al. [5] | 0.143 | 0.064
GAN-based [47] | 0.171 | 0.089
Lip2Wav [5] | 0.369 | 0.220
Hong et al. [28] | 0.463 | 0.304
RobustL2S | 0.493 | 0.277
Figure 2: MOS scores on Intelligibility, Quality and Naturalness with their 95
% confidence interval computed from their t-distribution on GRID-4S dataset
Figure 3: MOS scores on Intelligibility, Quality and Naturalness with their 95
% confidence interval computed from their t-distribution on Lip2Wav dataset

###  5.3 RobustL2S in Unconstrained settings

Table 4 provides a synopsis of RobustL2S’s performance on the Lip2Wav dataset.
This dataset includes a significant amount of silences between words, and
RobustL2S shows a notable improvement across all metrics. Despite Lip2Wav’s
data asynchrony issues, which may affect the quality of the generated speech,
RobustL2S demonstrates substantial performance gains in objective metrics,
highlighting its overall superiority in producing intelligible speech.
However, it is worth noting that RobustL2S performs similarly or slightly
worse than [28] on the hs and eh ESTOI metric. This could potentially be
attributed to the poor resolution ($480p$ and $360p$) of the original videos,
making it challenging to accurately recognize lip regions.

###  5.4 Subjective evaluation

Fig. 2 and Fig. 3 shows the MOS scores on intelligibility (MOS(I)), quality
(MOS(Q)), and naturalness (MOS(N)) of synthesized speech from evaluated
methods on Grid-4S and Lip2Wav datasets. We requested ten English proficient
subjects to score five randomly selected samples from different methods on the
Lip2Wav and GRID-4S datasets. It can be observed that our model outperforms
the evaluated methods, exhibiting higher Mean Opinion Score (MOS) values. This
demonstrates that the proposed approach inherits the advantages of
disentangled SSL features and the mapping of lip sequences to content-specific
information. As a result, our model not only inherently improves the
intelligibility aspect of synthesized speech but also generates speech that is
highly natural and of high quality.

##  6 Conclusions

We propose a novel framework for Lip-to-Speech system, called RobustL2S ,
which accurately synthesizes spoken content from silent videos. This is
accomplished by utilizing a New A Rev. based sequence-to-sequence model to
establish an inter-modality mapping, allowing us to learn a suitable decoding
space from the lips’ self-supervised (SSL) representations. We further
demonstrate the effectiveness of mapping SSL features rather than SSL units
for synthesizing intelligible speech. Both quantitative and qualitative
results showcase state-of-the-art performance in constrained settings (such as
GRID and TCD-TIMIT) and unconstrained settings (like Lip2Wav). In our future
work, we aim to introduce emotive effects in the synthesized speech,
considering that HuBERT embeddings are known to lack prosody information.
Additionally, we plan to explore diffusion-based speech vocoders and their
application in a multi-lingual setup. We commit to open-source our
implementation on acceptance.

## References

  * [1] Zixiong Su, Shitao Fang and Jun Rekimoto  “LipLearner: Customizable Silent Speech Interactions on Mobile Devices”  In _Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems_ , 2023, pp. 1–21
  * [2] Gaoxiang Cong et al.  “Learning to Dub Movies via Hierarchical Prosody Models”  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 14687–14697
  * [3] Randa El-Bialy et al.  “Developing phoneme-based lip-reading sentences system for silent speech recognition”  In _CAAI Transactions on Intelligence Technology_ 8.1 Wiley Online Library, 2023, pp. 129–138
  * [4] Hang Zhou et al.  “Vision-infused deep audio inpainting”  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2019, pp. 283–292
  * [5] KR Prajwal, Rudrabha Mukhopadhyay, Vinay P Namboodiri and CV Jawahar  “Learning individual speaking styles for accurate lip to speech synthesis”  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 13796–13805
  * [6] William H Sumby and Irwin Pollack  “Visual contribution to speech intelligibility in noise”  In _The journal of the acoustical society of america_ 26.2 Acoustical Society of America, 1954, pp. 212–215
  * [7] Lynne E Bernstein, Nicole Jordan, Edward T Auer and Silvio P Eberhardt  “Lipreading: A review of its continuing importance for speech recognition with an acquired hearing loss and possibilities for effective training”  In _American Journal of Audiology_ 31.2 ASHA, 2022, pp. 453–469
  * [8] Jeongsoo Choi, Minsu Kim and Yong Man Ro  “Intelligible Lip-to-Speech Synthesis with Speech Units”  In _arXiv preprint arXiv:2305.19603_ , 2023
  * [9] Martin Cooke, Jon Barker, Stuart Cunningham and Xu Shao  “An audio-visual corpus for speech perception and automatic speech recognition”  In _The Journal of the Acoustical Society of America_ 120.5 Acoustical Society of America, 2006, pp. 2421–2424
  * [10] Naomi Harte and Eoin Gillen  “TCD-TIMIT: An audio-visual corpus of continuous speech”  In _IEEE Transactions on Multimedia_ 17.5 IEEE, 2015, pp. 603–615
  * [11] Yongqi Wang and Zhou Zhao  “FastLTS: Non-Autoregressive End-to-End Unconstrained Lip-to-Speech Synthesis”  In _Proceedings of the 30th ACM International Conference on Multimedia_ , 2022, pp. 5678–5687
  * [12] Disong Wang et al.  “VCVTS: Multi-speaker Video-to-Speech synthesis via cross-modal knowledge transfer from voice conversion”  In _ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2022, pp. 7252–7256  IEEE
  * [13] Ariel Ephrat and Shmuel Peleg  “Vid2speech: speech reconstruction from silent video”  In _2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2017, pp. 5095–5099  IEEE
  * [14] Ariel Ephrat, Tavi Halperin and Shmuel Peleg  “Improved speech reconstruction from silent video”  In _Proceedings of the IEEE International Conference on Computer Vision Workshops_ , 2017, pp. 455–462
  * [15] Hassan Akbari, Himani Arora, Liangliang Cao and Nima Mesgarani  “Lip2audspec: Speech reconstruction from silent lip movements video”  In _2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2018, pp. 2516–2520  IEEE
  * [16] Ravindra Yadav, Ashish Sardana, Vinay P Namboodiri and Rajesh M Hegde  “Speech prediction in silent videos using variational autoencoders”  In _ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2021, pp. 7048–7052  IEEE
  * [17] Konstantinos Vougioukas, Pingchuan Ma, Stavros Petridis and Maja Pantic  “Video-driven speech reconstruction using generative adversarial networks”  In _arXiv preprint arXiv:1906.06301_ , 2019
  * [18] Rodrigo Mira et al.  “End-to-end video-to-speech synthesis using generative adversarial networks”  In _IEEE Transactions on Cybernetics_ IEEE, 2022
  * [19] Minsu Kim, Joanna Hong and Yong Man Ro  “Lip to speech synthesis with visual context attentional GAN”  In _Advances in Neural Information Processing Systems_ 34, 2021, pp. 2758–2770
  * [20] Rodrigo Mira et al.  “SVTS: scalable video-to-speech synthesis”  In _arXiv preprint arXiv:2205.02058_ , 2022
  * [21] Leyuan Qu, Cornelius Weber and Stefan Wermter  “LipSound2: Self-Supervised Pre-Training for Lip-to-Speech Reconstruction and Lip Reading”  In _IEEE transactions on neural networks and learning systems_ IEEE, 2022
  * [22] Minsu Kim, Joanna Hong and Yong Man Ro  “Lip-to-speech synthesis in the wild with multi-task learning”  In _ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2023, pp. 1–5  IEEE
  * [23] Zheng-Yan Sheng, Yang Ai and Zhen-Hua Ling  “Zero-shot personalized lip-to-speech synthesis with face image based voice control”  In _ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2023, pp. 1–5  IEEE
  * [24] Jonathan Shen et al.  “Natural tts synthesis by conditioning wavenet on mel spectrogram predictions”  In _2018 IEEE international conference on acoustics, speech and signal processing (ICASSP)_ , 2018, pp. 4779–4783  IEEE
  * [25] Jinzheng He et al.  “Flow-Based Unconstrained Lip to Speech Generation”  In _Proceedings of the AAAI Conference on Artificial Intelligence_ 36.1, 2022, pp. 843–851
  * [26] Durk P Kingma and Prafulla Dhariwal  “Glow: Generative flow with invertible 1x1 convolutions”  In _Advances in neural information processing systems_ 31, 2018
  * [27] Munender Varshney, Ravindra Yadav, Vinay P Namboodiri and Rajesh M Hegde  “Learning Speaker-specific Lip-to-Speech Generation”  In _2022 26th International Conference on Pattern Recognition (ICPR)_ , 2022, pp. 491–498  IEEE
  * [28] Joanna Hong, Minsu Kim, Se Jin Park and Yong Man Ro  “Speech reconstruction with reminiscent sound via visual voice memory”  In _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ 29 IEEE, 2021, pp. 3654–3667
  * [29] Adam Polyak et al.  “Speech resynthesis from discrete disentangled self-supervised representations”  In _arXiv preprint arXiv:2104.00355_ , 2021
  * [30] Chenpeng Du, Yiwei Guo, Xie Chen and Kai Yu  “VQTTS: high-fidelity text-to-speech synthesis with self-supervised VQ acoustic feature”  In _arXiv preprint arXiv:2204.00768_ , 2022
  * [31] Wei-Ning Hsu et al.  “ReVISE: Self-Supervised Speech Resynthesis With Visual Input for Universal and Generalized Speech Regeneration”  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 18795–18805
  * [32] Keith Ito and Linda Johnson  “The LJ Speech Dataset”, https://keithito.com/LJ-Speech-Dataset/, 2017
  * [33] Wei-Ning Hsu et al.  “Hubert: Self-supervised speech representation learning by masked prediction of hidden units”  In _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ 29 IEEE, 2021, pp. 3451–3460
  * [34] Bowen Shi, Wei-Ning Hsu, Kushal Lakhotia and Abdelrahman Mohamed  “Learning audio-visual speech representation by masked multimodal cluster prediction”  In _arXiv preprint arXiv:2201.02184_ , 2022
  * [35] Kaiming He, Xiangyu Zhang, Shaoqing Ren and Jian Sun  “Deep residual learning for image recognition”  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778
  * [36] Stavros Petridis et al.  “Audio-visual speech recognition with a hybrid ctc/attention architecture”  In _2018 IEEE Spoken Language Technology Workshop (SLT)_ , 2018, pp. 513–520  IEEE
  * [37] Ashish Vaswani et al.  “Attention is all you need”  In _Advances in neural information processing systems_ 30, 2017
  * [38] Yi Ren et al.  “Fastspeech 2: Fast and high-quality end-to-end text to speech”  In _arXiv preprint arXiv:2006.04558_ , 2020
  * [39] Jungil Kong, Jaehyeon Kim and Jaekyoung Bae  “Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis”  In _Advances in Neural Information Processing Systems_ 33, 2020, pp. 17022–17033
  * [40] Daniel Michelsanti et al.  “Vocoder-based speech synthesis from silent videos”  In _arXiv preprint arXiv:2004.02541_ , 2020
  * [41] Adrian Bulat and Georgios Tzimiropoulos  “How far are we from solving the 2D & 3D Face Alignment problem? (and a dataset of 230,000 3D facial landmarks)”  In _International Conference on Computer Vision_ , 2017
  * [42] Alec Radford et al.  “Robust speech recognition via large-scale weak supervision”  In _OpenAI Blog_ , 2022
  * [43] Ann Lee et al.  “Direct speech-to-speech translation with discrete units”  In _arXiv preprint arXiv:2107.05604_ , 2021
  * [44] Cees H Taal, Richard C Hendriks, Richard Heusdens and Jesper Jensen  “An algorithm for intelligibility prediction of time–frequency weighted noisy speech”  In _IEEE Transactions on Audio, Speech, and Language Processing_ 19.7 IEEE, 2011, pp. 2125–2136
  * [45] Jesper Jensen and Cees H Taal  “An algorithm for predicting the intelligibility of speech masked by modulated noise maskers”  In _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ 24.11 IEEE, 2016, pp. 2009–2022
  * [46] Minsu Kim, Joanna Hong, Se Jin Park and Yong Man Ro  “Multi-modality associative bridging through memory: Speech sound recollected from face video”  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 296–306
  * [47] Douglas Burnham, Ruth Campbell, G Away and BJ Dodd  “Hearing eye II: the psychology of speechreading and auditory-visual speech”  Routledge, 2013


65 | 5202800126 | 59765.892 | 349.0 | $582.1\pm 1.3$ | $2.7\pm 0.03$ | 10 | $4.5\pm 2.2$ | $0.03\pm 0.05$ | $1.24\pm 0.53$ | 0.71
66 | 5202800126 | 59765.933 | 43.0 | $618.1\pm 3.8$ | $2.73\pm 0.03$ | 1 |  $\cdots$ | $-1.08\pm 0.11$ | $0.41\pm 0.05$ | 0.59
67 | 5202800126 | 59765.934 | 140.0 | $588.6\pm 2.1$ | $2.73\pm 0.03$ | 3 | $5.0\pm 2.4$ | $-0.31\pm 0.08$ | $0.38\pm 0.13$ | 0.69
69 | 5202800127 | 59766.666 | 294.0 | $616.5\pm 1.4$ | $0.93\pm 0.01$ | 9 | $5.9\pm 2.2$ |  $\cdots$ | $0.0\pm 1.21$ | 0.96
70 | 5202800127 | 59766.731 | 314.0 | $615.2\pm 1.4$ | $0.93\pm 0.01$ | 9 | $5.2\pm 2.2$ | $0.04\pm 0.05$ | $1.32\pm 0.43$ | 0.83
71 | 5202800128 | 59767.291 | 613.0 | $634.9\pm 1.0$ |  $\cdots$ | 19 |  $\cdots$ | $0.04\pm 0.04$ | $1.26\pm 0.28$ | 0.89
73 | 5202800128 | 59767.612 | 107.0 | $602.4\pm 2.4$ | $2.11\pm 0.01$ | 3 |  $\cdots$ | $-0.3\pm 0.08$ | $0.46\pm 0.08$ | 0.67
74 | 5202800128 | 59767.614 | 864.0 | $585.9\pm 0.8$ |  $\cdots$ | 26 | $3.3\pm 2.1$ | $0.0\pm 0.01$ | $1.85\pm 0.75$ | 0.81
75 | 5202800129 | 59768.149 | 328.0 | $661.1\pm 1.4$ | $2.1\pm 0.02$ | 9 | $1.9\pm 2.1$ | $0.1\pm 0.06$ | $1.18\pm 0.22$ | 0.79
76 | 5202800129 | 59768.195 | 634.0 | $683.5\pm 1.0$ | $2.1\pm 0.02$ | 19 | $3.1\pm 2.0$ | $0.05\pm 0.04$ | $1.23\pm 0.23$ | 0.88
77 | 5202800130 | 59769.035 | 334.0 | $599.7\pm 1.3$ | $0.871\pm 0.004$ | 10 | $3.9\pm 2.2$ | $-0.06\pm 0.07$ | $0.18\pm 0.6$ | 1.08
78 | 5202800130 | 59769.099 | 548.0 | $648.0\pm 1.1$ | $0.862\pm 0.003$ | 15 | $5.7\pm 2.1$ | $0.08\pm 0.05$ | $0.98\pm 0.22$ | 0.87
79 | 5202800130 | 59769.168 | 199.0 | $659.6\pm 1.8$ | $0.862\pm 0.005$ | 6 | $1.3\pm 2.1$ | $-0.06\pm 0.31$ | $0.11\pm 1.11$ | 0.83
80 | 5202800131 | 59771.233 | 372.0 | $603.7\pm 1.3$ | $2.22\pm 0.01$ | 11 | $0.8\pm 2.1$ |  $\cdots$ | $0.0\pm 0.95$ | 0.79
81 | 5202800131 | 59771.297 | 375.0 | $548.7\pm 1.2$ | $2.22\pm 0.01$ | 11 |  $\cdots$ |  $\cdots$ | $0.01\pm 0.6$ | 0.93
82 | 5202800131 | 59771.362 | 373.0 | $608.2\pm 1.3$ | $0.776\pm 0.003$ | 11 | $3.9\pm 2.1$ | $-0.03\pm 0.04$ | $0.49\pm 0.77$ | 0.95
83 | 5202800132 | 59783.974 | 311.0 | $491.9\pm 1.3$ | $1.04\pm 0.02$ | 9 | $7.6\pm 2.4$ | $0.03\pm 0.05$ | $1.15\pm 0.68$ | 0.75
84 | 5202800133 | 59784.039 | 298.0 | $503.7\pm 1.3$ | $1.04\pm 0.02$ | 9 |  $\cdots$ | $-0.12\pm 0.06$ | $0.2\pm 0.28$ | 0.92
85 | 5202800133 | 59784.426 | 359.0 | $522.3\pm 1.2$ | $0.94\pm 0.01$ | 11 | $4.9\pm 2.3$ | $-0.03\pm 0.08$ | $0.16\pm 1.07$ | 0.99
86 | 5202800134 | 59785.071 | 394.0 | $510.1\pm 1.1$ | $1.03\pm 0.01$ | 12 | $3.7\pm 2.3$ | $-0.01\pm 0.04$ | $0.26\pm 1.86$ | 0.84
87 | 5202800134 | 59785.393 | 318.0 | $499.4\pm 1.3$ | $2.66\pm 0.01$ | 9 | $4.2\pm 2.4$ | $-0.03\pm 0.05$ | $0.35\pm 0.77$ | 0.63
89 | 5202800135 | 59793.183 | 550.0 | $410.0\pm 0.9$ | $1.0\pm 0.02$ | 17 | $2.8\pm 2.5$ |  $\cdots$ | $0.0\pm 0.44$ | 1.21
90 | 5202800136 | 59794.108 | 83.0 | $413.9\pm 2.2$ | $1.02\pm 0.02$ | 2 | $3.5\pm 3.0$ | $-0.63\pm 0.09$ | $0.38\pm 0.08$ | 0.88
91 | 5202800137 | 59795.721 | 43.0 | $409.9\pm 3.1$ | $1.08\pm 0.02$ | 1 | $0.3\pm 3.5$ | $-1.3\pm 0.11$ | $0.35\pm 0.05$ | 0.8
92 | 5202800138 | 59796.79 | 1159.0 | $392.9\pm 0.6$ | $2.41\pm 0.01$ | 36 |  $\cdots$ | $0.01\pm 0.02$ | $1.24\pm 0.78$ | 0.85
93 | 5202800139 | 59797.563 | 443.0 | $368.4\pm 0.9$ | $2.43\pm 0.01$ | 13 |  $\cdots$ | $-0.01\pm 0.02$ | $1.39\pm 0.99$ | 0.64
5202800140 | 5202800140 | 59804.593 | 2261.4 | $213.6\pm 0.3$ | $0.66\pm 0.02$ | 68 | $3.1\pm 3.4$ | $0.01\pm 0.02$ | $1.24\pm 0.51$ | 0.82
5202800141 | 5202800141 | 59805.496 | 1816.0 | $204.3\pm 0.3$ | $0.6\pm 0.02$ | 56 |  $\cdots$ | $-0.04\pm 0.08$ | $0.11\pm 0.46$ | 1.03
5202800142 | 5202800142 | 59806.98 | 664.0 | $182.8\pm 0.5$ | $0.62\pm 0.03$ | 20 | $2.4\pm 3.8$ | $-0.11\pm 0.2$ | $0.1\pm 0.31$ | 0.9
5202800143 | 5202800143 | 59807.496 | 1745.0 | $162.5\pm 0.3$ | $0.58\pm 0.02$ | 54 |  $\cdots$ | $-0.03\pm 0.02$ | $0.57\pm 0.25$ | 0.99
5202800144 | 5202800144 | 59808.657 | 1728.0 | $132.8\pm 0.3$ | $0.68\pm 0.02$ | 52 | $0.8\pm 4.4$ |  $\cdots$ | $0.0\pm 1.18$ | 1.02
5202800145 | 5202800145 | 59809.753 | 1032.1 | $89.6\pm 0.3$ | $1.62\pm 0.04$ | 31 | $6.9\pm 5.4$ | $0.06\pm 0.03$ | $0.7\pm 0.19$ | 1.0
5202800146 | 5202800146 | 59810.721 | 1092.0 | $52.4\pm 0.2$ | $1.45\pm 0.04$ | 33 | $8.4\pm 7.0$ | $-0.06\pm 0.32$ | $0.07\pm 0.57$ | 0.82
5202800147 | 5202800147 | 59811.689 | 525.0 | $32.8\pm 0.3$ | $1.17\pm 0.05$ | 16 |  $\cdots$ | $-0.09\pm 0.04$ | $0.58\pm 0.18$ | 1.04
5202800148 | 5202800148 | 59813.495 | 589.0 | $18.8\pm 0.2$ | $0.78\pm 0.04$ | 18 | $8.8\pm 11.8$ | $-0.08\pm 0.03$ | $0.63\pm 0.16$ | 1.14
5202800160 | 5202800160 | 59814.334 | 1726.0 | $9.3\pm 0.1$ | $0.77\pm 0.02$ | 75 | $20.9\pm 14.1$ | $0.01\pm 0.01$ | $1.53\pm 0.27$ | 0.9


# On radicals of Novikov algebras

A.S.Panasenko  Alexander Sergeevich Panasenko
iii Sobolev Institute of Mathematics,
iii pr. Koptyuga, 4,
iii 630090, Novosibirsk, Russia.
iii Novosibirsk State University,
iii Universitetskiy pr., 1,
iii 630090, Novosibirsk, Russia<EMAIL_ADDRESS>
Panasenko, A.S., On radicals of Novikov algebras

© 2023 Panasenko A.S

> Abstract.  We show that in a prime nonassociative Novikov algebra every
> nonzero ideal is non-associative. We prove that Baer (and Andrunakievich)
> radical and the largest left quasiregular ideal coincide in finite
> dimensional Novikov algebras over a field of characteristic 0 or
> algebraically closed field of odd characteristic. We show non-existence of
> right quasiregular radical in finite dimensional Novikov algebras.
>
> Keywords: Novikov algebra, radical, prime algebra, semiprime algebra, finite
> dimensional algebra, quasiregular ideal.

##  1\. Introduction

The structure theory of any variety of algebras is strongly related to the
theory of radicals that exist in this variety. For example, in alternative
algebras there are many useful radicals well known from the associative case:
the Jacobson, Baer, Köthe, Levitsky, Andrunakievich radicals. In Jordan
algebras the situation is slightly different: the quasiregular radical does
not have such a good characterization as in the associative case, and the
existence of the Baer radical has been an open question for more than half a
century. However, in Jordan algebras there are a lot of other radicals
(Andrunakievich, Köthe, Levitsky), as well as the McCrimmon radical, which is
specific for the Jordan case.

In recent years, an active study of the structure and combinatorial theories
of Novikov algebras has resumed. In the paper [1] L. A. Bokut, Y. Chen and Z.
Zhang proved an analogue of the Poincaré-Birkhoff-Witt theorem for Novikov
algebras and constructed the Gröbner-Shirshov basis for the free Novikov
algebra.

In the article [2] I. P. Shestakov and Z. Zhang proved that the following
three properties are equivalent for the Novikov algebra $A$: $A$ is solvable,
$A$ is right-nilpotent, and $A^{2}$ is nilpotent. In addition, in this paper
it is shown that the minimal ideal of the Novikov algebra is either simple or
has zero multiplication.

In [3], U. U. Umirbaev and V. N. Zhelyabin proved an analog of the Bergman-
Isaacs theorem for Novikov algebras: if $G$ is a finite abelian group and the
zero component in a $G$-graded Novikov algebra $A$ is solvable, then the
algebra $A$ is also solvable (if the characteristic of the field is zero or
does not divide $|G|$).

Every left-symmetric algebra (and hence every Novikov algebra) is Lie-
admissible. The paper [4] is devoted to the study of Lie-solvable Novikov
algebras. For example, in this work K. M. Tulenbaev, U. U. Umirbaev and V. N.
Zhelyabin showed that in a Lie-solvable Novikov algebra the commutator ideal
is right-nilpotent (if the characteristic of the field is not equal to 2). The
paper [5] is devoted to the study of simple Lie-solvable left-symmetric
algebras.

In the paper [6] the author shows that every ideal in a prime Novikov algebra
is a prime Novikov algebra. A similar statement was proved for semiprime
algebras, which made it possible to assert the existence of a Baer radical (in
the sense of Kurosh) in the variety of Novikov algebras.

In this paper, radicals of Novikov algebras are considered. First, we refine
the result from [6] by proving that in a prime non-associative Novikov
algebra, non-zero ideals are non-associative. This result will be used to
characterize the Andrunakievich radical.

In many varieties of algebras, all popular and useful radicals coincide with
the largest nilpotent ideal in the finite-dimensional case. It is well known
that in finite-dimensional Novikov algebras, nilpotency is not a property that
is stable under extensions. But solvability is. We prove that the
Andrunakievich and Baer radicals coincide with each other and with the largest
solvable ideal in the finite-dimensional Novikov algebra. Moreover, in the
case of a field of characteristic 0 or an algebraically closed field of odd
characteristic, the left quasiregular radical also coincides with the Baer
radical.

Throughout this paper, all algebras are considered over a field $F$ of
characteristic not 2. Definitions 2–7 are recalled from [7].

If $A$ is an algebra over a field, then for every elements $x,y,z\in A$ we use
the notation $(x,y,z)=(xy)z-x(yz)$ for the associator of elements $x,y,z$.

Definition 1 (see [8]). An algebra $A$ over a field $F$ is called a Novikov
algebra if the following identities hold for all $x,y,z\in A$:

$(x,y,z)=(y,x,z),$ $(xy)z=(xz)y.$

##  2\. Semiprime Novikov algebras

Let $A$ be a Novikov algebra. It is well known that the product of ideals in
$A$ is an ideal. If $I$ is an ideal of the algebra $A$, then by $T(I)$ we
denote the subspace in $I$ generated by all elements of the form $(ij)j$,
where $i,j\in I$. Denote by $P(I)$ the subspace of $I$ generated by all
elements of the form $(aj)j$, where $j\in I,a\in A$.

Lemma 1. Let $I$ be an ideal of the Novikov algebra $A$. Then the subspaces
$P(I)$ and $T(I)$ are ideals in the algebra $A$, $P(I)=(AI)I$, $T(I)=I^{2}I$,
and

$P(I)^{2}\subset T(I)\subset P(I).$

Proof. Let $i,j\in I$ and $a\in A$. Then $(a(i+j))(i+j),(ai)i,(aj)j\in P(I)$.
Hence

$2(ai)j=(ai)j+(aj)i=(a(i+j))(i+j)-(ai)i-(aj)j\in P(I).$

Thus, $(ai)j\in P(I)$ and $P(I)=(AI)I$. Similarly, $T(I)=I^{2}I$. In
particular, $T(I)$ and $P(I)$ are ideals of $A$.

It remains to note that if $i_{1}\in I,a_{1}\in A$, then

$((aj)j)((a_{1}j_{1})j_{1})=((a((aj_{1})j_{1}))j)j\in T(I).$

So, $P(I)^{2}\subset T(I)$. ∎

Definition 2. Recall that an algebra over a field is called semiprime if every
nonzero ideal $I$ has nontrivial multiplication, i.e. $I^{2}=0$ implies that
$I=0$. An algebra is prime if $IJ\neq 0$ for all nonzero ideals $I,J$.

Lemma 2. Let $I$ be a nonzero ideal of the semiprime Novikov algebra $A$. Then
$T(I)\neq 0$.

Proof. Assume that $T(I)=0$. Then, by Lemma 1, $P(I)^{2}=0$, whence, by
semiprimality, $P(I)=0$, i.e.

$(ai)j=0$

for every $a\in A$, $i,j\in I$ by Lemma 1. In particular, $(I^{2})^{2}=0$, so
$I^{2}=0$ and $I=0$. ∎

Definition 3. If $A$ is an algebra over a field, then its nucleus is the
following subset:

$N(A)=\\{n\in A\mid(n,A,A)=(A,n,A)=(A,A,n)=0\\}.$

In every algebra, the nucleus is a subalgebra, see [7]. In a Novikov algebra,
the nucleus is an ideal, see [6]. Moreover, in [6] it is proved that the
nucleus in a prime nonassociative Novikov algebra is equal to zero.

One can check that any Novikov algebra satisfies the following identities
(see, for example, [4]):

$(ad,b,c)=(a,bd,c)=(a,b,c)d.$

We will use the following notation for a left annihilator of a subset $M$ in
an algebra $A$:

$Ann_{l}(M)=\\{a\in A\,|\,aM=0\\}.$

Lemma 3. Let $I$ be an ideal in a Novikov algebra $A$, $n\in A$ and
$(I,I,n)=0$. Then $(A,A,n)\subset\mathrm{Ann}_{l}T(I)$. If $n\in N(I)$, then
$(A,n,A)=(n,A,A)\subset\mathrm{Ann}_{l}T(I)$.

Proof. Indeed, let $a,b\in A$, $i,j\in I$. We have

$(a,b,n)((ij)j)=(a((ij)j),b,n)=-((a,ij,j),b,n)+((a(ij))j,b,n)$
$-((a,i,j),bj,n)=0.$

It means that $(A,A,n)\subset\mathrm{Ann}_{l}T(I)$.

Let $n\in N(I)$. The next goal is to prove
$(n,A,A)\subset\mathrm{Ann}_{l}T(I)$. First, we recall ([6]) that
$n(I,I,I)=0$. Then we have

$(n,a,b)((ij)j)=((n(ij))j,a,b)=(nij,aj,b)=((n,aj,b)i)j$ $=(((n\cdot
aj)b)i)j-((n(aj\cdot b))i)j$ $=(n(aj),b,i)j+((n\cdot aj)(bi))j-((n(aj\cdot
b))i)j$ $=(n(aj)(bi))j-(n(aj\cdot b)i)j$ $=(n(aj\cdot bi))j-(n(aj\cdot
b)i)j=(n(aj\cdot bi)-n(aj\cdot b)i)j$ $=(n(a\cdot bi)j)j-(n(ab\cdot
j)i)j=(n(a\cdot bi-ab\cdot i)j)j=-(n(a,b,i)j)j$ $=-n(aj,bj,i)=0.$

Thus, $(n,a,b)((ij)j)=0$ and $(n,A,A)\subset\mathrm{Ann}_{l}T(I)$. ∎

Now we can strengthen one of the results of [6].

Theorem 1. Let $A$ be a prime nonassociative Novikov algebra with a nonzero
ideal $I$. Then $I$ is a prime non-associative Novikov algebra.

Proof. By Theorem 3 in [6], we obtain that $I$ is a prime Novikov algebra. Let
$n\in N(I)$. By Lemma 2, $T(I)\neq 0$. By [6] (Lemma 2), the left annihilator
of a left ideal is an ideal. Then, by the primality of the algebra $A$, we
obtain that $\mathrm{Ann}_{l}T(I)=0$. By Lemma 3
$(A,A,n)\subset\mathrm{Ann}_{l}T(I)=0$ and
$(n,A,A)\subset\mathrm{Ann}_{l}T(I)=0$. Thus $n\in N(A)=0$, so $N(I)=0$. In
particular, the algebra $I$ is non-associative. ∎

Definition 4. The intersection of all nonzero ideals of an algebra is called
its heart. An algebra is subdirectly irreducible if its heart is nonzero. The
heart $H$ of a subdirectly irreducible algebra is idempotent if $H^{2}=H$.
Since a subdirectly irreducible algebra with an idempotent heart is prime, the
following corollary is true. Part of this corollary followed earlier from the
results of [2].

Corollary 1. Let $A$ be a subdirectly irreducible nonassociative Novikov
algebra with an idempotent heart. Then the heart is a simple non-associative
Novikov algebra.

##  3\. Andrunakievich radical

Definition 5. A class of Novikov algebras $\mathcal{R}$ is called radical if
the following conditions are satisfied.

1) The homomorphic image of an algebra in $\mathcal{R}$ lies in $\mathcal{R}$.

2) Every Novikov algebra $A$ contains an ideal $\mathcal{R}(A)$ from
$\mathcal{R}$, which contains all ideals in $A$ from $\mathcal{R}$.

3) The quotient algebra $A/\mathcal{R}(A)$ does not contain nonzero ideals
from $\mathcal{R}$.

For completeness, we present the construction of the Andrunakievich radical
from [7], restricting it to Novikov algebras.

Let $\mathcal{B}$ be the class of all subdirectly irreducible Novikov algebras
with idempotent heart. An ideal $I$ of the Novikov algebra $A$ is called
$\mathcal{B}$-ideal if the quotient algebra $A/I$ belongs to the class
$\mathcal{B}$. The class $\mathcal{A}$ of Novikov algebras that do not map
homomorphically onto algebras in the class $\mathcal{B}$ is radical (proved in
[7]). The largest ideal of the algebra $A$ contained in the class
$\mathcal{A}$ is denoted by $\mathcal{A}(A)$ and is called the Andrunakievich
radical of the algebra $A$.

Definition 6. A radical class $\mathcal{R}$ is called hereditary if for every
ideal $I$ in $A$ we have $\mathcal{R}(I)=I\cap\mathcal{R}(A)$.

Let us prove some propositions. Their analogues for alternative algebras were
proved in [7] and our proofs are basically the same.

Proposition 1. Let $A$ be a Novikov algebra, $I$ is an ideal in $A$ and $J$ is
a $\mathcal{B}$-ideal in $I$. Then there exists a $\mathcal{B}$-ideal $K$ in
$A$ with a condition $K\cap I=J$.

Proof. An algebra $I/J$ is semiprime, so by Lemma 6 in [6] $J$ is an ideal in
$A$. We will use notations $\overline{A}=A/J$ and $\overline{I}=I/J$. The
algebra $\overline{I}$ is a subdirectly irreducible Novikov algebra with
idempotent heart. By Zorn’s lemma we have an ideal $\overline{K}$ in
$\overline{A}$, which is maximal with condition
$\overline{I}\cap\overline{K}=\overline{0}$. It means that in an algebra
$\overline{\overline{A}}=\overline{A}/\overline{K}$ all ideals have a nonzero
intersection with an image of $\overline{I}$. Then the heart
$H(\overline{\overline{A}})$ contains the heart of an image of $\overline{I}$
which is isomorphic to the heart of $\overline{I}$. But the heart of
$\overline{I}$ is nonzero and idempotent. So, $\overline{\overline{A}}$ is in
$\mathcal{B}$.

Let $K$ be a preimage of an ideal $\overline{K}$ in $A$. Then we have
$\overline{\overline{A}}=\overline{A}/\overline{K}\simeq A/K$. So, $K$ is a
$\mathcal{B}$-ideal in $A$ and $K\cap I=J$. ∎

Proposition 2. Let $A$ be a Novikov algebra and $I$ be an ideal in $A$. Then
$\mathcal{A}(I)$ is an ideal in $A$.

Proof. We will use a notation $M=\mathcal{A}(I)$.

a) Suppose that $M$ does not contain $Ma$ for some $a\in A$. Then $M\subsetneq
Ma+M$ and $Ma+M$ is an ideal in $I$ by ([6], Lemma 5.a). Suppose that
$(Ma)^{2}\subseteq M$. Let us define a map $\varphi:M\to(M+Ma)/M$,
$\varphi(m)=ma+M$. If $m,n\in M$ then

$\varphi(m+n)=\varphi(m)+\varphi(n).$

By right commutativity we have

$\varphi(mn)=(mn)a+M=(ma)n+M=0+M.$

We supposed that $(Ma)^{2}\subseteq M$, so

$\varphi(m)\varphi(n)=(ma)(na)+M=0+M.$

It means that $\varphi$ is a homomorphism of $M$ onto an ideal $(M+Ma)/M$ of
an algebra $I/M$. It is a contradiction because $\mathcal{A}(I/M)=0$ and
$M\in\mathcal{A}$. So $M$ does not contains $(Ma)^{2}$.

Let us prove that $M+(Ma)^{2}$ is an ideal in $I$ and $M+(Ma)^{2}\subseteq
M+Ma$. Indeed, $M+Ma$ is an ideal, so $M+(Ma)^{2}\subseteq M+Ma$. We have

$(M+(Ma)^{2}))I\subseteq M+((Ma)(Ma))I\subseteq M+((Ma)I)(Ma)\subseteq\\\
\subseteq M+(Ma)^{2}.$ $I(M+(Ma)^{2})\subseteq M+I((Ma)(Ma))\\\ \subseteq
M+(I,Ma,Ma)+(I(Ma))(Ma)\\\ \subseteq M+(M,I,Ma)a+(Ma+M)(Ma)\\\ \subseteq
M+(Ma)^{2}+((MI)(Ma))a+(M(I(Ma)))a\\\ \subseteq
M+(Ma)^{2}+(Ma)^{2}I+Ma(Ma+M)\subseteq M+(Ma)^{2}.$

So, $M+(Ma)^{2}$ is an ideal in $I$.

Lemma 5.c in [6] implies that $(M+(Ma)^{2})/M$ is a trivial ideal in an
algebra $I/M$. But $\mathcal{A}(I/M)=0$, so we have a contradiction. It means
that $M$ contains $Ma$.

b) Suppose that $M$ does not contain $aM$ for some $a\in A$. Then $M\subsetneq
aM+M$ and $aM+M$ is an ideal in $I$ by ([6], Lemma 5.a). Suppose that
$(aM)^{2}\subseteq M$. Let us define a map $\varphi:M\to(M+aM)/M$,
$\varphi(m)=am+M$. If $m,n\in M$ then

$\varphi(m+n)=\varphi(m)+\varphi(n).$

We have

$\varphi(mn)=a(mn)+M=(a,m,n)+(am)n+M=(m,a,n)+M=0+M.$

We supposed that $(aM)^{2}\subseteq M$, so

$\varphi(m)\varphi(n)=(am)(an)+M=0+M.$

It means that $\varphi$ is a homomorphism of $M$ onto an ideal $(M+aM)/M$ of
an algebra $I/M$. It is a contradiction because $\mathcal{A}(I/M)=0$ and
$M\in\mathcal{A}$. So $M$ does not contain $(aM)^{2}$.

Let us prove that $M+(aM)^{2}$ is an ideal in $I$ and $M+(aM)^{2}\subseteq
M+aM$. Indeed, $M+aM$ is an ideal, so $M+(aM)^{2}\subseteq M+aM$. We have

$(M+(aM)^{2}))I\subseteq M+((aM)(aM))I\subseteq M+((aM)I)(aM)\\\ \subseteq
M+(aM)^{2}.$ $I(M+(aM)^{2})\subseteq M+I((aM)(aM))\\\ \subseteq
M+(I,aM,aM)+(I(aM))(aM)\\\ \subseteq M+(a,I,aM)M+(aM+M)(aM)\\\ \subseteq
M+(aM)^{2}+((aI)(aM))M+(a(I(aM)))M\\\ \subseteq
M+(aM)^{2}+((aM)(aM))I+(aM)(I(aM))\\\ \subseteq M+(aM)^{2}+(aM)M\subseteq
M+(aM)^{2}$

So, $M+(aM)^{2}$ is an ideal in $I$.

Lemma 5.e in [6] implies that $(M+(aM)^{2})/M$ is a trivial ideal in an
algebra $I/M$. But $\mathcal{A}(I/M)=0$, so we have a contradiction. It means
that $M$ contain $aM$. ∎

Proposition 3. In the class of Novikov algebras, the Andrunakievich radical is
hereditary.

Proof. Theorem 8.3 in [7] states that a radical $\mathcal{R}$ is hereditary
iff it satisfies two conditions:
(I) if $\mathcal{R}(A)=0$ and $I$ is an ideal in $A$ then $\mathcal{R}(I)=0$;
(II) if $\mathcal{R}(A)=A$ and $I$ is an ideal in $A$ then $\mathcal{R}(I)=I$.

Let us prove (I). Let $A$ be a Novikov algebra, $\mathcal{A}(A)=0$ and $I$ is
a nonzero ideal in $A$. Then $\mathcal{A}(I)$ is an ideal in $A$ by
Proposition 2. But $\mathcal{A}(\mathcal{A}(I))=\mathcal{A}(I)$, so
$\mathcal{A}(I)\subseteq\mathcal{A}(A)=0$.

Let us prove (II). Let $A\in\mathcal{A}$ and $I$ is an ideal in $A$. If
$I\notin\mathcal{A}$ then $I$ contains a nonzero $\mathcal{B}$-ideal $J\neq
I$. By Proposition 1 we have a $\mathcal{B}$-ideal $K$ in $A$ such that $K\cap
I=J$. It means that $A/K\in\mathcal{B}$ and $K\neq A$. We have a contradiction
($\mathcal{A}(A/K)=A/K)$.

So, the Andrunakievich radical is hereditary in Novikov algebras. ∎

Proposition 4. Let $A$ be a Novikov algebra and $A\in\mathcal{B}$ and $I$ is a
nonzero ideal in $A$. Then $I\in\mathcal{B}$ and $H(I)=H(A)$.

Proof. An algebra $H(A)$ is simple by [2] and $H(A)\subseteq I$. Zorn’s lemma
allows us to choose a maximal ideal $K$ in $I$ with a condition $K\cap
H(A)=0$. An algebra $I/K$ is semiprime, because each ideal in $I/K$ contains
an image $\overline{H(A)}$ of $H(A)$. Then $K$ is an ideal in $A$ by ([6],
Lemma 6), so $K=0$. It means that each nonzero ideal in $I$ has a nonzero
intersection with $H(A)$. But $H(A)$ is simple, so each nonzero ideal in $I$
contains $H(A)$. ∎

Now we are ready to prove the following statement.

Theorem 2. The Andrunakievich radical $\mathcal{A}(A)$ of the Novikov algebra
$A$ is equal to the intersection of all its $\mathcal{B}$-ideals, and the
quotient algebra $A/\mathcal{A}(A)$ is a subdirect product of subdirectly
irreducible algebras with an idempotent heart.

Proof. If $C$ is an algebra in $\mathcal{B}$ and $L$ is an ideal in $C$ then
$L$ is in $\mathcal{B}$ by Proposition 4. It means that $\mathcal{A}(L)\neq L$
and $\mathcal{A}(C)=0$.

Let $J$ be a $\mathcal{B}$-ideal in $A$. By the statement above we have
$\mathcal{A}(A/J)=0$. It means that $\mathcal{A}(A)\subseteq J$.

Let $I$ be an intersection of all $\mathcal{B}$-ideals in an algebra $A$. Then
we have $\mathcal{A}(A)\subseteq I$.

If $I\neq\mathcal{A}(I)$ then $I$ contains a nonzero $\mathcal{B}$-ideal
$J\neq I$. By Proposition 1 there exists a $\mathcal{B}$-ideal $K$ in $A$,
such that $K\cap I=J\neq I$. We have a contradiction, $I$ is contained in all
$\mathcal{B}$-ideals in $A$. So, $I\subseteq\mathcal{A}(A)$. Last statement is
straightforward. ∎

Since the sum of ideals is an ideal, then in finite dimensional algebras one
can define a solvable radical as the largest solvable ideal. In the variety of
Novikov algebras this ideal coincides with the Baer radical $B(A)$ whose
existence was proved in [6].

Now we can prove that the Baer and Andrunakievich radicals coincide in the
finite dimensional case.

Theorem 3. In a finite dimensional Novikov algebra, the Baer radical coincides
with the Andrunakievich radical.

Proof. Let $B$ be a prime finite dimensional Novikov algebra. If $H(B)=0$ then
we have a finite set of nonzero ideals in $B$, which intersection is zero. It
means that there exist nonzero ideals $I$ and $J$ in $B$, such that $I\cap
J=0$. It is a contradiction with primality, so $H(B)\neq 0$ and
$B\in\mathcal{B}$.

Let $A$ be a Novikov algebra. By Theorem 2 $A/\mathcal{A}(A)$ is a subdirect
product of prime Novikov algebras, so $A/\mathcal{A}(A)$ is a semiprime
algebra and $B(A)\subseteq\mathcal{A}(A)$.

Consider the algebra $C=A/B(A)$. If $\mathcal{A}(A)$ does not lie in $B(A)$,
then the algebra $C$ has a nonzero Andrunakievich radical $\mathcal{A}(C)\neq
0$. The algebra $C$ is a subdirect product of prime algebras $C/C_{i}$, where
$C_{i}$ are ideals in $C$, $i\in\\{1,\dots,n\\}$. But $\mathcal{A}(C)$ has a
nonzero image in some $C/C_{k}$, so $\overline{C}=C/C_{k}$ is an algebra in
$\mathcal{B}$ and it has a nonzero Andrunakievich radical
$\mathcal{A}(\overline{C})$. It means that an algebra
$\mathcal{A}(\overline{C})$ is an algebra in $\mathcal{B}$ (by Proposition 4)
and in $\mathcal{A}$. It is a contradiction, so $\mathcal{A}(A)\subseteq
B(A)$. ∎

Note that in associative and commutative algebras the Andrunakievich radical
coincides with the Jacobson radical, so that in infinite-dimensional Novikov
algebras the Andrunakievich and Baer radicals differ.

##  4\. Quasiregular radical

Definition 7. An element $x\in A$ is called left quasiregular if there exists
$y\in A$ such that $x+y=yx$. An element $x\in A$ is called right quasiregular
if there exists $y\in A$ such that $x+y=xy$. An element is called quasiregular
if there exists $y\in A$ such that $x+y=xy=yx$. An algebra is called
quasiregular (right quasiregular, left quasiregular) if all its elements are
quasiregular (right quasiregular, left quasiregular).

Example 1. Consider the two-dimensional Novikov algebra $A=Fa+Fb$ with
multiplication $a^{2}=b^{2}=ba=0$, $ab=b$, [9]. It is easy to see that the
elements $\alpha a$ and $\beta b$ are quasiregular for every $\alpha,\beta\in
F$, that is, the spaces $Fa$ and $Fb$ are quasiregular. However, it is
directly verified that the element $a+b$ is not right quasiregular. Thus, the
largest quasiregular (right quasiregular) ideal in $A$ is $Fb$. In particular,
a solvable radical need not be quasiregular or right quasiregular in finite
dimensional Novikov algebras. Since an algebra with zero multiplication is
obviously quasiregular, the property of quasiregularity (right
quasiregularity) is not radical in the class of finite dimensional Novikov
algebras, namely, the following property does not hold: if $I$ and $A/I$ are
radical, then $A$ is radical.

Lemma 4. The right-nilpotent algebra is left quasiregular. A left-nilpotent
algebra is right quasiregular.

Proof. Indeed, let $A$ be a right-nilpotent algebra, $x\in A$. Define
$x^{m}=x^{m-1}\cdot x$ and let $x^{n}=0$. Then

$-x+(x-x^{2}+\dots+(-1)^{n}x^{n-1})=(x-x^{2}+\dots+(-1)^{n}x^{n-1})\cdot(-x).$

Thus the element $-x$ is left quasiregular. But this is true for every $x\in
A$. In other words, $y$ is left quasiregular for every $y\in A$. Similarly, it
can be shown that every element of a left-nilpotent algebra is right
quasiregular. ∎

I. P. Shestakov and Z. Zhang proved [2] that the right-nilpotency of the
Novikov algebra is equivalent to its solvability (and is equivalent to the
nilpotency of its square). Thus, Lemma 4 implies that the Baer radical of a
finite dimensional Novikov algebra is left quasiregular. In fact, under some
constraint on the field, the converse is also true.

Example 2. Let $F$ be a field of characteristic $p>2$. Consider the algebra
$A_{p^{n}}(a,b)$ with a basis $\\{y_{-1},y_{0},\dots,y_{p^{n}-2}\\}$ and the
following multiplication:

$y_{-1}y_{-1}=ay_{p^{n}-2},$ $y_{-1}y_{0}=y_{-1}+by_{p^{n}-2},$
$y_{i}y_{j}=C_{i+j+1}^{j}y_{i+j},$

where $a,b$ are fixed elements from $F$ and $y_{k}=0$ for $k>p^{n}-2$.

In [11] it is proved that over an algebraically closed field of characteristic
$p>2$ every simple Novikov algebra is isomorphic to some $A_{p^{n}}(a,b)$.

Theorem 4. Let $F$ be a field of characteristic 0 or an algebraically closed
field of characteristic $p>2$. A finite dimensional Novikov algebra over the
field $F$ is left quasiregular if and only if it is solvable.

Proof. In one direction, the statement is true by Lemma 4. Let $A$ be a left
quasiregular finite dimensional algebra. We prove by induction on dimension
that $A$ is solvable. If $A=Fx$ and $x^{2}=\beta x\neq 0$ then
$(\frac{1}{\beta}x)^{2}=\frac{1}{\beta^{2}}\beta x=\frac{1}{\beta}x$. It means
that $e=\frac{x}{\beta}$ is an idempotent, $e^{2}=e$ and $A=Fe$. There exists
$\alpha\in F$, such that $e+\alpha e=e\cdot\alpha e=\alpha e$, so $e=0$, it is
a contradiction. So $x^{2}=0$ and $A$ is trivial.

Suppose $A$ has its proper non-zero ideal $I$. Then, by the inductive
hypothesis, the algebras $I$ and $A/I$ are solvable, so that the algebra $A$
is solvable. Thus, we can assume that the algebra $A$ is simple. In [9] (see
also [10]) it is proved that over a field of characteristic 0 the simple
finite dimensional Novikov algebra is a field. It is easy to see that an
identity 1 is not left quasiregular, so a field is not left quasiregular. If
$F$ is an algebraically closed field of characteristic $p>2$, then $A\simeq
A_{p^{n}}(a,b)$ for some $n\in\mathbb{N}$, $a,b\in A$. Suppose $A$ is left
quasiregular. Then the element $y_{0}$ is left quasiregular, that is,
$y_{0}+x=xy_{0}$ for some $x\in A$. Let
$x=\sum\limits_{j=-1}^{p^{n}-2}\alpha_{j}y_{j}$. Then

$y_{0}+x=\sum\limits_{j=-1}^{p^{n}-2}\alpha_{j}y_{j}y_{0}=x+\alpha_{-1}by_{p^{n}-2},$

whence it follows that $y_{0}=\alpha_{-1}by_{p^{n}-2}$, which is impossible.
Contradiction. ∎

The question remains whether there exists the largest left quasiregular ideal
in infinite dimensional Novikov algebras and whether it is radical in the
class of Novikov algebras.

##  5\. Acknowledgements

The author expresses his gratitude to P. S. Kolesnikov, whose remarks helped
to improve this paper. The author is grateful to the referee, whose remarks
helped to improve this paper and to fix some mistakes.

## References

  * [1] L.A.Bokut, Y.Chen, Z.Zhang, On free Gelfand-Dorfman-Novikov-Poisson algebras and a PBW theorem, Journal of Algebra, 500, 2018, 153–170.
  * [2] I.P.Shestakov, Z.Zhang, Solvability and Nilpotency of Novikov Algebras, Comm. in algebra, 48:5, 2020, 1–9.
  * [3] U.Umirbaev, V.Zhelyabin, On the solvability of graded Novikov algebras, Int. Journal of Algebra and Computation, 31:7, 2021, 1405–1418.
  * [4] K.Tulenbaev, U.Umirbaev, V.Zhelyabin, On the Lie-solvability of Novikov algebras, Journal of Algebra and Its Applications, 2022, 2350117.
  * [5] V.N.Zhelyabin, A.P.Pozhidaev, U.U.Umirbaev, On simple Lie-solvable pre-lie algebras, Algebra and Logic, 61:2, 2022, 230–238.
  * [6] A.S.Panasenko, Semiprime Novikov algebras, Int. Journal of Algebra and Computation, 32:7, 2022, 1369–1378.
  * [7] K.A.Zhevlakov, A.M.Slin’ko, I.P.Shestakov, A.I.Shirshov, Rings that are nearly associative, Academic press, New York, 1982.
  * [8] J.M.Osborn, Novikov algebras, Nova Journal of Algebra and Geometry, 1, 1992, 1–14.
  * [9] E.I.Zel’manov, A class of local translation-invariant Lie algebras, Sov.Math.Dokl., 35, 1987, 216–218.
  * [10] V.Dotsenko, N.Ismailov, U.Umirbaev, Polynomial identities in Novikov algebras, 2022, arXiv:2209.13662.
  * [11] X.Xu, On Simple Novikov Algebras and Their Irreducible Modules, Journal of algebra, 185, 1996, 905–934.