# Leveraging generative artificial intelligence to simulate student learning
behavior

Songlin Xu  University of California San Diego<EMAIL_ADDRESS>Xinyu Zhang
University of California San Diego

###### Abstract

Student simulation presents a transformative approach to enhance learning
outcomes, advance educational research, and ultimately shape the future of
effective pedagogy. We explore the feasibility of using large language models
(LLMs), a remarkable achievement in AI, to simulate student learning
behaviors. Unlike conventional machine learning based prediction, we leverage
LLMs to instantiate virtual students with specific demographics and uncover
intricate correlations among learning experiences, course materials,
understanding levels, and engagement. Our objective is not merely to predict
learning outcomes but to replicate learning behaviors and patterns of real
students. We validate this hypothesis through three experiments. The first
experiment, based on a dataset of N = 145, simulates student learning outcomes
from demographic data, revealing parallels with actual students concerning
various demographic factors. The second experiment (N = 4524) results in
increasingly realistic simulated behaviors with more assessment history for
virtual students modelling. The third experiment (N = 27), incorporating prior
knowledge and course interactions, indicates a strong link between virtual
students’ learning behaviors and fine-grained mappings from test questions,
course materials, engagement and understanding levels. Collectively, these
findings deepen our understanding of LLMs and demonstrate its viability for
student simulation, empowering more adaptable curricula design to enhance
inclusivity and educational effectiveness.

###### keywords:

Artificial Intelligence, Digital Twin, Student Simulation, Large Language
Models

In an era defined by rapid technological advancement and an escalating
intricacy of global challenges, education stands as the key to unlocking human
potential, fostering innovation, and addressing the myriad issues facing our
world [1, 2]. However, in the evolution of traditional education, it is
increasingly clear that education cannot be a one-size-fits-all endeavor[3],
and the key to success lies in tailoring the learning experience to the unique
needs and characteristics of each student[4]. Student simulation [5, 6], like
a digital twin [7] in classroom, presents an innovative approach in this
regard, allowing us to bridge the gap between education research theory and
practice, providing educators with a powerful tool to develop adaptive,
personalized curricula [8] to enhance inclusivity and enrich the pedagogical
landscape.

However, prevailing approaches treat student simulation as a black-box machine
learning problem [9, 10, 11, 12], overlooking the intricacies of learning
process and failing to account for the impact of student learning experiences.
We posit that student simulation should not be reduced to a mere test score
prediction task, as it should encompass vital components of learning behaviors
[13], including students’ demographic information, understanding of course
concepts and classroom engagement. Furthermore, given the multifaceted nature
of human behaviors, achieving precise predictions of student behaviors becomes
exceedingly challenging [14].

To address these complexities, we propose the development of a digital twin to
replicate the nuanced learning behaviors and patterns exhibited by students,
as opposed to striving for exact predictions of their behaviors. For instance,
the digital twin need not precisely predict the same test scores as real
students but should exhibit similar responses when receiving feedback from
course instructors, mirroring the behavior of actual students. We examine the
potential of utilizing large language models (LLMs) [15], an achievement in
generative artificial intelligence, for simulating student learning behaviors.
LLMs, advanced AI systems pretrained on extensive datasets, possess the
capacity for large-scale natural language understanding and generation, thus
enabling the creation of a digital twin to emulate virtual students with
specified demographic data, observations, and actions.

We validate our hypothesis through three experiments. The first experiment is
based on an N = 145 dataset to simulate student learning outcomes by
instantiating virtual students with more than 20 kinds of demographic
information (such as high school level, living conditions, parents education,
etc). The results demonstrate consistent impact of various demographic factors
on virtual students’ learning outcomes compared with real students. In the
second experiment (N = 4524), we incorporate assessment history into the
simulation, and find increasingly realistic simulated student learning
outcomes as more assessment history is used to model virtual students. In the
third experiment (N = 27), we simulate both learning experiences and outcomes
at a fine-grained level by considering students’ engagement, prior knowledge,
and understanding levels when interacting with course materials. The results
indicate that virtual students’ learning experiences are linked to their past
experiences, and learning outcomes are more realistically simulated when
accounting for fine-grained mappings from test questions to corresponding
course materials and student understanding levels. In contrast to a coarse-
grained approach that aggregates all course materials and understanding
levels, this finer-grained analysis offers a more realistic simulation.

Cumulatively, these findings enrich our comprehension of large language models
and establish the feasibility and effectiveness of employing AI to simulate
student learning behaviors, thus enabling educators to design more engaging,
adaptable curricula that promote inclusivity and educational efficacy.
Subsequent sections present our primary results and in-depth analysis from the
three experiments.

Figure 1: (A). Example of input prompts (denoted as Q) and LLMs’ responses
(denoted as A) in experiment 1. Contents have been truncated. Full examples
can be found in Appendix. (B). Example distributions of demographic
information in the dataset of experiment 1. (C). Blue and orange dots
represent simulation and real human results, respectively. The Pearson
coefficient $r$ represents the average correlation in all running times.

## Results

We employed the OpenAI API [16], GPT-3.5-Turbo model (referred to as ChatGPT),
for conducting our student simulations. The temperature parameter was
consistently set at 0 to yield deterministic results, while all other
parameters remained at their default settings, unless explicitly specified.

### Experiment 1

The first experiment aims to examine whether LLMs could capture correlation
between student background and learning outcome. We utilized the dataset from
[17] that includes N = 145 students’ demographics (Fig. 1(B)) and final grade.
A complete list (about 30 kinds of factors) and distributions are depicted in
Appendix. We ask LLMs to mimic students with specific demographic backgrounds
and predict the virtual students’ final grade (one from 0: Fail, 1: DD, 2: DC,
3: CC, 4: CB, 5: BB, 6: BA, 7: AA). One example prompt and responses are
depicted in Fig. 1(A).

Intuitively, it is difficult to directly predict students’ final grade without
training data. However, as mentioned before, our goal is not to precisely
predict students’ learning outcomes but to test whether virtual students could
exhibit similar patterns as real students. Therefore, instead of directly
comparing the simulated final grade accuracy, we validate the correlation
between virtual and real students across different demographic factors (Fig.
1). Interestingly, we find highly aligned (Pearson correlation: $r$ $>$ 0.7)
behaviors in most factors such as age, transportation, living condition,
parents jobs, learning habits, etc, as is depicted in Fig. 1(C).

To check the robustness of the LLMs simulation, we set temperature parameter
to be default value (1 instead of 0) and run the experiment for four times.
The simulation results are consistent, as depicted in Fig. 1(C). More detailed
Pearson coefficients in each demographic factor are depicted in Appendix,
where we find significant consistency ($r$ $>$ 0.7) among different study
running times in most factors, suggesting the robustness of LLMs’ simulation.

These results demonstrate the effectiveness of harnessing LLMs to simulate
students’ learning outcomes. Specifically, LLMs could capture the correlation
between various demographics and students’ learning outcomes. This capacity
stems from LLMs’ robust in-context learning abilities and extensive knowledge
base [18]. Nonetheless, we observe inconsistent correlations between virtual
and real students in some demographic factors (see Appendix). Such
discrepancies are expected and justifiable given the complexity of human
behaviors, making it nearly impossible to replicate precise student behaviors
[14]. Additionally, limited training data contributes to this variability.
However, this diversity and divergence in simulation outcomes contribute to
the development of a more effective digital twin, as determinism alone may not
adequately represent the range of student learning behaviors.

Figure 2: (A). Example of input prompts (denoted as Q) and LLMs’ responses
(denoted as A) in experiment 2. Contents have been truncated. Full examples
can be found in Appendix. (B). Illustration of experiment 2 design: virtual
students are instantiated with demographics and past assessments history. (C).
Pearson correlation matrix in experiment 2 results. i to viii represent
simulated final exam scores using the simulation model from Type i to Type
viii, respectively. H represents real students’ final exam scores and A
represents average score of all assessments of real students. (D),(E),(F)
represent the distribution of final exam scores in different regions in course
id $CCC$ (D), course id $DDD$ (E) and both courses (F), respectively. Blue,
green, and orange represent real students, simulated students using only
demographics (Type i) and simulated students using both demographics and all
past assessments scores (Type iii), respectively. (G),(H),(I),(J),(K)
represent the distribution of student number (G), IMD band (H), average
assessments score (I), final exam scores of real students (J), and final exam
scores of simulated students using Type iii (K) in different regions,
respectively.

### Experiment 2

The second experiment aims to go into a finer-grained level by simulating
students with both demographics and past assessment performance(Fig.2(B)). We
use an N = 4524 dataset of students in UK with more than five assessments from
Open University Learning Analytics Dataset [19]. Fig. 2(G) depicts the student
number distribution in each region of UK. The final exam score is a numeric
value from 0 to 100. Each student is simulated with 8 configurations,
resulting in more than 40,716 responses of LLMs. We use temperature parameter
to be 0 in this experiment for more deterministic outputs. The example of
prompts to instantiate virtual students and predict final exam score according
to both demographic information and assessment history is described in Fig.
2(A). In order to investigate the impact of different factors on student
simulation performance, we use different configurations to instantiate virtual
students: Type 1: demographics, Type 2: all past assessments scores, Type 3:
demographics + all past assessments scores, Type 4 to Type 8 refers to the
configuration using only past one to past five assessments scores without
demographics (Fig. 3).

We find highly aligned correlation in final exam scores between virtual and
real students using Type ii ($r$ $=$ 0.69) and Type iii ($r$ $=$ 0.68), as
depicted in Fig. 2(C). More importantly, we find the correlation becomes
stronger when we use more assessments scores as input (from Type iv to Type
viii) (Fig. 2(C) and appendix Fig), suggesting that students simulation tends
to be more realistic as taking more student assessment history data for
instantiating virtual students.

Figure 3: (A)(B) depict an example of input prompts (denoted as Q) and LLMs’
responses (denoted as A) in experiment 3: task 1 and task 2, respectively.
Contents have been truncated. Full examples can be found in Appendix. (C).
Illustration of information used for student simulation in experiment 3.
(D)(E) depict Pearson correlation matrix in the first and last slide in
$birth$ course in task 1 for student understanding level simulation,
respectively. Hu represents real students’ understanding level and a,b,c
represent simulated understanding levels with Type a, Type b, Type c,
respectively. (F)(G) depict Pearson correlation matrix in the first and last
slide in $star$ course in task 1 for student understanding level simulation,
respectively. (H) depict Pearson correlation matrix in $birth$ course in task
2 for student post test performance simulation. Hu represents real students’
post test scores and 1a to 3c represent simulated post test scores with Type
1a to Type 3c, respectively. U, Pre, and ISC represent real students’ average
understanding levels, pre-test scores, and course engagement, respectively.

Surprisingly, we find that the simulation performance gets a little worse when
we add demographics (Type iii: $r$ $=$ 0.68) compared with only using
assessments (Type ii: $r$ $=$ 0.69). The reason is mainly because of limited
demographics in this dataset and some demographics may not directly reflect
learning performance, which is different from experiment 1 that has rich
demographics. This is supported by the inconsistent mapping results between
IMD band 111IMD (Indices of Multiple Deprivation) is a set of indices used in
the UK to measure multiple dimensions of deprivation, including income,
employment, health, education, crime, etc [20]. (Fig. 2(H)) and students’
final exam scores (Fig. 2(J)).

However, we find consistency between students’ average assessments (Fig. 2(I))
and final scores (Fig. 2(J)), which suggests and explains the capability of
leveraging assessments for final exam score simulation. As a result, we
observe highly aligned final scores between virtual students powered by Type
iii (demographics + assessments) (Fig. 2(K)) and real students (Fig. 2(J)) in
the distribution of regions in UK. However, if we only use demographics to
instantiate students, the simulation performance drops heavily, as depicted in
Fig. 2(D)(E)(F), echoing Type i results in Fig. 2(C) and suggesting importance
of assessments information.

Although the map results look quite similar between average assessments (Fig.
2(I)) and simulation results (Fig. 2(K)), the range of color bar is different.
More apparent difference could be found in Fig. 2(D)(E)(F), suggesting that
LLMs could capture correlation between assessments and final scores but will
not simply copying results from assessments for final score simulation. From
these figures, we also observe that Type iii simulation exhibits consistent
but not overlapped distribution among regions compared with real students.
This diversity and divergence in simulation outcomes could contribute to the
development of a more effective digital twin, as determinism alone may not
adequately represent the range of student learning behaviors.

Figure 4: (A). Each cell represents a specific real student’s (vertical axis)
understanding level (color depth) of specific slides related to the specific
post-test question (horizontal axis). (B). Each cell represents a specific
student. H represents real students’ post test scores and 1a to 3c represent
simulated post test scores with Type 1a to Type 3c, respectively. U, P, and I
represent real students’ average understanding levels, pre-test scores, and
course engagement, respectively. (C)(D)(E)(F)(G)(H) represent each simulated
student’s answer correctness in each post-test question (horizontal axis)
using Type 2a, 2b, 2c, 3a, 3b, 3c, respectively. Darker cell means virtual
students answer this question correctly and vice versa. All matrices share the
same color bar.

### Experiment 3

The third experiment aims to go into a much finer-grained level by simulating
students with not only demographic information but also each student’s
interactions with course materials. We use a dataset of N = 27 students from
[21] which collected students’ gaze data, pre-test and post-test scores while
students were watching course videos. The dataset[21] has suggested the
synchronized eye movements among engaged students in the course and
demonstrated the correlation between students’ intersubject correlation (ISC)
of gaze behaviors and post-test performance. Therefore, we use ISC to
represent the engagement of students in our simulation. In the dataset, we
also find a negative correlation between pupil size and student performance in
a preliminary analysis. In addition, larger pupil size has been shown to be
correlated with students’ increased stress and overwhelming cognitive load
[22]. Therefore, we use pupil size to represent students’ real-time
understanding level during video watching in the course and larger pupil size
will represent a lower understanding level in course concepts.

There are two courses named birth and star, which are segmented into 10 and 6
slides respectively according to course materials. Virtual students will not
only simulate post-test scores as learning outcome, but also understanding
levels after each slide in the course video. We use temperature parameter to
be 0 in LLMs for more deterministic outputs.

#### Task 1: understanding level simulation

We ask LLMs to mimic students with specific demographics and past
understanding levels in previous slides, and then predict the understanding
level in the current slide, which is a numeric value from 0 to 1. One example
prompt and LLMs’ responses are described in Fig. 3(A). We use different
configurations to instantiate virtual students to investigate the impact of
different factors: Type a simply inputs demographics and all course materials
to predict understanding level of all slides together. Type b emphasizes the
current slide material in addition to Type a’s input, and LLMs will just
predict understanding level of the individual slide. Type c will add all past
understanding levels before the current slide in addition to Type b’s input to
predict understanding level of the individual slide.

We find higher correlation in understanding levels between virtual and real
students using Type c compared with Type a and Type b (Fig. 2), suggesting
that past understanding levels (instead of demographics) will significantly
affect the understanding level in the current slide. Additionally, we observe
increasingly improved simulation performance with the increase of slide number
(Appendix Figure). For example, in Fig. 2, the correlation between virtual
(Type c) and real students increases significantly from the first slide to the
last slide in both course birth ($r$ $=$ 0.30 to $r$ $=$ 0.89) and course star
($r$ $=$ -0.17 to $r$ $=$ 0.86). This suggests that more past understanding
levels could provide more in-context information to help LLMs mimic more
realistic students learning behaviors.

#### Task 2: learning outcome simulation

We use prompts to instantiate virtual students and simulate post-test scores
as learning outcomes according to demographics, understanding levels per
slide, pre-test score (prior knowledge background), and engagement. One
example is depicted in Fig. 3(B). Similarly, we use different configurations
to instantiate virtual students: Type 1a, 1b, 1c ask LLMs to predict the
average accuracy in all post-test questions. Type 1a inputs demographics, all
slides, and post-test questions. Type 1b adds past understanding levels of all
slides in addition to Type 1a’s input. Type 1c adds pre-test questions and
average score in addition to Type 1b’s input. The input of Type 2a, 2b, 2c
follows the design of Type 1a, 1b, 1c, respectively. But Type 2a, 2b, 2c ask
LLMs to predict accuracy of each question in post-test. However, note that
Type 1a to 1c still input all post-test questions together. By contrast, Type
3a, 3b, 3c go into a much finer-grained level where LLMs will treat each
question individually for single input and predict accuracy of each specific
question. Type 3a inputs demographics, slides related to the specific post-
test question, understanding levels of related slides, and the current post-
test question. Type 3b adds pre-test questions and average score compared with
Type 3a’s input. Type 3c adds course engagement in addition to Type 3b’s
input. More details are depicted in appendix.

We find best aligned correlation in Type 3c between virtual and real students
in course birth: $r$ $=$ 0.59 and star: $r$ $=$ 0.47. Fig. 3(H) and appendix
depicts the Pearson correlation matrix in birth and star respectively.
Specifically, introducing pre-test information will significantly improve
simulation performance from Type 1b ($r=-0.15$) to Type 1c ($r=0.36$), Type 2b
($r=0.16$) to Type 2c ($r=0.36$), and Type 3a ($r=0.28$) to Type 3b
($r=0.57$). This is evidenced by correlation ($r=0.40$) between pre-test and
post-test performance. Additionally, we observe importance of understanding
level and course engagement for more realistic simulation, from Type 2a
($r=0$) to Type 2b ($r=0.16$) and from Type 3b ($r=0.57$) to Type 3c
($r=0.59$), evidenced by correlation between understanding level and post-test
performance ($r=0.28$) and correlation between engagement and post-test
performance ($r=0.62$). A more intuitive visualization of simulation
performance across all types for each student is depicted in Fig. 4(B). These
findings suggest the importance of incorporating students’ previous knowledge
background, real-time concept understanding level and course engagement for
more realistic simulation.

More importantly, we observe that simulation gets more realistic when we treat
each question individually (Type 3a to 3c: best $r$ $=$ 0.59) compared with
treating all questions as a whole (Type 1a to 2c: best $r$ $=$ 0.36). To
reveal such individual differences, we visualize virtual students’ answers
(correct or not) for each question in each cell with different types(Fig.
4(C,D,E,F,G,H)). We find that simulation just fails if we only input
demographics and course materials (Type 2a) since almost all questions are
predicted to be wrongly answered. This is due to the requirement of more in-
context information during course interactions. This explanation is supported
by the improvement of simulation when understanding levels are introduced
(Type 2b). Finally, the introduction of pre-test information and course
engagement also further improves the variety of student simulation in Type 2c,
3b, 3c (Fig. 4), contributing to developing a more effective digital twin.

## Discussion

Education is fundamental to the success of humans and society [23]. Student
simulation stands as the key for adaptive learning experience tailoring to
meet unique needs of diverse students[24]. Inspired by success of LLMs in
cognitive studies [25], we leverage LLMs to mimic students with specific
demographics and fine-grained interactions with course materials, underscoring
the potential for digital twins to revolutionize the educational landscape in
the future.

Specifically, we observe that LLMs could capture the correlation between
students’ demographics and their potential success in the future exam. In
addition, past assessments history could provide more helpful insights to
support more realistic learning behaviors simulation. Furthermore, finer-
grained simulation in the interaction with course materials could better
predict the success of students in the future post-tests. Since we do not use
training data to fine-tune LLMs, one question is why LLMs could work so well
in such scenarios. One answer stems from LLMs’ robust in-context learning
abilities [18] and extensive knowledge base with human feedback [26, 27],
making it understand the potential correlation between various factors and
students’ experience and success.

We highlight our unique contributions in two significant ways: First, we
pioneer the validation of leveraging generative AI to simulate student
learning behaviors in large-scale datasets and fine-grained course interaction
experience. Second, rather than merely testing whether LLMs could solve a task
or not, we emphasize exploring and explaining how LLMs could work well in
student simulation via carefully designed probes and ablation study
configurations in micro benchmarks, revealing versatile capabilities of LLMs
across different contexts. As a result, our research significantly complements
existing work.

We acknowledge potential limitations and open questions. First, the simulation
is based on general understanding of LLMs so the accuracy may vary for
individual cases, although this might not pose issues in practical scenarios
like new teacher training. Second, LLMs’ sensitivity to prompt design [28]
could lead to varied results when information order changes. Lastly, token
size constraints [29] might limit applications in extended educational
contexts. We anticipate that future developments in LLMs may address these
challenges.

In conclusion, we demonstrate the feasibility of leveraging generative AI to
create digital twins for simulating student learning behaviors. Our research
reveals strong alignment between virtual and real students’ behaviors. By
enabling the simulation and study of learning behaviors, this work has the
potential to enhance learning experiences and outcomes, leading to
transformative impacts on education and training. These findings offer
educators deeper insights into dynamics of the learning environment, driving
innovation in pedagogical strategies and inspiring the development of adaptive
and personalized educational technologies. As such, our research contributes
to the evolving landscape of online education.

## Methods

More details are depicted in Appendix.

### Data and Code Availability

All data and codes to reproduce the findings are available at:
https://github.com/songlinxu/EduTwin.

## References

  * [1] Trilling, B. & Fadel, C.  _21st century skills: Learning for life in our times_ (John Wiley & Sons, 2009).
  * [2] Tracey, B., Florian, K. _et al._ _Educational research and innovation governing education in a complex world_ (OECD Publishing, 2016).
  * [3] Gašević, D., Dawson, S., Rogers, T. & Gasevic, D.  Learning analytics should not promote one size fits all: The effects of instructional conditions in predicting academic success.  _The Internet and Higher Education_ 28, 68–84 (2016).
  * [4] Salend, S. J.  _Creating inclusive classrooms_ (Pearson Education, 2010).
  * [5] VanLehn, K., Ohlsson, S. & Nason, R.  Applications of simulated students: An exploration.  _Journal of artificial intelligence in education_ 5, 135–135 (1994).
  * [6] Cherryholmes, C. H.  Some current research on effectiveness of educational simulations: Implications for alternative strategies.  _American Behavioral Scientist_ 10, 4–7 (1966).
  * [7] Barricelli, B. R., Casiraghi, E. & Fogli, D.  A survey on digital twin: Definitions, characteristics, applications, and design implications.  _IEEE Access_ 7, 167653–167671, DOI: 10.1109/ACCESS.2019.2953499 (2019).
  * [8] Kinsner, W.  Digital twins for personalized education and lifelong learning.  In _2021 IEEE Canadian Conference on Electrical and Computer Engineering (CCECE)_ , 1–6 (IEEE, 2021).
  * [9] Piech, C. _et al._ Deep knowledge tracing.  _Advances in neural information processing systems_ 28 (2015).
  * [10] Beck, J. E. & Woolf, B. P.  High-level student modeling with machine learning.  In _International Conference on Intelligent Tutoring Systems_ , 584–593 (Springer, 2000).
  * [11] Hussain, M., Zhu, W., Zhang, W., Abidi, S. M. R. & Ali, S.  Using machine learning to predict student difficulties from learning session data.  _Artificial Intelligence Review_ 52, 381–407 (2019).
  * [12] Xu, J., Moon, K. H. & Van Der Schaar, M.  A machine learning approach for tracking and predicting student performance in degree programs.  _IEEE Journal of Selected Topics in Signal Processing_ 11, 742–753 (2017).
  * [13] Shelton, B. E., Hung, J.-L. & Lowenthal, P. R.  Predicting student success by modeling student interaction in asynchronous online courses.  _Distance Education_ 38, 59–69 (2017).
  * [14] Cziko, G. A.  Unpredictability and indeterminism in human behavior: Arguments and implications for educational research.  _Educational researcher_ 18, 17–25 (1989).
  * [15] Zhao, W. X. _et al._ A survey of large language models.  _arXiv preprint arXiv:2303.18223_ (2023).
  * [16] Openai api.  https://platform.openai.com/overview (2023).  Accessed: 2023-10-14.
  * [17] Yılmaz, N. & Sekeroglu, B.  Student performance classification using artificial intelligence techniques.  In _International Conference on Theory and Application of Soft Computing, Computing with Words and Perceptions_ , 596–603 (Springer, 2019).
  * [18] Dong, Q. _et al._ A survey for in-context learning.  _arXiv preprint arXiv:2301.00234_ (2022).
  * [19] Kuzilek, J., Hlosta, M. & Zdrahal, Z.  Open university learning analytics dataset.  _Scientific data_ 4, 1–8 (2017).
  * [20] Imd band.  https://en.wikipedia.org/wiki/Multiple_deprivation_index (2023).  Accessed: 2023-10-25.
  * [21] Madsen, J., Júlio, S. U., Gucik, P. J., Steinberg, R. & Parra, L. C.  Synchronized eye movements predict test scores in online video education.  _Proceedings of the National Academy of Sciences_ 118, e2016980118 (2021).
  * [22] Chen, S. & Epps, J.  Using task-induced pupil diameter and blink rate to infer cognitive load.  _Human–Computer Interaction_ 29, 390–413 (2014).
  * [23] Felten, P. & Lambert, L. M.  _Relationship-rich education: How human connections drive success in college_ (Jhu Press, 2020).
  * [24] Truong, H. M.  Integrating learning styles and adaptive e-learning system: Current developments, problems and opportunities.  _Computers in human behavior_ 55, 1185–1193 (2016).
  * [25] Binz, M. & Schulz, E.  Using cognitive psychology to understand gpt-3.  _Proceedings of the National Academy of Sciences_ 120, e2218523120 (2023).
  * [26] Lampinen, A. K. _et al._ Can language models learn from explanations in context?  _arXiv preprint arXiv:2204.02329_ (2022).
  * [27] Stiennon, N. _et al._ Learning to summarize with human feedback.  _Advances in Neural Information Processing Systems_ 33, 3008–3021 (2020).
  * [28] Zhao, Z., Wallace, E., Feng, S., Klein, D. & Singh, S.  Calibrate before use: Improving few-shot performance of language models.  In _International Conference on Machine Learning_ , 12697–12706 (PMLR, 2021).
  * [29] Jozefowicz, R., Vinyals, O., Schuster, M., Shazeer, N. & Wu, Y.  Exploring the limits of language modeling.  _arXiv preprint arXiv:1602.02410_ (2016).


[affiliation=1]ChenChen [affiliation=2]ZehuaLiu [affiliation=2]XiaolouLi
[affiliation=2]LantianLi [affiliation=1]DongWang

# CNVSRC 2023: The First Chinese Continuous Visual Speech Recognition
Challenge

###### Abstract

The first Chinese Continuous Visual Speech Recognition Challenge aimed to
probe the performance of Large Vocabulary Continuous Visual Speech Recognition
(LVC-VSR) on two tasks: (1) Single-speaker VSR for a particular speaker and
(2) Multi-speaker VSR for a set of registered speakers. The challenge yielded
highly successful results, with the best submission significantly
outperforming the baseline, particularly in the single-speaker task. This
paper comprehensively reviews the challenge, encompassing the data profile,
task specifications, and baseline system construction. It also summarises the
representative techniques employed by the submitted systems, highlighting the
most effective approaches. Additional information and resources about this
challenge can be accessed through the official website at
_http://cnceleb.org/competition_.

###### keywords:

CNVSRC, Visual speech recognition, Lip reading, Chinese VSR

##  1 Introduction

Visual Speech Recognition (VSR), commonly called lip reading, is a technology
that utilizes lip movements to infer speech content. It has broad
applications, including public surveillance, support for elderly and disabled
individuals, and fake video detection. Traditionally, VSR has been primarily
focused on recognizing isolated words or phrases. For example, Martinez et al.
[1] developed a model that extracts visual features using 3D convolution,
ResNet-18, and a multi-scale temporal convolutional network (MS-TCN). This was
further enhanced by simple average pooling and a softmax layer for inferring
word posteriors, resulting in commendable performance on the LRW [2] and
LRW-1000 datasets [3], which are the largest publicly available benchmark
datasets for unconstrained isolated word lip-reading in English and Mandarin,
respectively. Ma et al. [4] adopted a similar architecture but introduced a
densely connected temporal convolutional network (DC-TCN) to achieve improved
performance.

Recently, research in lip reading has progressed beyond word and phrase
recognition to focus on large vocabulary continuous visual speech recognition
(LVC-VSR), a more challenging task but with more realistic merit. Significant
advancements have been made in English benchmarks, partly attributable to the
availability of large-scale English visual-speech datasets such as LRS2 [5],
LRS3 [5], AVSpeech [6], and VoxCeleb [7, 8]. While earlier studies addressed
this issue using hand-crafted features and sequential models like HMM [9] or
RNN [10], a significant breakthrough occurred with the end-to-end approach,
which processes raw video frames and generates a word sequence. LipNet [11] is
perhaps the first end-to-end model, integrating spatiotemporal convolution
layers and bi-directional gated recurrent unit (BGRU), trained using the
connectionist temporal classification(CTC) loss [12]. The model underwent
testing on GRID, a dataset with limited grammar and vocabulary [13]. A similar
architecture was adopted by Jeon et al. [14], with their approach involving
the integration of multiple CNN-based streams into the feature extraction
process.

While promising, the GRID corpus is limited in its ability to capture real-
world complexity due to the constrained grammar and vocabulary of the
sentences. A more challenging task involves large vocabulary continuous visual
speech recognition based on datasets gathered from online media repositories
like YouTube. A seminal work [5]introduced the first comprehensive visual
speech datasets LRS2 and LRS3, along with the first transformer-based system
trained with either the CTC loss (TM-CTC) or the sequence-to-sequence loss
(TM-seq2seq). In a parallel endeavour, Google [15] developed an end-to-end
model (CNN/BLSTM backbone and CTC loss) that transcribes videos into phone
sequences and utilizes an FST-based decoder to obtain word sequences. The
research was expanded upon in a subsequent study [16], which introduced a
hybrid CTC/Attention model utilizing a ResNet/Conformer encoder and a
Transformer-based language model (LM). This work was further developed by
incorporating time-masking data augmentation and an auxiliary reconstruction
loss in a subsequent publication [17].

In addition to employing complex structures, a simple yet effective approach
is increasing the training data volume. However, a major challenge arises from
the dearth of well-labeled data. One promising strategy to address this issue
involves utilizing a pre-trained automatic speech recognition (ASR) model to
transcribe unlabelled videos. This methodology was extensively explored in
[18], wherein a ResNet/Conformer model was trained on both unlabelled datasets
like VoxCeleb2 [8] and AVSpeech [6], as well as text-labelled datasets such as
LRW [2], LRS2 [5], and LRS3 [5]. The study revealed significant performance
enhancements with auto-labelled data, and further improvements were observed
with increased incorporation of unlabelled data.

For VSR on Chinese data, the research progress has been significantly impeded
by the scarcity of data resources. Despite the presence of LRW-1000 [3] as the
sole large-scale dataset in Chinese, it consists solely of isolated words. In
2023, the release of the CN-CVS dataset [19] marked the debut of the first
large-scale continuous visual-speech dataset in Chinese, thereby presenting an
opportunity to propel research in Chinese LVC-VSR. This also motivated the
first Chinese Continuous Visual Speech Recognition Challenge (CNVSRC 2023),
hosted as a special session on NCMMSC 2023 to estimate the performance
boundary of existing LVC-VSR techniques with Chinese data and attracting
research interest in this domain. The CN-CVS [19] dataset was used as the
primary training data, supplemented by two additional datasets - CNVSRC-Single
and CNVSRC-Multi - introduced by the organizers to facilitate system
development and evaluation. The organizers also published the model and code
of the baseline systems so that the participants could use them as references
when developing their systems. This paper summarises the challenge,
emphasizing the predominant findings gleaned from the submitted systems.

The structure of the rest of this paper is outlined as follows: Section 2
introduces the tasks and data of the challenge. In Section 3, a comprehensive
description of the baseline system is provided, covering the model structure,
training strategies, and performance evaluation. Section 4 reports the
challenge result and summarizes the representative technologies utilized by
the participants. Lastly, the paper is concluded in Section 5.

##  2 Tasks and Data

###  2.1 Tasks

The CNVSRC 2023 challenge encompasses two distinct tasks: Single-speaker VSR
(T1) and Multi-speaker VSR (T2). Task T1 emphasizes the performance of large-
scale tuning for a specific speaker, whereas T2 focuses on the fundamental
performance of the system for non-specific but _registered_ speakers, i.e.,
speakers seen in the data for system development. In both tasks, the system is
fed with silent facial videos featuring a single individual, and it is
required to generate the spoken content in written form.

Each task is further categorized into a `fixed track' and an `open track'. The
fixed track permits the use of data and additional resources that have been
agreed upon by the organizing committee. Conversely, the open track allows
participants to employ any resources except the evaluation set.

Character Error Rate (CER) was used as the main metric to evaluate VSR
performance, formulated as follows:

$CER=\frac{S+D+I}{N}$ (1)

where $S$, $D$, and $I$ represent the number of substitutions, deletions, and
insertions in the output transcription, respectively. $N$ is the number of
characters in the ground truth transcription.

###  2.2 Data profile

CNVSRC 2023 utilized the CN-CVS dataset [19] in addition to two supplementary
datasets: CNVSRC-Single and CNVSRC-Multi, which served as the development and
evaluation data for the single-speaker VSR task (T1) and multi-speaker VSR
task (T2), respectively. All the data was transcribed into text. Table 1
presents the data profile of the three datasets. Note that both CNVSRC-Single
and CNVSRC-Multi were split into a development set and an evaluation set. The
development data was transparent to the participants (including the video,
audio, and text), while the text and audio of the evaluation data were kept
secret during the entire challenge process. The participants could use the
development data in any way, e.g., freely splitting it into a subset for model
fine-tuning and a subset for model validation/selection.

Table 1: Data profile used in CNVSRC 2023.

| CN-CVS | CNVSRC-Single | CNVSRC-Multi
---|---|---|---
DataSet | Train | Dev | Eval | Dev | Eval
# Spks | 2,557 | 1 | 43
# Videos | 206,261 | 25,947 | 2,881 | 20,450 | 10,269
# Hours | 308.00 | 94.00 | 8.41 | 29.24 | 14.49

####  2.2.1 CN-CVS

The CN-CVS dataset [19] comprises visual-speech data from over 2,557 speakers,
totalling more than 300 hours of videos. It encompasses various scenarios,
including news broadcasts and public speeches. So far, CN-CVS is the largest
open-source Chinese visual-speech dataset. This dataset was used as the
primary training data for the CNVSRC 2023 challenge. Note that the original
publication of CN-CVS is for the video-to-speech synthesis (VTS) task and thus
does not involve text transcription. To support the CNVSRC 2023 challenge, we
labelled the video with a semi-automatic pipeline that involves ASR
transcribing and human check, as will be presented shortly.

####  2.2.2 CNVSRC-Single

The CNVSRC-Single dataset, designed for a single-speaker VSR task (T1), is
obtained from a broadcaster's online channel. It includes over 800 speech
videos of that broadcaster, with a cumulative duration of over 100 hours. The
pipeline used in [19] was employed for the collection and processing.

####  2.2.3 CNVSRC-Multi

The CNVSRC-Multi dataset was designed as the development/evaluation data for
the multi-speaker VSR task (T2). It encompasses two scenarios: reading in a
recording studio and speeches downloaded from the internet.

In the recording studio scenario, facial videos of speakers were captured from
three different camera angles (0°,30°,60°), while their speech audio was
recorded using a high-quality microphone. The speakers were prompted a
sentence at each time via a computer screen and were asked to read the
sentence clearly with a neutral emotion. The video data from the _front
camera_ was used in CNVSRC 2023, which was transcoded to 25 frames per second
(FPS) and scaled to an appropriate size. A total of 23 speakers participated
in the recording, each reading 1,000 sentences. In the speeches from the
internet scenario, videos of public speeches from 20 speakers were collected
from the internet, again following the same collection and processing pipeline
as [19]. To ensure the integrity of the collected videos from the internet, a
face recognition tool111https://pypi.org/project/face-recognition/ was
employed to check whether there is only one face in each video frame and if
the face is the target face. A manual check was then conducted to double-check
that each extracted video only contained the target face.

###  2.3 Text annotation

To generate text transcriptions, a Paraformer-based ASR system [20] was
employed to transcribe the speech of all the videos. Furthermore, the manual
check was conducted to ensure that the CER of the transcriptions remains below
2%.

##  3 Baseline System

Table 2: Training Details of the Pretraining (P1 & P2) and Fine-tuning (FT)
steps when constructing the baseline systems.

Experiment | P1 | P2 | FT (Single-Speaker) | FT (Multi-Speaker)
---|---|---|---|---
Initialize | Random | P1 Saved Model | P2 Saved Model | P2 Saved Model
Warmup Epochs | 5 | 5 | 2 | 2
Learning Rate | 0.0002 | 0.001 | 0.0003 | 0.0002
Training Epochs | 75 + Early stop | 75 | 80 | 80
Saved Model | Top 10 average | Last 10 epochs average | Last 5 epochs average | Last 5 epochs average

Leveraging the state-of-the-art model used in Auto-AVSR [18], we trained two
baseline systems, one for the single-speaker VSR task (T1) and the other for
the multi-speaker VSR task (T2). Only the datasets provided in this challenge
were used. In other words, these systems conform to the specifications of the
fixed tracks.

###  3.1 Model structure

The model structure is duplicated from Auto-AVSR [18]. Specifically, it
comprises three components: visual frontend, encoder, and decoder. As in [18],
the visual frontend adopts ResNet18 as its backbone, except that the first
2D-CNN layer is replaced with a 3D-CNN layer to capture local spatiotemporal
correlation. The encoder adopts a Conformer structure with 12 layers, while
the decoder employs a Transformer structure with 6 layers.

Upon receiving the input video data, the visual frontend performs initial
local spatiotemporal feature extraction. Subsequently, the Conformer encoder
further extracts context-dependent features. A projection layer and a
transformer decoder are employed to predict the output class labels. The
entire model was trained with the joint CTC/Attention loss, where the CTC loss
is back-propagated through the projection layer, while the Attention loss is
back-propagated through the decoder. Refer to [18] for details.

###  3.2 Data pre-processing

The provided datasets of the challenge include the faces of the target
speakers, so a pre-processing pipeline was designed to extract the lip region.
Note that the pipeline was equally applied to both the training data (CN-CVS)
and the development/evaluation data (CNVSRC-Single and CNVSRC-Multi).

Initially, we utilized RetinaFace [21] to detect the facial regions in each
frame. Subsequently, FAN [22] was employed to extract facial landmarks, with
which each detected face was aligned with a mean reference face. Finally, we
extracted the lip region through the alignment for each video frame, which
served as the input to the visual frontend. The modelling units are subword
tokens, and the tokenizer was trained using the SentencePiece [23] tool in the
unigram mode. During the training of this tokenizer, only the text data from
CN-CVS was utilized. Subsequently, we employed this tokenizer to process the
CN-CVS, CNVSRC-Single, and CNVSRC-Multi datasets, obtaining the token
sequences used to train the recognition models.

###  3.3 Training strategy

We followed a two-step process to build the baseline systems. Firstly, we
performed pre-training with the CN-CVS dataset. Subsequently, the development
set was split into an `adaptation set' and a `validation set', with a ratio of
8:1 for the single-speaker dataset and 3:1 for the multi-speaker dataset. The
adaptation set was used to fine-tune the pre-trained model, while the
validation set was used to select the appropriate checkpoint. The training
process was summarized in Table 2, and the details are as follows.

The initial training phase (P1) selected CN-CVS videos with a duration of less
than 4 seconds to train an `easy model'. The maximum number of training epochs
was 75, and early stopping was triggered if the model's performance on the
validation set started to drop. Once the training stopped, we selected the top
10 models based on their accuracy on the validation set, averaged their
parameters to obtain the P1 model. Next, a full pre-training phase (P2) was
evoked using the complete CN-CVS dataset, and the training was conducted for
75 epochs. Note that a warmup stage of 5 epochs was designed, by which the
learning rate was gradually increased from 0 to 0.001. The average of the
models of the last 10 epochs was used as the P2 model, i.e., the pre-trained
model.

The fine-tuning step started from the P2 model and ran 80 epochs, including 2
epochs of warmup. This process is the same for the models trained for the
single-speaker task and the multi-speaker task, but there are indeed some
differences. Besides the learning rate (see Table 2), the most notable
difference is that for the single-speaker model, we randomized the parameters
of the classification layer of the P2 model, to provide sufficient space for
the adaptation with the large amount of single-speaker data. The average of
the models of the last 5 epochs was used as the final model, for both the
single-speaker task and the multi-speaker task.

###  3.4 Performance

The performance of the baseline models was evaluated on the respective
validation set and evaluation set for both the single-speaker task and multi-
speaker task, using the TorchMetrics
tool222https://lightning.ai/docs/torchmetrics/stable/. The CER results are as
shown in Table 3.

Table 3: Performance of the baseline systems.

| Character Error Rate
---|---
Task | T1: Single-speaker VSR | T2: Multi-speaker VSR
Valid | 48.57% | 58.77%
Eval | 48.60% | 58.37%

Table 4: Leaderboard of CNVSRC2023. Team ID and CER are reported. Task | T1: Single-speaker VSR | T2: Multi-speaker VSR
---|---|---
Track | Fixed Track | Open Track | Fixed Track | Open Track
Baseline | 48.60% | 48.60% | 58.37% | 58.37%
Rank 1 | T237 | 34.76% | T237 | 34.76% | T244 | 53.68% | T237 | 41.05%
Rank 2 | T266 | 38.09% |  |  | T267 | 54.56% | T244 | 53.68%
Rank 3 | T290 | 39.47% |  |  |  |  |  |
Rank 4 | T238 | 40.52% |  |  |  |  |  |
Rank 5 | T267 | 41.62% |  |  |  |  |  |

##  4 CNVSRC 2023 Report

###  4.1 Leaderboard

CNVSRC 2023 received 10 valid submissions from 6 teams. Most teams chose to
submit their results to the single-speaker task, suggesting that single-
speaker VSR is more suitable for the current stage of technical development,
and multi-speaker VSR is over-challenging. The leaderboard is reported in
Table 4.

Overall, T237 achieved the best performance in 3/4 of the tasks & tracks, and
their results outperformed the baseline systems by a large margin. Their
proposed system consists of a ResNet-3D visual frontend, an E-Branchformer
encoder [24], and a Transformer decoder. The Chinese characters were used as
the modelling units, and multiple data augmentation methods, including speed
perturbation, random rotation, and horizontal flipping, were applied during
training. Additionally, a ROVER-based system fusion [25] was performed during
the inference procedure.

###  4.2 Technical summary

We summarize here the promising techniques demonstrated by the results of the
submissions, highlighting the most effective methods in bold font.

####  4.2.1 Data pre-processing

Many teams adhered to the baseline system for data pre-processing. A notable
observation is that T237 extracted lip regions of varying sizes and
resolutions as inputs to their model and discovered that larger regions (lip +
mouth around) yielded clear performance improvement.

####  4.2.2 Data augmentation

The participants extensively used data augmentation techniques, including
random erase, random crop, random flip, and adaptive time masking. Notably,
speed perturbation and generative data augmentation were reported to yield
unexceptionally remarkable results. Speed perturbation adjusts the speed of
the original videos by a factor ranging from 0.9 to 1.1. T237 and T238
observed notable enhancements in model performance (4.86% relative
improvement) by applying speed perturbation. Generative data augmentation
involves generating speech-driven lip videos, hence producing extra video-text
training pairs. T238 utilized facial images from CNVSRC-Single and speech from
CN-CVS and CNVSRC-Single to create an extra set of video-text pairs and
reported a relative CER reduction of 6.98%.

####  4.2.3 Model structure

Most of the participating teams adopted the model architecture of the baseline
systems, though some teams chose a more complicated backbone to pursue better
performance. For instance, T237 achieved superior results using a ResNet3D
structure. Moreover, T237 employed two advanced encoder structures:
Branchformer [26] and E-Branchformer [24]. All these advanced structures lead
to notable performance improvements. T266 introduced an inner CTC residual
module [27, 28, 29] that resides in the Conformer block of the encoder. This
module back-propagated a CTC loss through the shallow layers thus facilitating
more effective parameter updates for the shallow layers of the model.
Furthermore, taking inspiration from [30, 31], T266 utilized a bi-transformer
structure to construct the decoder. This modification enhances the model's
ability to capture contextual information from both the past and future
segments.

####  4.2.4 Modeling units

Most participating teams used Chinese characters as the modelling units, and
showed better performance than the subword tokens used by the baseline
systems. In addition to subword tokens, T238 used phonemes as supplementary
modelling units and designed a separate decoder for phoneme recognition. This
approach achieved performance improvement, for which a hypothesis is that
phonemes contain less semantic variation and thus are more closely related to
lip movement, which may stabilize the training, especially in the early stage.

####  4.2.5 Cross-modality modeling

Some teams designed various approaches to leverage the cross-modal dependency.
T290 invented an ASR-VSR joint system that forces the video representations to
predict not only the text labels but also the speech representations produced
by the middle layer of the ASR system. Following the same inspiration, T244
trained an audio-visual recognition system where the ASR and VSR have their
respective independent encoders and decoders, and an AVSR decoder is
constructed on top of the fused ASR and VSR features.

####  4.2.6 Decoding strategy

Several teams integrated RNN-based or Transformer-based language models to
enhance the decoder, and mild performance improvement was reported. Moreover,
system fusion was widely employed by teams to improve the performance of their
systems.

##  5 Conclusion

This paper comprehensively details the inaugural Chinese Continuous Visual
Speech Recognition Challenge (CNVSRC 2023). A key motivation of the challenge
is to investigate the performance bound of VSR under the present data
resource, e.g., 300 hours of training data from 2,557 speakers. The overall
results suggest that the performance is far from satisfactory, even for the
single-speaker scenario where about 100 hours of video is available for one
person. The poor performance is certainly attributed to the lack of data, but
whether it is related to the special linguistic properties of Chinese, e.g.,
the ubiquitous homophones, is unknown.

Based on these technical reports from participants, we have summarized the key
techniques that might be crucial in constructing Chinese VSR systems. The most
effective methods, ordered by their merit in terms of CER reduction: Chinese
characters as modelling units, rich data augmentation, fully 3D-CNN visual
frontend, cross-modality modelling, system fusion. Leveraging the technical
insights provided by the participants, we have established a cutting-edge
benchmark for Chinese LVC-VSR. We aspire that these resources will strengthen
the burgeoning field of LVC-VSR research.

## References

  * [1] B. Martinez, P. Ma, S. Petridis, and M. Pantic, ``Lipreading using temporal convolutional networks,'' in _IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP)_. IEEE, 2020, pp. 6319–6323.
  * [2] J. S. Chung and A. Zisserman, ``Lip reading in the wild,'' in _Asian Conference on Computer Vision (ACCV)_. Springer, 2017, pp. 87–103.
  * [3] S. Yang, Y. Zhang, D. Feng, M. Yang, C. Wang, J. Xiao, K. Long, S. Shan, and X. Chen, ``Lrw-1000: A naturally-distributed large-scale benchmark for lip reading in the wild,'' in _IEEE International Conference on Automatic Face & Gesture Recognition (FG)_. IEEE, 2019, pp. 1–8.
  * [4] P. Ma, Y. Wang, J. Shen, S. Petridis, and M. Pantic, ``Lip-reading with densely connected temporal convolutional networks,'' in _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , 2021, pp. 2857–2866.
  * [5] T. Afouras, J. S. Chung, A. Senior, O. Vinyals, and A. Zisserman, ``Deep audio-visual speech recognition,'' _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 44, no. 12, pp. 8717–8727, 2018.
  * [6] A. Ephrat, I. Mosseri, O. Lang, T. Dekel, K. Wilson, A. Hassidim, W. T. Freeman, and M. Rubinstein, ``Looking to listen at the cocktail party: a speaker-independent audio-visual model for speech separation,'' _ACM Transactions on Graphics_ , vol. 37, no. 4, pp. 1–11, 2018.
  * [7] A. Nagrani, J. S. Chung, and A. Zisserman, ``Voxceleb: a large-scale speaker identification dataset,'' in _INTERSPEECH_. ISCA, 2017.
  * [8] J. Chung, A. Nagrani, and A. Zisserman, ``Voxceleb2: Deep speaker recognition,'' in _INTERSPEECH_. ISCA, 2018.
  * [9] A. J. Goldschen, O. N. Garcia, and E. D. Petajan, ``Continuous automatic speech recognition by lipreading,'' in _Motion-Based Recognition_. Springer, 1997, pp. 321–343.
  * [10] S. Petridis and M. Pantic, ``Deep complementary bottleneck features for visual speech recognition,'' in _IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP)_. IEEE, 2016, pp. 2304–2308.
  * [11] Y. M. Assael, B. Shillingford, S. Whiteson, and N. De Freitas, ``Lipnet: End-to-end sentence-level lipreading,'' _arXiv preprint arXiv:1611.01599_ , 2016.
  * [12] A. Graves, S. Fernández, F. Gomez, and J. Schmidhuber, ``Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks,'' in _International Conference on Machine learning (ICML)_ , 2006, pp. 369–376.
  * [13] M. Cooke, J. Barker, S. Cunningham, and X. Shao, ``An audio-visual corpus for speech perception and automatic speech recognition,'' _The Journal of the Acoustical Society of America_ , vol. 120, no. 5, pp. 2421–2424, 2006.
  * [14] S. Jeon, A. Elsharkawy, and M. S. Kim, ``Lipreading architecture based on multiple convolutional neural networks for sentence-level visual speech recognition,'' _Sensors_ , vol. 22, no. 1, p. 72, 2021.
  * [15] B. Shillingford, Y. Assael, M. W. Hoffman, T. Paine, C. Hughes, U. Prabhu, H. Liao, H. Sak, K. Rao, L. Bennett _et al._ , ``Large-scale visual speech recognition,'' _arXiv preprint arXiv:1807.05162_ , 2018.
  * [16] P. Ma, S. Petridis, and M. Pantic, ``End-to-end audio-visual speech recognition with conformers,'' in _IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP)_. IEEE, 2021, pp. 7613–7617.
  * [17] P. Ma, S. Petridis, and M. Pantic, ``Visual speech recognition for multiple languages in the wild,'' _Nature Machine Intelligence_ , vol. 4, no. 11, pp. 930–939, 2022.
  * [18] P. Ma, A. Haliassos, A. Fernandez-Lopez, H. Chen, S. Petridis, and M. Pantic, ``Auto-avsr: Audio-visual speech recognition with automatic labels,'' in _IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP)_. IEEE, 2023, pp. 1–5.
  * [19] C. Chen, D. Wang, and T. F. Zheng, ``Cn-cvs: A mandarin audio-visual dataset for large vocabulary continuous visual to speech synthesis,'' in _IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP)_. IEEE, 2023, pp. 1–5.
  * [20] Z. Gao, S. Zhang, I. McLoughlin, and Z. Yan, ``Paraformer: Fast and accurate parallel transformer for non-autoregressive end-to-end speech recognition,'' in _INTERSPEECH_. ISCA, 2022.
  * [21] J. Deng, J. Guo, Y. Zhou, J. Yu, I. Kotsia, and S. Zafeiriou, ``Retinaface: Single-stage dense face localisation in the wild,'' _arXiv preprint arXiv:1905.00641_ , 2019.
  * [22] A. Bulat and G. Tzimiropoulos, ``How far are we from solving the 2d & 3d face alignment problem?(and a dataset of 230,000 3d facial landmarks),'' in _IEEE International Conference on Computer Vision (ICCV)_ , 2017, pp. 1021–1030.
  * [23] T. Kudo and J. Richardson, ``Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing,'' in _Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , 2018, pp. 66–71.
  * [24] K. Kim, F. Wu, Y. Peng, J. Pan, P. Sridhar, K. J. Han, and S. Watanabe, ``E-branchformer: Branchformer with enhanced merging for speech recognition,'' in _IEEE Spoken Language Technology Workshop (SLT)_. IEEE, 2023, pp. 84–91.
  * [25] J. G. Fiscus, ``A post-processing system to yield reduced word error rates: Recognizer output voting error reduction (rover),'' in _IEEE Workshop on Automatic Speech Recognition and Understanding Proceedings_. IEEE, 1997, pp. 347–354.
  * [26] Y. Peng, S. Dalmia, I. Lane, and S. Watanabe, ``Branchformer: Parallel mlp-attention architectures to capture local and global context for speech recognition and understanding,'' in _International Conference on Machine Learning (ICML)_. PMLR, 2022, pp. 17 627–17 643.
  * [27] J. Nozaki and T. Komatsu, ``Relaxing the Conditional Independence Assumption of CTC-Based ASR by Conditioning on Intermediate Predictions,'' in _INTERSPEECH_ , 2021, pp. 3735–3739.
  * [28] J. Lee and S. Watanabe, ``Intermediate loss regularization for ctc-based speech recognition,'' in _IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP)_. IEEE, 2021, pp. 6224–6228.
  * [29] M. Burchi and R. Timofte, ``Audio-visual efficient conformer for robust speech recognition,'' in _IEEE/CVF Winter Conference on Applications of Computer Vision_ , 2023, pp. 2258–2267.
  * [30] D. Wu, B. Zhang, C. Yang, Z. Peng, W. Xia, X. Chen, and X. Lei, ``U2++: Unified two-pass bidirectional end-to-end model for speech recognition,'' _arXiv preprint arXiv:2106.05642_ , 2021.
  * [31] B. Zhang, D. Wu, Z. Peng, X. Song, Z. Yao, H. Lv, L. Xie, C. Yang, F. Pan, and J. Niu, ``Wenet 2.0: More productive end-to-end speech recognition toolkit,'' _arXiv preprint arXiv:2203.15455_ , 2022.