ЂЅЈЊЎАВДЖИКМОРТФЦШЪЬЮавджикмортфцшъьюёєіљћџѣѫѵґғҕҗҙқҝҟҡңҥҧҩҫҭүұҳҵҷҹһҽҿӃӅӇӋӍӔәӡө

# Automatic Speech Recognition of Low-Resource Languages Based on Chukchi

Cydnie Davenport
Linguistic Theory and Language Description Masters program
Higher School of Ecomonics
Moscow, Russia
<EMAIL_ADDRESS>
Emil Nadimanov
Computational Linguistics Masters program
Higher School of Ecomonics
Moscow, Russia
<EMAIL_ADDRESS>
Anastasia Safonova
Computational Linguistics Masters program
Higher School of Ecomonics
Moscow, Russia
<EMAIL_ADDRESS>
Tatiana Yudina
Computational Linguistics Masters program
Higher School of Ecomonics
Moscow, Russia
<EMAIL_ADDRESS>

##  1 Introduction

The following paper presents a project focused on the research and creation of
a new Automatic Speech Recognition (ASR) and Text to Speech (TTS) system based
in the Chukchi language. The aim of this is to develop a system that makes the
language more accessible to speakers of Chukchi - such as annotating subtitles
on videos and movies, providing more accessible data for research and
analysis, or the creation of chat-bots for online users. This system should
consist of; an acoustic model for receiving an audio signal fragment and which
gives the probability of various phonemes based on the fragment analyzed; a
language model for determining which suggestions are more or less likely; and
a decoder which will determine the most likely prediction. Predictive
automatic speech recognition models already exist and are a popular focus in
the realm of Natural Language Processing, however, the most challenging
adversaries are low-resource languages due to extreme data deficits.

This project is centered around a multi-step research process. Initially, we
began by analyzing the Chukchi language from a linguistic perspective, but for
the sake of clarity regarding motivations for making this system, it also must
be looked at from a cultural and sociolinguistic perspective. What is known
about the language, are there any cultural influences within the language, why
is such a system necessary, and so on. Once there was extensive understanding
of the subject at hand, the next step would be finding data that is usable.
For this project, this included broadcasts from a Russian-based Chukchi radio
station, videos and lessons from YouTube, written translations of the Bible,
and the Higher School of Economics’ set of Chukchi-based corpora known as
Chuklang. Once enough data is collected, there then comes the task of cleaning
it. This included labeling and segmenting audio data for training, cleaning
and filtering out unnecessary symbols (mainly Russian) from text, and
determining which data would be used for pre-training and which would be used
for testing the resultant model. Once enough data has been collected and
cleaned for our model, we sample and train various models to understand how
they process data. Additionally, we must try various encoders to understand
how well they clean out noise and extra acoustic audio. Extra research must be
conducted in order to compare models designed for both high- and low-resource
languages. Various designs and tools for training ASR models include VQ-VAE,
XLSR, the toolkit Kaldi, wav2vec, and more. The intended result of this
project is an automatic speech-recognition system that can seamlessly work
with Chukchi and provides us with the potential to be used for other low-
resource languages.

##  2 BACKGROUND

###  2.1 The Chukchi Language

The Chukotko-Kamchatkan family of languages is said to contain two branches by
default. The northern branch is referred to as the Chukotian branch (or
“Luorevetlan”, based on the Chukchi ethnonym) and consists of Chukchi, Koryak,
Alutor and Kerek (now extinct). The second branch is known as Itelmen, and
contains the language Western Itelmen, which itself consists of two dialects:
Khajrjusovo and Sedanka [5]. The language of focus for this paper is Chukchi,
a polysynthetic language spoken primarily within the Chukotka Autonomous
Okrug, which is located in the easternmost part of Siberia. Chukchi itself is
an endangered indigenous language with less than 10,000 speakers at present,
and most speakers are bilingual with a primary language of Russian. There are
only less than 100 speakers who don’t speak Russian at all. Instances and
usages of this language are difficult to come by, and is not a language taught
in schools. The decreasing use of this language in general everyday life, as
well prominence of Russian within the community demonstrates the necessity for
an automatic speech recognition system, so that we may provide more
accessibility to such an endangered and very low-resource language and its
community.

###  2.2 What is a Low-Resource Language?

In the field of NLP, research tends to have a large focus on languages where
data and native speakers are easily accessible, and the language is relatively
well-known. These are referred to as high-resource languages, and as such,
produce a large quantity of data. On the other hand, low-resource languages
(occasionally referred to as LRLs) are usually “..less studied, resource
scarce, less computerized, less privileged, less commonly taught, or low
density..” [8] and therefore are not prioritized in the realm of NLP research.
However, this is actually one of the more major motivating factors for our
project. Chukchi is an incredibly low-resource language, an example of which
is that most of the up-to-date information regarding the language and its
speakers is most easily accessed from a detailed article found on
Wikipedia111Link to the article in question:
https://en.wikipedia.org/wiki/Chukchi_language#:~:text=Chukchi%20%2F%CB%88t%CA%83%CA%8Ak,mainly%20in%20Chukotka%20Autonomous%20Okrug.&text=In%20the%20UNESCO%20Red%20Book,the%20list%20of%20endangered%20languages.
The low-resourcedness of Chukchi is what inspired this project, as it is an
endangered language, and one that is not particularly accessible in terms of
media, education, and history. By creating a new automatic speech recognition
system, not only can accessibility be provided for this language, but it also
creates new opportunities for the same achievements in other low-resource
languages.

####  2.2.1 What is an ASR System?

Traditionally, modern automatic speech recognition systems are typically made
up of three different parts: a lexicon, an acoustic model, and a language
model222Information about the basic components of an automatic speech
recognition system is widely available, one of the more easily understandable
sources can be found here: https://voximplant.com/blog/what-is-automatic-
speech-recognition. The lexicon contains the information that an ASR system
needs to be able to understand the input it receives on the base level. This
includes things such as phonetic transcription codes that are used for the
target language’s phonemes. For English,
ARPABET333https://en.wikipedia.org/wiki/ARPABET and
TIMIT444https://en.wikipedia.org/wiki/TIMIT are the most commonly used codes
and transcriptions, developed by the Defense Advanced Research Projects Agency
(DARPA).

The second component of an ASR system is the acoustic model, which is
responsible for forging the relationships between the phonemes of a language
(such as the ones provided in the lexicon) and an audio signal. This
interaction is supported by the use of transcripts along with their respective
audio files555Microsoft conducts extensive research regarding acoustic models,
more information as well as links to other sources and publications can be
found here: https://www.microsoft.com/en-us/research/project/acoustic-
modeling/, and are thus supposed to be able to map statistical representations
for feature vector sequences of a particular phoneme (or sound unit) and
classify it [11]. This allows the system to recognize and distinguish this
particular sound unit from the rest of the phonemes that it may encounter in
both training data and experimental data.

Finally, there is the language model, which helps to provide clearer contexts
and allows the model to view the language in a naturally occurring form. Thus,
this is where training comes in. By training the language model, contexts
become more comprehensive and coherent when interacting with the system, and
are thus understandable. By design, the system, with the help of all of these
aforementioned components, is then supposed to be able to predict speech
patterns.

###  2.3 Previous Research

Regarding previous research focused on low-resource languages and ASR, there
have been multiple approaches to finding the most efficient and effective
model for processing such a limited amount of available data. The basic
framework for processing speech was typically based on a few components. For
example, these components could have included an autoencoder (denoising or
otherwise), dual transformation for both text and speech, bidirectional
sequence modeling, typically with a major focus on unsupervised pre-training
[10]. In addition to this, many approaches also included a Transformer-based
unified model structure [10], [7]. The goal was to have a system that could
sample the language evenly and return feedback to the model, learning as it
continued to sample more data [13]. These components are crucial to the
creation of our model, and will be utilized in this project.

Both universities and major corporations alike (e.g., Google with Strope et
al, 2011) have also researched the most effective ways to implement the most
ideal features for training both acoustic models as well as language models.
In many cases, it is incredibly difficult to create a pre-training environment
that is entirely unsupervised, but the key here is that it is almost
unsupervised. The benefit of unsupervised data pre-training is that it makes
data much more usable. Without the need for supervision, the amount of usable
data increases significantly, which gives us much more accessibility to
languages that lack a significant amount of data (i.e., being able to utilize
data in a more efficient way). The key to much of the unsupervised training
that already exists is the technique implemented. Discriminative, in which a
dual unigram and trigram language model was used to interpret relative truth,
active or passive. Passive learning was a technique and algorithm that
dominated the realm of automatic speech recognition for much of its lifespan.

Passive learning was the initial algorithm used for training language models.
This meant that a model was trained based on a single implementation of a set
of data, fixed in time. As a result, there was no room allowed for the model
to improve. Additionally, all the data being used was usually transcribed
under human supervision, and training a model to work with language data ended
up being a very time consuming process. By taking the workload off of the
researchers and volunteers who transcribe this data and manually check the
model, more effective and efficient means of training an ASR system can be
developed. Active learning mechanisms, as a result, are particularly useful in
cases like these. With a feedback system that allows for the model to learn
from itself and ultimately use less data. This can prove invaluable in
developing an ASR system for Chukchi and other low-resource languages.

##  3 DATA COLLECTION

Given that Chukchi is a very low-resource language with very few speakers,
finding usable data proved difficult, as was discussed above. Samples of both
spoken and written Chukchi were selected from any source that could be found.
This included the Charles Weinstein website of Chukchi with translations and
descriptions in both French and Russian, recent news broadcasts in Chukchi
(December 2020 and January 2021) from the Anadyr’-based radio station Radio
“Purga”, videos from Youtube, corpora from Chuklang.ru, as well as translated
parts of the Bible from Bible.is.

###  3.1 Radio ‘Purga’

One of the main sources of high-quality annotated data was the Chukchi radio
station Radio “Purga,” which has a special feature at their station in which
they report news in Chukchi on a regular (almost daily) basis. A
representative of this radio station provided our research team with a total
of 2.53 hours of audio data from 30 episodes of morning news. These audio
files then had to be manually split into shorter chunks of both audio
recording and text pairs in order to be used further. Fortunately, each
broadcast came with its own script. However, there were a few issues with this
data in that the real recording and the script would sometimes differ.
Additionally, the script also contained several sentences of pure Russian
speech, which makes some part of the audio file unusable. Both of these issues
were solved by excluding such recording-script pairs from the dataset.

###  3.2 YouTube Videos

Youtube was another primary resource to find any instances of Chukchi audio
samples. All videos found were then converted into WAV format. This portion of
the corpus contains:

  * •

stories;

  * •

Chukchi online language lessons;

  * •

interviews with native speakers;

  * •

lessons from the project “Vetgav. Chuckhi lessons”;

  * •

cartoons;

  * •

news;

  * •

and more.

All links to the video data used for this project and statistical sources can
be found at the project’s GitHub666Project GitHub:
https://github.com/ftyers/fieldasr/blob/main/DATA.md. The video data of the
corpus totaled at 14 hours, 12 minutes and 13 seconds.

###  3.3 Bible Audio

The resource Bible.is777Bible.is: https://www.faithcomesbyhearing.com/audio-
bible-resources/bible-is contains chapters from the Bible in a variety of
languages from around the world, including Chukchi. There we found audio
recordings totaled at 3 hours, 36 minutes and 52 seconds in wav format. Some
Bible chapters do have a text annotation (for example, The Gospel of Luke),
and some don’t (The Book of Jonah).

###  3.4 Chuklang Corpora

The Chuklang Corpora888Chuklang corpora: https://chuklang.ru was created by
professors and students of the National Research University - Higher School of
Economics in Moscow, Russia, during the linguistic expeditions to the village
of Amguema in the Iultinsky District of the Chukotka Autonomous Okrug. It
consists of annotated audio recordings with a total length of 1 hour, 14
minutes and 18 seconds.

###  3.5 All audio data

Table 1 displays information about the duration of all audio data, broken down
by resource type.

Table 1: Distribution of audio Resource |  Duration |  Transcription availability
---|---|---
Radio ‘Purga’ |  2:32:00 |  +
Youtube |  14:11:13 |  -
Bible |  3:36:52 |  -
Chuklang |  1:14:18 |  +

Duration of all unannotated data - 17:48:05
Duration of all annotated data - 3:46:18
Total duration of all audio data - 21:34:23

###  3.6 Text

For a linguistic model in an automatic speech recognition system, any kind of
texts are useful, even if they do not have their own audio annotations. The
most commonly used lexical items (ынкъам, гивик, ымы, ытлён, ынан, Чукоткакэн,
гатвален, ынӄэн, лыги, вальыт, вагыргыттитэ, гатваленат) were used to assist
in finding written examples of Chukchi. In addition to this,
Yandex.XML999Yandex.XML: https://yandex.ru/dev/xml/, which is a service that
allows a user to send a search query to the Yandex search engine and receive
any answers found in XML format, was also used. No more than 200 results per
query were allowed per its own limitations, however, by using various filters
and sorting mechanisms, 1800 URLs were found and 462 unique links were
extracted. From these selected URLs, the largest and most useful sources for
parsing were determined to be:

  * •

news outlets

  * •

stories and riddles

  * •

fictional literature

  * •

grammar and thematic dictionary of the Chukchi language, a collection of
Chukchi literary texts

The full text corpus is composed of 112,719 sentences, and 2,068,273 words.
The obtained URL links and text corpus can be found on the GitHub repository
for this project101010Project GitHub:
https://github.com/ftyers/fieldasr/blob/main/DATA.md. Table 2 below displays
the distribution of texts according to subcorpus. From these numbers, the
necessity for automatic deletion of Russian from a few sub corpora becomes
quite noticeable.

Table 2: Distribution of texts by subcorpus (before the removal of the Russian language) Subcorpus |  Number of texts/ pages |  Number of sentences |  Number of words |  % of words |  Russian language presence
---|---|---|---|---|---
Internet newspaper "Extreme North" |  118 |  6187 |  82661 |  4,00% |  +
A special supplement to the newspaper "Extreme North" |  11 |  303 |  27466 |  1,33% |  +
"Portal of National Literatures" |  4 |  569 |  4729 |  0,23% |  -
The book "По аргишному пути канчаланского чаучу" |  1 |  444 |  3936 |  0,19% |  +
Lingvoforum: fairy tales, stories |  8 |  251 |  1297 |  0,06% |  -
VK: fairy tales |  11 |  573 |  2840 |  0,14% |  -
Puzzles |  1 |  1234 |  8112 |  0,39% |  +
The Wayback Machine |  47 |  3079 |  16952 |  0,82% |  -
Charles Weinstein |  10 |  96765 |  1891705 |  91,46% |  +
Radio ‘Purga’ |  29 |  836 |  7742 |  0,37% |  +
Chuklang Corpora |  1006 |  1006 |  4414 |  0,21% |  -
Bible.is |  25 |  1472 |  16419 |  0,79% |  -
All data |  1271 |  112719 |  2068273 |  100% |

##  4 DATA PROCESSING

###  4.1 Preprocessing: Text

The Chukchi alphabet contains a special symbol (‘) which was implemented in
different forms in writing (‘, ’, ‘, “, ”, etc) across different sources of
Chukchi. Therefore, the first step for preprocessing was to identify all
instances of this apostrophe and determine a single variant for its
representation, with the result being: (’). Following this were substitutions
of letters к’ - ӄ, K’- Ӄ, н’ - c, Н’ - Ӈ, where the rightmost form is the
final representation needed.

Additionally, we cleaned up the data collected from internet-sources on a more
global scale: that is, deletion of hyperlinks, illegible symbols, email
addresses (these can frequently be found in news articles), and occasionally,
double spaces between words. All data preprocessing was carried out with the
help of the standard Python library re111111Documentation on the Python
library re can be found here: https://docs.python.org/3/library/re.html, which
is used for work with regular expressions.

###  4.2 Removing Russian from the Text Corpus

Due to the writing system of both Chukchi and Russian being incredibly
similar, it became necessary to clean the Russian out of the text corpus. This
is because the similarities between the visible forms of the languages can be
problematic to the model, and thus skew our results. Binary sentence
classification with the help of the most frequently used words in both Russian
and Chukchi. Based on the collected text data that only contains Chukchi, we
calculated the most frequently occurring words. The “New Russian Lexical
Frequency Dictionary”121212New Russian Lexical Frequency Dictionary:
http://dict.ruslang.ru/freq.php was used to create a list of the most
frequently occurring Russian words. In order to predict which class a sentence
belonged to, each word in the text, along with its normal form determined by
the morphological analyzer pymorphy2131313Documentation for pymorphy2:
https://pymorphy2.readthedocs.io/en/stable/, were checked for entries in the
frequency list. The corresponding variable-counters were increased and then
compared. If a Russian word was found more than a Chukchi word, then that
sentence was considered Russian, and vice versa.

The pre-training model “lid.176.bin” was used for fastText141414Documentation
for fastText: https://fasttext.cc/ for language recognition. This model is
supported for 176 languages. With the assistance of this model, the
likelihoods of sentences belonging to certain languages were isolated,
followed by the use of the model KMeans151515Documentation for KMeans:
https://pymorphy2.readthedocs.io/en/stable/ for segmenting data into 2
clusters: Chukchi and not-Chukchi. For the given model the f1-score value was
0.91. After deletion of Russian from the corpus, the size of the resulting
corpus was 15309 sentences and 117567 words. This corpus can be found on the
GitHub repository for this project161616Project GitHub:
https://github.com/ftyers/fieldasr/blob/main/data/text_corpus.txt.

Table 3: Distribution of texts by subcorpus (after the removal of the Russian language Subcorpus |  Number of texts/ pages |  Number of sentences |  Number of words |  % of words
---|---|---|---|---
Internet newspaper "Extreme North" |  118 |  3063 |  33219 |  28,26%
A special supplement to the newspaper "Extreme North" |  11 |  3236 |  25160 |  21,40%
"Portal of National Literatures" |  4 |  574 |  4732 |  4,02%
The book "По аргишному пути канчаланского чаучу" |  1 |  208 |  1425 |  1,21%
Lingvoforum: fairy tales, stories |  8 |  304 |  1297 |  1,10%
VK: fairy tales |  11 |  755 |  2840 |  2,42%
Puzzles |  1 |  358 |  1511 |  1,29%
The Wayback Machine |  47 |  3127 |  16904 |  14,38%
Charles Weinstein |  10 |  332 |  1989 |  1,69%
Radio ‘Purga’ |  29 |  863 |  7654 |  6,51%
Chuklang Corpora |  1006 |  1006 |  4416 |  3,76%
Bible.is |  25 |  1483 |  16420 |  13,97%
All data |  1271 |  15309 |  117567 |  100%

###  4.3 Attempts at Denoising

Due to Chukchi being a very low-resource language, the amount of high quality
audio recordings in our dataset were minimal. Most of the audio that dominated
the collected data were those of fairly poor quality. In order to achieve
better results with the data on hand, a denoising autoencoder was applied to
the data to filter out extra noise.

However, either due to flaws in denoising methodology or due to the high level
of noise pollution in the collected audio data, the application of a denoising
autoencoder was fruitless, and did not appear to aid our system. In the
future, we would like to find something that will clean our audio in a more
efficient and effective way.

###  4.4 Segmenting and Labeling Radio Data

The samples from Radio “Purga” were collected for the months of December 2020
and January 2021. These files typically consisted of a short news segment,
usually no longer than 5 minutes each. With about 15 of these samples
collected for each month, we had 200 minutes of audio. However, these audio
files needed to be segmented, as it was unfortunately not all Chukchi. These
audio recordings were segmented by sentences, and labeled using the software
ELAN171717ELAN is a linguistic annotator that allows for work with audio
files, including segmentation, and labeling.. Any instances of Russian were to
be extracted and removed from the files. In addition to this, the audio
broadcasts were also provided with personal scripts from the speaker’s notes.
As a result, we were able to take these scripts, and segment the audio by
sentences, which was both necessary for processing data and training the
model. However, some of these scripts appeared to be used more as notes as
opposed to actual scripts, with some portions of the broadcast entirely
missing from the documents we had been provided with. In these cases, the
audio was not usable, and was removed from the dataset.

###  4.5 Audio Data Slicing

All received audio was cut into small fragments. Sometimes it was possible to
cut by pauses, so as not to cut off parts of words that may be important for
ASR model training. In this case, the Python libraries
pyAudioAnalysis181818https://github.com/tyiannak/pyAudioAnalysis and
pydub191919https://github.com/jiaaro/pydub were used. Sometimes it was
impossible to cut by pauses (for example, if there was music playing in the
background), if this was the case, then the files were cut by 5 minutes.

##  5 METHODOLOGY

This project was centered around 3 experiments with 3 different models: VQ-
VAE, XLSR, and wav2vec. These models were selected specifically for their
previously displayed use for low-resource languages. Each model was pre-
trained with a portion of the cleaned, segmented, annotated and unannotated
Chukchi data, and then tested with the remaining portion.

###  5.1 VQ-VAE

####  5.1.1 Theory

One approach used in creation of ASR systems for low-resource languages is to
use a model that can learn from latent representations in language. For this
task the best option would be to use VQ-VAE [9]. This model has two important
components: VAE (Variational Auto-Encoders) [6] and VQ (Vector Quantized). VAE
is a type of NN architecture which does inference by retrieving statistics
(mean and variance) for latent random variables. Model learns separate iand
ifor a random variable. VAE works for continuous data, а вот VQ-VAE learns
discrete representations (embeddings). It is assumed that the model VQ-VAE
will learn embeddings by constructing its own inputs. Therefore, only audio
data is required for learning. Thus, if the learned latent representations are
learned well, then they can be used for unsupervised or semi-supervised
models.

Figure 1: Left: A figure describing the VQ-VAE. Right: Visualisation of the
embedding space. (Oord, Vinyals, et al. 2017)

####  5.1.2 Experiment

Two experiments were carried out for the collected audio data: Chuklang,
Bible.is, Radio “Purga,” and YouTube. The data corpus used in the first
experiment contained audio with a length of 4-5 minutes (as a result a large
part of this audio was cut down to 5-20 seconds), therefore a decision was
made to conduct a second experiment in which the audio that had lengths of
longer than 20 seconds were converted into smaller audio fragments, segmented
by pauses. This was done as the length of the largest piece of audio in the
dataset influences the padding parameter value of the model: the model can
have a significant bias in the data, as it will fill all the audio-recording
vectors with zeros to fit the size of the largest audio piece.

The second experiment that was carried out focused on the use of data
augmentation. We attempted to apply various effects from the first experiment
to the audiodata, thereby doubling the size of the training dataset. Regarding
effects, the following were used:

  * •

single-pole lowpass filter (["lowpass", "-1", "300"]),

  * •

reduce the speed (["speed", "0.8"]),

  * •

after that it was necessary to add the ‘rate‘ effect with the original sample
rate (["rate", f"{sample_rate1}"]),

  * •

reverberation (["reverb" "w"]).

In the present work we used the model VQ-VAE for PyTorch. The following
parameters were used for the VQ-VAE model:

  * •

input_dim=39,

  * •

hid_dim=256,

  * •

enc_dim=64,

  * •

K=512.

The VQ-VAE model was trained for 1000 epochs. The following values were
selected as the batch-size for the data: 128 (train), 10 (validation), 16
(test). PyTorch’s Adam Optimizer was also implemented with a learning rate of
2e-4.

####  5.1.3 Results

The validation loss function was used in order to rate the quality of the VQ-
VAE model. The results from the change in the function can be observed on
Table 4.

Table 4: Validation loss values for experiments with VQ-VAE Experiment |  Validation loss (1 epoch) |  Validation loss (1000 epoch)
---|---|---
Experiment 1.1 (without augmentation) |  3249.4058 |  3132.565
Experiment 1.2 (with augmentation) |  3247.8926 |  3131.7913
Experiment 2 |  3249.5176 |  3128.0374

It is visible from this Table 4 that even after 1000 epochs, the value of the
validation loss was left practically unchanged. As a result, we have
determined that this model was unsuccessful for our research purposes due to 2
main reasons: the small amount of data available for training, and the poor
sound-quality of these audio files. As mentioned previously, the denoising
autoencoder proved ineffective, resulting in audio that still contained a
significant amount of noise, likely interfering with the model.

###  5.2 XLSR

####  5.2.1 Theory

Another approach to this task is using a model trained on a high-resource
language and fine-tuning it via low-resource language data, in our case,
Chukchi. A model that is suitable for such a cross-language approach is
referred to as XLSR or Cross-Lingual Speech Representations. XLSR is a
transformer-based multilingual model that was trained on 56 thousand hours of
speech data shared between 53 languages. It is supposed to represent latent
features that are shared between languages. This means that XLSR can be used
as a solid base for fine-tuning for low-resource languages. Therefore,
Chukchi, being extremely low-resource, is a great candidate for this
experiment. For a broader description of the model and this approach we
suggest addressing the original article by Facebook AI team [4].

We hypothesized that through using a model that has learned generalized
natural language representation, it will be easier to train a model that
recognizes Chukchi speech - basically, a big part of the training process is
skipped, and the whole training process is essentially condensed to fine-
tuning. Moreover, the authors of the mentioned article have proven that
multilingual models may even outperform monolingual ones for some languages.

Figure 2: A shared quantization module over feature encoder representations
produces multilingual embeddings, through which the model learns to share
discrete tokens across languages, creating bridges across languages [4]

####  5.2.2 Experiment

After the necessary initial steps, such as feature extraction, normalization,
sampling rate adjustment, etc., the model was further trained on the annotated
Chukchi data at hand. The CNN part from Figure 2 in section 5.2.1 does not
need fine-tuning, because it was trained by the authors of the model where the
multilingual features are encoded. Moreover, the fine-tuning process is
relatively short, and only lasts for 6 epochs, 400 steps each, because the
model has shown to be prone to overfitting on such a small dataset. The
training rate was set to a static value of 3e-4, which fits in the range
suggested by the authors of the original paper. It seems to make little sense
to change learning rate dynamically, given the low amount of training data.

The model will be publicly available at our team’s GitHub
repository202020Project GitHub:
https://github.com/ftyers/fieldasr/blob/main/DATA.md.

####  5.2.3 Results

The data that was used for fine-tuning consisted of annotated data that had
the structure shown on Table 5.

Table 5: Training and Testing times for the XLSR model TRAIN |  02:49:36.35
---|---
TEST |  00:17:09.25
TOTAL |  03:06:45.00

This allowed the fine-tuned model to achieve a WER212121WER is a metric that
is commonly used for speech modelling and speech recognition. It is computed
at word-level as (Deletions + Insertions + Substitutions) / N, where N is the
number of words in a text, and the three components of the dividend are the
numbers of corresponding operations, applied to words, that are needed to
reconstruct the original text. of 0.758395 and CER222222CER is computed the
same way as WER, but on character level. That means that the same formula is
used and the same operations are counted, but in appliance to characters in
the original text. 0.186895 - surprisingly low for such a small dataset. We
plan to prepare more data and further improve this result, as it has appeared
to be the most fruitful approach among others. Additionally, it appears to be
useful to compare these metrics with the results achieved in the original
article (see Table 6).

Table 6: Word Error Rate Language |  Assamese |  Tagalog |  Swahili |  Georgian |  Our: Chukchi
---|---|---|---|---|---
WER |  44.1 |  33.2 |  26.5 |  31.1 |  75.8
Fine-tuning data, hrs |  55 |  76 |  30 |  46 |  3

WER proved to not be the best metric for training in Chukchi, as it accounts
for any mistake in a word. Considering the nature of this language is
polysynthetic (e.g. it contains long “sentence-words”), it is easy to achieve
a high error rate due to a large number of substitutions. CER is, in our
opinion, a more suitable metric in these circumstances. Expectedly, in
comparison with the results in the Table 7 above, our model does not perform
satisfactorily on the word-level.

Table 7: Character Error Rate Language |  Assamese |  Tagalog |  Swahili |  Lao |  Ours: Chukchi
---|---|---|---|---|---
CER |  17.9 |  13.1 |  21.3 |  22.4 |  18.7
Fine-tuning data, hrs |  55 |  76 |  30 |  59 |  3

Character error rate accounts for character-level mistakes. Our model has
achieved a comparable result, given much less data. This result is very
satisfying, however, low WER definitely means that this model is not suitable
for production use. However, if one takes into consideration the morphological
complexity of Chukchi and its incredibly scarce data, one can conclude that
XLSR has proven to be an astoundingly powerful base for low-resource ASR.

As a demonstration, we would like to finish this part of the article with an
example of the model’s output on Table 8

Table 8: Output from the XLSR model Original |  Recognised
---|---
ӈутингивик ымыӆьычукоткак ӈыраӄ аӄатвагыргын гатваленат яама нымытваӆьа милгэрти ыннэнчьэн о’равэтӆьан егтэлытваркын ӄутти ӈыроӄ гэвъилинэт |  ӈутингивик м чукуткак ӈыаӄалтвагыргыт гатваленат яма нымытваӆьа милгэри ынэчьэнноравэтлан егтэлвркын отиӈыргэвилиэт

###  5.3 Wav2Vec Unsupervised

####  5.3.1 Theory

Another model suitable for our task, wav2vec Unsupervised (wav2vec-U), was
presented in May 2021 by Facebook AI [1]. The main feature of this model is
that only separate audio and texts that are not related to each other are
needed (not annotated ones), which makes it possible to apply it to low-
resource languages that lack text and audio pairs. A good result (about 85%
for low-resource languages, such as Tatar) is obtained due to the self-
supervised model (wav2vec 2.0), a simple k-means clustering method and a
Generative Adversarial Network (GAN). The illustration of wav2vec-U can be
found in Figure 3. The wav2vec-U training procedure consists of three main
steps:

  * •

Preparation of speech representations and text data;

  * •

Generative Adversarial Training (GAN);

  * •

Iterative self-training + Kaldi LM-decoding.

Figure 3: Illustration of wav2vec Unsupervised (Baevski, et al. 2021)

####  5.3.2 Experiment

As mentioned above, using the model requires specific preprocessing of the
data, which requires wav2vec 2.0 (open-source framework for self-supervised
learning of representations from raw audio data), eSpeak (a compact open
source software speech synthesizer for English and other languages) and
fastText (a library for efficient learning of word representations and
sentence classification) pre-trained models for Chukchi, which do not exist.
It is possible to train these models from scratch, however, it takes a lot of
time, and at the moment of writing this article it was not possible to
preprocess the data properly for the wav2vec-U model.

####  5.3.3 Results

No results for wav2vec-U have been obtained at this time, this will be a task
for the next study.

##  6 CONCLUSION

In this work we introduced our project, focused on creating a new automatic
speech recognition (ASR) system for low-resource languages, with a focus on
Chukchi. Our goal was to create a system as unsupervised as possible. In order
to run experiments and train our selected models to work for Chukchi, we
collected a sizable corpus of both audio and text data in Chukchi, a feat that
was rather unexpected for such a low-resource language. In total, there were
15,309 sentences, and 117,567 words for the full text corpus, and 21 hours, 34
minutes, 23 seconds worth of audio data. We then proceeded to conduct 3
different experiments centered around VQ-VAE, XLSR, and wav2vec models.

The first experiment, regarding Vector Quantized - Variational Auto-Encoder
(VQ-VAE), demonstrated very little change in validation loss between the first
and thousandth epoch that had been executed. This result was found in all
three sub-experiments that were run with both augmented and unaugmented data.
This minimal change is likely a result of how little data we had for this
model, as well as the poor sound quality of the audio files. The second
experiment, focused on XLSR, was shown to be more promising than the first,
with a surprisingly low WER of 0.758395 and CER 0.186895. XLSR actually proved
to be the most powerful base for development in low-resource ASR. These
results on their own may not appear particularly noteworthy, however, we must
take into account the fact that Chukchi is a polysynthetic language, meaning
that words and sentences tend to have more of an overlapping appearance when
manifested in written language. With this in mind, our interpretation of the
CER result becomes much more significant. Finally, the third experiment
conducted through wav2vec, unfortunately, proved fruitless. There are no
results to interpret at this time, as preprocessing of the data was
unsuccessful. This experiment will be pursued in further studies.

In summary, we were able to train 2 separate models for work in Chukchi, and
will be attempting to train the third in the near future. Despite some
unfortunate setbacks, we have made an incredible amount of progress in our
understanding of low-resource automatic speech recognition, have learned what
works for this particular set of data and what proves ineffective, and set
tasks that can be completed in future research. Ultimately, we consider this
to be a successful project with a promising outlook on the future of low-
resource automatic speech recognition.

## References

  * [1] Baevski, Alexei, et al..  Unsupervised Speech Recognition.  arXiv preprint arXiv:2105.11084, 24 May 2021.
  * [2] Baevski, Alexei, et al..  wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations.  arXiv preprint arXiv:2006.11477v3, 22 Oct. 2020.
  * [3] Chorowski, Jan, et al.  Unsupervised Speech Representation Learning Using WaveNet Autoencoders.  In IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 27, no. 12, pp. 2041–2053, 2019.
  * [4] Conneau, Alexis, et al.  Unsupervised Cross-Lingual Representation Learning for Speech Recognition.  arXiv preprint arXiv:2006.13979v2, 15 Dec. 2020.
  * [5] Fortescue, Michael.  Comparative Chukotko-Kamchatkan Dictionary.  In Mouton De Gruyter, 2005.
  * [6] Kingma, Diederik P, and Max Welling.  Auto-Encoding Variational Bayes.  arXiv preprint arXiv:1312.6114v5, 9 Jan. 2014.
  * [7] Kriman, Samuel, et al.  Quartznet: Deep Automatic Speech Recognition with 1D Time-Channel Separable Convolutions.  In ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2020.
  * [8] Magueresse, Alexandre, et al.  Low-Resource Languages: A Review of Past Work and Future Challenges.  arXiv preprint arXiv:2006.07264v1, 12 June 2020.
  * [9] Oord, Aaron van den, et al.  Neural Discrete Representation Learning.  arXiv preprint arXiv:1711.00937v2, 30 May 2018.
  * [10] Ren, Yi, et al.  Almost Unsupervised Text to Speech and Automatic Speech Recognition.  arXiv preprint arXiv:1905.06791, 26 July 2020.
  * [11] Sarma, Mousmita, and Kandarpa Kumar Sarma.  Acoustic Modeling of Speech Signal Using Artificial Neural Network.  In Advances in Systems Analysis, Software Engineering, and High Performance Computing, pp. 282–299, 2015.
  * [12] Schneider, Steffen, et al.  wav2vec: Unsupervised Pre-Training for Speech Recognition.  arXiv preprint arXiv:1904.05862v3, 2 July 2019.
  * [13] Xiao, Yubei, et al.  Adversarial Meta Sampling for Multilingual Low-Resource Speech Recognition.  arXiv preprint arXiv:2012.11896, 12 Apr. 2021.


# Assessing the efficacy of large language models in generating accurate
teacher responses

Yann Hicke, Abhishek Masand, Wentao Guo, Tushaar Gangavarapu
Cornell University

###### Abstract

Tack et al. (2023) organized the shared task hosted by the 18th Workshop on
Innovative Use of NLP for Building Educational Applications on generation of
teacher language in educational dialogues. Following the structure of the
shared task, in this study, we attempt to assess the generative abilities of
large language models in providing informative and helpful insights to
students, thereby simulating the role of a knowledgeable teacher. To this end,
we present an extensive evaluation of several benchmarking generative models,
including GPT-4 (few-shot, in-context learning), fine-tuned GPT-2, and fine-
tuned DialoGPT. Additionally, to optimize for pedagogical quality, we fine-
tuned the Flan-T5 model using reinforcement learning. Our experimental
findings on the Teacher-Student Chatroom Corpus subset indicate the efficacy
of GPT-4 over other fine-tuned models, measured using BERTScore and DialogRPT.

We hypothesize that several dataset characteristics, including sampling,
representativeness, and dialog completeness, pose significant challenges to
fine-tuning, thus contributing to the poor generalizability of the fine-tuned
models. Finally, we note the need for these generative models to be evaluated
with a metric that relies not only on dialog coherence and matched language
modeling distribution but also on the model’s ability to showcase pedagogical
skills.

##  1 Introduction

The advent of powerful open-source generative language models such as GPT-2
Radford et al. (2019), T5 Raffel et al. (2020), OPT Zhang et al. (2022), BLOOM
Scao et al. (2022), Flan-T5 Chung et al. (2022) or LLAMA Touvron et al. (2023)
has led to significant developments in conversational agents, opening avenues
for various applications in education Wollny et al. (2021). Such AI-driven
educational dialogues offer the potential for skill improvement and
personalized learning experiences, with intelligent tutoring systems
increasingly gaining traction Bibauw et al. (2022). However, deploying AI-
based teachers in the educational ecosystem demands the careful modeling and
evaluation of these agents to ensure their capability to address critical
pedagogical concerns.
Tack and Piech (2022) created the AI teacher test challenge which follows the
recommendations from Bommasani et al. (2021) (pp. 67-72) stating that, if we
want to put generative models into practice as AI teachers, it is imperative
to determine whether they can (a) speak to students like a teacher, (b)
understand students, and (c) help students improve their understanding.
Taking inspiration from the AI teacher test challenge which asks whether
state-of-the-art generative models are good AI teachers, capable of replying
to a student in an educational dialogue this paper seeks to investigate the
applicability of reinforcement learning (RL) techniques in the generation of
AI teacher responses within educational dialogues. The AI teacher test
challenge emphasizes the need for a systematic evaluation of generative models
to ensure that they can effectively communicate with students, comprehend
their needs, and facilitate their academic improvement. Can we guide the
language generator with RL to help it focus on these pedagogical requirements?
Tack et al. (2023) organized the shared task hosted by the 18th Workshop on
Innovative Use of NLP for Building Educational Applications on generation of
teacher language in educational dialogues. Following the structure of the
shared task, in this study, we aim to evaluate the potential of combining
state-of-the-art generative language models with reinforcement learning
algorithms to generate AI teacher responses in the context of real-world
educational dialogues sourced from the Teacher Student Chatroom Corpus Caines
et al. (2020, 2022). The natural baselines for the task at hand are SOTA
closed-source models such as GPT-4, and fine-tuned open-source pre-trained
models such as GPT-2 Radford et al. (2019). We will evaluate these natural
baselines before evaluating fine-tuned pre-trained models using RL techniques,
that optimize for pedagogical quality.
By exploring the role of reinforcement learning in guiding the generation of
AI teacher responses, we aim to advance the discourse on the utilization of
conversational agents in educational settings and contribute innovative ideas
to the ongoing shared task on the generation of teacher language in
educational dialogues at the 18th Workshop on Innovative Use of NLP for
Building Educational Applications.
The rest of this paper is structured as follows. Section 2 offers a
comprehensive review of relevant literature in the areas of AI-driven
educational dialogues and reinforcement learning-based language generation.
Section 3 discusses the analysis and processing of the dataset prior to
conducting any language modeling tasks. In Section 4, the proposed model and
its methodology for generating AI teacher responses in educational
interactions are introduced. Section 5 evaluates the effects of our approach
on the quality and relevance of the generated AI teacher responses and
highlights key observations. Finally, Section 6 concludes the paper and
explores potential directions for future research.

##  2 Related Work

A variety of related literature exists in the realm of conversational teaching
between a student and a teacher. In this section, we review several notable
works addressing aspects of teacher-student dialogues, foundation models, and
conversational datasets, which have contributed to the progress and
understanding of generative models in educational contexts.

Teacher-Student Dialogues
One prominent resource in educational dialogues is the National Council of
Teachers of English (NCTE) dataset Kane et al. (2022). It includes numerous
examples of teacher-student interactions, which can serve as a valuable
resource for the training and evaluation of generative models in an
educational context.
The SimTeacher dataset Cohen et al. (2020) is an assemblage of information
obtained through a "mixed-reality" simulation platform. This unique
environment aids beginner educators in honing essential skills for classroom
settings by employing student avatars managed by human actors. All aspiring
teachers from a prominent public university participate in several brief
simulation sessions throughout their educational preparation program, focusing
on improving their ability to encourage more profound textual understanding
among students. The original researchers annotated a variable called "quality
of feedback" within the transcript, determining how effectively teachers
proactively assist students.
In Chen et al. (2019), we can find a dataset collected from an education
technology company that provides on-demand text-based tutoring for math and
science. With a mobile application, a student can take a picture of a problem
or write it down and is then connected to a professional tutor who guides the
student to solve the problem. The dataset represents, after some selection,
108 tutors and 1821 students. Each session is associated with two outcome
measures: (1) student satisfaction scores (1-5 scale) and (2) a rating by the
tutor manager based on an evaluation rubric (0-1 scale).

Foundation Models
Bommasani et al. (2021) provided a comprehensive analysis of the opportunities
and risks of foundation models, including insights into their use in
educational applications. They identified potential benefits, such as
personalized learning and accessibility, while also highlighting the major
risks, such as unfair biases and the generation of harmful content. This work
establishes the need for carefully crafted benchmarks and evaluations to
assess the potential of generative models in education.
The AI Teacher Test Tack and Piech (2022) builds on this idea by examining the
performance of generative models such as GPT3 Brown et al. (2020) and Blender
Roller et al. (2020) in generating appropriate and informative responses in a
teacher-student dialogue.
Kasneci et al. Kasneci et al. (2023) conducted an investigation to understand
the effectiveness of ChatGPT Team (2022) as a tool for educational support.
They analyzed the model’s performance in a student-tutoring context, examining
its ability to provide accurate, relevant, and engaging responses for
learners. By identifying the strengths and weaknesses of ChatGPT in this
specific setting, they contributed to a better understanding of how generative
models can be successfully deployed in educational applications. Our work
builds on these foundations by evaluating the potential of combining
reinforcement learning with generative models to enhance the performance of AI
teacher agents in educational dialogues.

Conversational Uptake
Collins (1982) introduced the concept of uptake as a way to comprehend the
effectiveness of conversational responses in a teacher-student dialogue. It
laid the groundwork for the evaluation of generative models in dialogues by
taking into account the relevance and appropriateness of model-generated
responses.
Demszky et al. Demszky et al. (2021) further explored the concept of
Conversational Uptake by proposing metrics to assess the success of responses
in maintaining and advancing a conversation. By applying these metrics to AI-
generated responses, their work contributes to the evaluation of models in
realistic conversation settings, including teacher-student dialogues. Our work
attempts to guide the language generation process with similar goals in mind.
We hope to find proxies of pedagogical quality through NLP metrics such as
BERTScore combined with DialogRPT.

We continue by reviewing the literature utilizing reinforcement learning as a
guide for language generation.

Reinforcement Learning for language generation
Policy gradient-based algorithms and their variants have been widely used in
text generation to optimize sequence-level metrics Ranzato et al. (2015); Shen
et al. (2015); Norouzi et al. (2016); Pasunuru and Bansal (2018). Off-policy
Reinforcement Learning (RL) is also commonly used in dialogue applications
where online interaction with users is expensive Serban et al. (2017); Jaques
et al. (2019). The main difference in our work is that we take advantage of
demonstrations and design generic reward functions for generation tasks. We
extend this concept to educational contexts by employing reinforcement
learning to guide the generation of AI teacher responses in educational
dialogues. We focus on optimizing the responses of fine-tuned generative
models based on a reward system designed to enhance the pedagogical quality of
the generated responses. Recently, Ramamurthy et al. Ramamurthy et al. (2022)
explored the efficacy of using RL to optimize language models in several
natural language processing tasks, including text classification, sentiment
analysis, and language generation. They developed a library, RL4LMs, which
provides a generic framework for deploying RL-based language models for
various tasks. We build on top of the RL4LMs framework by adding a new task to
its existing array of tasks which we hope can be added as a standard for any
future RLHF benchmark.

##  3 Data

The shared task for BEA 2023 is based on the Teacher-Student Chatroom Corpus
(TSCC) Caines et al. (2020). This corpus comprises data collected from 102
chatrooms where English as a Second Language (ESL) teachers interact with
students to work on language exercises and assess the students’ English
language proficiency.

###  3.1 Data Extraction and Format

From each dialogue in the TSCC, several shorter passages were extracted. Each
passage is at most 100 tokens long, consisting of several sequential teacher-
student turns (i.e., the preceding dialogue context) and ending with a teacher
utterance (i.e., the reference response). These short passages are the data
samples used in this shared task.

The data samples are formatted using a JSON structure inspired by the ConvoKit
Chang et al. (2020). Each training sample is represented as a JSON object with
three fields:

  * •

id: a unique identifier for the sample.

  * •

utterances: a list of utterances corresponding to the preceding dialogue
context. Each utterance is a JSON object with a "text" field containing the
utterance and a "speaker" field containing a unique label for the speaker.

  * •

response: a reference response, which corresponds to the final teacher’s
utterance. This utterance is a JSON object with a "text" field containing the
utterance and a "speaker" field containing a unique label for the speaker.

Each test sample is represented as a JSON object that uses the same format as
the training sample but excludes the reference response. As a result, each
test sample has two fields:

  * •

id: a unique identifier for the sample.

  * •

utterances: a list of utterances, which corresponds to the preceding dialogue
context. Each utterance is a JSON object with a "text" field containing the
utterance and a "speaker" field containing a unique label for the speaker.

###  3.2 Data Distribution and Characteristics

The TSCC corpus is divided into three sets: train, dev, and test, each
comprising 2747, 305 and 273 of the samples, respectively. The corpus has 3325
samples, and each sample has an average length of 7.52 turns, with about 7.33
tokens per turn on average. Table 1 presents a summary of the statistics of
the TSCC corpus across the training, development, and testing sets.

Table 1: Summary of the statistics for the TSCC corpus across the train, dev, and test sets. Dataset |  Train |  Dev |  Test
---|---|---|---
Num Samples |  2747 |  305 |  273
Avg Turns |  7.7 |  7.92 |  5.23
Avg Tokens Per Turn |  7.29 |  7.21 |  8.27

The TSCC corpus exhibits several characteristics that are specific to
educational dialogues and pose challenges to natural language generation
models. For instance, the dialogues often include technical vocabulary and
idiomatic expressions related to English language learning. Additionally, the
dialogues can be highly varied in terms of topic, complexity, and participant
proficiency. Finally, the dialogues can include challenging responses which
are based on out-of-context information, posing challenges for conversational
agents. These characteristics must be taken into consideration when selecting
and evaluating generative models for the TSCC corpus.

###  3.3 Data Overlap and Challenges

It is worth noting that the released development and training sets in the TSCC
dataset have some overlaps, as individual conversation samples within these
sets have been generated by creating chunks from larger conversations. This
overlap may lead to potential biases and overfitting when training and
evaluating models on this dataset. However, the test set for the BEA 2023
shared task is free of overlaps, allowing for a more accurate assessment of
the model’s performance in generating AI teacher responses.

The presence of overlaps in the development and training sets posed a
challenge, as models inadvertently learned to predict teacher responses based
on the similarities between the samples rather than genuinely understanding
the context and dynamics of the teacher-student interaction. It is essential
to be aware of this issue and devise strategies to mitigate the risks
associated with such overlaps and ensure that the models are robust and
capable of handling diverse and unseen scenarios.

To ensure the validity of our model on the validation set, we employed an
iterative inclusion process to create a train-val split without any overlap
between them. This process involved carefully selecting and excluding samples
from the training set that had any similarity or overlap with the samples in
the development set. This approach aimed to minimize the risk of data leakage
and ensure that our model was evaluated on a truly unseen set of dialogues.

##  4 Methods

The primary objective of our study is to investigate the potential of using
in-context learning, supervised fine-tuning, and reinforcement learning to
generate AI teacher responses in educational dialogues. Our proposed methods
will be evaluated using the Teacher Student Chatroom Corpus (TSCC) dataset. In
this section, we provide an overview of the three main parts of our
methodology: in-context learning using GPT-4, supervised fine-tuning with
existing models such as GPT-2 and DialoGPT, and supervised fine-tuning with
Reinforcement Learning using the RL4LMs library Ramamurthy et al. (2022).

###  4.1 In-context Learning

####  4.1.1 GPT-4

As a preliminary step, we investigate the potential of in-context learning
using GPT-4, a state-of-the-art language model. It generates educational
dialogues based on its pre-trained knowledge, which has been acquired from a
vast corpus of text data during its training process (the pre-training data
might have included the test set; we will address this issue in the discussion
section).

To evaluate the performance of GPT-4, we prompted GPT-4 in a few-shot fashion.
We retrieved 5 most similar teacher-student conversations from the TSCC
dataset and provided them to the model in addition to the current conversation
and instructions about the model’s role as a teacher. Details about the prompt
construction that helps guide the model toward generating suitable responses
as a teacher can be found in the Appendix A.

###  4.2 Supervised Fine-tuning

To further adapt pre-trained language models to the specific educational
context and generate more accurate and context-aware teacher responses, we
explore supervised fine-tuning using GPT-2 and DialoGPT models.

####  4.2.1 GPT-2

GPT-2 Radford et al. (2019) is a decoder-only large language model pre-trained
on WebText, and we used GPT-2 Large, which has 24 transformer decoder blocks
with 774 million parameters.

We fine-tune the GPT-2 model Radford et al. (2019) using the Huggingface
Library on the Teacher Student Chatroom Corpus (TSCC) dataset. For each
educational dialogue, we concatenated all dialogue turns into a single string
with additional information of speaker roles i.e. students or teachers. As a
result, the input to the GPT-2 model consists of a sequence of text
representing the conversation history, culminating in the teacher’s response.
We then finetuned GPT-2 Large Radford et al. (2019) with a causal language
modeling task. Details of the exact hyperparameters used during the fine-
tuning process can be found in the Appendix.

After the fine-tuning process, we evaluated the fine-tuned GPT-2 model’s
performance on the test set by comparing its generated teacher responses to
reference responses, assessing the model’s ability to generate context-aware
and educationally relevant responses in line with the teacher’s role in the
TSCC dataset.

####  4.2.2 DialoGPT

DialoGPT Zhang et al. (2019) is a dialogue model based on the GPT-2
architecture, specifically designed for generating conversational responses.
DialoGPT is trained with 147M conversation pieces extracted from Reddit Zhang
et al. (2019), and it is trained with causal language modeling objectives with
multi-turn dialogue. We adapt our training dataset with the same format as
that of DialoGPT during pretraining and then prompt the DialoGPT to generate
an educational dialogue of teachers in the validation set. After training, we
follow the same methodology for evaluation as GPT-2 which we discussed in the
earlier section.

###  4.3 Supervised Fine-tuning with Reinforcement Learning

####  4.3.1 Flan-T5 Fine-tuned with RL4LMs

To optimize the generative models for pedagogical quality, we explore the use
of reinforcement learning techniques in the fine-tuning process. We employ the
RL4LMs library Ramamurthy et al. (2022), which provides an efficient and
scalable framework for reinforcement learning-based language model fine-
tuning.

The RL4LMs library incorporates Proximal Policy Optimization (PPO) Schulman et
al. (2017) as the reinforcement learning algorithm, which is known for its
stability and sample efficiency. The library also supports the integration of
custom reward functions, allowing us to design rewards that encourage the
generation of pedagogically sound teacher responses.

To implement the reinforcement learning-based fine-tuning, we first fine-tune
the Flan-T5 Chung et al. (2022) model on the TSCC dataset using supervised
learning, as described in the previous section. Next, we utilize the RL4LMs
library to fine-tune the model further using the PPO algorithm. We use an
equal division of the F1 as calculated by the roberta-large version of
BERTScore and DialogRPT-updown as the reward function. More Details about the
reinforcement learning fine-tuning process can be found in the Appendix.

The subsequent evaluation of the fine-tuned Flan-T5 model reveals the benefits
of incorporating reinforcement learning into the fine-tuning process,
contributing to more context-aware, relevant, and pedagogically effective AI
teacher responses.

##  5 Results

In this section, we present the results and discuss the performance of GPT-4,
fine-tuned GPT-2, and fine-tuned DialoGPT models on the TSCC dataset. We
analyze the strengths and weaknesses of each approach and provide insights
into their potential applications and limitations in an educational context.

Table 2: Validation set results Model |  BERTScore |  DialogRPT
---|---|---
GPT-4 |  0.82 |  0.69
Finetuned GPT-2 Large |  0.94 |  0.63
Finetuned DialoGPT Large |  0.94 |  0.64
Finetuned Flan-T5 w/ RL |  0.89 |  0.71
Table 3: Test set results Model |  BERTScore |  DialogRPT
---|---|---
GPT-4 |  0.8 |  0.70
Finetuned Flan-T5 w/ RL |  0.66 |  0.34

###  5.1 GPT-4

The GPT-4 model, without fine-tuning on the TSCC dataset, demonstrates a
relatively strong performance in generating educational dialogues. The
generated teacher responses are generally fluent and contextually relevant,
indicating that GPT-4 has a good understanding of the educational context
based on its pre-trained knowledge. However, some limitations are observed in
the model’s ability to generate accurate and pedagogically sound responses
consistently.

The carefully crafted prompt provided to the model plays a crucial role in
guiding GPT-4 toward generating suitable responses as a teacher. Although the
model is capable of generating contextually relevant and linguistically
correct responses, it may not always produce the most pedagogically sound or
helpful responses for the students. This limitation highlights the importance
of fine-tuning the model on a specific educational dataset, such as TSCC, to
further enhance its performance in generating AI teacher responses.

Additionally, due to the nature of the dataset, where conversations were often
cut off, the model sometimes lacked the full context needed to generate
meaningful responses that accurately represented the ground truth. Despite
this limitation, GPT-4’s responses were generally sensible and appropriate
given the available context.

###  5.2 Finetuned GPT-2

We observe that compared with DialoGPT, GPT-2 usually generates longer and
more formal responses, even with the same generation hyperparameters.

###  5.3 Finetuned DialoGPT

We observe that DialoGPT usually generates shorter and more vernacular
responses. It fits better in a conversational setting, but sometimes the
educational uptakes are not satisfactory since the responses are not guiding
students to learn the language.

###  5.4 Finetuned Flan-T5 w/ RL

We observe that the results of Flan-T5 w/ RL on the validation set are really
good suggesting that the model was able to hack the metrics designed as the
reward. On the contrary, it is performing poorly on the test set suggesting
that it overfits the validation set. We hypothesize two reasons for this to be
the case: the way conversations are split into chunks in the training dataset
or the difference in distribution between the training set and the test set.

##  6 Discussion

Conversational agents have the potential to revolutionize the teaching
landscape by addressing several challenges and enhancing the overall learning
experience for both students and educators Wollny et al. (2021). However,
developing conversational agents that can behave like human teachers requires
addressing several challenges Tack and Piech (2022).

Data challenges. As noted in the subsections above, the generations from the
GPT-$4$ model outperformed all the fine-tuned models, with and without
reinforcement learning. To this end, we put forward the proposition that an
array of dataset features plays a crucial role in posing significant
challenges to the fine-tuning process of generative models. These features
include several dataset characteristics, including sampling,
representativeness, prompt and response lengths, and dialogue
completeness—upon manual inspection, we identified several dialogues to be cut
off—pose serious challenges in achieving superior performance with fine-
tuning. Furthermore, upon random inspection of the generations from the fine-
tuned models, we identified that these models seem to have learned simple,
generic, often inappropriate yet correct responses such as “thank you” and
“okay.” While more recent language models have been shown to have high few-
shot performance, we believe that fine-tuned models could adapt better to
provide domain-specific responses in comparison. To achieve this, we emphasize
the need for extending the current dataset to include longer prompts with more
context.

[student] | someone plugged the charger in
---|---
[teacher] | that’s bad, charger must be $\\_\\_\\_$?
[student] | umm $\dots$
[model] | (a) plugged in $\leftarrow$ score: $0.91$
| (b) disconnected $\leftarrow$ score: $0.90$
[reference] | plugged out
Figure 1: An example dialog demonstrating that two opposing responses, (a) and
(b), ranked alike using the BERTScore metric.

It is important to acknowledge that these models might not be as effective as
desired in their response generation due to these intricacies. The current
efforts made by the research community to collect and build quality datasets
encompassing enough information about the educational task to enable AI
teacher generative models to fully generalize in any context is what we assess
to be the main focus that the community should adopt Jarratt (2023).

Evaluation metrics. In addition, we emphasize that to truly gauge the
efficiency of these AI-powered teaching models, it is vital to go a step
further and examine their ability to comprehend the unique nuances in the
students’ queries and cater to their particular educational requirements. This
implies the need for a pedagogically meaningful evaluation metric. We believe
that it is crucial for the research community to embrace this as the second
primary focus. While common evaluation metrics such as BERTScore and DialogRPT
are commonly used in several language and dialog modeling tasks, it is
important to note that these metrics were not fundamentally designed to
capture the level of pedagogical meaningfulness in the generated responses. As
an example, consider the dialog shown in Figure 1—depending on the given
context, only one of the responses (option (b): disconnected) is correct,
while both the responses are ranked as equally correct by the BERTScore
metric. Commonly-used domain-agnostic metrics often serve as a proxy for how
coherent and human-like the generated responses are. However, for more goal-
oriented tasks such as modeling teacher-student conversational dialogues,
these metrics seem to fall short. This generalization gap becomes more
apparent on analyzing the results from the fine-tuned Flan-T$5$ model with a
feedback loop based on BERTScore and DialogRPT scores—despite the model
performing significantly well on training and validation sets, it failed to
generalize on unseen test data. In an effort to advance research on this
front, we note the need for auxiliary training-level metrics, including the
faithfulness of the generation to the true response, to ensure that the
generations are context-aware and factually accurate (e.g., correct option (b)
vs. incorrect option (a) in Figure 1).

GPT-4 unknown pre-training data. We understand that the use of GPT-4 as a
baseline in our study presents challenges due to its unknown training data.
Yet, whether GPT-4 has seen parts of the TSCC dataset during its pre-training
or not, the improvement of performance compared to the reference with regard
to the DialogRPT scores and human evaluation scores attached to the
leaderboard of the shared task suggests that the potential of using such high-
performing models in this domain warrants further exploration.

##  7 Conclusion

In this paper, we explored the potential of using large pre-trained language
models and reinforcement learning for generating AI teacher responses in an
educational context. We first presented a few-shot approach using the GPT-4
model, which demonstrated promising results in generating contextually
relevant and fluent responses, but with limitations in generating
pedagogically sound responses consistently. We then fine-tuned GPT-2 and
DialoGPT on the TSCC dataset and evaluated their performance using BERTScore
and DialogRPT metrics. We also proposed an approach using RL to optimize
directly for pedagogical values. We hypothesized that several dataset
characteristics (e.g., dialog completeness, sampling) pose challenges to
achieving superior performance with fine-tuning. To this end, we recommend the
extension of the dataset to include longer prompts with extended context.
Finally, we also draw attention to the need for more domain-specific metrics
(in both evaluation and reward-based training) in enabling the generation of
accurate, context-aware, and factually correct teacher responses.

## References

  * Bibauw et al. (2022) Serge Bibauw, Thomas François, and Piet Desmet. 2022.  Dialogue systems for language learning: Chatbots and beyond.  In _The Routledge handbook of second language acquisition and technology_ , pages 121–135. Routledge.
  * Bommasani et al. (2021) Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. 2021.  On the opportunities and risks of foundation models.  _arXiv preprint arXiv:2108.07258_.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901.
  * Caines et al. (2022) Andrew Caines, Helen Yannakoudakis, Helen Allen, Pascual Pérez-Paredes, Bill Byrne, and Paula Buttery. 2022.  The teacher-student chatroom corpus version 2: more lessons, new annotation, automatic detection of sequence shifts.  In _Proceedings of the 11th Workshop on NLP for Computer Assisted Language Learning_ , pages 23–35.
  * Caines et al. (2020) Andrew Caines, Helen Yannakoudakis, Helena Edmondson, Helen Allen, Pascual Pérez-Paredes, Bill Byrne, and Paula Buttery. 2020.  The teacher-student chatroom corpus.  _arXiv preprint arXiv:2011.07109_.
  * Chang et al. (2020) Jonathan P Chang, Caleb Chiam, Liye Fu, Andrew Z Wang, Justine Zhang, and Cristian Danescu-Niculescu-Mizil. 2020.  Convokit: A toolkit for the analysis of conversations.  _arXiv preprint arXiv:2005.04246_.
  * Chen et al. (2019) Guanliang Chen, Rafael Ferreira, David Lang, and Dragan Gasevic. 2019.  Predictors of student satisfaction: A large-scale study of human-human online tutorial dialogues.  _International Educational Data Mining Society_.
  * Chung et al. (2022) Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2022.  Scaling instruction-finetuned language models.  _arXiv preprint arXiv:2210.11416_.
  * Cohen et al. (2020) Julie Cohen, Vivian Wong, Anandita Krishnamachari, and Rebekah Berlin. 2020.  Teacher coaching in a simulated environment.  _Educational evaluation and policy analysis_ , 42(2):208–231.
  * Collins (1982) James Collins. 1982.  Discourse style, classroom interaction and differential treatment.  _Journal of reading behavior_ , 14(4):429–437.
  * Demszky et al. (2021) Dorottya Demszky, Jing Liu, Zid Mancenido, Julie Cohen, Heather Hill, Dan Jurafsky, and Tatsunori Hashimoto. 2021.  Measuring conversational uptake: A case study on student-teacher interactions.  _arXiv preprint arXiv:2106.03873_.
  * Jaques et al. (2019) Natasha Jaques, Asma Ghandeharioun, Judy Hanwen Shen, Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Gu, and Rosalind Picard. 2019.  Way off-policy batch deep reinforcement learning of implicit human preferences in dialog.  _arXiv preprint arXiv:1907.00456_.
  * Jarratt (2023) Daniel Jarratt. 2023.  Chatgpt: The double-edged sword of ai in education.
  * Kane et al. (2022) Thomas Kane, Heather Hill, and Douglas Staiger. 2022.  National center for teacher effectiveness main study.
  * Kasneci et al. (2023) Enkelejda Kasneci, Kathrin Seßler, Stefan Küchemann, Maria Bannert, Daryna Dementieva, Frank Fischer, Urs Gasser, Georg Groh, Stephan Günnemann, Eyke Hüllermeier, et al. 2023.  Chatgpt for good? on opportunities and challenges of large language models for education.  _Learning and Individual Differences_ , 103:102274.
  * Norouzi et al. (2016) Mohammad Norouzi, Samy Bengio, Navdeep Jaitly, Mike Schuster, Yonghui Wu, Dale Schuurmans, et al. 2016.  Reward augmented maximum likelihood for neural structured prediction.  _Advances In Neural Information Processing Systems_ , 29.
  * Pasunuru and Bansal (2018) Ramakanth Pasunuru and Mohit Bansal. 2018.  Multi-reward reinforced summarization with saliency and entailment.  _arXiv preprint arXiv:1804.06451_.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _The Journal of Machine Learning Research_ , 21(1):5485–5551.
  * Ramamurthy et al. (2022) Rajkumar Ramamurthy, Prithviraj Ammanabrolu, Kianté Brantley, Jack Hessel, Rafet Sifa, Christian Bauckhage, Hannaneh Hajishirzi, and Yejin Choi. 2022.  Is reinforcement learning (not) for natural language processing?: Benchmarks, baselines, and building blocks for natural language policy optimization.  _arXiv preprint arXiv:2210.01241_.
  * Ranzato et al. (2015) Marc’Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. 2015.  Sequence level training with recurrent neural networks.  _arXiv preprint arXiv:1511.06732_.
  * Roller et al. (2020) Stephen Roller, Emily Dinan, Naman Goyal, Da Ju, Mary Williamson, Yinhan Liu, Jing Xu, Myle Ott, Kurt Shuster, Eric M Smith, et al. 2020.  Recipes for building an open-domain chatbot.  _arXiv preprint arXiv:2004.13637_.
  * Scao et al. (2022) Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. 2022.  Bloom: A 176b-parameter open-access multilingual language model.  _arXiv preprint arXiv:2211.05100_.
  * Schulman et al. (2017) John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017\.  Proximal policy optimization algorithms.  _arXiv preprint arXiv:1707.06347_.
  * Serban et al. (2017) Iulian V Serban, Chinnadhurai Sankar, Mathieu Germain, Saizheng Zhang, Zhouhan Lin, Sandeep Subramanian, Taesup Kim, Michael Pieper, Sarath Chandar, Nan Rosemary Ke, et al. 2017.  A deep reinforcement learning chatbot.  _arXiv preprint arXiv:1709.02349_.
  * Shen et al. (2015) Shiqi Shen, Yong Cheng, Zhongjun He, Wei He, Hua Wu, Maosong Sun, and Yang Liu. 2015\.  Minimum risk training for neural machine translation.  _arXiv preprint arXiv:1512.02433_.
  * Tack et al. (2023) Anaïs Tack, Ekaterina Kochmar, Zheng Yuan, Serge Bibauw, and Chris Piech. 2023\.  The BEA 2023 Shared Task on Generating AI Teacher Responses in Educational Dialogues.  In _Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications_ , page to appear, Toronto, Canada. Association for Computational Linguistics.
  * Tack and Piech (2022) Anaïs Tack and Chris Piech. 2022.  The ai teacher test: Measuring the pedagogical ability of blender and gpt-3 in educational dialogues.  _arXiv preprint arXiv:2205.07540_.
  * Team (2022) OpenAI Team. 2022.  Chatgpt: Optimizing language models for dialogue.
  * Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023.  Llama: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_.
  * Wollny et al. (2021) Sebastian Wollny, Jan Schneider, Daniele Di Mitri, Joshua Weidlich, Marc Rittberger, and Hendrik Drachsler. 2021.  Are we there yet?-a systematic literature review on chatbots in education.  _Frontiers in artificial intelligence_ , 4:654924.
  * Zhang et al. (2022) Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022.  Opt: Open pre-trained transformer language models.  _arXiv preprint arXiv:2205.01068_.
  * Zhang et al. (2019) Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and Bill Dolan. 2019.  Dialogpt: Large-scale generative pre-training for conversational response generation.  _arXiv preprint arXiv:1911.00536_.

##  Appendix A Appendix

###  A.1 GPT-4 Prompt Construction

To evaluate the performance of GPT-4, we provided it with a few-shot prompt
that includes a selection of similar teacher-student conversations from the
TSCC dataset. This approach helps guide the model toward generating suitable
responses as a teacher. The prompt is constructed as follows:

  * •

We direct the system role to act as a teacher and encourage learning by using
the prompt as given below.

  * •

Retrieve the 5 most similar teacher-student conversations from the TSCC
dataset. This is done by computing the cosine similarity between the input
conversation context and the current conversation context in the dataset using
embeddings generated by the text-embedding-ada-002 model.

  * •

Concatenate the selected conversations with the input conversation, separated
by special tokens to indicate the beginning and end of a new sample
conversation.

This prompt construction aims to provide GPT-4 with the necessary context and
guidance to generate accurate and pedagogically relevant responses in the
context of teacher-student dialogues. The prompt is designed as follows:

> You are acting as a teacher, and you are helping a student learn. Be
> patient, helpful, and kind. Don’t be superimposing; give short responses to
> encourage learning. Make the student feel comfortable and confident, and
> help them learn. Now, join the following conversation: <conversation
> context>

The prompt is designed using the following directives in mind:-

  * •

We instruct the system with several indicators to act as a teacher and provide
helpful advice to the student.

  * •

To mitigate the challenge of generating teacher-like responses, we advise the
model to be patient, kind, and helpful to the student.

  * •

Through the directive to keep responses short and encouraging, we guide the
model toward generating suitable responses that might help the student learn
effectively.

  * •

The model is also instructed to make the student feel comfortable and
confident in their learning process, providing an overall supportive
environment for the student.

  * •

Finally, the conversation context is provided to the model to set the context
for the given student query, allowing the model to generate appropriate
responses given the conversation context.

Combining all these aspects together, we aim to guide the model toward
generating contextually relevant and pedagogically meaningful responses in the
given teacher-student dialogue.

We use the following hyperparameters for querying the GPT-4 model:

  * •

Model: gpt-4-0314

  * •

Temperature: 1

  * •

Max Tokens: 100

  * •

Top p: 1

###  A.2 Fine-tuning Exact Parameters

For our supervised fine-tuning experiments, we used the following
hyperparameters:

####  A.2.1 GPT-2

  * •

Learning rate: 1e-5

  * •

Batch size: 32

  * •

Epochs: 10

  * •

Max sequence length: 1024

  * •

Optimizer: AdamW

  * •

Scheduler: linear learning rate scheduler

####  A.2.2 DialoGPT

  * •

Learning rate: 1e-5

  * •

Batch size: 32

  * •

Epochs: 10

  * •

Max sequence length: 1024

  * •

Optimizer: AdamW

  * •

Scheduler: linear learning rate scheduler

###  A.3 Supervised Fine-tuning with Reinforcement Learning Details

To implement the reinforcement learning-based fine-tuning using the RL4LMs
library, we first fine-tuned the Flan-T5 model on the TSCC dataset using
supervised learning. After this initial fine-tuning step, we utilized the
RL4LMs library to fine-tune the model further using reinforcement learning. We
used an equal division of the BERTScore and DialogRPT as the reward function
to optimize the model for pedagogical quality. The following hyperparameters
were used for the reinforcement learning fine-tuning process:

  * •

Learning rate: 1e-6

  * •

Batch size: 64

  * •

Epochs: 5

  * •

Max prompt length: 512

  * •

Max episode length: 100

  * •

Optimizer: AdamW

  * •

Scheduler: linear learning rate scheduler

The YAML file for the RL4LMs scipt is as follows:

⬇

tokenizer:

model_name: google/flan-t5-small

padding_side: left

truncation_side: left

pad_token_as_eos_token: False

reward_fn:

id: dialog_rpt_bert

args:

BERTScore_coeff: 0.5

DialogRPT_coeff: 0.5

datapool:

id: bea

truncate: False

args: {}

env:

n_envs: 1

args:

max_prompt_length: 100

max_episode_length: 20

terminate_on_eos: True

context_start_token: 0

prompt_truncation_side: "right"

alg:

id: ppo_separate

args:

n_steps: 20

batch_size: 64

verbose: 1

learning_rate: 0.000001

clip_range: 0.2

n_epochs: 1

value_update_epochs: 3

# batchify: False

gae_lambda: 0.95

gamma: 0.99

ent_coef: 0.01

kl_div:

coeff: 0.001

target_kl: 2.0

policy:

id: seq2seq_lm_actor_critic_policy

args:

model_name: google/flan-t5-small

apply_model_parallel: True

prompt_truncation_side: "right"

generation_kwargs:

do_sample: True

top_k: 0

min_length: 9

max_new_tokens: 20

train_evaluation:

eval_batch_size: 64

n_iters: 200

eval_every: 20

save_every: 10

metrics:

- id: bert_score

args:

language: en

- id: dialog_rpt

args:

model_name: "microsoft/DialogRPT

-updown"

label_ix: 0

batch_size: 1

# - id: uptake

# args:

# model_name: None

# label_ix: 0

# batch_size: 1

generation_kwargs:

num_beams: 5

min_length: 9

max_new_tokens: 20