# Fast FullSubNet: Accelerate Full-band and Sub-band Fusion Model for Single-
channel Speech Enhancement

###### Abstract

FullSubNet is our recently proposed real-time single-channel speech
enhancement network that achieves outstanding performance on the Deep Noise
Suppression (DNS) Challenge dataset. A number of variants of FullSubNet have
been proposed recently, but they all focus on the structure design towards
better performance and are rarely concerned with computational efficiency.
This work proposes a new architecture named Fast FullSubNet dedicated to
accelerating the computation of FullSubNet. Specifically, Fast FullSubNet
processes sub-band speech spectra in the mel-frequency domain by using
cascaded linear-to-mel full-band, sub-band, and mel-to-linear full-band models
such that frequencies involved in the sub-band computation are vastly reduced.
After that, a down-sampling operation is proposed for the sub-band input
sequence to further reduce the computational complexity along the time axis.
Experimental results show that, compared to FullSubNet, Fast FullSubNet has
only 13% computational complexity and 16% processing time, and achieves
comparable or even better performance.

Index Terms—  Fast FullSubNet, FullSubNet, computational cost, sub-band,
speech enhancement

##  1 Introducation

Speech enhancement aims to improve speech intelligibility and perceptual
quality in noisy environments [1]. Recent Deep Noise Suppression (DNS)
Challenge [2, 3, 4] have significantly contributed to advances in the speech
enhancement field and fostered many state-of-the-art (SOTA) methods [5, 6, 7].
Among these methods, FullSubNet [6] attracted broad attention because of its
excellent perceptual quality of enhanced speech, especially for the
reverberant speech case. Unlike the mainstream methods, which only process the
full-band spectra, FullSubNet integrates a full-band model and a sub-band
model and performs joint optimization. In FullSubNet, the full-band model
extracts global spectral information and long-distance cross-band
dependencies. Meanwhile, the sub-band model processes the frequency bands
independently and focuses on local spectral patterns and signal stationarity.
Experiments show that these two kinds of models are complementary and can be
efficiently integrated into one framework.

The fusion scheme proposed by FullSubNet is compatible with other advanced
techniques employed in SOTA speech enhancement methods. In the past year, a
number of FullSubNet variants [8, 9, 10, 11, 12] have been proposed. DCCRN-
SUBNET [11] combines a deep complex convolution recurrent network (DCCRN) and
attention gates as an improved full-band model and keeps the sub-band model
unchanged. DPT-FSNET [9] proposes a dual-path transformer-based full-band and
sub-band fusion network. FullSubNet+ [8] replaces the LSTM layers in the
original full-band model with stacked temporal convolutional network blocks.
STSubNet [12] uses a novel sub-band network to corporate an efficient spectro-
temporal receptive field extractor to achieve simultaneous denoising and
dereverberation. Through the sophisticated structure design, these variants
achieve remarkable noise suppression performance. Nevertheless, the
computational cost of the full-band and sub-band fusion models remains a blind
spot due to the hundreds of times running the sub-band model for processing
one signal clip.

This work aims at accelerating the computation of FullSubNet. Instead of
straightforwardly seeking to prune or quantize neural networks, we introduce a
new architecture named Fast FullSubNet to reduce the complexity caused by the
sub-band model while maintaining the speech enhancement performance. Unlike
FullSubNet, which works in the linear-frequency domain, this work proposed to
work in the mel-frequency domain to largely reduce the number of frequency
bands. Mel-frequency presents speech spectra more compactly and meanwhile
without losing spectral information in the sense of human auditory perception.
Specifically, the linear-frequency spectra are first transformed into the mel-
frequency domain, then processed with cascaded full-band and sub-band models
following the spirit of FullSubNet. Afterward, an extra mel-to-linear full-
band model is added to transform back to the linear-frequency domain, which is
similar to the neural vocoders used in recent text-to-speech (TTS) systems
[13, 14] that perform a mel-to-linear transformation. Besides, to further
reduce the complexity of the sub-band model, we propose to downsample the
feature sequence processed by the sub-band model, and then leverage the mel-
to-linear full-band model to interpolate the output of the sub-band model.
These models and strategies are efficiently integrated together. Experimental
results show that Fast FullSubNet reduces the computational complexity and
processing time to about 13% and 16% of that of FullSubNet. Moreover, Fast
FullSubNet achieves comparable or even better speech enhancement performance
than FullSubNet due to the use of mel-frequency and post mel-to-linear model.
We believe that the design scheme of Fast FullSubNet is also suitable for
other full-band and sub-band fusion models [8, 9, 10, 11, 12]. Code and audio
samples are available at https://github.com/haoxiangsnr/FullSubNet.

##  2 Method

Fig. 1: Diagram of Fast FullSubNet. The right parts of rectangle boxes show
feature dimensions, e.g., “$1~{}(F)$” represents a $F$-dimension vector.
“$F_{\text{mel}}(2N+2)$” denotes $F_{\text{mel}}$ independent
($2N+1$)-dimensional vectors.

This work processes speech signals in the short-time Fourier transform (STFT)
domain. The observed noisy speech signals are given by

$x(t,f)=s(t,f)+n(t,f)$ (1)

where $x(t,f)$, $s(t,f)$ and $n(t,f)$ represent the complex-valued time-
frequency (T-F) bins of noisy speech, noise-free speech (can be the
reverberant image signal received at the microphone) and interference noise,
respectively, with $t\in[1,\cdots,T]$, $f\in[0,\cdots,F-1]$, $T$ and $F$ being
the time frame, discrete frequency bin, number of frames and number of
discrete frequencies, respectively. Note that this work only focuses on the
denoising task, which means the purpose of this work is to suppress noise
$n(t,f)$ and recover the reverberant speech signal $s(t,f)$.

The proposed Fast FullSubNet reduces the computational complexity of
FullSubNet by decreasing the number of frequencies and time frames involved in
the sub-band model. Figure 1 shows the workflow of Fast FullSubNet, which
keeps the motivation and logic of the original FullSubNet unchanged. To
decrease the number of frequency bands, we first perform speech enhancement in
the mel-frequency domain with a linear-to-mel full-band model
$\mathcal{F}_{\text{l2m}}$ and a sub-band model $\mathcal{S}$, since the mel-
frequency representation of speech is more compact and still informative in
terms of human auditory perception. Then, the output of $\mathcal{S}$ is
transformed back to the linear-frequency domain with a mel-to-linear full-band
model $\mathcal{F}_{\text{m2l}}$. Each of the three models consists of a two-
layer LSTM network.

###  2.1 Full-band model $\mathcal{F}_{\text{l2m}}$

The mel-scale spectral magnitude is first processed by the full-band model
$\mathcal{F}_{\text{l2m}}$ for extracting global spectral information and
long-distance cross-band dependencies. Formally, the linear frequency signal
$x(t,f)$ is transformed to the mel-frequency domain as
$x_{\text{mel}}(t,f),f\in[0,\cdots,F_{\text{mel}}-1]$, where $F_{\text{mel}}$
is the number of mel frequencies. The input vector of
$\mathcal{F}_{\text{l2m}}$ at time $t$ is

$\displaystyle\mathbf{x}(t)=[x_{\text{mel}}(t,0),\cdots,x_{\text{mel}}(t,F_{\text{mel}}-1)]^{T}\in\mathbb{R}^{F_{\text{mel}}}.$
(2)

The sequence of this feature vector is processed with two layers of LSTM. This
full-band model outputs a spectral embedding with the same size as
$\mathbf{x}(t)$, namely one hidden unit for each mel frequency. This spectral
embedding provides complementary information to the following sub-band model.

###  2.2 Sub-band model $\mathcal{S}$

The sub-band model processes the mel frequencies independently, and all
frequencies share the same network. The sub-band model predicts clean speech
leveraging the signal stationarity and the local spectral pattern of speech.
Specifically, the input of the sub-band model contains two sources. For one
frequency $f$, the first source is the noisy mel-spectra of this frequency and
$N$ adjacent frequencies at each frequency side. The second source is the
output of the full-band model at frequency $f$, denoted as
$\mathcal{F}_{\text{l2m}}(\mathbf{x}(t))(f)$. We concatenate these two sources
as the input of the sub-band model

$\displaystyle\mathbf{x}_{\text{sub}}(t,f)$
$\displaystyle=[x_{\text{mel}}(t,f-N),\cdots,x_{\text{mel}}(t,f),\cdots,$
$\displaystyle
x_{\text{mel}}(t,f+N),\mathcal{F}_{\text{l2m}}(\mathbf{x}(t))(f)]^{T}\in\mathbb{R}^{2N+2}.$
(3)

The sequence of this feature vector is processed with the same two layers of
LSTM for all frequencies. For each mel frequency, the sub-band model outputs a
one-dimensional hidden unit.

###  2.3 Full-band model $\mathcal{F}_{\text{m2l}}$

The full-band model $\mathcal{F}_{\text{m2l}}$ transforms mel-frequency back
to linear-frequency. The output of full-band model $\mathcal{F}_{\text{l2m}}$
and sub-band model $\mathcal{S}$ are concatenated as the input of
$\mathcal{F}_{\text{m2l}}$:

$\displaystyle\mathbf{x}_{\text{m2l}}(t)=[$
$\displaystyle\mathcal{F}_{\text{l2m}}(\mathbf{x}(t))^{T},\mathcal{S}(\mathbf{x}_{\text{sub}}(t,0)),\cdots,$
$\displaystyle\mathcal{S}(\mathbf{x}_{\text{sub}}(t,F_{\text{mel}}-1))]^{T}\in\mathbb{R}^{2F_{\text{mel}}}.$
(4)

Two layers of LSTM followed by one linear layer predict the final linear-
frequency output. The complex-valued Ideal Ratio Mask (cIRM) [15] is taken as
the learning target. Denote cIRM as $y(t,f)\in\mathbb{C}$ for one T-F bin.
$\mathcal{F}_{\text{m2l}}$ predicts the real-valued cIRM vector at time $t$ as

$\displaystyle\mathbf{y}(t)=[$
$\displaystyle\text{R}\\{y(t,0)\\},\text{I}\\{y(t,0)\\},\cdots,$
$\displaystyle\text{R}\\{y(t,F-1)\\},\text{I}\\{y(t,F-1)\\}]^{T}\in\mathbb{R}^{2F},$
(5)

where $\text{R}\\{\\}$ and $\text{I}\\{\\}$ denote the real and imaginary
parts of complex number, respectively. This mel-to-linear model performs a
similar function as the neural vocoders used in TTS [14, 16], as they both
perform mel-frequency to linear-frequency transformation, except that speech
enhancement can use the noisy signal phase. As shown in Figure 1, to employ a
look-ahead of $\tau$ frames, the target sequence could be set to be delayed
$\tau$ frames relative to the input sequence.

###  2.4 Sub-band down-sampling

One key characteristic of speech signals is that the samples are ordered in
time, and successive samples are dependent/redundant [17], which means we may
process only a part of the samples without performance degradation. To further
reduce the computational complexity of the sub-band model, we down-sample the
feature sequence of the sub-band model by a factor of $m$. Down-sampling is
conducted by non-overlapped averaging the input, i.e.,
$\mathbf{x}_{\text{sub}}(t,f)$, for every $m$ frames. For frequency $f$, the
down-sampled input is denoted as $\tilde{\mathbf{x}}_{\text{sub}}(n,f)$, where
$n\in[1,\cdots,\lceil\frac{T}{m}\rceil]$ is the index of down-sampled time
frames. When $m=1$, there will be no downsampling. The down-sampled sequence
may lose information employed by the sub-band model, such as the temporal
dynamic of local spectral pattern, and thus degrades the quality of the sub-
band output.

The output of the sub-band model will be down-sampled accordingly. The down-
sampled sub-band output is first copied for $m$ times and then fed to the
following full-band model $\mathcal{F}_{\text{m2l}}$. For this case,
$\mathcal{F}_{\text{m2l}}$ is not only used for mel-to-linear transformation
but also for interpolating the sub-band output leveraging the dependence of
adjacent frames. Note that, also as the input of $\mathcal{F}_{\text{m2l}}$,
the output of $\mathcal{F}_{\text{l2m}}$ is not down-sampled, which may
alleviate the difficulty of interpolation.

As shown in Figure 1, to conduct down-sampling and meanwhile guarantee online
processing, at a one-time step, the averaging of input frames only uses the
previous $m-1$ time steps, while the output is copied to the future $m-1$ time
steps.

Table 1: Performance on DNS Challenge (INTERSPEECH 2020) dataset. For
comparison methods, the scores are directly quoted from their original papers,
and the missing scores in the original papers are shown as blanks.

Method | Down-sampling |  |  |  |  |  |  |  |  |  |  |
---|---|---|---|---|---|---|---|---|---|---|---|---
factor $m$ | With Reverb |  |  |  | No Reverb |  |  |  | # Param |  |  |
(M) | MACs |  |  |  |  |  |  |  |  |  |  |
(G/s) | RTF |  |  |  |  |  |  |  |  |  |  |
|  | WB-PESQ | NB-PESQ | STOI | SI-SDR | WB-PESQ | NB-PESQ | STOI | SI-SDR |  |  |
Noisy |  | 1.822 | 2.753 | 86.62 | 9.033 | 1.582 | 2.454 | 91.52 | 9.071 | - | - | -
DTLN [18] |  |  | 2.70 | 84.68 | 10.53 |  | 3.04 | 74.76 | 16.34 | 0.99 | 0.11 | 0.043
DCCRN-E [5] |  |  | 3.077 |  |  |  | 3.266 |  |  | 3.74 | 6.56 | 0.128
Conv-TasNet [19] |  | 2.75 |  |  |  | 2.73 |  |  |  | 8.68 | 5.97 | 0.659
Sub-band Model [20] | $384\times 2$ | 2.650 | 3.274 | 90.53 | 14.67 | 2.369 | 3.052 | 94.24 | 16.15 | 1.30 | 21.68 | 0.401
FullSubNet [6] |  | 2.969 | 3.473 | 92.62 | 15.75 | 2.888 | 3.305 | 96.11 | 17.29 | 5.64 | 30.73 | 0.511
Full-band Model |  | 2.726 | 3.388 | 91.15 | 14.75 | 2.831 | 3.354 | 96.10 | 16.58 | 8.15 | 0.53 | 0.026
Fast FullSubNet | 1 | 3.031 | 3.511 | 93.14 | 15.68 | 2.865 | 3.375 | 96.29 | 17.11 | 6.84 | 7.79 | 0.147
| 2 | 3.016 | 3.497 | 92.96 | 15.85 | 2.808 | 3.353 | 96.11 | 16.98 | 6.84 | 4.12 | 0.082
| 4 | 2.896 | 3.438 | 92.35 | 15.51 | 2.707 | 3.294 | 95.85 | 16.35 | 6.84 | 2.29 | 0.053
| 8 | 2.862 | 3.414 | 92.11 | 15.31 | 2.692 | 3.380 | 95.77 | 16.38 | 6.84 | 1.39 | 0.042
| $+\infty$ | 2.865 | 3.419 | 92.18 | 15.12 | 2.763 | 3.325 | 95.96 | 16.60 | 4.91 | 0.32 | 0.016

##  3 Experiments

###  3.1 Experimental setup

For a fair comparison with FullSubNet, all experiments are conducted on the
DNS challenge (INTERSPEECH 2020) dataset. This dataset consists of a clean
speech set including about 500-hour clips from 2150 speakers and a noise
dataset including over 180-hour clips from 150 classes. The synthesized noisy-
clean speech pairs follow the dynamic mixing strategy of FullSubNet. Before
the start of each training epoch, 75% of the clean speech clips are mixed with
randomly selected room impulse responses (RIR) from (1) Multichannel Impulse
Response Database [21] with three reverberation times 0.16 s, 0.36 s, and 0.61
s. (2) Reverb Challenge dataset [22] with three reverberation times 0.3 s, 0.6
s and 0.7 s. Then, based on a randomly selected SNR between -5 and 20 dB, the
reverberated or no-reverberated speech will be mixed with a randomly selected
noise. The total data “seen” by the model is over 5000 hours after ten epochs
of training. We use the test dataset of the DNS Challenge dataset for
evaluation, including two categories of synthetic clips, i.e., without and
with reverberations. Each category has 150 noisy clips with SNR levels
distributed between 0 dB to 20 dB.

The sampling rate of audio signals is 16000 Hz. STFT uses a 32 ms (512
samples) Hanning window and a 16 ms hop size. The number of mel-frequency bins
is set to 64. For training, we adopt the Adam optimizer with a learning rate
of 0.001. For a fair comparison, same as the parameters mentioned in [20, 6],
we set output delay $\tau$ to two frames so that the model exploits $16\times
2=32$ ms future information. The sequence length for training is set to the
$T=192$ frames (about 3 s). According to preliminary experiments, the number
of neighbor frequencies $N$ in Equation (2.2) is set to 5. The three models
all consist of two stacked unidirectional LSTM layers and a linear layer. The
two full-band models and one sub-band model, have 384/257, 512/512, and
384/384 hidden units for their own two LSTM layers, respectively.

###  3.2 Results

Table 1 shows the experimental results in terms of perceptual quality (WB-/NB-
PESQ [23]), speech intelligibility (STOI [24]), and SI-SDR [25]. Besides, the
“# Param” column shows the number of parameters. We also report the Mult-Add
calculations (MACs) calculated using the torchinfo tool
111https://github.com/TylerYep/torchinfo to show the computational complexity.
In addition, real-time factor (RTF), a metric for measuring the inference
speed, is also measured on a platform with Intel (R) Core (TM) i7-9700 CPU @
3.00 GHz and PyTorch 1.12.

Effectiveness of reducing frequencies. When the downsampling factor $m$ of
Fast FullSubNet is set to one, there is no time downsampling. Compared with
the original FullSubNet, decreasing the frequencies for sub-band processing
(from 257 to 64) reduces MACs and RTF to about 25% and 29 % of that of
FullSubNet. Moreover, almost all performance measures are comparable or even
better, possibly due to the use of mel-frequency and a post mel-to-linear
full-band model. The advantage of using mel-frequency will be more clear
later.

Effectiveness of sub-band downsampling. Another set of experiments is
conducted with increasing downsampling factors. Compared with $m=1$, $m=2$
achieves a comparable enhancement performance, and MACs and RTF further reduce
to about 13% and 16% of that of FullSubNet, respectively. This result fits our
expectation that the successive time frames are relatively
dependent/redundant, and the mel-to-linear full-band model can well
interpolate the down-sampled sub-band output. Decreasing $m$ to 4 and 8, MACs
and RTF will be further reduced but at the cost of enhancement performance
degradation. Finally, when $m=+\infty$, the sub-band model is totally removed,
which achieves similar speech enhancement performance as $m=8$, which means
setting up to $m=8$, the sub-band model is no longer useful for speech
enhancement. We would like to note that our preliminary experiments show that
the performance degradation caused by sub-band downsampling is related to the
number of look-ahead frames, as the look-ahead frames provide more recent
information for the interpolation of the output of the sub-band model. This
means a smaller/larger number of look-ahead frames will allow using a
smaller/larger down-sampling factor, without suffering from performance
degradation.

When the sub-band model is removed with $m=+\infty$, the network consists of
two layers of mel-frequency and two layers of mel-to-linear full-band LSTMs.
To testify the use of mel-frequency, we train a Full-band Model (as shown in
Table 1) composed of four layers of 512-dim LSTMs with linear-frequency input
and output. It can be seen that the mel-frequency network performs better than
the Full-band Model in terms of both speech enhancement performance and
computational complexity. The possible reason is that mel-frequency represents
speech spectra in a more compact way, which eases the learning of mapping
between noisy and clean spectra.

Comparsion with SOTAs. We compare Fast FullSubNet with several recent SOTA
methods that provide results on the DNS Challenge dataset and open-source
their network implementation to compare MACs and RTF fairly. It can be seen
that FullSubNet has already outperformed these methods in terms of speech
enhancement performance. The proposed Fast FullSubNet with $m=2$ has smaller
MACs and RTF than these methods except for DTLN [18]. Notably, the flexible
strategies for reducing computational complexity provided by Fast FullSubNet
are also suitable for other FullSubNet variants.

##  4 Conclusion

This paper proposes a new architecture named Fast FullSubNet to accelerate the
computation of FullSubNet by reducing the number of frequencies and time
frames involved in the computation of the sub-band model. Experimental results
show that compared with the original FullSubNet, Fast FullSubNet achieves
comparable or better performance with significantly smaller complexity.
Importantly, the flexible strategies for reducing computational complexity
provided by Fast FullSubNet are also suitable for other FullSubNet variants.

## References

  * [1] Philipos C. Loizou,  Speech Enhancement: Theory and Practice,  CRC Press, Inc., USA, 2nd edition, 2013.
  * [2] Chandan K.A. Reddy, Vishak Gopal, Ross Cutler, Ebrahim Beyrami, Roger Cheng, Harishchandra Dubey, Sergiy Matusevych, Robert Aichner, Ashkan Aazami, Sebastian Braun, Puneet Rana, Sriram Srinivasan, and Johannes Gehrke,  “The INTERSPEECH 2020 Deep Noise Suppression Challenge: Datasets, Subjective Testing Framework, and Challenge Results,”  in Interspeech 2020. Oct. 2020, pp. 2492–2496, ISCA.
  * [3] Chandan K. A. Reddy, Harishchandra Dubey, Vishak Gopal, Ross Cutler, Sebastian Braun, Hannes Gamper, Robert Aichner, and Sriram Srinivasan,  “ICASSP 2021 Deep Noise Suppression Challenge,”  in ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), June 2021, pp. 6623–6627,  ISSN: 2379-190X.
  * [4] Harishchandra Dubey, Vishak Gopal, Ross Cutler, Ashkan Aazami, Sergiy Matusevych, Sebastian Braun, Sefik Emre Eskimez, Manthan Thakker, Takuya Yoshioka, Hannes Gamper, and Robert Aichner,  “Icassp 2022 Deep Noise Suppression Challenge,”  in ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2022, pp. 9271–9275,  ISSN: 2379-190X.
  * [5] Yanxin Hu, Yun Liu, Shubo Lv, Mengtao Xing, Shimin Zhang, Yihui Fu, Jian Wu, Bihong Zhang, and Lei Xie,  “DCCRN: Deep Complex Convolution Recurrent Network for Phase-Aware Speech Enhancement,”  in Interspeech 2020. Oct. 2020, pp. 2472–2476, ISCA.
  * [6] Xiang Hao, Xiangdong Su, Radu Horaud, and Xiaofei Li,  “Fullsubnet: A Full-Band and Sub-Band Fusion Model for Real-Time Single-Channel Speech Enhancement,”  in ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), June 2021, pp. 6633–6637,  ISSN: 2379-190X.
  * [7] Andong Li, Wenzhe Liu, Xiaoxue Luo, Guochen Yu, Chengshi Zheng, and Xiaodong Li,  “A Simultaneous Denoising and Dereverberation Framework with Target Decoupling,”  in Interspeech 2021. Aug. 2021, pp. 2801–2805, ISCA.
  * [8] Jun Chen, Zilin Wang, Deyi Tuo, Zhiyong Wu, Shiyin Kang, and Helen Meng,  “FullSubNet+: Channel Attention Fullsubnet with Complex Spectrograms for Speech Enhancement,”  in ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2022, pp. 7857–7861,  ISSN: 2379-190X.
  * [9] Feng Dang, Hangting Chen, and Pengyuan Zhang,  “DPT-FSNet: Dual-Path Transformer Based Full-Band and Sub-Band Fusion Network for Speech Enhancement,”  in ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2022, pp. 6857–6861,  ISSN: 2379-190X.
  * [10] Zhuangqi Chen and Pingjian Zhang,  “Lightweight Full-band and Sub-band Fusion Network for Real Time Speech Enhancement,”  in Interspeech 2022. Sept. 2022, pp. 921–925, ISCA.
  * [11] Xin yuan, Qun Yang, and Shaohan Liu,  “DCCRN-SUBNET: A DCCRN and SUBNET Fusion Model for Speech Enhancement,”  in 2021 7th International Conference on Computer and Communications (ICCC), 2021, pp. 525–529.
  * [12] Feifei Xiong, Weiguang Chen, Pengyu Wang, Xiaofei Li, and Jinwei Feng,  “Spectro-Temporal SubNet for Real-Time Monaural Speech Denoising and Dereverberation,”  in Interspeech 2022. Sept. 2022, pp. 931–935, ISCA.
  * [13] Jonathan Shen, Ruoming Pang, Ron J. Weiss, Mike Schuster, Navdeep Jaitly, Zongheng Yang, Zhifeng Chen, Yu Zhang, Yuxuan Wang, Rj Skerrv-Ryan, Rif A. Saurous, Yannis Agiomvrgiannakis, and Yonghui Wu,  “Natural TTS Synthesis by Conditioning Wavenet on MEL Spectrogram Predictions,”  in 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Apr. 2018, pp. 4779–4783,  ISSN: 2379-190X.
  * [14] Wei Ping, Kainan Peng, Andrew Gibiansky, Sercan O. Arik, Ajay Kannan, Sharan Narang, Jonathan Raiman, and John Miller,  “Deep Voice 3: Scaling Text-to-Speech with Convolutional Sequence Learning,”  Feb. 2022.
  * [15] Donald S. Williamson, Yuxuan Wang, and DeLiang Wang,  “Complex Ratio Masking for Monaural Speech Separation,”  IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 24, no. 3, pp. 483–492, Mar. 2016,  Conference Name: IEEE/ACM Transactions on Audio, Speech, and Language Processing.
  * [16] Takuhiro Kaneko, Hirokazu Kameoka, Kou Tanaka, and Nobukatsu Hojo,  “Maskcyclegan-VC: Learning Non-Parallel Voice Conversion with Filling in Frames,”  in ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), June 2021, pp. 5919–5923,  ISSN: 2379-190X.
  * [17] Dimitris G. Manolakis, Vinay K. Ingle, and Stephen M. Kogon,  Statistical and adaptive signal processing: spectral estimation, signal modeling, adaptive filtering, and array processing,  Artech House signal processing library. Artech House, Boston, 2005.
  * [18] Nils L. Westhausen and Bernd T. Meyer,  “Dual-Signal Transformation LSTM Network for Real-Time Noise Suppression,”  in Interspeech 2020. Oct. 2020, pp. 2477–2481, ISCA.
  * [19] Yuichiro Koyama, Tyler Vuong, Stefan Uhlich, and Bhiksha Raj,  “Exploring the Best Loss Function for DNN-Based Low-latency Speech Enhancement with Temporal Convolutional Networks,” Aug. 2020,  arXiv:2005.11611 [cs, eess].
  * [20] Xiaofei Li and Radu Horaud,  “Online Monaural Speech Enhancement Using Delayed Subband LSTM,”  in Interspeech 2020. Oct. 2020, pp. 2462–2466, ISCA.
  * [21] Elior Hadad, Florian Heese, Peter Vary, and Sharon Gannot,  “Multichannel audio database in various acoustic environments,”  in 2014 14th International Workshop on Acoustic Signal Enhancement (IWAENC), Sept. 2014, pp. 313–317.
  * [22] Keisuke Kinoshita, Marc Delcroix, Sharon Gannot, Emanuël A. P. Habets, Reinhold Haeb-Umbach, Walter Kellermann, Volker Leutnant, Roland Maas, Tomohiro Nakatani, Bhiksha Raj, Armin Sehr, and Takuya Yoshioka,  “A summary of the REVERB challenge: state-of-the-art and remaining challenges in reverberant speech processing research,”  EURASIP Journal on Advances in Signal Processing, vol. 2016, no. 1, pp. 7, Jan. 2016.
  * [23] A.W. Rix, J.G. Beerends, M.P. Hollier, and A.P. Hekstra,  “Perceptual evaluation of speech quality (PESQ)-a new method for speech quality assessment of telephone networks and codecs,”  in 2001 IEEE International Conference on Acoustics, Speech, and Signal Processing. Proceedings (Cat. No.01CH37221), 2001, vol. 2, pp. 749–752 vol.2,  ISSN: 1520-6149.
  * [24] Cees H. Taal, Richard C. Hendriks, Richard Heusdens, and Jesper Jensen,  “An Algorithm for Intelligibility Prediction of Time–Frequency Weighted Noisy Speech,”  IEEE Transactions on Audio, Speech, and Language Processing, vol. 19, no. 7, pp. 2125–2136, Sept. 2011,  Conference Name: IEEE Transactions on Audio, Speech, and Language Processing.
  * [25] Jonathan Le Roux, Scott Wisdom, Hakan Erdogan, and John R. Hershey,  “SDR – Half-baked or Well Done?,”  in ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2019, pp. 626–630,  ISSN: 2379-190X.


11institutetext: University of Massachusetts Amherst
Contact Email: 11email<EMAIL_ADDRESS>

# SmartPhone: Exploring Keyword Mnemonic with Auto-generated Verbal and Visual
Cues

Jaewook Lee  Andrew Lan

###### Abstract

In second language vocabulary learning, existing works have primarily focused
on either the learning interface or scheduling personalized retrieval
practices to maximize memory retention. However, the learning content, i.e.,
the information presented on flashcards, has mostly remained constant. Keyword
mnemonic is a notable learning strategy that relates new vocabulary to
existing knowledge by building an acoustic and imagery link using a keyword
that sounds alike. Beyond that, producing verbal and visual cues associated
with the keyword to facilitate building these links requires a manual process
and is not scalable. In this paper, we explore an opportunity to use large
language models to automatically generate verbal and visual cues for keyword
mnemonics. Our approach, an end-to-end pipeline for auto-generating verbal and
visual cues, can automatically generate highly memorable cues. We investigate
the effectiveness of our approach via a human participant experiment by
comparing it with manually generated cues.

###### Keywords:

Keyword Mnemonic Vocabulary Learning Large Language Models

##  1 Introduction

Learning vocabulary is key to learning second (mostly foreign) languages, but
also a difficult task. One of the most well-known and effective methods is
flashcards, i.e., writing the L2 (a second language word) word on the front
and writing down the corresponding L1 word (a first or native language word)
on the back, with content such as mnemonic or context. Moreover, one may
manage flashcards by putting the cards in boxes to follow the Leitner system
[13] to recall the word regularly following the forgetting curve [8]. However,
both writing down every word and managing a bunch of cards require significant
effort and can take a lot of effort from learners.

Technology advances have enabled vocabulary learning to shift from manually
writing down the words to using software systems such as Anki [10] and Quizlet
[21], which make language learning more efficient and engaging. Some systems
use ideas behind intelligent tutoring systems to model the learner’s knowledge
state to intervene in the retrieval practice [18, 22, 23]. Many studies have
shown that managing retrieval practice and designing personalized schedules
using cognitive models can significantly improve learning efficiency [7, 12].
Many systems also use gamified interfaces and enable learners to share decks
with others, making the learning process more interactive and socially
relevant [1, 10, 21]. However, despite these advances, the learning _content_
, i.e., what is written on the flashcard, has mostly stayed the same
throughout the years.

Regarding the content for second language learning, keyword mnemonic [3] is a
notable memory encoding strategy that uses interactive visual imagery with a
keyword that sounds like part of a foreign word. Forming the keyword-based
interactive image takes a two-step approach: creating first an acoustic and
then an imagery link. Imagine a native English speaker is learning the Spanish
word pato, which means duck. The keyword that sounds like the word is pot.
Using the keyword, the learner first creates an acoustic link between the
keyword and the Spanish word. Then, the learner builds an imagery link that
connects the sound and its meaning by using a verbal cue, such as “A duck
wearing a pot on its head.” By relating new information to existing knowledge,
learners have an easier time memorizing the word and can retain it in memory
for a longer time.

Previous studies on keyword mnemonics have shown their effectiveness compared
with different learning strategies. Comparing keyword mnemonic with rote
rehearsal and combining both strategies showed that the keyword group
outperformed the other two groups [5]. Comparing the keyword mnemonic group
with verbal and visual cues with mixed methods of contextual clues, word
structure analysis, and opposite word pairs showed that the keyword group
performed better in both short-term and long-term retention [20]. However,
since the cues given in these studies are manually generated by experts, it is
difficult to employ this approach at a large scale in the systems mentioned
above.

In 2014, Savva et al. introduced an automatic keyword generation approach
based on a cross-lingual system, TransPhoner [19]. It evaluates candidate
keywords in the second language using the following measures for a given input
word: imageability, phonetic similarity, orthographic similarity, and semantic
similarity. The authors experimented on the effectiveness of TransPhoner using
an evaluation set of 36 German words [9] with three other conditions: no
keywords, randomly sampled keywords, and manually generated keywords. The
result shows that the TransPhoner-generated condition achieved the highest
score and the manually-generated keyword condition had no significant
difference from randomly generated keywords.

Despite TransPhoner’s success in automatically generating keywords as cues,
other forms of richer verbal or visual cues that could further help learners
build an imagery link cannot be automatically generated. The learner (or
teacher) still needs to manually develop them to connect the keyword and the
L1 word, which requires a lot of effort on their part. Moreover, it takes an
expert to come up with an image as the visual cue that corresponds to the
verbal cue. Using image APIs such as Google Image API, one can juxtapose
images of a keyword and an L1 word, but doing is not as effective as showing
both words together in a single image. To make keyword mnemonic scalable, we
need an end-to-end solution that takes words as input and generates keyword,
verbal and visual cues.

Contributions. In this paper, we detail a pipeline for automatically
generating verbal and visual cues in one shot via text generator and text-to-
image generator. Our contributions are as follows:

  * •

We propose a large language model (LLM)-based pipeline that automatically
generates highly memorable verbal and visual cues for an L1 word in language
learning. We believe that our automated approach will significantly reduce
content development costs by enhancing time efficiency and reducing manual
generation effort. To the best of our knowledge, we are the first to apply
LLMs in the context of keyword mnemonic.

  * •

We implement a web application for human participant studies and use it to
compare our approach with existing ones. We analyze the effectiveness of four
approaches: automatically generated keyword only, automatically generated
keyword with a verbal cue, automatically generated keyword with both verbal
and visual cues, and manually generated keyword and verbal cues. We also
outline avenues for future work that could stem from our approach.

##  2 Methodology

In this section, we detail our pipeline for automatically generating cues. Our
work is driven by the following two research questions:

  * •

Can we automatically generate human-level verbal cues for the keyword?

  * •

Can we generate a visual cue that may facilitate building an imagery link that
is described in a verbal cue?

We narrow the scope of automatically generating verbal and visual cues to the
experiments conducted in previous studies [9, 19] in this preliminary effort.
We use the evaluation set of 36 German words and keywords from previous
studies for both manually and automatically generated cues as baselines. Since
verbal cues only exist for manually generated keywords, our task boils down to
automatically generating verbal cues using TransPhoner-generated keywords and
generating visual cues using verbal cues.

###  2.1 Pipeline for Auto-generating Verbal and Visual Cues

Figure 1: Our end-to-end pipeline for automatically generating verbal and
visual cues for an L2 word.

We propose a pipeline consisting of two LLMs that generate verbal and visual
cues in two steps: First, we use a text generator to automatically generate a
sentence containing the TransPhoner keyword as the verbal cue. Second, we use
a text-to-image generator to generate an image as the visual cue. LLMs, pre-
trained with massive datasets, have shown human-level performance on the tasks
we described above through prompts. This is because LLMs are good for
controllable text generation [17] and following instructions [16]. With proper
prompts, models show their ability to solve the tasks with zero-shot or few-
shot setups. We use zero-shot setup LLMs for both generating verbal and visual
cues. We detail the pipeline through an example in Fig. 1 where we need to
generate cues for the German word flasche, which means a bottle. The keyword
generated by TransPhoner is flashy; Using the keyword and the meaning of the
word, we create the prompt: “Write a short, catchy sentence that connects
flashy and bottle.” Additionally, we add a constraint on verbal cues to start
with “Imagine” for two reasons. First, verbal cues in the previous study [9]
are in that format. Since we are trying to answer whether we could achieve
human-level verbal cues, we match the format. Second, we follow grammatical
characteristics that come after the word imagine. After the word “Imagine”,
usually a noun or gerund comes out; we found that the generated verbal cue
contains fewer ambiguous pronouns, which makes the cue more descriptive. This
feature is key to linking the text generator and text-to-image generator
within the same pipeline. Using the prompt, our text generator, GPT-3 [6]
(text-davinci-003, temp=0.5), generates the verbal cue. Then, we reuse the
verbal cue as the prompt for our text-to-image generator, DALL-E 2 [15], by
removing the word “Imagine”. One can freely choose any LLMs to automatically
generate these verbal and visual cues. We present the gray region in Fig. 1 to
the participant as learning content.

##  3 Experimental Evaluation

In this section, we detail our experiments on presenting different content to
different participants to explore whether automatically generated verbal and
visual cues are effective in vocabulary learning.

###  3.1 Experimental Design

In the experiment, participants learn 36 German words and are tested on
recalling both the German word (generation) and its English meaning
(recognition). The words are split into three sets, which means that each
participant goes through the learning, recognition, and generation cycle three
times. Words in each set are randomly shuffled for each participant. At the
end of the experiment, we also ask participants to rate the helpfulness of the
cues.

####  3.1.1 Learning and Testing

We provide each participant with both instructions on the study and the
content that helps them learn the word; see Section 3.2 for details. Each word
has a 30-second limit for the participant to memorize, and the participant can
choose to move on to the next word after 15 seconds. After 30 seconds, we
automatically move on to the next word. German words are pronounced twice,
after 2 seconds and 7 seconds, respectively, after being displayed. We show a
timer to participants to make them aware of the time remaining for each word.
Participants have 15 seconds for both recognition and generation during
testing. To avoid confusion between the two tests, we provide instructions
such as “What is this in English?” and “What is this in German?”. For
generation, we also ask participants to use a, o, u, s instead of Umlaut ä, ö,
ü, ß. We show a timer to participants as well. Words in both tasks are
randomized in order.

####  3.1.2 Participants

We recruit participants from Amazon Mechanical Turk [2]. We require
participants to be native English speakers with no German language experience.
Considering the experiment takes about 40 minutes, we paid each participant
$7.25 and added a bonus of $2.75 for those who got a score of over 70% on the
final test. The bonus encourages participants to do their best. However, we
acknowledge that some participants may cheat on tests to achieve a high score
by using an external dictionary, which we have no control of.

####  3.1.3 Web Interface

We implement a React web application as our participant interface, which is
designed based on the previous study [19]. We place an IRB-approved consent
form on the front page and only participants who agree can participate in the
experiment; the form explains in detail how the experiment is structured. We
also show an example with a German word not in our evaluation set to clarify
the procedure to participants. We collect metadata on time spent both during
learning and testing, along with the responses to further investigate
participant behavior.

###  3.2 Experimental Conditions

We first divide participants into two groups based on how the keyword was
generated: automatically (auto-cue) and manually (manual-cue). Among many
combinations of verbal and visual cues that can be presented to the
participants, we choose conditions that enable both intra- and inter-group
comparisons. We recruit a total of 80 participants for our study, with 20 in
each condition.

As shown in Fig. 2, we show the example of our web interface on how the
content is displayed in different conditions. For intra-group comparisons, we
further divide the auto-cue group into three conditions: Condition I is only
provided with the TransPhoner-generated keyword, Condition II is provided with
the keyword and the verbal cue generated by our pipeline, and Condition III is
provided with the keyword and both the verbal and visual cues generated by our
pipeline. For the inter-group comparisons, we provide both the auto-cue group
and manual-cue group with information in Condition II. We note that the
previous study [19] compared the groups with Condition I by not including
verbal cues that were originally presented with the manually generated
keywords [9]. The manually generated verbal cue and keyword should be
considered as a whole since the keyword might have been chosen to provide a
verbal cue with the best imageability among many keyword candidates.

Figure 2: A snapshot of our web interface shown to experiment participants.

We refer to these four conditions as Auto-I, Auto-II, Auto-III, and Manual-II.
The instructions for each condition are shown in Table 1. We use the same
instructions for Condition I from Savva et al. Our instructions for Condition
II tell participants to create an imagery of a scene specified in a verbal
cue. Our instructions for Condition III tell participants to remember the
image, which is based on the verbal cue that describes a specific scene.

Table 1: Cues and instructions we used for different experimental conditions. Cond. | Cue | Instruction
---|---|---
|  Keyword |  Verbal |  Visual
I |  yes |  no |  no |  Imagine a visual scene connecting the given keyword with the English meaning, and the sound of the German word.
II |  yes |  yes |  no |  Imagine a specific scene described in the verbal cue that connects the given keyword with the English meaning, and the sound of the German word.
III |  yes |  yes |  yes |  Remember the image by following the verbal cue that connects the given keyword with the English meaning, and the sound of the German word.
|  |  |  |

###  3.3 Evaluation Metrics

We use different metrics to score recognition and generation. For recognition,
we use cosine similarity between the word embeddings [4] between the answer
and the response. We also consider responses that miss “to” for
“to”-infinitives to be correct. Unlike recognition, as a novice German
learner, generation is bounded to the orthographic feature of vocabulary.
Therefore, we use a standardized (subtracting 1 and normalizing to 1)
Levenshtein distance to score generation, following previous studies [19]. We
also ask participants to evaluate the helpfulness of the cues using a 5-point
Likert scale, which is provided along with the entire 36 words and the cues.

###  3.4 Results and Discussion

After we exclude participants who did not understand the experiment properly,
such as those who wrote down the keyword when recalling the English meaning,
we have a total of 72 participants: Auto-I (20) with an average age of 25.4
years (SD = 2.3), Auto-II (17) with an average age of 24.2 years (SD = 1.7),
Auto-III (18) with an average age of 24.8 years (SD = 1.6), and Manual-II (17)
with an average age of 25.3 years (SD = 1.1).

Fig. 3 shows per-participant experimental data in box plots averaged among 36
German words. Learning time is time spent memorizing a word, while testing
time is the average time on recognition and generation of the word. Similarly,
the combined score is an average of recognition and generation scores.
Learning time, testing time, and Likert scale are normalized by their maximum
value.

The median of time spent on learning was 19.8, 18.9, 18.6, and 19.2 seconds,
respectively, for the four conditions out of the 30 seconds time limit, which
may suggest that cognitive load across different conditions is similar. The
median of time spent on testing, i.e., the average time spent on recognition
and generation, was 8.85, 9.75, 8.7, and 7.95 seconds out of the 15 seconds
time limit. The median of the 5-point Likert scale was 4.2, 3.95, 4.25, and
4.4.

Now, we analyze the combined score based on the per-word combined score, as
shown in Fig. 4. We perform a one-tailed Welch’s t-test assuming unequal
variances on the hypotheses of one condition being better than another. We set
our level of significance to 5%. We detail each hypothesis below. Case A, B,
and C in Fig. 4 are words we present with content generated through our
pipeline for qualitative analysis.

Figure 3: Box plots of per-participant data for each experimental condition.

####  3.4.1 Auto-I vs. Auto-II: Does a verbal cue help learning?

We hypothesize that Auto-II, with additional verbal cues, will result in
better recognition and generation scores than Auto-I, which uses only
keywords. We define our null hypothesis ($H_{0}$) and alternate hypothesis
($H_{a}$) as follows:

  * •

$H_{0}$: $\mu_{Auto-II}\leq\mu_{Auto-I}$

  * •

$H_{a}$: $\mu_{Auto-II}>\mu_{Auto-I}$

A right-tailed test shows there is no significant effect of verbal cues,
$t(33)=-1.79,p=0.96$; we cannot reject $H_{0}$. On the contrary, a left-tailed
test shows statistical significance in favor of the keyword-only condition,
$t(33)=1.79,p=0.04$. This result can be explained by several factors: The
participants might have done rote rehearsals instead of building links as
instructed in Table 1. Moreover, participants may come up with their own
verbal cues that are more memorable than automatically generated ones.
Personalized by default, participants’ own verbal cues may be a better fit for
each individual’s own experience.

####  3.4.2 Auto-II vs. Manual-II: Are automated verbal cues effective?

We hypothesize Manual-II to be an upper bound of Auto-II since the former cues
are generated by experts in psycholinguistics. Therefore, we define our null
hypothesis and alternate hypothesis as follows:

  * •

$H_{0}$: $\mu_{Manual-II}\leq\mu_{Auto-II}$

  * •

$H_{a}$: $\mu_{Manual-II}>\mu_{Auto-II}$

A right-tailed test shows that there is no significant difference between the
two conditions, $t(24)=-0.32,p=0.62$; we cannot reject $H_{0}$. In Fig. 4, we
show three words where participants perform better in the Auto-II condition
than Manual-II (case A) and otherwise (case B), respectively. Case A in Table
2 shows that auto-generated cues are more memorable than manual cues even with
a grammatical error (risen should be raised) or not realistic (Reuben sandwich
calling your name). Case B, on the other hand, contains keywords where auto-
generated cues are not frequently used (Triton, frizzy) or making it hard to
imagine (a wagon with stories). This result implies that although we can
automatically generate high-quality verbal cues, choosing appropriate keywords
remains crucial. Therefore, we need to add keyword generation to the pipeline
and evaluate the quality of both generated keywords and the verbal cue.

Figure 4: Per-word combined score for all four experimental conditions, with three cases highlighting some words that work especially well with certain cues. Table 2: Examples of automatically and manually generated verbal cues. A keyword is represented in italic, while a meaning is in bold. Case | Word |  Auto |  Manual
---|---|---|---
A | Treten |  Imagine stepping into treason, a treacherous path that can never be undone. |  Imagine you step on a stair tread.
Rasen |  Imagine a risen lawn that is lush and green! |  Imagine your lawn covered in raisins.
Rufen |  Imagine Reuben calling out your name! |  Imagine you call a friend to put a new roof on a cottage.
B | Streiten |  Imagine Triton and his trident quarreling with the waves. |  Imagine you quarrel about the Menai straits.
Sagen |  Imagine a wagon full of stories just waiting to be told! |  Imagine you tell someone sago is good for them.
Friseur |  Imagine a hairdresser who can tame even the most frizzy hair! |  Imagine your hairdresser inside a freezer.
C | Nehmen |  Imagine Newman taking the initiative to take action! |  Imagine you take a name in your address book.
Brauchen |  Imagine needing to fix a broken heart. |  Imagine brokers need much experience.

####  3.4.3 Auto-II vs. Auto-III: Does a visual cue help learning?

We hypothesize better performance by Auto-III, which uses additional visual
cues, than Auto-II. Therefore, we define our null hypothesis and alternate
hypothesis as follows:

  * •

$H_{0}$: $\mu_{Auto-III}\leq\mu_{Auto-II}$

  * •

$H_{a}$: $\mu_{Auto-III}>\mu_{Auto-II}$

A right-tailed test shows that there is no significant difference between the
two conditions, $t(32)=0.39,p=0.35$; we cannot reject $H_{0}$. In Fig. 4, we
show three words for the cases where participants perform better in the Auto-
III condition than in Auto-II (case B) and two for when it does not (case C),
respectively. Case B shows that Auto-III, which has additional visual cues
than Auto-II, performs similarly as Manual-II. Considering the previous
comparison that Auto-II has a lower score than Manual-II, we see that Auto-III
does somewhat outperform Auto-II. Therefore, we can conclude that visual cues
help participant build the imagery link to some degree.

For a more qualitative analysis, Fig. 5 shows visual cues generated by our
pipeline. Fig. 5 (a-c) shows that visual cues may be helpful in cases where
keywords that lack imageability and are not frequently used (Triton, frizzy)
or in cases where auto-generated verbal cues are hard to imagine (a wagon with
stories). However, as shown in case C, visual cues for abstract words (to
take, to need) do not help much. Fig. 5 (d-e) shows that in these cases the
generated image is not descriptive enough to facilitate the imagery link.
Interestingly, the Likert scale score was higher for Auto-III than Auto-II in
every word except one. This result implies that participants think it is
helpful to have additional visual cues. However, we cannot create effective
visual cues for every word. Generating descriptive visual cues, especially for
abstract words, remains a challenging task.

Case B |  |  |
---|---|---|---

(a) Streiten

(b) Sagen

(c) Friseur

Case C |  |
---|---|---

(d) Nehmen

(e) Brauchen

Figure 5: Examples of visual cues generated by our pipeline in cases where
they are helpful to participants and cases where they are not.

##  4 Conclusions and Future Work

In this paper, we explored the opportunity of using large language models to
generate verbal and visual cues for keyword mnemonics. A preliminary human
experiment suggested that despite showing some promise, this approach has
limitations and cannot reach the performance of manually generated cues yet.

There are many avenues for future work. First, we need a larger-scale
experiment in a real lab study, which provides us a controlled environment to
test both short-term and long-term retention. Since we only tested short-term
retention, it is possible that no approach can significantly outperform
others. We also need more psycholinguistics perspectives on constraining time
spent on learning and testing. By conducting the research in a more controlled
environment, we can use additional information (e.g., demographics, language
level) to help us conduct a deeper analysis of the results. We do clarify that
using Amazon’s Mechanical Turk to conduct experiments is standard in prior
work, which is part of the reason why we chose this experimental setting. To
track long-term retention, we likely have to resort to knowledge tracing
models that handle either memory decay [11] or open-ended responses [14].
Second, we can extend our pipeline by generating the keyword automatically as
well instead of using TransPhoner-generated keywords, which may make our
approach even more scalable. One important aspect that must be studied is how
to evaluate the imageability of the keywords and verbal cue that contains both
keywords and vocabulary, which remains challenging. Third, we can generate
personalized content for each participant. We may provide additional
information in the text generator about the topic they are interested in that
we could use to generate a verbal cue. Moreover, we can generate a story that
takes all words into account. It is also possible to generate verbal cues in
L2 as well, which may help learners by providing even more context. Fourth,
instead of the pronunciation of the word, we can use other features in
language to generate verbal cues. For example, when learning Mandarin,
memorizing Chinese characters is as important as learning how to pronounce the
word. The Chinese character 休 means rest, which is xiū in Mandarin. The
character is called a compound ideograph, a combination of a person (人) and a
tree (木), which represents a person resting against a tree. Combined with a
keyword, shoe, for example, we could accomplish two goals with one verbal cue,
“A person is resting by a tree, tying up their shoe.” This way, we can make
visual cues more descriptive for abstract words.

##  5 Acknowledgements

The authors thank the NSF (under grants 1917713, 2118706, 2202506, 2215193)
for partially supporting this work.

## References

  * [1] Ahn, L.v.: Duolingo, https://www.duolingo.com
  * [2] Amazon: Amazon mechanical turk, https://www.mturk.com
  * [3] Atkinson, R.C., Raugh, M.R.: An application of the mnemonic keyword method to the acquisition of a russian vocabulary. Journal of experimental psychology: Human learning and memory 1(2), 126 (1975)
  * [4] Bojanowski, P., Grave, E., Joulin, A., Mikolov, T.: Enriching word vectors with subword information. Transactions of the association for computational linguistics 5, 135–146 (2017)
  * [5] Brahler, C.J., Walker, D.: Learning scientific and medical terminology with a mnemonic strategy using an illogical association technique. Advances in physiology education 32(3), 219–224 (2008)
  * [6] Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J.D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al.: Language models are few-shot learners. Advances in neural information processing systems 33, 1877–1901 (2020)
  * [7] Carrier, M., Pashler, H.: The influence of retrieval on retention. Memory & cognition 20, 633–642 (1992)
  * [8] Ebbinghaus, H.: Memory: A contribution to experimental psychology. Annals of neurosciences 20(4), 155 (2013)
  * [9] Ellis, N.C., Beaton, A.: Psycholinguistic determinants of foreign language vocabulary learning. Language learning 43(4), 559–617 (1993)
  * [10] Elmes, D.: Anki, http://ankisrs.net
  * [11] Ghosh, A., Heffernan, N., Lan, A.S.: Context-aware attentive knowledge tracing. In: Proc. ACM SIGKDD. pp. 2330–2339 (2020)
  * [12] Larsen, D.P., Butler, A.C., Roediger III, H.L.: Repeated testing improves long-term retention relative to repeated study: a randomised controlled trial. Medical education 43(12), 1174–1181 (2009)
  * [13] Leitner, S.: So lernt man lernen. Herder (1974), https://books.google.com/books?id=opWFRAAACAAJ
  * [14] Liu, N., Wang, Z., Baraniuk, R., Lan, A.: Open-ended knowledge tracing for computer science education. In: Conference on Empirical Methods in Natural Language Processing. pp. 3849–3862 (2022)
  * [15] OpenAI: Dall-e 2, https://openai.com/dall-e-2
  * [16] Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al.: Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems 35, 27730–27744 (2022)
  * [17] Prabhumoye, S., Black, A.W., Salakhutdinov, R.: Exploring controllable text generation techniques. arXiv preprint arXiv:2005.01822 (2020)
  * [18] Reddy, S., Labutov, I., Banerjee, S., Joachims, T.: Unbounded human learning: Optimal scheduling for spaced repetition. In: Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining. pp. 1815–1824 (2016)
  * [19] Savva, M., Chang, A.X., Manning, C.D., Hanrahan, P.: Transphoner: Automated mnemonic keyword generation. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. pp. 3725–3734 (2014)
  * [20] Siriganjanavong, V.: The mnemonic keyword method: Effects on the vocabulary acquisition and retention. English Language Teaching 6(10), 1–10 (2013)
  * [21] Sutherland, A.: Quizlet, http://quizlet.com
  * [22] Ye, J., Su, J., Cao, Y.: A stochastic shortest path algorithm for optimizing spaced repetition scheduling. In: Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. pp. 4381–4390 (2022)
  * [23] Zylich, B., Lan, A.: Linguistic skill modeling for second language acquisition. In: LAK21: 11th International Learning Analytics and Knowledge Conference. pp. 141–150 (2021)