11institutetext: Dublin City University, Ireland 22institutetext: VNU-HCM,
University of Science, Vietnam

# An Improved Subject-Independent Stress Detection Model Applied to Consumer-
grade Wearable Devices

Van-Tu Ninh Both authors contributed equally to this research.11 Manh-Duy
Nguyen 11 ⋆ ⋆ Sinéad Smyth 11 Minh-Triet Tran 22 Graham Healy 11 Binh T.
Nguyen 22 Cathal Gurrin 11

###### Abstract

Stress is a complex issue with wide ranging physical and psychological impacts
on human daily performance. Specifically, acute stress detection is becoming a
valuable application in contextual human understanding. Two common approaches
to training a stress detection model are subject-dependent and subject-
independent training methods. Although subject-dependent training methods have
proven to be the most accurate approach to build stress detection models,
subject-independent models are a more practical and cost-efficient method, as
they allow for the deployment of stress level detection and management systems
in consumer-grade wearable devices without requiring training data for the end
user. To improve the performance of subject-independent stress detection
models, in this paper, we introduce a stress-related bio-signal processing
pipeline with a simple neural network architecture using statistical features
extracted from multimodal contextual sensing sources including Electrodermal
Activity (EDA), Blood Volume Pulse (BVP), and Skin Temperature (ST) captured
from a consumer-grade wearable device. Using our proposed model architecture,
we compare the accuracy between stress detection models that use measures from
each individual signal source, and one model employing the fusion of multiple
sensor sources. Extensive experiments on the publicly available WESAD dataset
demonstrate that our proposed model outperforms conventional methods as well
as providing $1.63\%$ higher mean accuracy score compared to the state-of-the-
art model while maintaining a low standard deviation. Our experiments also
show that combining features from multiple sources produces more accurate
predictions than using only one sensor source individually.

###### Keywords:

Affective Computing, Stress Detection Model, Human Context, Multimodal Sensing

##  1 Introduction

The development of sensor technology in recent years has lead to the
availability of both consumer-grade and medical-grade wearable devices which
has facilitated research into personal sensing with applications in self-
quantification, lifelogging, and healthcare [11]. This has also resulted in
the creation of many large multimodal personal datasets [11] comprising of
different data types (e.g, passive visual capture, mobile device context,
physiological data) [8] that enables research community to develop intelligent
systems to track individual’s health and gain more insights of an individual’s
personal data such as daily-life event segmentation [9], activities of daily-
living identification as an indicator in health tracking systems [10], etc.
Although multiple data sources are recorded in multimodal personal datasets
[12], only the combination of visual and related metadata including semantic
locations, daily-life activities, date and time are employed extensively in
research [10, 20] while others has not yet been exploited. Typically,
physiological signals are usually ignored due to the limited amount of
research conducted using this type of data as well as the limitations of
recording devices in terms of the granularity signal measurement. Since
consumer-grade wearable devices for health tracking increasingly allow the
capture of real-time physiological signals (e.g Empatica E4 wristband, Fitbit
sensors, Garmin watches), researchers are now able to gather multi-sensor-
source datasets as input for the study of developing automatic emotion
recognition system and automatic stress detection models. Despite the
possibilities, there has been a limited amount of research in this field to
date due to three commonly-known challenging mentioned in [6].

There are two conventional approaches to building stress detection models,
which correspond to two different training methods: subject-independent models
and subject-dependent models. The hypothesis of the subject-dependent stress
detection model is that the physiological response to stress stimuli is
different for each individual and the stress monitoring systems need to adapt
to the stress pattern of each person [24]. Therefore, stress detection models
are likely to perform more accurately when they are trained with each
individual’s data instead of using external data from various individuals.
Nkurikiyeyezu et al. found that this hypothesis holds true by comparing the
accuracy score of both subject-dependent and subject-independent stress
detection models trained on high-resolution EDA and ECG signals using the
SWELL [17] and WESAD [23] datasets [22]. Hence, most research to date has
concentrated on the application of subject-dependent stress detection models
while ignoring subject-independent models despite their practicality and cost-
efficiency for consumer-grade application scenarios. In order to address these
issues, we investigate the usage of physiological data recorded from consumer-
grade wearable devices for automatic stress detection and propose a new model
that improves the accuracy of subject-independent stress detection. In
summary, we present three main contributions of this paper:

  1. 1.

Through extensive experiments, we prove that the fusion of multiple sensor
sources of the consumer-grade wearable device can enhance the accuracy of
stress detection models compared to when using each signal individually.

  2. 2.

We propose a bio-signal processing pipeline with a novel training method for
the subject-independent models that learn stress/non-stress patterns of EDA,
BVP, ST, and their fusion.

  3. 3.

Our proposed model outperforms traditional Machine Learning methods as well as
being $1.63\%$ more accurate than the state-of-the-art model on the same
experiment dataset.

##  2 Related Work

Various human contextual data sources, such as Heart rate (HR), Heart Rate
Variability (HRV), and Electrodermal Activity (EDA) are found to be
discriminative signals for stress level measurement [1]. Using such multi-
modal contextual signals, Nkurikiyeyezu et al., Schmidt et al., and Siirtola
manage to build high-accuracy subject-dependent stress detection models [26,
22, 23]. Results of these works show that subject-dependent models
outperformed subject-independent ones and the gap between the performance of
the two models is huge.

In 2018, Schmidt et al. released a public multimodal dataset named WESAD which
captured both high-resolution and low-resolution physiological contextual
signals of 15 participants under different conditions [23]. They also provided
preliminary work on their dataset by training a subject-independent stress
detection model. In a binary classification task, they achieved an average
accuracy score of $88.33\%$ ($0.25$) using Random Forest classifiers trained
on combinations of low-resolution sensor signals (EDA, BVP, TEMP). However, as
the number of stress and non-stress samples in the WESAD dataset is unequal,
this accuracy score cannot reflect the stress detection capability of the
model completely as the model can achieve a high accuracy score by predicting
the value of the majority class for all predictions.

Siirtola continued to evaluate the performance of subject-independent stress
detection models using the same features as in the preliminary work of Schmidt
et al. but with another appropriate evaluation metric and different window
size [25]. The best model using Linear Discriminant Analysis (LDA) trained on
three signals which include Skin Temperature (ST), BVP, and HR achieves the
highest average balanced accuracy score of $87.4\%$ ($10.4$). This result is
high for subject-independent stress detection model. However, they suggested
that the significant variation in recognition accuracy between study subjects
can be alleviated by building subject-dependent stress detection instead.

In 2019, Nkurikiyeyezu et al. compared the performance of these two models
using high-resolution EDA and HRV signals from both WESAD [23] and SWELL [17]
dataset [22]. The accuracy score was chosen as the appropriate evaluation
metric for their works since they down-sampled the dataset randomly to balance
the number of samples in both classes. They provided evidence that the
subject-dependent stress detection model outperformed the subject-independent
one. They also proposed a hybrid calibrated model to improve the performance
of the subject-independent model from $42.5\%\pm 19.9\%$ to $95.2\%\pm 0.5\%$
by including a small number of samples of the unseen subject $(n=100)$ [22].
However, their proposed hybrid calibrated model is not an enhanced version of
the subject-independent model but a different way for subject-dependent model
training as it requires a small amount of stress/non-stress annotated samples
from the targeted subject. Additionally, their work was only limited to the
use of high-resolution signals, which are usually recorded using laboratory
devices only, without analysing the performance of their models on low-
resolution signals captured from consumer-grade wearable devices. More work is
needed to analyse the possibility of improving the performance of the subject-
independent stress detection model trained on low-resolution physiological
signals.

##  3 Stress Detection Dataset

To improve the stress prediction accuracy of the subject-independent model, we
conducted experiments on the benchmark dataset that is used extensively in
related works [22, 25, 13] to evaluate the performance of stress detection
model. The benchmark dataset named WESAD [23] consists of four different types
of low-resolution physiological data collected from 15 participants under two
different study protocols in a laboratory environment. The low-resolution
physiological signals including accelerometer (ACC), skin temperature (ST),
Blood Volume Pulse (BVP), and Electrodermal Activity (EDA) are recorded using
the Empatica E4 medical-grade wearable sensor, which facilitates real-time
physiological data acquisition regardless of user context. Among the four
signals, only three bio-signals which are related directly to the response of
acute stress includes EDA, BVP, and ST. In our work, we concentrate on
analysing the use of low-resolution EDA, BVP, and ST signals, which are
recorded with a sampling rate of 4 Hz, 64 Hz, and 4 Hz respectively, to
improve the stress prediction accuracy of a subject-independent model. Each
study protocol in the dataset comprises of amusement, stress, meditation, and
baseline conditions in different orders for each participant. Details of the
four affective conditions are as follows:

  1. 1.

Baseline Condition: This condition lasts for 20 minutes which aims to capture
the neutral affective state of the participant. The participant is asked to
sit or stand at a table with neutral reading material.

  2. 2.

Amusement Condition: The participant watches a set of eleven funny video
clips. A short neutral time period of 5 seconds is presented between the video
clips. The total length for this condition is 392 seconds.

  3. 3.

Stress Condition: The participant is exposed to the Trier Social Stress Test
(TSST) [16], where they are required to provide a five-minute speech on their
strengths and weaknesses in front of a panel that includes three human
resource specialists. Finally, the participant counts down from 2023 in
decrements of 17, and is requested to start over if they make a mistake. The
total length of this condition is about 10 minutes.

The total duration of the study protocol is about two hours, which is
considered to be long enough to capture sufficient physiological data for
stress detection model training. Since previous works using this dataset
employ study protocol is used as the ground-truth for constructing the stress
detection model [22, 23, 25, 21], we also use the same ground-truth as in
previous works for consistent comparison of stress prediction accuracy of the
proposed models. In detail, the baseline and amusement condition are
classified into non-stress class while the stress condition is considered as
the stress one.

##  4 Experiments Description

In this research, we employ the bio-signals of the WESAD dataset, EDA, BVP,
and ST signals, that can be recorded from separate sensors integrated on a
low-cost consumer-grade wearable device to predict the stress pattern of the
individual. We also propose a bio-signal processing pipeline for each signal
individually before extracting statistical features, which is described in
4.1. Several statistical features identified in other researcher’s findings
are extracted from these signals, and then concatenated together to build our
prediction models. We then conduct many experiments with different training
approaches to evaluate the effectiveness of combining signals in subject-
independent models.

###  4.1 Bio-signal Processing and Statistical Feature Extraction of EDA, BVP,
and ST

For both EDA and BVP, we extract statistical features using NeuroKit2
package111https://github.com/neuropsychology/NeuroKit [18] and HRV-analysis
library222https://github.com/Aura-healthcare/hrv-analysis for each 60-second
segment. The window shift used in our experiment is 0.25 second. The values of
the window size and window shift are the same as in the original paper of
WESAD dataset for consistency when comparing the prediction results of the
models [23]. As the physiological signals vary from person to person, we
employ feature normalisation method to reduce the difference people’s
physiological responses. In addition, since the signals recorded using
consumer-grade wearable device such as EDA, BVP, etc. contain many types of
noise, we utilise many signal processing techniques to remove noises, baseline
drifts, and outliers in the raw signal. These steps are combined together to
clean the raw signal before extracting statistical feature, which is
considered to be a bio-signal processing flow to improve the quality of the
extracted feature.

For the EDA, the raw signal in each 60-second segment is firstly pre-processed
to remove motion artifacts using the wavelet-based adaptive denoising
procedure as described in [2]. The signal is then filtered by a fourth-order
Butterworth low-pass filter with cut off frequency of 0.5 Hz to remove line
noise. The min-max normalization is then applied to the cleaned signal to
remove the inter-individual difference before it is inputted into the
NeuroKit2 package to decompose into Skin Conductance Response (SCR) and Skin
Conductance Level (SCL) using the cvxEDA method [7]. Other characteristics of
SCR including SCR Peaks, SCR Onsets, and SCR Amplitude are also extracted.
Finally, the statistical EDA features from three related works [3, 22, 23] are
computed which resulted in a 36-dimensional vector.

For the BVP, we firstly clean the raw signal in each window segment by
removing the outlier values over the $98^{\text{th}}$ and below the
$2^{\text{th}}$ percentile using winsorisation method as in [5] and removing
the baseline drift using Butterworth high-pass filter with cut-off frequency
of $0.5$ Hz as in [14]. We then apply min-max normalization to the cleaned
signal to minimise the physiological signal difference between individuals
before following the previous research [21] to employ the Elgandi processing
pipleline [4] to clean the photoplethysmogram (PPG) signal [19] and detect
systolic peaks. The systolic peaks are used to compute a list of RR-intervals,
which are then pre-processed using the hrv-analysis package to remove outliers
and ectopic beats [28] as well as interpolating missing values. The cleaned
RR-intervals are used to compute the NN-intervals, which are main items to
compute time-domain, frequency-domain, geometrical, and Poincare-plot
features. For frequency-domain HRV features, we employ the same parameters of
low (LF: 0.04-0.15 Hz) and high (HF: 0.15-0.4 Hz) frequency bands as in [23].
The range of very-low frequency band used in our work is the same as in HRV-
analysis package (0.003-0.04 Hz). In summary, we inherit most of the HRV
features from [22, 23] and combined them into a 30-dimensional vector.

For the ST, the statistical features are extracted on the raw 60-second
segment signal as in [23]. The fusion of statistical features from three
signal-sources is a 72-dimensional vector. The detail of extracted features is
shown in Table 1.

Table 1: List of extracted features. Abbreviations: # = number of, $\sum$ = sum of, STD = standard deviation, RMS = Root Mean Square. |  Feature |  Description
---|---|---
EDA |  $\mu_{EDA}$, $\sigma_{EDA}$, $\min_{EDA}$, $\max_{EDA}$ |  Mean, STD, min, max of the EDA
$\partial_{EDA}$ |  Slope of the EDA
$\text{range}_{EDA}$, $\text{range}_{SCR}$ |  Dynamic range of EDA and SCR
$\mu_{SCL}$, $\sigma_{SCL}$ |  Mean, STD of the SCL
$\text{corr}(SCL,t)$ |  Correlation btw SCL and time
$\\#_{Peak}$ |  $\\#$ identified SCR peaks
$\sum_{SCR}^{Amp}$, $\sum_{SCR}^{t}$ |  $\sum$ SCR startle magnitudes and response durations
$\int_{SCR}$ |  Area under the identified SCRs
$\mu_{SCR}$, $\sigma_{SCR}$, $\max_{SCR}$, $\min_{SCR}$ |  Mean, STD, min, max of the SCR
$\mu_{\nabla_{SCR}}$, $\sigma_{\nabla_{SCR}}$, $\mu_{\nabla(\nabla_{SCR})}$, $\sigma_{\nabla(\nabla_{SCR})}$ |  Mean and STD of the 1st and second derivative of the SCR
$\mu_{Peak}$, $\sigma_{Peak}$, $\max_{Peak}$, $\min_{Peak}$ |  Mean, STD, min, max of SCR Peaks
$\text{kurtosis}(SCR)$, $\text{skewness}(SCR)$ |  Kurtosis and skewness of SCR
$\mu_{Onset}$, $\sigma_{Onset}$, $\max_{Onset}$, $\min_{Onset}$ |  Mean, STD, min, max of SCR Onsets
ALSC = $\sum_{n=2}^{N}\sqrt{1+(r[n]-r[n-1])^{2}}$ |  Arc length of the SCR
INSC = $\sum_{n=1}^{N}\left|r[n]\right|$ |  Integral of the SCR
APSC = $\dfrac{1}{N}\sum_{n=1}^{N}r[n]^{2}$ |  Normalized average power of the SCR
RMSC = $\sqrt{\dfrac{1}{N}\sum_{n=1}^{N}r[n]^{2}}$ |  Normalized RMS of the SCR
BVP |  $\mu_{HR}$, $\sigma_{HR}$ |  Mean and STD of Heart Rate
$\mu_{HRV}$, $\sigma_{HRV}$ |  Mean and STD of HRV
$\text{kurtosis}(HRV)$, $\text{skewness}(HRV)$ |  Kurtosis and Skewness of HRV
$f^{VLF}_{HRV}$, $f^{LF}_{HRV}$, $f^{HF}_{HRV}$, $f^{LFNorm}_{HRV}$, $f^{HFNorm}_{HRV}$ |  Very low (VLF), Low (LF), and High (HF) frequency band in the HRV power spectrum and their normalized values.
$f^{LF/HF}_{HRV}$ |  Ratio of HRV LF and HRV HF.
$\sum\limits_{x\in\\{\text{VLF, LF, HF}\\}}^{f}$ |  $\sum$ of the freq. components in VLF-HF
$\text{NN}50$, $\text{pNN}50$, $\text{NN}20$ $\text{pNN}20$ |  # and percentage of HRV intervals differing more than 50 ms and 20ms.
HTI |  HRV Triangular index
$\text{rms}_{HRV}$ |  RMS of the HRV
SD1, SD2 |  Short and long-term poincare plot descriptor of HRV
RMSSD, SDSD |  RMS and STD of all interval of differences between adjacent RR intervals.
SDSD_RMSSD |  Ratio of SDSD over RMSSD.
RELATIVE_RR ($\mu$, median, $\sigma$, RMSSD, kurtosis, skewness) |  Mean, median, STD, RMSSD, kurtosis, and skewness of the relative RR.
ST |  $\mu_{ST}$, $\sigma_{ST}$, $\min_{ST}$, $\max_{ST}$ |  Mean, STD, min, max of ST
$\text{range}_{ST}$, $\partial_{ST}$ |  Range and slope of ST

###  4.2 Stress Detection Model Training Methodology

In this research, we build different classifiers to detect the stress
condition of each participant in the WESAD dataset. Two conventional machine
learning classifiers that are widely applied in this field which are Random
Forest (RF) and Support Vector Machine (SVM) are applied as baseline models
using our proposed feature extraction pipeline. The feature used for these
machine learning models is either a feature vector combined from three signals
(dimension of 72) or a feature vector of each signal only (dimension of 30 for
BVP, 36 for EDA, and 6 for ST). Additionally, we introduce a neural network
(NN) architecture that captures not only the local detail of EDA, BVP, and ST
separately but also the fusion of these signals. The neural network model, as
depicted in Figure 1, contains three distinct embedding modules for each
signal then a concatenating layer to learn the joint encoded features. The
model is then added with three different classification layers for three
branches which aims to optimise the performance of embedding stages of EDA,
BVP, and ST signals prior to the concatenating step. The overall loss used to
train the NN model is the sum of losses of all branches. In the testing phase,
the NN model makes a prediction based on the average of all branches in order
to gather the detail of each signal and their combined information. We also
integrate batch normalization and dropout techniques to make the model
converge faster and to address any over-fitting concerns.

Figure 1: The structure of our proposed neural network model. The numbers
indicate the number of features of a signal and also the number of units in
each layer.

We train two models (RF and SVM) using the Leave-One-Subject-Out (LOSO) scheme
as in [23]. For the NN model, the Leave-One-Subject-Out (LOSO) scheme is also
employed, however, the data is split into train set ($80\%$) and validation
set ($20\%$) using the Stratified Shuffle Split to deal with the imbalanced
nature of the ground-truth distribution in the dataset. Additionally, the
effectiveness of combining signals in stress detection is also considered. For
each scenario described above, we run four trials either using features of
each signal individually or using the fusion features from multiple sources.
In the first three trials, the NN model is trained and tested only with its
respective branch as illustrated in Figure 1. By conducting these runs, we
assess the effectiveness of using each signal in the stress prediction problem
and whether fusing signals can produce better prediction results.

###  4.3 Experimental Configuration

In our experiments, we set $250$ trees for the RF model with enabled out-of-
bag, bootstrap samples, max depth of 8, min sample splits of 2, and min sample
leaves of 4. The radial basis function kernel was used in the SVM model with
regularization parameter of 10. The remaining parameters of both RF and SVM
model are kept default as in sklearn library333https://scikit-
learn.org/stable/. Both models are set up to take the imbalance of the dataset
into account by enabling balanced weights for each class when training. These
are configurations that achieve the highest accuracy score after we conduct
extensive experiments. The NN model is trained with an Adam optimiser [15]
with a learning rate of $0.003$ while the dropout level and the batch size are
set at $10\%$ and 2048 accordingly. Regarding the evaluation metrics, we
report both balanced accuracy and accuracy scores due to the imbalance between
number of samples in the two classes in the dataset. Based on the analysis of
Straube et al., balanced accuracy (BA) is both an appropriate choice and an
intuitive metric to evaluate prediction results of a binary classification
problem when an imbalanced dataset is used for testing [27]. The balanced
accuracy is calculated by taking the average of the recall measured on each
class separately.

##  5 Results

In Table 2 and Table 3, we show the effect of combining statistical features
from three sources of signal and compare the results of our proposed stress
detection model with previous works. As can be seen from both tables, all
models of ours obtain higher evaluation scores when using the combination
features from multiple sources of signals than using only each of them
individually. According to Table 2, compared to the original work in [23],
employing our proposed bio-signal processing pipeline before feature
extraction step increases the mean accuracy score of conventional Machine
Learning model (RF model) around $3.2\%$. Our proposed NN model improves the
performance of state-of-the-art (SOTA) subject-independent stress detection
model around $1.63\%$ using the same number of bio-signals. However,
conventional Machine Learning models also achieve competitive accuracy scores
compared to the SOTA model when using the fusion feature of three sensor
sources with appropriate signal processing before feature extraction. The mean
accuracy scores of our SVM and RF models using the combined features as input
are $92.71\%$ and $91.53\%$ with standard deviation of $\pm 7.90\%$ and $\pm
7.24\%$ respectively while the one of our NN model is $94.50\%$ $(\pm
5.64\%)$. These results indicate that our proposed NN model not only increases
the prediction accuracy of subject-independent stress detection model in
average, but it also does not result in a large difference of accuracy score
between each subject’s model.

Table 2: Comparison of the mean accuracy score between different subject-independent stress detection models in previous works and ours using biosensor signals (window size = 60s, window shift = 0.25s). Sensor Combinations | Schmidt [23] | Lam [13] | Proposed
---|---|---|---
RF | LDA | StressNAS | SVM | RF | NN
EDA+BVP+ST | 88.33 | 86.46 | 92.87 | 92.71 | 91.53 | 94.50
EDA | 76.29 | 78.08 | 79.24 | 76.32 | 75.53 | 77.57
BVP | 84.18 | 85.83 | 81.16 | 86.95 | 87.39 | 89.94
ST | 67.82 | 69.24 | 71.46 | 69.21 | 69.23 | 74.07

In terms of imbalanced-data insensitive evaluation metrics, we report the
balanced accuracy scores of our models and compare them with corresponding
related work [25]. For consistency in comparison the balanced accuracy score
with [25], we use the window size and window shift of 120 seconds and 0.25
second respectively. According to the results in Table 3, our proposed NN
model outperforms conventional Machine Learning approaches reported in [25].
In detail, the balanced accuracy score of our NN models is $94.16\%$ $(\pm
6.90\%)$, which is higher than the QDA model of [25] around $12.56\%$ using
the same number of bio-signals. In addition, our RF model achieves higher
balanced score than the one in [25] approximately $12.09\%$, which proves that
our bio-signal processing pipeline is efficient and necessary for feature
extraction step to enhance subject-independent stress detection model. The
balanced accuracy score of our SVM and RF model trained with features combined
from different sensor sources are $93.36\%$ $(\pm 9.14\%)$ and $93.03$ $(\pm
10.19\%)$ respectively.

To facilitate for other future research to compare results with ours, we also
report both balanced accuracy and accuracy scores of our models with different
settings of window size and window shift that are not reported in Table 2 and
Table 3. For window size of 120s, the accuracy scores of our SVM, RF, and NN
models using combined feature are $95.23$ $(\pm 5.32)$, $94.57$ $(\pm 6.42)$,
and $95.26$ $(\pm 4.68)$ correspondingly For window size of 60s and window
shift of 0.25s, the balanced accuracy scores of our SVM, RF, and NN models are
$90.85$ $(\pm 9.99)$, $90.32$ $(\pm 11.21)$, and $92.66$ $(\pm 9.06)$
correspondingly.

Table 3: Comparison of the mean balanced accuracy score between different subject-independent stress detection models in previous work and ours using biosensor signals (window size = 120s, window shift = 0.25s). Sensor Combinations | Siirtola [25] | Proposed
---|---|---
RF | LDA | QDA | SVM | RF | NN
EDA+BVP+ST | 81.00 | 78.80 | 81.60 | 93.36 | 93.09 | 94.16
EDA | 78.30 | 73.50 | 69.70 | 71.34 | 70.12 | 77.52
BVP | 81.40 | 81.20 | 67.90 | 87.57 | 90.92 | 88.28
ST | 66.90 | 75.20 | 68.30 | 71.07 | 71.13 | 77.97

##  6 Conclusion

In this paper, we build a model that uses a neural network (NN) architecture
for subject-independent stress detection using three types of bio-signals that
can be captured from a consumer-grade low-cost device. The model contains four
different NN modules where each of the three modules learns the embedding
features from each bio-signal individually while the remaining one learns the
joint embedded feature of the three modules by concatenating the latent-space
representation of each signal. The proposed model is evaluated against Random
Forest and Support Vector Machine models on the WESAD dataset using balanced
accuracy and accuracy score. Our experiments show that using statistical
features from multiple sensor sources can produce more accurate stress
prediction results. Additionally, our experiments also show that our proposed
NN model outperforms conventional Machine Learning approaches for subject-
independent model training for both evaluation metrics. In detail, our NN
model achieves higher evaluation scores than the SOTA model and conventional
Machine Learning models while maintaining a low standard deviation score. We
believe that our findings could help promote and guide future efforts in
improving subject-independent stress detection models, in order to enhance the
integration of stress detection and management system into consumer-grade low-
cost wearable devices in a practical and cost-efficient manner.

#### Acknowledgments.

This publication is funded as part of Dublin City University’s Research
Committee and research grants from Science Foundation Ireland under grant
numbers SFI/13/RC/2106, SFI/13/RC/2106_P2, and 18/CRT/6223.

## References

  * [1] Can, Y.S., Chalabianloo, N., Ekiz, D., Ersoy, C.: Continuous stress detection using wearable sensors in real life: Algorithmic programming contest case study. Sensors (Basel, Switzerland) 19 (2019)
  * [2] Chen, W.V., Jaques, N., Taylor, S., Sano, A., Fedor, S., Picard, R.W.: Wavelet-based motion artifact removal for electrodermal activity. 2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) pp. 6223–6226 (2015)
  * [3] Choi, J., Ahmed, B., Gutierrez-Osuna, R.: Development and evaluation of an ambulatory stress monitor based on wearable sensors. IEEE transactions on information technology in biomedicine 16(2), 279–286 (2011)
  * [4] Elgendi, M., Norton, I., Brearley, M., Abbott, D., Schuurmans, D.: Systolic peak detection in acceleration photoplethysmograms measured from emergency responders in tropical conditions. PLoS One 8(10), e76585 (2013)
  * [5] Gjoreski, M.: Continuos Stress Monitoring using a Wrist Device and a Smartphone. Ph.D. thesis, Jožef Stefan International Postgraduate School, Ljubljana, Slovenia (09 2016)
  * [6] Gjoreski, M., Luštrek, M., Gams, M., Gjoreski, H.: Monitoring stress with a wrist device using context. Journal of Biomedical Informatics 73, 159–170 (2017). https://doi.org/https://doi.org/10.1016/j.jbi.2017.08.006, https://www.sciencedirect.com/science/article/pii/S1532046417301855
  * [7] Greco, A., Valenza, G., Lanata, A., Scilingo, E.P., Citi, L.: cvxeda: A convex optimization approach to electrodermal activity processing. IEEE Transactions on Biomedical Engineering 63(4), 797–804 (2016). https://doi.org/10.1109/TBME.2015.2474131
  * [8] Gurrin, C., Albatal, R., Joho, H., Ishii, K.: A privacy by design approach to lifelogging. In: O’Hara, K., Nguyen, C. and Haynes, P., (eds.) Digital Enlightenment Yearbook 2014. pp. 49–73. IOS Press, The Netherlands (2014)
  * [9] Gurrin, C., Joho, H., Hopfgartner, F., Zhou, L., Albatal, R.: Overview of ntcir-12 lifelog task. In: NTCIR (2016)
  * [10] Gurrin, C., Joho, H., Hopfgartner, F., Zhou, L., Ninh, V.T., Le, T.K., Albatal, R., Dang-Nguyen, D.T., Healy, G.: Overview of the ntcir-14 lifelog-3 task. In: Proceedings of the 14th NTCIR Conference on Evaluation of Information Access Technologies (2019)
  * [11] Gurrin, C., Smeaton, A., Doherty, A.: Lifelogging: Personal big data. Found. Trends Inf. Retr. 8, 1–125 (2014)
  * [12] Gurrin, C., Joho, H., Hopfgartner, F., Zhou, L., Albatal, R., Healy, G., Nguyen, D.T.D.: Experiments in Lifelog Organisation and Retrieval at NTCIR, pp. 187–203. Springer Singapore, Singapore (2021)
  * [13] Huynh, L., Nguyen, T., Nguyen, T., Pirttikangas, S., Siirtola, P.: StressNAS: Affect State and Stress Detection Using Neural Architecture Search, p. 121–125. Association for Computing Machinery, New York, NY, USA (2021), https://doi.org/10.1145/3460418.3479320
  * [14] Kher, R.: Signal processing techniques for removing noise from ecg signals. In: Journal of Biomedical Engineering and Research (2019)
  * [15] Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. In: Bengio, Y., LeCun, Y. (eds.) 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings (2015), http://arxiv.org/abs/1412.6980
  * [16] Kirschbaum, C., Pirke, K.M., Hellhammer, D.H.: The ‘trier social stress test’–a tool for investigating psychobiological stress responses in a laboratory setting. Neuropsychobiology 28(1-2), 76–81 (1993)
  * [17] Koldijk, S., Sappelli, M., Verberne, S., Neerincx, M.A., Kraaij, W.: The swell knowledge work dataset for stress and user modeling research. In: Proceedings of the 16th International Conference on Multimodal Interaction. p. 291–298. ICMI ’14, Association for Computing Machinery, New York, NY, USA (2014). https://doi.org/10.1145/2663204.2663257
  * [18] Makowski, D., Pham, T., Lau, Z.J., Brammer, J.C., Lespinasse, F., Pham, H., Schölzel, C., Chen, S.A.: Neurokit2: A python toolbox for neurophysiological signal processing. Behavior Research Methods pp. 1–8 (2021)
  * [19] Nabian, M., Yin, Y., Wormwood, J., Quigley, K.S., Barrett, L.F., Ostadabbas, S.: An open-source feature extraction tool for the analysis of peripheral physiological data. IEEE journal of translational engineering in health and medicine 6, 1–11 (2018)
  * [20] Ninh, V.T., Le, T.K., Zhou, L., Piras, L., Riegler, M., Halvorsen, P., Lux, M., Tran, M., Gurrin, C., Dang-Nguyen, D.T.: Overview of imageclef lifelog 2020: Lifelog moment retrieval and sport performance lifelog. In: CLEF (2020)
  * [21] Ninh, V.T., Smyth, S., Tran, M.T., Gurrin, C.: Analysing the performance of stressdetection models on consumer-grade wearable devices. In: SoMeT (2021)
  * [22] Nkurikiyeyezu, K., Yokokubo, A., Lopez, G.: Effect of person-specific biometrics in improving generic stress predictive models. Sensors and Materials 32, 703–722 (02 2020). https://doi.org/10.18494/SAM.2020.2650
  * [23] Schmidt, P., Reiss, A., Duerichen, R., Marberger, C., Van Laerhoven, K.: Introducing wesad, a multimodal dataset for wearable stress and affect detection. In: Proceedings of the 20th ACM international conference on multimodal interaction. pp. 400–408 (2018)
  * [24] Schmidt, P., Reiss, A., Dürichen, R., Laerhoven, K.V.: Wearable affect and stress recognition: A review. ArXiv abs/1811.08854 (2018)
  * [25] Siirtola, P.: Continuous stress detection using the sensors of commercial smartwatch. Adjunct Proceedings of the 2019 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2019 ACM International Symposium on Wearable Computers (2019)
  * [26] Siirtola, P., Röning, J.: Comparison of regression and classification models for user-independent and personal stress detection. Sensors (Basel, Switzerland) 20 (2020)
  * [27] Straube, S., Krell, M.M.: How to evaluate an agent’s behavior to infrequent events?—reliable performance estimation insensitive to class distribution. Frontiers in Computational Neuroscience 8, 43 (2014)
  * [28] V., K.M., L, F.E.: Correction of the heart rate variability signal for ectopics and missing beats. In: Heart Rate Variability, eds M. Malik, Camm A. J (1995)


# Word Order in English-Japanese Simultaneous Interpretation:
Analyses and Evaluation using Chunk-wise Monotonic Translation

Kosuke Doi1, Yuka Ko1, Mana Makinae1, Katsuhito Sudoh1, 2, Satoshi Nakamura1,
3

1Nara Institute of Science and Technology,
2Nara Women’s University, 3The Chinese University of Hong Kong, Shenzhen
{doi.kosuke.de8, sudoh<EMAIL_ADDRESS>
###### Abstract

This paper analyzes the features of monotonic translations, which follow the
word order of the source language, in simultaneous interpreting (SI). The word
order differences are one of the biggest challenges in SI, especially for
language pairs with significant structural differences like English and
Japanese. We analyzed the characteristics of monotonic translations using the
NAIST English-to-Japanese Chunk-wise Monotonic Translation Evaluation Dataset
and found some grammatical structures that make monotonic translation
difficult in English-Japanese SI. We further investigated the features of
monotonic translations through evaluating the output from the existing speech
translation (ST) and simultaneous speech translation (simulST) models on NAIST
English-to-Japanese Chunk-wise Monotonic Translation Evaluation Dataset as
well as on existing test sets. The results suggest that the existing SI-based
test set underestimates the model performance. We also found that the
monotonic-translation-based dataset would better evaluate simulST models,
while using an offline-based test set for evaluating simulST models
underestimates the model performance.

Word Order in English-Japanese Simultaneous Interpretation:
Analyses and Evaluation using Chunk-wise Monotonic Translation

Kosuke Doi1, Yuka Ko1, Mana Makinae1, Katsuhito Sudoh1, 2, Satoshi Nakamura1,
3 1Nara Institute of Science and Technology, 2Nara Women’s University, 3The
Chinese University of Hong Kong, Shenzhen {doi.kosuke.de8, sudoh,
<EMAIL_ADDRESS>

##  1 Introduction

Simultaneous Interpreting (SI) is the task of translating speech from a source
language into a target language in real time. SI is a cognitively demanding
task, and human simultaneous interpreters are known to employ strategies such
as segmentation, summarization, and generalization He et al. (2016).
Maintaining the word order in a source language is also an important strategy,
especially for language pairs whose word order is different (_e.g._ , English
and Japanese), to shorten delay and reduce cognitive load. Because of these
features, SI sentences are different from offline translation sentences, but
most of the studies in automatic SI Oda et al. (2014); Ma et al. (2019); Liu
et al. (2020); Papi et al. (2023) have been used offline translation corpora
(_e.g._ , MuST-C; Di Gangi et al., 2019) for both training and evaluation of
the models due to the limited amount of simultaneous interpretation corpora
(SICs).

For the English-Japanese language pair, several SICs have been constructed
Toyama et al. (2004); Shimizu et al. (2014); Matsushita et al. (2020); Doi et
al. (2021). Based on NAIST Simultaneous Interpretation Corpus (NAIST-SIC; Doi
et al., 2021), Zhao et al. (2024)111The dataset was released in 2023 (see the
version 3 of the paper). created an automatically-aligned parallel SI dataset,
named NAIST-SIC-Aligned. Since the sentences in the dataset are aligned at the
sentence level, they can be used for model training. Actually, Ko et al.
(2023) and Zhao et al. (2024) trained SI models using SI data from NAIST-SIC-
Aligned. The model performance was evaluated through automatic evaluation
metrics such as BLEU Papineni et al. (2002) using a small test set curated
based on SI sentences generated by professional human simultaneous
interpreters.

The scores reported in Ko et al. (2023) and Zhao et al. (2024) were relatively
low but the test set used in the two studies might have underestimated the
model performance. Since human simultaneous interpreters use strategies such
as summarization and generalization, phrases that do not affect the main idea
are not necessarily translated into the target language. If an SI model
generates translation for the phrases that a human interpreter did not, the
output sentence might not be evaluated properly, even when it is _correct_
translation.

Source |  (1) The US Secret Service, / (2) two months ago, / (3) froze the Swiss bank account / (4) of Mr. Sam Jain right here, / (5) and that bank account / (6) had 14.9 million US dollars in it / (7) when it was frozen.
---|---
Offline |  (1) 米国のシークレットサービスは / (2) 2ヶ月前に / (4) サム・ジェイン氏の / (3) スイス銀行口座を凍結しました / (5) その口座には / (6) 米ドルで1490万ドルありました
|  [The US Secret Service / two months ago / Mr. Sam Jain’s / froze the Swiss
bank account / that bank account / had 14.9 million US dollars]
SI |  (1) アメリカのシークレッドサービスが、/ (3) スイスの銀行の口座を凍結しました。 / (4) サムジェインのものです。 / (5) この銀行口座の中には、 / (6) 一千四百九十万ドルが入っていました。
|  [The US Secret Service / froze the Swiss bank account / it is Sam Jain’s
one / in this bank account / had 14.9 million dollars]
CMT |  (1) アメリカ合衆国シークレットサービスは、 / (2) 2ヶ月前に、 / (3) スイスの銀行口座を凍結しました、 / (4) ここにいるサム・ジェイン氏の口座です、 / (5) そしてその銀行口座には / (6) 490万米ドルが入っていました、 / (7) 凍結された時。
|  [The US Secret Service / two months ago / froze the Swiss bank account /
the account of Mr. Sam Jain right here / and that bank account / had 14.9
million US dollars in it / when it was frozen]
Table 1: Comparison of target sentences in each translation mode. Examples of
offline, SI, and CMT are from the subtitle of TED talks, NAIST-SIC, and NAIST
English-to-Japanese Chunk-wise Monotonic Translation Evaluation Dataset,
respectively. “/” shows the boundaries of the chunks. The numbers preceding
the chunks in the source sentence represent the order of appearance. The
numbers preceding the chunks in the target sentences correspond to the numbers
in the source sentence.

Fukuda et al. (2024) pointed out the difficulty for SI models to learn which
phrases in source speech are less important and called for constructing SI
models that only employ the strategy of maintaining the word order in a source
language. As the first step, they created NAIST English-to-Japanese Chunk-wise
Monotonic Translation Evaluation Dataset222https://dsc-
nlp.naist.jp/data/NAIST-SIC/Aligned-Chunk_Mono-EJ/. The source sentences in
the test set used in Ko et al. (2023) were automatically segmented into
chunks, and each chunk was translated in a way that did not include the
content of the subsequent chunks. Unlike in SI sentences by human
interpreters, where not all information in the source sentences is translated,
chunk-wise monotonic translation (CMT) was developed so that all information
is translated (Table 1)333Precisely, omission to maintain the fluency of the
sentence was allowed. See Section 3.1 for the details about the dataset..
Fukuda et al. (2024) have investigated the quality of CMT through human
evaluation, but have not analyzed the characteristics of it. In addition, they
have not conducted any evaluation experiments in which model outputs are
evaluated on the CMT Evaluation dataset.

In this paper, we analyze CMT sentences qualitatively and quantitatively. In
the process of developing CMT, it was allowed to repeat, defer, and omit
phrases in the source sentences to keep the translation fluent. We suppose
that there are factors (_e.g._ , syntactic structures) that prevent monotonic
translation if phrases were repeated, deferred, or omitted in CMT sentences
since the sentences were translated without time constraints. In addition, we
evaluate the output from (1) a speech translation (ST) model trained on
offline data, (2) a simultaneous speech translation (simulST) model trained on
offline data, and (3) a simulST model trained on both offline and SI data. The
contributions of this paper are as follows:

  * •

We analyze CMT sentences and show that they tend to be longer than offline
translation primarily because of repetition.

  * •

We investigate what makes the phrases in source sentences to be repeated,
deferred, and omitted, and show that most cases occur because of particular
grammatical structures. When a phrase in a chunk is a dependent of a phrase in
the preceding chunk, the head phrase is typically repeated or deferred.

  * •

We evaluate the output from three different models on NAIST English-to-
Japanese Chunk-wise Monotonic Translation Evaluation Dataset: (1) an ST model
trained on offline data, (2) a simulST model trained on offline data, and (3)
a simulST model trained on both offline and SI data. We show that the existing
SI-based test set Ko et al. (2023); Zhao et al. (2024) underestimates the
model performance. The results also show that the CMT evaluation dataset would
better evaluate simulST models while using an offline-based test set for
evaluating simulST models underestimates the model performance.

##  2 Related Work

###  2.1 Simultaneous Interpretation Corpora

SICs are valuable resources both for developing automatic SI models and for
analyzing the characteristics of SI. For the English-Japanese language pair,
several SICs are publicly available Toyama et al. (2004); Shimizu et al.
(2014); Matsushita et al. (2020); Doi et al. (2021) although the amount of
such corpora is very limited compared to offline translation corpora.

Using these corpora, SI sentences have been analyzed from various
perspectives, such as strategies and interpreting patterns used by
interpreters, latency, translation quality, and word order Tohyama and
Matsubara (2006); Ono et al. (2008); Cai et al. (2018, 2020); Doi et al.
(2021). Furthermore, SI models have also been developed using SICs Ryu et al.
(2004); Shimizu et al. (2013); Ko et al. (2023).

###  2.2 Word Order in Simultaneous Interpreting

When dealing with language pairs whose sentence structures are different,
including English and Japanese (SVO and head-initial vs. SOV and head-final),
reducing the word order difference between the source and the target language
is important to minimize the delay.

Murata et al. (2010) segmented source sentences into semantically meaningful
units with a maximum length of 4.3 seconds, and translated those units from
the viewpoint of SI. He et al. (2015) designed syntactic transformation rules
for Japanese-English simultaneous machine translation. By applying the rules
to target language sentences (_i.e._ , English), they generated more monotonic
translations, while preserving the meaning of source sentences and maintaining
the grammaticality of the target language. In English-Japanese SI, Futamata et
al. (2020) reordered Japanese sentences to make the word order closer to the
original English sentences. They further applied style transfer to make the
sentence fluent and obtained sentences close to SI sentences by human
interpreters. Han et al. (2021) proposed an algorithm to reorder and refine
the target sentences so that the target sentences were aligned largely
monotonically. They trained SI models for four language pairs, including
English-Japanese. Nakabayashi and Kato (2021) segmented sentences into chunks
and created bilingual pairs of the chunks with explicit annotations of context
information. The SI model trained on the data translated the source sentences
while referencing preceding chunks but connecting chunks naturally remained a
challenge. Higashiyama et al. (2023) have constructed a large-scale English
$\leftrightarrow$ Japanese SIC with information of chunk boundaries in source
and target sentences and phrases that can be omitted in target sentences.
NAIST English-to-Japanese Chunk-wise Monotonic Translation Evaluation Dataset
Fukuda et al. (2024), which is similar to Higashiyama et al.’s (2023), is
relatively small and intended for the evaluation purpose.

The word order differences among different translation modes have also been
investigated. Okamura and Yamada (2023) quantitatively compared the degree to
which the word order of the source sentences was maintained and found that SI
sentences retained the order better than consecutive interpreting and offline
translation sentences. Cai et al. (2020) found syntactic and non-syntactic
factors that affect interpreters’ word order decisions through the statistical
analyses of an SIC. In this paper, we analyze what makes monotonic translation
difficult. While Cai et al. (2020) analyzed actual SI data generated by human
simultaneous interpreters, we use the CMT dataset, which is created without
time constraints, and in which all information in the source sentences is
translated into the target sentences.

##  3 Chunk-wise Monotonic Translation

In SI between language pairs with different sentence structures, interpreters
are known to segment source sentences into chunks and translate them in the
order of the chunks444Reordering may occur within a chunk. Okamura and Yamada
(2023). This section describes the details of NAIST English-to-Japanese Chunk-
wise Monotonic Translation Evaluation Dataset, which is used in our analyses.

###  3.1 Data

NAIST English-to-Japanese Chunk-wise Monotonic Translation Evaluation Dataset
consists of 511 pairs of source sentences and their corresponding chunk-wise
monotonic translation (CMT) sentences with information of chunk boundaries in
source and target sentences. Source (_i.e._ , English) sentences, which were
used as the test set in Ko et al. (2023), were segmented following the five
rules that reflected interpreter’s strategies555See the original paper or the
code for chunking: https://github.com/ahclab/si_chunker based on the syntactic
analysis results from spaCy. The source sentences come from 8 TED talks.

Data | Sum | Per Sent.
---|---|---
# sentence pairs | 511 | –
# chunks | 1,677 | 3.28
# source words | 8,104 | 15.86
# target words | 13,981 | 27.36
Table 2: Statistics for chunk-wise monotonic translation data. The number of
words in the target sentence was calculated by us. The other values are cited
from Fukuda et al. (2024).

Translators were provided with source sentences with chunk boundaries and were
asked to (1) translate them in the order of the chunks while (2) connecting
the chunks naturally and (3) not including the content of the subsequent
chunks. It was allowed to (1) repeat, (2) defer, and (3) omit phrases in the
source sentences to keep the translation fluent, but was instructed to avoid
using the operations with larger number as much as possible. Data statistics
are shown in Table 2.

##  4 Data Analysis

Fukuda et al. (2024) have examined the quality of CMT sentences through human
evaluation but have not analyzed the characteristics of them. We suppose that
there are factors that prevent monotonic translation if a phrase in the source
sentences is repeated, deferred, and omitted since the CMT sentences were
generated without time constraints. We qualitatively and quantitatively
analyze the CMT sentences with these operations and reveal the factors.

To better understand the characteristics of CMT sentences, we also compare
them with SI sentences from NAIST-SIC and NAIST-SIC-Aligned, and offline
translation sentences from the subtitles of TED talks. Since the SI sentences
in NAIST-SIC were not aligned at the sentence level, we performed alignment
manually. Some source sentences did not match across the datasets, and we
excluded those 10 sentences from the analyses. In addition, 25 sentences were
not translated in NAIST-SIC666For example, due to time constraints,
interpreters may not be able to translate a whole sentence. See Doi et al.
(2021)., which were also excluded from the analyses. As a result, 476
sentences were used for our analyses.

Type | Tag |  Meaning
---|---|---
Span | repeat |  Phrases that are repeated
| zero-repeat |  Target phrases that are repeated; No corresponding source phrases (_e.g._ , zero that-clause)
| defer |  Phrases that are not translated within the current chunk but in a subsequent chunk
| omit |  Source phrases that are not translated in the target sentences
| ahead |  Target phrases translated using the subsequent chunks
| add |  Target phrases that have no corresponding source phrases
| error |  Phrases with translation errors
| sep |  Boundaries of source and target sentences
Relation | rel-repeat |  Connect source and target phrases with repeat tags
| repeat_d# |  Connect target phrases with repeat tags
| defer_d# |  Connect source and target phrases with defer tags
| ahead_d# |  Connect source and target phrases with ahead tags
| rel-err |  Connect source and target phrases with error tags
Table 3: List of tags used for the annotations. “d#” (#=1, 2, …) represents
the distance between chunks.

###  4.1 Annotations

To analyze the characteristics of CMT sentences, we annotated tags to source
and CMT sentences. The list of tags is shown in Table 3. Prior to the
annotation, we tokenized the English sentences using spaCy777https://spacy.io/
and the Japanese sentences using MeCab Kudo et al. (2004) with unidic. Then,
the source and CMT sentences were concatenated with special token [SEP] and
annotated using an open-source data labeling tool,
doccano888https://github.com/doccano/doccano. Annotation examples are shown in
Figure 1.

Figure 1: Annotation examples. Repeat tags were assigned even if the strings
did not exactly match but referred to the same entity or had the same meaning.

The first and second authors collaboratively annotated the first 50 examples
while discussing. As sufficient agreement was expected, the remainder of the
data were annotated by the first author.

###  4.2 Results of Analyses

####  4.2.1 Comparison among Different Translation Modes

We compared the length of sentences in the four datasets. Fukuda et al. (2024)
have also conducted similar comparisons based on the number of characters.
However, there were variations in spelling and differences in transcription
systems (_e.g._ , numbers), and we made comparisons based on the number of
words segmented by MeCab.

Table 4 show that CMT sentences were the longest, followed by offline, NAIST-
SIC, and NAIST-SIC-Aligned, which matched the results reported in Fukuda et
al. (2024). Long translation sentences can pose some problems. As discussed in
Fukuda et al. (2024), it may increase the cognitive load on the
listeners/readers of the translations. In addition, longer output may cause a
delay even though CMT aims to reduce it.

Dataset | Sum | Per Sent.
---|---|---
CMT | 13,508 | 28.38
NAIST-SIC | 8,914 | 18.73
NAIST-SIC-Aligned | 8,072 | 16.96
Offline | 9,907 | 20.81
Table 4: Comparison of the number of words in different translation modes

####  4.2.2 Factors that Make CMT Sentences Longer

To reveal what made the CMT sentences longer, we first analyzed them
qualitatively. Our analyses suggest that (1) CMT sentences contain many
formulaic expressions for the end of sentences as sentences are segmented into
small chunks, and (2) words that are often omitted in Japanese (_e.g._ ,
pronouns) are explicitly translated since translators were instructed to avoid
omitting phrases in the source language, as in the following examples.

> (En) It’s when we warmed it up, / and we turned on the lights / and looked
> inside the box, / we saw that the piece metal / was still there in one
> piece.
>
> (CMT) 私たちがそれを暖めるときです、 / 電気をつけて、 / 箱の中を見たときです、 / 私たちはその金属片を見たんです、 /
> それはまだ一つの塊としてそこにありました。
> (offline) 物体を暖め明かりをつけ 箱の中を見たところ金属片はまだそこに存在していました

> (En) He only has use of his eyes.
>
> (CMT) 彼は目だけを使えます。
> (offline) 目だけしか動かせません

In addition to the above characteristics, there were many repetitions observed
in particularly _long_ sentences. To verify this, we further analyzed
particularly _long_ and _short_ sentences, chosen based on the length ratio of
CMT sentences to offline ones. The _long_ and _short_ sentences were defined
as those with a length ratio greater/smaller than the average $\pm$ 0.5
standard deviations (avg.=1.39, SD=0.43). We subjectively judged if those
sentences contained many repetitions. We also identified the sentences whose
offline translations were short.

Table 5 shows that _long_ sentences contained more repetitions than _short_
ones. The offline translation sentences were short, probably because they were
originally subtitles, for which limited space was allowed. We also checked it
quantitatively using the number of repeat tags assigned to the sentences, and
found that the frequency of repetition tags was higher in _long_ sentences
(Table 5).

Type | N | Repeat | Repeat | Short
---|---|---|---|---
|  | (subjective) | (tag) | offline
Long | 131 | 54 (41.22%) | 3.35 | 53
Short | 171 | 22 (12.87%) | 0.81 | –
Table 5: Comparison between _long_ and _short_ CMT sentences. The number of
repeat tags is per sentence.

####  4.2.3 Omission in SI Sentences

To find techniques to shorten translations, we analyzed SI sentences in NAIST-
SIC. Based on the length ratio of SI sentences to offline ones, we defined SI
sentences that potentially involved reasonable omissions (omission; $0.6\leq$
the ratio $<0.9$), and SI sentences that may not fully convey the meaning of
the source sentences (undertranslation; the ratio $<0.6$), following the
criteria in Higashiyama et al. (2023).

We expected to find some trends (_e.g._ , part-of-speech) in phrases that were
omitted, but we was not able to. In addition, we found a certain number of
_unacceptable_ translations in both categories (43.12% and 60.00% for omission
and undertranslation, respectively). The results suggest that human
simultaneous interpreters judge the importance of the phrases based on context
and decide whether to translate them, which are sometimes correct and
sometimes not.

####  4.2.4 Factors that Make Monotonic Translation Difficult

With the help of tags annotated to source and CMT sentences, we analyzed
factors that make monotonic translation difficult. Table 6 shows the number of
source phrases that were repeated, deferred, and omitted. The values are based
on the number of rel-repeat, defer_d#, and omit tags, respectively. We counted
the number of relation tags for repeat and defer because the span tags for the
two operations were assigned to both source and CMT sentences. The results
show that the translators used repeat most frequently, followed by defer and
omit, as they were instructed (see Section 3.1).

Operation | N
---|---
repeat | 301
defer | 173
omit | 36
Table 6: Comparison of the number of operations used in CMT

For these phrases, we explored what makes monotonic translations difficult.
Our analyses revealed that most cases of repeat and defer occurred because of
particular grammatical structures. Table 7 lists major structures along with
their frequencies in the data and examples. In these structures, a phrase in a
chunk is typically a dependent of a phrase in the preceding chunk. In the
example of a post-modifier (Table 7), the relative pronoun clause is a
dependent of the noun phrase _a device_ , which is in a preceding chunk. When
phrases with a dependency relation exist across multiple chunks, CMT would be
difficult because Japanese is a strongly head-final language. The examples in
Table 7 show how human translators address these structures by repeating or
deferring some phrases in the subsequent chunks.

Prepositions, post-modifiers, and dependent clauses have also been identified
as syntactic factors that affect interpreters’ word order decisions in Cai et
al. (2020). Human interpreters find those structures challenging for SI and
adopt a strategy to maintain the word order of the source language.

Structures | # repeat | # defer |  Examples
---|---|---|---
Noun with a post-modifier | 88 | 12 |  And now we’ve created a device / that has absolutely no limitations.
|  |  |  さて、私たちはデバイスを作り出しました、 / 全く制限のないものです。 [And we’ve created a device / one that has absolutely no limitations]
Head followed by multiple dependents | 35 | 6 |  … / allows for deep squats, / crawls and high agility movements.
|  |  |  … / 深いスクワットを可能にし、 / クロールや高い敏捷性の動きを可能にします。 [… / allows for deep squats / allows for crawls and high agility movements]
Dependent conjunction | 26 | 6 |  … / when he’s covered / in four feet of snow.
|  |  |  … / 彼が / 四フィートの雪に覆われてしまっていた時には。 [he / when was covered in for feet of snow]
Chunk boundary before a clause | 15 | 13 |  … / you know that this isn’t / how it normally goes.
|  |  |  / あなたは分かるはずです、これは / 通常の進行ではないということが。 [you know that this / isn’t how it normally goes]
Chunk boundary before a preposition | 10 | 7 |  … / providing totalitarian governments with tools / to do this / against their own citizens.
|  |  |  … / 全体主義政府にツールを提供しているということです、 / これを行うための、 / 自国の市民に対してこれを行うためのツールを。 [providing totalitarian governments with tools / to do this / tools to do this against their own citizens]
Table 7: Syntactic factors that prevent monotonic translations

In addition, we observed that inappropriate segmentation was addressed through
repeating and deferring the phrases. Most of the inappropriate segmentation
was found in phrasal verbs, -ing forms of verbs, and to-infinitives.

In SI data, we also found human interpreters repeat phrases to maintain the
word order of the source language. For example, in the example in Table 1, a
noun modified by a preposition phrase is repeated.

> (En) … / froze the Swiss bank account / of Mr. Sam Jain right here, / …
>
> (SI) … / スイスの銀行の口座を凍結しました。 / サムジェインのものです。 / …
>  [… / froze the Swiss bank account / it is Sam Jain’s one / …]

In addition, Okamura and Yamada (2023) reported that the order of the chunks
is swapped about once on average in an SI sentence. These things suggest that
human interpreters address the word order differences that make monotonic
translation difficult through repeating and deferring some phrases.

##  5 Evaluation Using CMT Data

To investigate the impact of using CMT sentences for evaluating the
translation quality, we evaluated the output from existing ST and simulST
models using NAIST English-to-Japanese Chunk-wise Monotonic Translation
Evaluation Dataset as well as existing test sets.

###  5.1 Data

We used the following datasets as the reference for the automatic evaluation
metrics:

  * •

n-cmt: CMT sentences from NAIST English-to-Japanese Chunk-wise Monotonic
Translation Evaluation Dataset

  * •

si_hum: SI sentences from NAIST-SIC, which were manually aligned to the source
speech

  * •

si_auto: SI sentences from NAIST-SIC-Aligned, which were aligned automatically

  * •

offline: offline translation sentences from the subtitles of TED talks.

###  5.2 Speech Translation Models

We used three existing ST and simulST models:

  * •

ST_offline: an ST model trained on offline data Fukuda et al. (2023)

  * •

simulST_offline: a simulST model trained on offline data Ko et al. (2023)

  * •

simulST_si_offline: a simulST model trained on both offline and SI data Ko et
al. (2023)

All the models were built by connecting two pre-trained models, HuBERT-Large
Hsu et al. (2021) for their speech encoder and the decoder of mBART50 Tang et
al. (2020) for their text decoder. The encoder and decoder were connected via
Inter-connection Nishikawa and Nakamura (2023) and a length adapter Tsiamas et
al. (2022). Both SimulST models used bilingual prefix pairs extracted using
Bilingual Prefix Alignment Kano et al. (2022) for the model training and
employed a decoding policy called local agreement Liu et al. (2020). For
ST_offline, we used the model with checkpoint averaging (Inter-connection +
Ckpt Ave. in Fukuda et al. (2023)). For simulST_offline and
simulST_si_offline, we used the models which satisfy the task requirement of
the simultaneous track in IWSLT 2023 Evaluation
Campaign999https://iwslt.org/2023/simultaneous, latency measured by Average
Lagging Ma et al. (2019) $\leq 2$ seconds (Offline FT and Mixed FT + Style in
Ko et al. (2023), respectively).

###  5.3 Metrics

We evaluated the translation quality of the output from the ST and simulST
models (see Section 5.2) using BLEU101010BLEU was calculated using sacreBLEU.
Post (2018) https://github.com/mjpost/sacrebleu,
BLEURT111111https://github.com/google-research/bleurt Sellam et al. (2020),
COMET121212https://github.com/Unbabel/COMET Rei et al. (2020), and
BERTScore131313https://github.com/Tiiiger/bert_score Zhang et al. (2020).
BERTScore was calculated using bert-base-multilingual-cased. We used the four
datasets described in Section 5.1 as the reference.

###  5.4 Evaluation Results

Model | BLEU |  | BLEURT |  | COMET
---|---|---|---|---|---
| n-cmt | si_hum | si_auto | offline | n-cmt | si_hum | si_auto | offline | n-cmt | si_hum | si_auto | offline
ST_offline | 14.487 | 8.856 | 8.637 | 17.775 | 0.553 | 0.447 | 0.414 | 0.538 | 0.838 | 0.797 | 0.781 | 0.833
simulST_offline | 15.406 | 8.446 | 7.773 | 17.907 | 0.556 | 0.442 | 0.406 | 0.531 | 0.826 | 0.780 | 0.763 | 0.821
simulST_si_offline | 15.982 | 12.031 | 11.020 | 13.191 | 0.567 | 0.493 | 0.460 | 0.519 | 0.807 | 0.774 | 0.761 | 0.789

Model | BERTScore (Pre.) |  | BERTScore (Rec.) |  | BERTScore (F1)
---|---|---|---|---|---
| n-cmt | si_hum | si_auto | offline | n-cmt | si_hum | si_auto | offline | n-cmt | si_hum | si_auto | offline
ST_offline | 0.801 | 0.735 | 0.722 | 0.789 | 0.769 | 0.739 | 0.735 | 0.788 | 0.784 | 0.737 | 0.728 | 0.788
simulST_offline | 0.799 | 0.730 | 0.717 | 0.783 | 0.770 | 0.738 | 0.734 | 0.786 | 0.783 | 0.734 | 0.725 | 0.784
simulST_si_offline | 0.817 | 0.764 | 0.746 | 0.759 | 0.784 | 0.766 | 0.760 | 0.757 | 0.800 | 0.764 | 0.752 | 0.757

Table 8: Results of quality evaluation metrics across ST and simulST models

Table 8 shows the results of quality evaluation metrics across ST and simulST
models. When the translation quality was evaluated using BLEU with n-cmt as
the reference, simulST_si_offline achieved the highest score. On the SI-based
test sets (_i.e._ , si_hum and si_auto), simulST_si_offline also had the
highest score. On the offline-based test set, in contrast, models trained on
only offline data achieved much higher score than simulST_si_offline. The same
tendencies were observed in BLEURT and BERTScore. These results suggest that
the model trained on both SI and offline data generated more SI-like
translations, and that such models should be evaluated using a reference
closer to SI sentences. In addition, using an offline-based test set might
underestimate the performance of the model trained on both SI and offline
data.

Comparing n-cmt, si_hum, and si_auto, the scores were highest for n-cmt,
followed by si_hum and si_auto on all metrics and models. Because si_hum is
based on SI sentences generated by human simultaneous interpreters, some
content in a source speech might be omitted or inadequately translated (under-
translation). SI sentences in si_auto, which were automatically created based
on human SI sentences, might contain less source speech content than those in
si_hum due to the alignment and filtering techniques applied (see Zhao et al.,
2024). In fact, BERTScore precision was higher than recall on n-cmt, in which
there are almost no omissions, while the recall was higher than the precision
on si_auto and the precision and the recall were almost equal on si_hum. These
results suggest that the existing SI-based test sets Ko et al. (2023); Zhao et
al. (2024) underestimate the model performance.

However, the results on COMET were different from those on other metrics
(Table 8). On all four test sets, ST_offline achieved the highest score,
followed by simulST_offline and simulST_si_offline. One of the possible
reasons is that COMET uses source sentences to calculate scores.

To examine the impact of source sentences, we also calculated the reference-
free COMET-QE using wmt22-cometkiwi-da and got similar results (0.813, 0.798,
0.766 for ST_offline, simulST_offline, simulST_si_offline, respectively). We
further calculated COMET-QE for n-cmt and offline, regarding them as oracle
data, and found that n-cmt had a higher score than offline (n-cmt: 0.832,
offline: 0.812). Because some translation sentences in offline include under-
translation, these results suggest that the COMET score tends to become higher
when more content in the source sentences is covered in the target sentences.
This feature does not fit the nature of SI, where human interpreters use
sophisticated strategies (see He et al., 2016; Cai et al., 2020, for example).
We need to carefully interpret the COMET score when we use it for evaluating
simulST models.

##  6 Conclusion

This paper focuses on monotonic translations in English-Japanese SI. Our
analyses revealed some syntactic factors that make monotonic translations
difficult and that human interpreters/translators address these challenges by
repeating or deferring some phrases in the source language in the subsequent
chunks. The grammatical structures that might cause delay would be useful
information for developing segmentation or decoding policies of simultaneous
machine translation systems.

We also evaluated the output from the existing ST and simulST models on NAIST
English-to-Japanese Chunk-wise Monotonic Translation Evaluation Dataset as
well as on existing SI-based and offline-based test sets. The results suggest
that monotonic translation data would better evaluate simulST models trained
using SI data, and that the existing test sets underestimate the performance
of such models. A future direction would be using monotonic translation data
for developing simulST models.

## Acknowledgments

A part of this work was supported by JSPS KAKENHI Grant Numbers JP21H05054 and
by JST, the establishment of university fellowships towards the creation of
science technology innovation, Grant Number JPMJFS2137.

## References

  * Cai et al. (2018) Zhongxi Cai, Koichiro Ryu, and Shigeki Matsubara. 2018.  Statistical analysis of missing translation in simultaneous interpretation using a large-scale bilingual speech corpus.  In _Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)_ , Miyazaki, Japan. European Language Resources Association (ELRA).
  * Cai et al. (2020) Zhongxi Cai, Koichiro Ryu, and Shigeki Matsubara. 2020.  What affects the word order of target language in simultaneous interpretation.  In _Proceedings of 2020 International Conference on Asian Language Processing (IALP)_ , pages 135–140.
  * Di Gangi et al. (2019) Mattia A. Di Gangi, Roldano Cattoni, Luisa Bentivogli, Matteo Negri, and Marco Turchi. 2019.  MuST-C: a Multilingual Speech Translation Corpus.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 2012–2017, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Doi et al. (2021) Kosuke Doi, Katsuhito Sudoh, and Satoshi Nakamura. 2021.  Large-scale English-Japanese simultaneous interpretation corpus: Construction and analyses with sentence-aligned data.  In _Proceedings of the 18th International Conference on Spoken Language Translation (IWSLT 2021)_ , pages 226–235, Bangkok, Thailand (online). Association for Computational Linguistics.
  * Fukuda et al. (2024) Ryo Fukuda, Kosuke Doi, Katsuhito Sudoh, and Satoshi Nakamura. 2024.  原発話に忠実な英日同時機械翻訳の実現に向けた順送り訳評価データ作成 [Creation of Evaluation Data for Monotonic Translation toward the Realization of Simultaneous English-Japanese Machine Translation Faithful to the Source Speech].  In _Proceedings of the 259th meeting of Special Interest Group of Natural Language Processing (IPSJ-SIGNL), 2024-NL-259(14)_ , pages 1–6.  (in Japanese).
  * Fukuda et al. (2023) Ryo Fukuda, Yuta Nishikawa, Yasumasa Kano, Yuka Ko, Tomoya Yanagita, Kosuke Doi, Mana Makinae, Sakriani Sakti, Katsuhito Sudoh, and Satoshi Nakamura. 2023.  NAIST simultaneous speech-to-speech translation system for IWSLT 2023.  In _Proceedings of the 20th International Conference on Spoken Language Translation (IWSLT 2023)_ , pages 330–340, Toronto, Canada (in-person and online). Association for Computational Linguistics.
  * Futamata et al. (2020) Kosuke Futamata, Katsuhito Sudoh, and Satoshi Nakamura. 2020.  英日同時通訳システムのための疑似同時通訳コーパス自動生成手法の提案[Proposal of a Method for Automatically Generating Pseudo-simultaneous Interpretation Corpora for English-Japanese Simultaneous Interpretation Systems].  In _Proceedings of the 26th Annual Meeting of the Association for Natural Language Processing_ , pages 1281–1284.  (in Japanese).
  * Han et al. (2021) HyoJung Han, Seokchan Ahn, Yoonjung Choi, Insoo Chung, Sangha Kim, and Kyunghyun Cho. 2021.  Monotonic simultaneous translation with chunk-wise reordering and refinement.  In _Proceedings of the Sixth Conference on Machine Translation_ , pages 1110–1123, Online. Association for Computational Linguistics.
  * He et al. (2016) He He, Jordan Boyd-Graber, and Hal Daumé III. 2016.  Interpretese vs. translationese: The uniqueness of human strategies in simultaneous interpretation.  In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 971–976, San Diego, California. Association for Computational Linguistics.
  * He et al. (2015) He He, Alvin Grissom II, John Morgan, Jordan Boyd-Graber, and Hal Daumé III. 2015.  Syntax-based rewriting for simultaneous machine translation.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_ , pages 55–64, Lisbon, Portugal. Association for Computational Linguistics.
  * Higashiyama et al. (2023) Shohei Higashiyama, Kenji Imamura, Masao Utiyama, and Eiichiro. Sumita. 2023.  GCP 同時通訳コーパスの構築[Construction of GCP Simultaneous Interpretation Corpus].  In _Proceedings of the 29th Annual Meeting of the Association for Natural Language Processing_ , pages 1405–1410.  (in Japanese).
  * Hsu et al. (2021) Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. 2021.  Hubert: Self-supervised speech representation learning by masked prediction of hidden units.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , 29:3451–3460.
  * Kano et al. (2022) Yasumasa Kano, Katsuhito Sudoh, and Satoshi Nakamura. 2022.  Simultaneous neural machine translation with prefix alignment.  In _Proceedings of the 19th International Conference on Spoken Language Translation (IWSLT 2022)_ , pages 22–31, Dublin, Ireland (in-person and online). Association for Computational Linguistics.
  * Ko et al. (2023) Yuka Ko, Ryo Fukuda, Yuta Nishikawa, Yasumasa Kano, Katsuhito Sudoh, and Satoshi Nakamura. 2023.  Tagged end-to-end simultaneous speech translation training using simultaneous interpretation data.  In _Proceedings of the 20th International Conference on Spoken Language Translation (IWSLT 2023)_ , pages 363–375, Toronto, Canada (in-person and online). Association for Computational Linguistics.
  * Kudo et al. (2004) Taku Kudo, Kaoru Yamamoto, and Yuji Matsumoto. 2004.  Applying conditional random fields to Japanese morphological analysis.  In _Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing_ , pages 230–237, Barcelona, Spain. Association for Computational Linguistics.
  * Liu et al. (2020) Danni Liu, Gerasimos Spanakis, and Jan Niehues. 2020.  Low-Latency Sequence-to-Sequence Speech Recognition and Translation by Partial Hypothesis Selection.  In _Proceedings of Interspeech 2020_ , pages 3620–3624.
  * Ma et al. (2019) Mingbo Ma, Liang Huang, Hao Xiong, Renjie Zheng, Kaibo Liu, Baigong Zheng, Chuanqiang Zhang, Zhongjun He, Hairong Liu, Xing Li, Hua Wu, and Haifeng Wang. 2019.  STACL: Simultaneous translation with implicit anticipation and controllable latency using prefix-to-prefix framework.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 3025–3036, Florence, Italy. Association for Computational Linguistics.
  * Matsushita et al. (2020) Kayo Matsushita, Masaru Yamada, and Hiroyuki Ishizuka. 2020.  An overview of the Japan National Press Club (JNPC) Interpreting Corpus.  _Invitation to Interpreting and Translation Studies_ , 22:87–94.  (in Japanese).
  * Murata et al. (2010) Masaki Murata, Tomohiro Ohno, Shigeki Matsubara, and Yasuyoshi Inagaki. 2010.  Construction of chunk-aligned bilingual lecture corpus for simultaneous machine translation.  In _Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC’10)_ , Valletta, Malta. European Language Resources Association (ELRA).
  * Nakabayashi and Kato (2021) Akiko Nakabayashi and Tsuneaki Kato. 2021.  同時機械翻訳のための文脈を考慮したセグメントコーパス[Context-Aware Segment Corpus for Simultaneous Machine Translation].  In _Proceedings of the 27th Annual Meeting of the Association for Natural Language Processing_ , pages 1659–1663.  (in Japanese).
  * Nishikawa and Nakamura (2023) Yuta Nishikawa and Satoshi Nakamura. 2023.  Inter-connection: Effective Connection between Pre-trained Encoder and Decoder for Speech Translation.  In _Proceedings of INTERSPEECH 2023_ , pages 2193–2197.
  * Oda et al. (2014) Yusuke Oda, Graham Neubig, Sakriani Sakti, Tomoki Toda, and Satoshi Nakamura. 2014.  Optimizing segmentation strategies for simultaneous speech translation.  In _Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 551–556, Baltimore, Maryland. Association for Computational Linguistics.
  * Okamura and Yamada (2023) Yuki Okamura and Masaru Yamada. 2023.  「順送り訳」の規範と模範 同時通訳を模範とした教育論の試論 [Norms and Canon of Progressive Translation - An Exploratory Study on Educational Theories Using Simultaneous Interpretation as a Canon].  In Hiroyuki Ishizuka, editor, _Word Order in English-Japanese Interpreting and Translation: The History, Theory and Practice of Progressive Translation_ , pages 217–250. Hitsuji Syobo.  (in Japanese).
  * Ono et al. (2008) Takahiro Ono, Hitomi Tohyama, and Shigeki Matsubara. 2008.  Construction and analysis of word-level time-aligned simultaneous interpretation corpus.  In _Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC’08)_ , Marrakech, Morocco. European Language Resources Association (ELRA).
  * Papi et al. (2023) Sara Papi, Marco Turchi, and Matteo Negri. 2023.  AlignAtt: Using Attention-based Audio-Translation Alignments as a Guide for Simultaneous Speech Translation.  In _Proceedings of INTERSPEECH 2023_ , pages 3974–3978.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  Bleu: a method for automatic evaluation of machine translation.  In _Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics_ , pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
  * Post (2018) Matt Post. 2018.  A call for clarity in reporting BLEU scores.  In _Proceedings of the Third Conference on Machine Translation: Research Papers_ , pages 186–191, Brussels, Belgium. Association for Computational Linguistics.
  * Rei et al. (2020) Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020.  COMET: A neural framework for MT evaluation.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 2685–2702, Online. Association for Computational Linguistics.
  * Ryu et al. (2004) Koichiro Ryu, Atsushi Mizuno, Shigeki Matsubara, and Yasuyoshi Inagaki. 2004.  Incremental japanese spoken language generation in simultaneous machine interpretation.  In _Proceedings of Asian Symposium on Natural Language Processing to Overcome language Barriers_ , pages 91–95.
  * Sellam et al. (2020) Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020.  BLEURT: Learning robust metrics for text generation.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7881–7892, Online. Association for Computational Linguistics.
  * Shimizu et al. (2013) Hiroaki Shimizu, Graham Neubig, Sakriani Sakti, Tomoki Toda, and Satoshi Nakamura. 2013.  Constructing a speech translation system using simultaneous interpretation data.  In _Proceedings of the 10th International Workshop on Spoken Language Translation: Papers_ , Heidelberg, Germany.
  * Shimizu et al. (2014) Hiroaki Shimizu, Graham Neubig, Sakriani Sakti, Tomoki Toda, and Satoshi Nakamura. 2014.  Collection of a simultaneous translation corpus for comparative analysis.  In _Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14)_ , pages 670–673, Reykjavik, Iceland. European Language Resources Association (ELRA).
  * Tang et al. (2020) Yuqing Tang, Chau Tran, Xian Li, Peng-Jen Chen, Naman Goyal, Vishrav Chaudhary, Jiatao Gu, and Angela Fan. 2020.  Multilingual translation with extensible multilingual pretraining and finetuning.  _arXiv_ , arXiv:2008.00401.
  * Tohyama and Matsubara (2006) Hitomi Tohyama and Shigeki Matsubara. 2006.  Collection of simultaneous interpreting patterns by using bilingual spoken monologue corpus.  In _Proceedings of the Fifth International Conference on Language Resources and Evaluation (LREC’06)_ , Genoa, Italy. European Language Resources Association (ELRA).
  * Toyama et al. (2004) Hitomi Toyama, Shigeki Matsubara, Koichiro Ryu, Nobuo Kawaguchi, and Yasuyoshi Inagaki. 2004.  CIAIR Simultaneous Interpretation Corpus.  In _Proceedings of Oriental COCOSDA_.
  * Tsiamas et al. (2022) Ioannis Tsiamas, Gerard I. Gállego, Carlos Escolano, José Fonollosa, and Marta R. Costa-jussà. 2022.  Pretrained speech encoders and efficient fine-tuning methods for speech translation: UPC at IWSLT 2022.  In _Proceedings of the 19th International Conference on Spoken Language Translation (IWSLT 2022)_ , pages 265–276, Dublin, Ireland (in-person and online). Association for Computational Linguistics.
  * Zhang et al. (2020) Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020.  Bertscore: Evaluating text generation with bert.  In _International Conference on Learning Representations_.
  * Zhao et al. (2024) Jinming Zhao, Yuka Ko, Kosuke Doi, Ryo Fukuda, Katsuhito Sudoh, and Satoshi Nakamura. 2024.  NAIST-SIC-Aligned: Automatically-Aligned English-Japanese Simultaneous Interpretation Corpus.  _arXiv_.  ArXiv:2304.11766, version 4.