# Dialogue Understandability:
Why are we streaming movies with subtitles?

Helard Becerra, Alessandro Ragano, Diptasree Debnath, Asad Ullah, Crisron
Rudolf Lucas, Martin Walsh, Andrew Hines This publication has emanated from
research conducted with the financial support of Xperi Inc. and Science
Foundation Ireland (SFI) under Grant Number 12/RC/2289_P2. For the purpose of
Open Access, the author has applied a CC BY public copyright licence to any
Author Accepted Manuscript version arising from this submission.H. Becerra, A.
Ragano, D. Debnath, A. Ullah, C. R. Lucas, and A. Hines are with the School of
Computer Science, University College Dublin, Dublin, Ireland e-mail:
<EMAIL_ADDRESS>Walsh is with Xperi Inc., San Jose, California
e-mail<EMAIL_ADDRESS>
###### Abstract

Watching movies and TV shows with subtitles enabled is not simply down to
audibility or speech intelligibility. A variety of evolving factors related to
technological advances, cinema production and social behaviour challenge our
perception and understanding. This study seeks to formalise and give context
to these influential factors under a wider and novel term referred to as
Dialogue Understandability. We propose a working definition for Dialogue
Understandability being a listener’s capacity to follow the story without
undue cognitive effort or concentration being required that impacts their
Quality of Experience (QoE). The paper identifies, describes and categorises
the factors that influence Dialogue Understandability mapping them over the
QoE framework, a media streaming lifecycle, and the stakeholders involved. We
then explore available measurement tools in the literature and link them to
the factors they could potentially be used for. The maturity and suitability
of these tools is evaluated over a set of pilot experiments. Finally, we
reflect on the gaps that still need to be filled, what we can measure and what
not, future subjective experiments, and new research trends that could help us
to fully characterise Dialogue Understandability.

###### Index Terms:

Dialogue Understandability, streaming, QoE, movie subtitles, dialogue
intelligibility.

##  I Introduction

Do you always have the subtitles on when watching Netflix? If so you are
amongst the 85% of Netflix users who turn on closed captions when streaming
movies or TV series [1]. Closed captions or subtitles have been in use since
the 1970s and are an invaluable accessibility tool for deaf or hard-of-hearing
individuals [2]. They are also commonly used for foreign content or even in
noisy environments, such as rolling news channels in airports.

Ideal Insight carried out a survey and found that 85% of Netflix (54% Amazon
Prime and 37% Disney+) customers opt to watch with captions [3]. Another
survey from YouGov reported that younger people seemed to enable subtitles
more often than older people (63% of adults under 30 years old) [4]. Such high
percentages cannot be explained based on hearing impairments, age, language
proficiency or noisy environments, so why are so many switching on subtitles?
The answer lies in the fact that streaming has changed not just what we watch,
but how it is made, distributed and consumed. Can we identify, explain and
potentially quantify the effects of these factors?

In this study, we explore the wide range of factors that can influence whether
we follow the story or get frustrated and potentially give up on watching a
movie. We focus the analysis on factors related to 1) the production of movies
(e.g., sound recording, actor speaking style, etc), 2) the distribution of
movies (e.g., sound coding, streaming quality), and 3) the listener
environment (e.g., room acoustics, tv integrated speaker, headphones). We
introduce the concept of Dialogue Understandability as a term to encompass
more than just intelligibility or basic word understanding. We will use the
term movie as a generic term for content, but the concept could equally be
applied to any cinematic, TV or streaming content. We explore Dialogue
Understandability from a universal perspective, acknowledging that many of the
issues and factors are further exacerbated for individuals with hearing
impairments.

In this paper, we propose a working definition for Dialogue Understandability
in the context of streamed content. To support this, we review, identify and
categorise the factors that impact Dialogue Understandability (Section III).
We show the links between the defined Dialogue Understandability factors and
the established Quality of Experience (QoE) framework, the lifecycle stages of
media streaming, and the involved stakeholders (Section IV). We explore
potential methods for automatic characterisation of Dialogue Understandability
factors and present a set of pilot studies to illustrate the feasibility of
applying these methods over movie-like content (Section V). Finally, we
reflect on Dialogue Understandability and its utility to characterise the
influence of sound and subtitles on the QoE of our movie consumption (Section
VI).

##  II Dialogue Understandability

###  II-A Working Definition

Several speech and dialogue aspects (e.g., quality, intelligibility, content,
etc.) have been defined and studied over the years, resulting in well-defined
assessment procedures. However, Dialogue Understandability for movies is a
broader concept. For instance, the traditional concept of intelligibility is
usually measured as the listener’s capacity to hear words as a percentage of
the overall dialogue [5]. Yet, over a streamed movie session, high levels of
intelligibility might not guarantee a viewer’s complete understanding of the
story plot. Using a traditional speech quality approach to describe and
measure Dialogue Understandability would also fall short, as the levels of
degradation captured by speech metrics might not be entirely relevant to
measuring the engagement of users and their perception of quality [6].

In this study, the term Dialogue Understandability is used in the context of
cinematic, TV and streamed content and is defined as follows:

”The listener’s capacity to follow a story plot without undue cognitive effort
or concentration being required which impacts the Quality of Experience.”

###  II-B Root causes

Dialogue Understandability is constantly exposed to different factors that
affect it at different levels. In [7], these factors are referred to as
situational factors, such as mumbling actors, sound effects/content background
noise, loud consumption environments (e.g., metro, bus, crowded street),
lexical complexity, languages/accents, and consumption habits. Some of these
factors were also reported as anecdotal examples in video articles that
highlighted the use of subtitles as a means to improve Dialogue
Understandability [8].

Although a range of studies analyses some of these factors individually from
different angles[9, 10, 11], to our knowledge, there is no formal study
listing and categorising factors affecting movie consumption at their
different stages. To fill this gap, we explore the root causes, identify the
main influential factors and organise them into six sub-categories: 1)
artistic and production style, 2) language, accents and dialogue content, 3)
sound mixing techniques, 4) coding and streaming technology, 5) equipment and
consumption environment, and 6) social, immersion and multitasking. Figure 1
presents these categories mapping them to the influential factors defined in
the QoE framework (human, system, and context) [12]. This mapping process was
inspired by previous studies that used the QoE framework to organise and
facilitate the analysis for tasks like the quality assessment of sound
archives and the listening effort analysis [13, 14]. Dialogue
Understandability factors are also traced back to three main stages in a
common movie consumption pipeline (creation, delivery, and consumption).
Figure 1 illustrates the spectrum of factors affecting streaming media
consumption, linking them to the lifecycle stages of movie consumption. Using
a lifecycle pipeline for analysis was done previously in a study presented by
Ragano et al. [13]. to analyse and assess the perceived quality of restored
sound archives throughout different lifecycle stages (digitization,
restoration, and consumption).

Figure 1: The Dialogue Understandability Influencing Factors mapped to the QoE
Influencing Factors .

##  III Related Concepts

Here, we review concepts that are related to aspects of Dialogue
Understandability but do not individually capture it. We briefly define each
concept, their relevance in the context of Dialogue Understandability, and how
they are measured (summary in Table I).

TABLE I: Summary of related concepts affecting Dialogue Understandability
(DU).

Concept | Media | Aim | Limitations
---|---|---|---
QoE | Audio, speech, video, image, text | Provides conceptual framework | DU not systematically studied within QoE framework
Speech quality | Speech | Assess listening experience |  Designed for speech communication.
Not capturing artistic choices in entertainment content
Speech intelligibility | Speech | Measure word recognition rates | Intelligibility levels don’t always correspond to DU levels
Loudness | Audio, speech | Measure sound pressure levels (intensity) |  Determines the dynamic range and affects the dialogue loudness.
Not sufficient to describe DU
Speech rate | Speech | Measure words per second |  Affects perception of clarity, comprehension and fluency of speech.
Not sufficient to describe DU
Speaker identification | Speech | Identification based on speech characteristics |  Identify actors that are more challenging to understand.
Not sufficient to describe DU
Immersion | Audio, speech, video, image, text | Measure cognitive attention | Not sufficient to describe DU
Multimodal interactions | Audio, speech, video, image, text | Assess quality perception of multimodal media |  Explores complex interactions in dialogue and visual cues.
Not sufficient to describe DU

###  III-A Quality of Experience (QoE)

QoE is defined as the degree of delight or annoyance of the user of an
application or service. It results from the fulfilment of his or her
expectations with respect to the utility and/or enjoyment of the application
or service in the light of the user’s personality and current state [12]. QoE
uses a conceptual framework and categorises the influencing factors affecting
how users perceive the quality of a system or service into three classes:
human, system, and context (see Table II). This provides researchers with a
consistent yet flexible framework they can use to guide the analysis and
assessment of services and applications following a user-centric approach.
Although several studies have targeted applications like audio-visual signal
transmission, speech quality and intelligibility, cognitive and listening
effort, etc [15], which add to the Dialogue Understandability concept, QoE
analysis was commonly restricted to the delivery and consumption stages [16],
omitting the creation stage, where critical choices that can affect Dialogue
Understandability are made. QoE measurement adopts two main approaches:
subjective assessments that rely on human opinions and objective assessments
that use computational algorithms to quantify the perceived quality.

TABLE II: Illustrative classification of Quality of Experience Influencing
Factors in the context of telecommunications from [17].

Infuential Factors | Examples
---|---
Human | Low-level processing (visual and auditory acuity, gender, age, mood)
|  Higher-level processing (cognitive processes, socio-cultural and
economic background, expectations, needs and goals, other personality traits)
System | Content
| Media (encoding, resolution, sample rate)
| Network (bandwidth, delay, jitter)
| Device (screen resolution, display size)
Context | Physical (location and space)
| Temporal (time of day, frequency of use)
| Social (inter-personal relations during experience)
| Economic
| Task (multitasking, interruptions, task type)
| Technical and information (relationship between systems)

###  III-B Speech Quality

Speech sounds can be distinguished on attributes like pitch, duration,
loudness, timber, content and spaciousness. The sum of these components
describes the perceived composition of the speech sound and its perceived
quality [18]. On its own, speech quality is a broad concept related to the
overall listening experience for a speech signal without focusing on the
information that speech conveys [19]. For instance, if speech is sampled at 8
kHz, it can still be understood, but its quality might be perceived as low.
Another major concern for speech quality is noise (or background noise), as
its presence at the send side could lead to masking effects lowering speech
intelligibility [19]. Speech quality assessment evolved for technology
assessment, that is, evaluating speech codecs and telephone speech. This
resulted in several methods for automatic assessment [18]. However, content or
aspects like artistic choices or actor voices are not part of the scope of
these methods. For example, the voice of Darth Vader in Star Wars may be
incorrectly labelled as poor quality when, in fact, it represents an
intentional artistic decision. Thus, it could be argued that despite the
speech quality capacity to describe and measure some aspects affecting
Dialogue Understandability, the current objectives of speech quality
assessment clash with the artistic intentions underlying the creation of
movies.

###  III-C Speech Intelligibility

Speech intelligibility measures how the content of a speech utterance is
correctly identified, that is, the number of words or phonemes correctly
recalled in a speech sample [18]. Ratings can be obtained through listening
tests such as the speech reception threshold (SRT) [20] or the word error rate
(WER) [21]. These measures, combined with voice activity detection (VAD)
algorithms, are important in speech communications, especially for individuals
with speech and hearing impairments and communication systems where speech
intelligibility must be preserved. Speech intelligibility, while a
contributing factor to Dialogue Understandability, is not sufficient on its
own to fully quantify it. A high or low intelligibility rating does not
necessarily correspond to high or low Dialogue Understandability. In fact, it
can be argued that even if certain words are unclear, the overall meaning of a
movie dialogue can still be grasped. This situation is commonly observed in
interactions between native and non-native speakers, where native listeners
rely on contextual cues to comprehend conversations and anticipate sentences
[22]. Conversely, high levels of intelligibility might not always guarantee
Dialogue Understandability, for instance, it has been pointed out that
intelligibility does not account for irony or sarcasm which might be critical
to understanding context [23].

###  III-D Loudness

Loudness refers to the subjective perception of the sound pressure level. It
is expressed in decibels (dB) and can be measured using specialised equipment
such as sound level meters or acoustic analysis software. The role of loudness
is a highly influencing factor in movies. It will affect the speech-to-
background ratio (SBR) which measures the balance between dialogue and non-
speech elements in a sound mix audio track; and it will also determine the
dynamic range which is defined as the intensity range between the softest and
the loudest sound within a film scene. According to Netflix, content that
avoids excessive dynamic range provides a better experience for customers
[24]. At its encoding stage, Netflix performs loudness normalisation for the
movies’ audio master tracks where speech activity is detected [25]. Dialogue
loudness can be a determinant factor for Dialogue Understandability, as
mumbling actors have been identified as one of the main factors that
contribute to the reliance on subtitles [26]. Computational algorithms to
measure speech loudness have been developed over the years [27], while it can
help describe key aspects in Dialogue Understandability, it alone is not
sufficient to fully address it.

###  III-E Speech Rate

Speech rate refers to the speed or pace at which an individual speaks,
typically measured in terms of words per minute (WPM), syllables per second
(SPS) or phonemes per second (PPS) [28]. Speech rate can be measured by
analysing the duration of an audio clip or a specific speech segment and
counting the number of words, syllables or phonemes within that timeframe
[29]. Alternatively, automatic speech recognition (ASR) systems can be used to
transcribe speech and estimate speech rate based on the recognised words.
Within movies, speech rate can vary depending on different aspects like the
character’s role, an actor’s intrinsic characteristic or the scene context.
High speech rate could increase cognitive effort and the ability of viewers to
keep up with the subtitles, which, in turn, might affect Dialogue
Understandability.

Figure 2: Evolution of sound mixes from 1920 to 2020.

###  III-F Speaker Identification

Speaker identification aims to determine the identity of an individual based
on their voice and speech characteristics [30]. The application of ML-based
techniques have improved the accuracy of speaker identification models [31].
From a Dialogue Understandability standpoint, speaker characteristics are
important as some actors may be more challenging to comprehend than others. A
recent survey conducted in the United States revealed that the top three
actors who were reported as difficult to understand were Tom Hardy, Sofia
Vergara, and Arnold Schwarzenegger [32]. Difficulties and increased cognitive
load associated with certain actors can also arise from language variations,
for instance, American-English viewers listening to British-English actors or
foreign accents. Being able to identify speakers within movies might help spot
potential issues related to Dialogue Understandability and could guide the
implementation of additional assistance tools to ensure Dialogue
Understandability. For instance, personalised dialogue enhancement algorithms
or selectively enabled subtitles, where users could select which character(s)
or language(s) to target.

###  III-G Immersion

Immersion can be defined as the cognitive response of the user to the
characteristics of a system or media content [33]. Several studies have
explored how to measure the levels of immersion of a subject for a given
stimulus. The Qualinet’s White Paper on Definitions of Immersive Media
Experience (IMEx) [34] narrowed the methods to measure immersive media
experience to three main categories: subjective, behavioural, and psycho-
physiological assessments. For Dialogue Understandability, the levels of
immersion experienced during a movie session can determine whether a user is
able to follow or not the content’s plot, which will determine the user
engagement (whether the user gets frustrated and stops watching that movie).
That immersion is partially shaped by technological advances and evolving
consumption habits (e.g., watching a movie on a smart TV while sending an
instant message over a smartphone). Such a level of immersion can also be
affected by other elements like the screen size, the environment, or narrative
factors. From a sound mix perspective, sound formats for home and cinema have
evolved, expanding the number of channels available and adding new components
to deliver a more immersive movie-watching experience for consumers [35].
Figure 2 depicts the evolution of different sound mix techniques over time,
going from Mono/Stereo, multichannel, 3D and object-based formats. IA-based
formats are now being explored to fit consumers and the way they perceive
sound [36]. Being able to measure such levels of immersion (subjective
experiments) is critical to describe Dialogue Understandability.

###  III-H Multimodal Interactions

Research on multimodal interaction explores how different types of media
affect each other and, in turn, the overall perceived quality [35, 37]. Audio
and visual components are the most common modalities for media content with
several studies exploring their mutual influence and their contribution to the
overall quality perception [38, 39]. However, multimodal research is not
restricted to just these two components. Subtitles (closed captions) are
traditionally included in audio-visual content to assist viewers with hearing
loss while they consume specific media content [40]. For Dialogue
Understandability, the interaction between these components gets more complex
with the inclusion of techniques like shot-cuts between scenes, speakers off-
camera, or visual cues to prompt speech intelligibility. In a video article,
the portal Vox provides compelling examples of why we have an increasing
reliance on subtitles to keep up with complex movie plots, unintelligible
dialogue sequences, and sound mixing limitations [8].

##  IV Influencing Factors

The six categories of factors affecting Dialogue Understandability presented
in this study are mapped to the human, system, and context QoE factors in
Figure 1. Classifying the Dialogue Understandability factors using the QoE
framework will assist in identifying and characterising methodologies and
tools to quantify Dialogue Understandability for movie content. We also
consider how these factors interact temporally across the different stages of
a movie lifecycle. We break the lifecycle into three main stages:

  1. 1.

Creation: Covers the production stages related to film development, pre-
production, film shooting, and post-production [41];

  2. 2.

Delivery: Considers the processes by which media is encoded, transmitted and
adapted to be reproduced over a particular device [42];

  3. 3.

Consumption: Covers the actual reproduction of the media by the consumer [11].

Figure 3 illustrates how Dialogue Understandability is influenced across the
stages of a movie pipeline and can be broadly grouped into six factors. It
also captures the stakeholders, technologies and measurement tools, and
clusters the factors by QoE classes: human, context and system. This helps us
to identify complex interactions during the movie lifecycle; for instance, we
can see that the language, accent and dialogue content can be considered as
both human and context factors. We can also see that Dialogue
Understandability factors are not restricted to one unique stage in this
lifecycle. For instance, sound mixing at the creation stage can affect
Dialogue Understandability, where a sound engineer combines the different
sound elements of the movie, but it can also occur at delivery stage, where
the sound mix is adapted (downmixed) to a sound mix format of a particular
consumption device. Below, each of the Dialogue Understandability influencing
factors are briefly described using illustrative examples.

Figure 3: The Dialogue Understandability lifecycle and key stakeholders. The
QoE Influencing Factors (top left) are mapped to stages of the lifecycle
(creation, delivery, consumption) for each of the Dialogue Understandability
Influencing Factors. Key stakeholders and systems at each stage are also
illustrated.

###  IV-A Artistic/Production Style

Audio-visual production has advanced in realism since broadcast TV of the 20th
Century. Long-running TV shows like Doctor Who or Star Trek illustrate the
evolution from an almost theatrical stage play with dialogue spoken to camera
to immersive dialogue on the run [43]. The artistic/production style factors
were mapped to a context QoE factor at the creation stage (Figure 3). At this
stage, content creators (producers, writers, directors) will come up with the
idea for the intended content and bring their artistic perspective to the
project (film, TV show, documentary, etc.). During this stage, the content
idea, script draft, the desired film locations and the artistic and creative
guidelines are defined. Despite the effect that they could have on the
Dialogue Understandability, artistic/production guidelines are crucial to
creators as they consider them as key to reflect the intended ‘meaning’ they
try to imprint on the content and the desired experience they want to deliver
to the target audience [12]. Artistic and production style features can be
sourced out through metadata available (e.g., genre, director, cast, etc).
Other speech and visual informative features could be obtained using automatic
methods [27, 44].

###  IV-B Language, Accents and Dialogue Content

War of the Worlds [45] is an example of a multilingual series co-produced by
Canal+, Fox and Star where, unless you are fluent in several languages, you
will need some subtitles. Audio-visual media regulations [46], streaming
companies’ investment strategies [47], and the desire for natural dialogues
have resulted in national productions with multiple languages and/or strong
local accents that people are not familiar with. The language, accent and
dialogue content factors were mapped to both context and human QoE factors,
and they were traced to the creation and consumption stages at the streaming
lifecycle (Figure 3). At the creation stage, the context is guided by local
content regulations, investment strategies [46, 47] and the need for more
natural and realistic productions. The human factor is set by the actors’
speech characteristics and their capacity for delivering natural dialogues
(e.g., accent and dialect mastery). At the consumption stage, the context is
determined by the area/country where the content is being consumed (e.g.,
English-language content over non-anglophone regions) [48]. In turn, the human
factor is determined by the consumers’ traits. Being a native speaker (L1) or
having a second language (L2) would put a consumer in a better position while
engaging with certain content. Whether someone sticks with a movie to the end
or continues to watch a series is influenced by more than the plot.
Understanding the effect of language, accent, and dialogue content from a
Dialogue Understandability perspective will help identify regions where
subtitles or dubbing would make a difference in the level of consumption and
acceptance. It can also guide the implementation of personalised tools for
dialogue enhancement or smart subtitling for specific languages or accents.
Language and accent features can be outsourced through the metadata available.
More recently, progress in ML-based techniques have allowed the use of
automatic tools for language and accent recognition [49].

###  IV-C Sound Mix

Film director Christopher Nolan (Tenet, Interstellar) is well known for his
special effects but also for his sound mixing. Some dialogue is meant to be
unintelligible and difficult to understand, but it is hard for the audience to
know whether this is intentional or not [50]. The balance between dialogue and
non-speech elements in a sound mix, the Speech-to-Background Ratio (SBR), has
a great impact on people’s ability to discern dialogue, especially for hearing
impaired consumers. Experiments highlighted the variance of ratio preferences
among subjects and content type (e.g., news, sports, drama) [51]. The sound
mixing factor was mapped to a context QoE factor and traced to the creation
and delivery stages of the streaming lifecycle (Figure 3). At the creation
stage, content creators (directors and producers) choose certain mixing
strategies and format standards that they consider important for transmitting
certain experiences. During the delivery stage, hardware manufacturers must
provide compatible devices to adapt the mixing formats available. The
evolution of microphones and sound recording equipment resulted in several
directors guiding their sound mixing decisions to target primarily the widest
surround sound formats available (e.g., Dolby Atmos allowing up to 128 audio
tracks). From a Dialogue Understandability standpoint, these choices come at
the cost of decreased intelligibility for users that consume content using
mobile devices or standard home sound mixing settings (e.g., 5.1, stereo and
mono mixing formats). At the creation stage, mixing engineers often have
access to the script or have listened to the dialogue several times. For them,
this could lead to content familiarity and reduce their sensitivity to poor
Dialogue Understandability. For dubbed material, streaming service providers
often require material for several languages. Due to time and budget
constraints, there is often minimal or no quality assurance on how the new
language melds with the original stem. Sound mixing characteristics are
commonly available through metadata and the technical specs from the content.
Collecting this information can help guide the implementation of dedicated
audio processing assistance tools for consumption devices to improve the
consumers’ experience (e.g., dialogue enhancement fitted for specific content
and devices) [51].

###  IV-D Coding and Streaming

With the rise of the over-the-top (OTT) model, which aims to deliver movies
and TV content via the Internet, streaming companies are now compelled to
provide their customers with services and equipment that guarantee high levels
of perceived quality in order to keep themselves as relevant alternatives in
the OTT market. The coding and streaming factor was mapped to the QoE system
factor, and it was traced to the delivery stage in the streaming lifecycle
(Figure 3). At the delivery stage, system factors involve the media
characteristics (e.g., video frame rate, sound mixing setup), the transmission
channel characteristics (e.g., the bandwidth and network characteristics), and
the equipment and device characteristics (e.g., high-resolution smartphone)
[12, 38]. Streaming service providers are responsible for storing the media
content in data centres equipped with high-speed internet connection for
efficient distribution. Meanwhile, hardware manufacturers must provide devices
that adhere to coding, network and transmission standards for audio playback.
Tools to compute coding and streaming features have received much attention
from the research community with several objective tools being developed and
standardised throughout the years [52]. They have served as a base for the
implementation of adaptive streaming technology to automatically tune content
quality based on the users’ network conditions, thus ensuring a better
experience without disruption in reproduction [53, 54].

###  IV-E Equipment and Environment

Advances in the consumer electronic sector have shaped the way we consume and
experience movies outside cinema theatres. For movie consumption, users now
have at their disposal a large variety of devices (e.g., smartphones, personal
computers, smart TVs, gaming consoles, home theatre systems, etc.), each one
supporting different sound mixing formats (e.g., mono, 2.0 stereo, 5.1 and 7.1
surround), making the sound mixing adaptation process harder. Modern TV
manufacturers aim for thinner designs with small speakers and paper-thin
drivers. To compensate for these drivers, a combination of psychoacoustics and
digital signal processing methods are commonly used to make the sound
perceived as louder, affecting the dialogue intelligibility. The equipment and
environment factors were mapped to system and context QoE factors, and they
were traced to the delivery stage in the streaming lifecycle (Figure 3). At
the delivery stage, the system factor is guided by hardware manufacturers and
their engagement to meet the wide range of requirements for sound downmixing
and format adaptation. Meanwhile, the context is guided by the type,
complexity, and usability of new consumption devices (e.g., mobility provided
by smartphones, tablets and other portable devices) [55]. Sound downmixing and
format adaptation for consumption devices pose several challenges to guarantee
good levels of Dialogue Understandability. Sound elements, potentially key for
a dialogue sequence, might get lost or overlapped due to the downmixing
process. Acquiring expensive equipment (e.g., high-quality soundbars) for a
better experience might not be a feasible solution if companies are interested
in reaching large audiences. Device mobility of new consumption devices comes
at the expense of environmental noise and distractions, which, depending on
the service and the content being consumed (e.g., a phone call in the street,
music while running, movies at the metro), could affect Dialogue
Understandability [56]. How and where the content is consumed affects Dialogue
Understandability, this could explain why people are much less likely to
enable subtitles when listening with headphones. Equipment and environmental
information might not be available at the consumer end. Questionnaires over
subjective experiments might be the best alternative to capture this
information and analyse its effect over the perceived Dialogue
Understandability.

###  IV-F Social/Immersion/Multitasking

Social behaviour has evolved throughout the years, and so has the way we watch
movies. Studies have reflected on multitasking and its effect while consuming
streamed content (e.g., watching a movie while chatting on social media apps)
[57]. The social, immersion and multitasking factors were mapped to the system
and human QoE factors, and they were traced to the delivery and consumption
stages (Figure 3). At a delivery stage, hardware manufacturers are compelled
to design devices that fulfil the needs of more demanding users and the so-
called ’digital-native’ generation [58]. At the consumption stage, the human
factor is guided by users’ technological demands (e.g., constant connectivity,
on-demand services), consumption habits (e.g., watching content at home while
doing chores) and social traits (e.g., watching a movie at night with low
volume). Social and cultural traits have also influenced the design of our
living spaces, where media is commonly consumed. Modern homes have shifted
from cluttered interiors to minimalist designs, often with bare walls, large
windows, and hard floors. This acoustic transformation has introduced room
modes, early reflections, and lingering echoes, hindering dialogue
comprehension in living areas. Additionally, open-plan layouts expose viewers
to household noises from appliances, often masking dialogue altogether. From a
Dialogue Understandability perspective, these aspects could lead to users
missing key events or dialogue sequences that could affect their ability to
follow the plot development. Collecting information at the consumers’ end that
could help describe and analyse these factors can be done through subjective
experiments using surveys and questionnaires [11].

The evolving trends in cinema art (production styles and naturalistic
content), technology advances (sound mixing, communication networks, and
equipment), and social behaviour (multitasking users and media consumption
habits) have impacted the dialogue perception of movies. To keep up with new
competitors, involved stakeholders need to consider these factors to deliver
high-perceived quality to their consumers and subscribers.

##  V Quantifying Dialogue Understandability

The definition of Dialogue Understandability that we propose in this paper is
novel. As such, there is no quantitative metric that is designed to measure
it. However, each Dialogue Understandability factor introduced in the previous
section is associated with numerous tools in the literature that can be
potentially used. This section provides a comprehensive overview of these
tools, linking them to each Dialogue Understandability factor (Table III).
Pilot experiments were conducted to verify the suitability of these tools over
movie-like content. We are particularly interested in seeing which tools are
mature enough to be used with real streamed media content, which ones need to
be adjusted and tuned to meet streaming context requirements, and which ones
are not at all contributing to capturing the Dialogue Understandability
factors they are targeting.

###  V-A Tools for measuring Dialogue Understandability

TABLE III: Mapping tools to Dialogue Understandability Factors. Language:l,
Accent:a.

Factors | Intelligibility Metrics | Quality Metrics | Languages and Accent Classification | Scene Characteristics | Metadata
---|---|---|---|---|---
| SRMR | NISQA-MOS | NISQA-noisiness | NISQA-intensity | NISQA-colour | VoxLingua (L) | CommonAccent (A) | P.910-Spatial | P.910-Temporal |
Creation |  |  |  |  |  |  |  |  |  |
Artistic/Production Style | $\blacksquare$ | $\blacksquare$ | $\blacksquare$ | $\blacksquare$ | $\blacksquare$ |  |  | $\blacksquare$ | $\blacksquare$ |
Language, Accent and Dialogue |  |  |  |  |  | $\blacksquare$ | $\blacksquare$ |  |  |
Sound Mix | $\blacksquare$ |  |  |  | $\blacksquare$ |  |  |  |  | $\blacksquare$
Delivery |  |  |  |  |  |  |  |  |  |
Coding and Streaming | $\blacksquare$ | $\blacksquare$ | $\blacksquare$ | $\blacksquare$ | $\blacksquare$ |  |  |  |  |
Equipment and Environment |  |  |  |  |  |  |  |  |  |
Consumption |  |  |  |  |  |  |  |  |  |
Social/Immersion/Multitasking |  |  |  |  |  |  |  |  |  |

####  V-A1 Intelligibility Metrics

Computational intelligibility metrics can be divided into intrusive (i.e. use
both test and reference signal) [59] and non-intrusive (i.e. use only the test
signal). The former group is not suitable for dialogue movies since a
reference signal is missing, and its definition can be ambiguous in this
context. Existing non-intrusive intelligibility metrics can be used for the
following Dialogue Understandability factors:

  * •

Creation: artistic/production style, sound mix;

  * •

Delivery: coding and streaming.

The Speech to Reverberation Modulation energy Ratio (SRMR) [60] is a non-
intrusive metric designed to assess quality and intelligibility for
reverberant speech conditions. The SRMR metric was evaluated in several speech
conditions, showing good performance and outperforming intrusive metrics [61].
SRMR can be used to measure the intelligibility of movie dialogues; however,
it is advised to use it with caution for certain conditions. For instance, it
has been observed that its performance decreases when speech is degraded with
multiple impairments such as speech-shaped noise, ideal binary masks, and
whenever different signal-to-noise ratios (SNRs) are combined together [62].
This means that the complexity of different sounds that occur in movie scenes,
such as foreground music, might mislead SRMR scores. Collaboration with
creators would allow more intrusive testing over dialogue-only tracks,
commonly available at a creation stage. This could also be done at a consumer
end through object-based audio or ML-based sound separation tools [63, 64].

####  V-A2 Quality Metrics

Speech quality can also be measured intrusively or non-intrusively. As with
intelligibility prediction, using intrusive metrics to predict quality in
dialogue movies is not feasible due to the ambiguity that might arise from
defining what a reference signal is. Research on non-intrusive speech quality
metrics has seen remarkable advances by applying deep learning techniques
[27]. These metrics aim at predicting the mean opinion score (MOS), which is
the average rating from several human raters for the same media stimulus. MOS
predictors can be useful for understanding scenes where dialogue quality
varies, which would allow us to investigate the root causes behind quality
variations. These causes, as mentioned above, might not necessarily be related
to network issues or low-quality codecs but can be due to artistic choices.
Quality metrics can be used for the following Dialogue Understandability
factors:

  * •

Creation: artistic/production style, sound mix;

  * •

Delivery: coding and streaming.

The NISQA model [27] is a non-intrusive metric that can be useful to collect
features beyond MOS. Besides the MOS prediction, the NISQA model provides
measures for noisiness, colouration, discontinuity, and loudness. Being able
to separate different quality factors allows us to explore characteristics
that go beyond the general MOS value. For instance, speech discontinuity or
colouration are traits that can be influenced by artistic choices. Sound mix
choices during the creation phase affect dialogue colouration due to timbre
interference between speech and other sounds present in the scene. Measuring
speech colouration can help to identify where speech complexity could increase
due to the sound mix.

####  V-A3 Scene Characteristics

How much someone can compress video scenes strongly depends on the content.
Content variability can be quantified in terms of spatial and temporal
information from video sequences. Such variability will affect the visual
perception, which, in turn, will influence the perceived audio quality.
Studies have shown that audio quality is strongly influenced by video quality
and that video quality tends to dominate the integrated audio-visual QoE [39].
The presence of lip movement has also shown a degree of influence over speech
perception. Indeed, it is accepted that speech perception is inherently
multimodal [65] with evidence suggesting that sighted humans exploit visual
cues such as lip movement when interacting [65]. Scene characteristics can be
quantified for the following Dialogue Understandability factors:

  * •

Creation: artistic/production style.

ITU-T recommendations P.910 suggest quantifying scene characteristics in terms
of spatial and temporal information [44]. The spatial information captures the
amount of detail in a single picture, a higher value indicates a more complex
scene. Meanwhile, the temporal information measures the temporal changes of a
video (a sequence of pictures). A higher value usually corresponds to a high
motion video sequence. Due to their low computational complexity, they are
very popular for classifying video databases for encoding or processing
purposes.

####  V-A4 Language and Accent Classification

Open-sourced multilingual automatic speech recognisers (ASR) can help quantify
the diversity of languages and accents in movies. For spoken language
identification (SLID), the use of deep neural networks has been demonstrated
to improve performance and report accurate predictions [66]. In terms of
accent identification, several models that deal with this task have been
proposed in recent years [67]. In multilingual ASR, identifying the accent has
been suggested to improve recognition accuracy by mitigating accented speech
errors [49]. In this context, existing language and accent classification
methods can be used for the following Dialogue Understandability factors:

  * •

Creation: language, accent and dialogue content.

Both language and accent automatic identification have benefited from the rise
of new machine learning approaches such as Multi-Task Learning (MTL) and Large
Acoustic Models (LAM). LAMs have gained popularity with the implementation of
Meta’s wav2vec 2.0 Self-Supervised Learning models for speech representation.
For language identification, the authors in [68] presented the VoxLingua
model, which was trained using audio segments collected from YouTube videos
covering almost 107 languages. In a recent work by Zuluaga-Gomez et al. [49],
the authors fine-tuned and improved two models: ECAPA-TDNN and wav2vec 2.0
model for accent identification.

####  V-A5 Metadata

The use of metadata to complement signal features has shown promising results
for predicting the perceived quality. In [69], Chinen et al. used the rater
group and synthesis system identifiers to complement the feature vector
descriptor generated by the wav2vec 2.0 model to predict synthesised speech
quality. In terms of Dialogue Understandability factors, complementary
information regarding the media content could help describe aspects that are
not entirely captured with objective methods. For instance, besides
information like the movie genre or country of production, which can inform
Dialogue Understandability, complementary information like the director’s name
or the movie cast might lead to better descriptors regarding the artistic and
production styles that will affect Dialogue Understandability. In this study,
we recommend using metadata to inform the following Dialogue Understandability
factors:

  * •

Creation: artistic/production style, sound mix.

Metadata is transmitted alongside the audio, video, subtitles, and other
components of a streamed movie, and its available at the streaming service
interface. The Internet Movie Database (IMDb) is an online dataset containing
detailed information for films, TV series, podcasts and streaming content.
This information includes the director, production crew, cast, plot summaries,
year of release, genre, user ratings/critical reviews, and technical
information like sound mix and aspect ratio. This information could help
characterise several Dialogue Understandability factors like the sound mix
format or even use the user’s rating to inform about the acceptance of the
movie.

###  V-B Quantifying Dialogue Understandability features

Some of the tools presented in this section were evaluated to verify their
suitability for characterising Dialogue Understandability. To achieve this,
pilot experiments were carried out using available movie clips and non-
modified versions of available objective tools. For each experiment, Table IV
contains the Dialogue Understandability factor the experiments are associated
with, the methods employed, the dataset source and a brief description and
technical specifications about the data used in each experiment.

TABLE IV: Pilot studies summary. DU: Dialogue Understandability.

Experiment | DU Factor | Methods | Dataset source | Dataset description | Suitability level
---|---|---|---|---|---
Candidate tools |  Artistic/production style,
Coding and streaming,
Sound mix |  Objective metrics
SRMR (speech intelligibility)[60]
NISQA (speech quality)[27]
Spatial/Temporal Index (scene characteristics)[44]
Silero VAD (additional feature)
Google WER (additional feature)

Subjective scores
Subjects: 3 speech experts
Material: 4 movie trailer clips
Equipment: Desktop computer with commercial speakers
Ranking scale: 0-5 (0 - no dialogue, 1 - bad quiality dialogue, 5 - high quality dialogue) | YouTube |  Four YouTube videoclips (movie trailers)
Audio: stereo, 44.1 kHz sampling rate, 128 kbps bitrate
Video: MPEG-4, 720p spatial resolution, 23.976 temporal resolution, 4:2:0
color sampling
Trailers: Casablanca (2 mn 22 s), Spider-Man 3 (2 mn 27 s), The Lord of the Rings-The fellowship of the ring (2 mn 29 s), Slumdog Millionaire (2 mn 15 s) |  Mid to low
-Needs adaptation for movie content
Language classification | Language, accent and dialogue |  Voxlingua [68]
107 Languages, 6628 hrs (62 hrs per language) | YouTube |  Three YouTube videoclips (10 s segmented movie trailers)
Audio: stereo, 44.1 kHz sampling rate, 128 kbps bitrate
Video: MPEG-4, 720p spatial resolution, 23.976 temporal resolution, 4:2:0
color sampling
Trailers: Narcos (11 clips, English, Spanish), Squid Game (12 clips, Korean), 1899 (11 clips, English, Spanish, French, German) |  Low
-Needs adaptation for movie content
-Add source separation + VAD component
Accent classification | Language, accent and dialogue |  CommonAccent [49]
Accents: 16 English (African, Australia, Bermuda, Canada, England, Hong Kong, Indian, Ireland, Malaysia, New Zealand, Philippines, Scotland, Singapore, South Atlantic, US, and Wales), 4 German, 6 Spanish, and 5 Italian | YouTube |  Two YouTube videoclips (segmented movie trailers)
Audio: stereo, 44.1 kHz sampling rate, 126 kbps bitrate
Video: Advance video codec, 640x360 spatial resolution, 24 temporal
resolution, 4:2:0 color sampling
Trailers: Sense8 (7 clips, 2 to 9 s), John Wick 4 (6 clips, 3 to 14 s) |  Low
-Needs adaptation for movie content
-Add source separation + VAD component

####  V-B1 Measuring Dialogue Understandability: Candidate Tools

In this experiment, we test some instrumental objective metrics (presented in
Section V-A) for Dialogue Understandability. We try to explore how these
metrics, which were developed to capture specific aspects from speech/audio
and video, will perform over movie-type content. The films were selected to
represent different film styles, genres, languages and contemporaneity (year
of release). Although we try to represent different types of content, the main
target is to gather information on how the available tools perform over movie-
like content. Overall, ten different features were extracted to measure
aspects affecting Dialogue Understandability. The audio WAV files were
segmented into clips of 10 s duration. Objective measurements were computed
for each segment separately and they were aligned later for each segment.
Three (3) expert listeners (with normal hearing) were presented with the four
(4) movie trailers (only once) and they were asked to rate the Dialogue
Understandability levels. The listening environment was kept the same for each
listening experimental session. Details regarding the movie clips, the metrics
tested, and the collection of subjective data are presented in Table IV.

Figure 4: Feature analysis for movie trailers. Slumdog Millionaire (Danny
Boyle, Loveleen Tandan, 2008).

Figure 4 presents the report for ten objective measurements plus the
subjective scores for one movie clip. This representative report aims to show
how these features vary in relation to a subjective Dialogue Understandability
score. Results from this experiment suggest that, by themselves, these
objective measurements are not able to fully characterise the perceived
Dialogue Understandability. However, variations in their scores, which were
also observed among the subjective ratings, indicate that they are capturing
variations in certain characteristics related to the perceived Dialogue
Understandability. Further experiments, including more human participants and
a curated dataset, are necessary to confirm that, in fact, these features have
the potential to be used to feed a predictive model for Dialogue
Understandability.

####  V-B2 Language identification

This experiment seeks to evaluate the application of language identification
models over movie-like content. Language identification could potentially help
the implementation of personalised assistance tools. For instance, speech
enhancement and smart captioning whenever a specific language is detected. We
are interested in evaluating the suitability of available tools for the
purpose of Dialogue Understandability characterisation. In this experiment, we
evaluate the pre-trained version of the VoxLingua model [68] over audio
samples extracted from YouTube video clips. Three video clips were selected to
represent different languages over streamed TV content (e.g., English, German,
Spanish, Korean). The audio component of the clips was extracted and segmented
into 10 s sequences. Details for the clip samples can be found in Table IV.

Results from this experiment showed the difficulty of the VoxLingua to
correctly predict some languages, such as English, Welsh, and Dutch.
Interestingly, the model predicted Latin and Javanese languages when there was
no speech activity in the given audio segment (background music and sound
effects). This highlights the need to add a VAD component (plus source
separation for overlapping sound) to avoid incongruous outcomes. Although the
model showed good performance over controlled speech segments [68], the
results from our experiment suggest that the model is sensitive to more
challenging segments that include noisy and competing sound effects.

####  V-B3 Accent identification

For this experiment, we tested the CommonAccent pre-trained model presented in
[49]. The model implementation takes an audio sample and returns percentage
values for the detected accents. The model was tested over YouTube clips
selected for their diversity of English accents in their dialogues. The audio
component for the two clips was extracted and segmented into clips of 2 to 14
s duration. Details for the movie clips and their content can be found in
Table IV. Results showed the model’s difficulty in identifying the target
accent in movies. Moreover, the acoustic and linguistic information (e.g.
accent portrayal of the actor, and movie script) also has a high effect on the
accuracy of the accent identification. Although the model was able to
recognise UK and US English accents, the model tended to force prediction when
no dialogue was present or when the target accent was outside the 16 accents
from which the system was trained.

####  V-B4 Modelling Dialogue Understandability

Results from pilot experiments showed that current methods had trouble
performing over movie-like audio clips. These results confirm the need for
adaptation and possibly additional tools (e.g., VAD component plus source
separation) before using them over movie-like content. Having looked at the
suitability of the different tools, Figure 5 presents a working diagram for
modelling Dialogue Understandability. The diagram shows the main components of
a streaming movie (e.g., video, audio, subtitles, metadata) that could act as
potential sources to extract features describing Dialogue Understandability.
The extracted features would act as input for a data-driven model trained to
predict the levels of Dialogue Understandability for streaming movies. It is
important to note that, although an approximation of the perceived Dialogue
Understandability could be attained using this type of model, it still won’t
be able to capture the entire range of factors affecting it. The diagram only
presents what is currently feasible, considering the available material and
tools. A large portion of information, mainly from the creation and
consumption stages, is still needed to fully measure the perceived Dialogue
Understandability. In that sense, this diagram should be taken as an initial
approximation of what modelling Dialogue Understandability constitutes.

Figure 5: Measuring Dialogue Understandability. Feature sources and potential
metrics to predict Dialogue Understandability.

##  VI Discussion

###  VI-A Sound mix format evolution

The timeline presented in Figure 2 shows how sound mix formats have evolved
and produced new formats that target a more realistic and immersive audio
experience. Multichannel-based formats have dominated the cinema sound
landscape and they were the standard for several cinema and home theatre
settings until the 2010s. The introduction of the Auro-3D format as an initial
three-dimensional audio experience attempt (height sound channels), served as
base to the development for object-based formats which allow audio elements to
move freely around the listener in a three-dimensional space. More recently,
AI-based formats are being developed trying to adjust their setup to match how
the listener would perceive the sound [36]. This timeline shows the efforts of
the sound cinema industry to create more immersive formats, yet; the
downmixing process, where these formats are adapted to fit different devices,
has received less attention. The design of these devices (e.g., thinner TVs
and smaller speakers), their functional characteristics (e.g., smartphones
plus headphones), and the diversity of devices (e.g., smartphones, personal
computers, smart TVs, gaming consoles) could affect the resulting sound mix
perceived by users consuming movies through such devices. Overall, we can
observe an increase in sound mixing formats, especially in the last 25 years.
Although some of them have not been widely adopted, such diversity still poses
a challenge to sound mixing engineers, streaming service companies and
hardware manufacturers wherever they need to decide to implement sound mixing
adaptations for different devices and consumption settings.

###  VI-B Utility of Dialogue Understandability

In this paper, we presented a working definition for Dialogue
Understandability and a list of factors influencing it. A Dialogue
Understandability perspective, which relates to the QoE framework and the
movie lifecycle stages, facilitates the analysis and understanding of extent
and effectiveness of the methods used to improve Dialogue Understandability
such as dialogue enhancement, sound mix, and smart subtitles.

Dialogue enhancement is an active area of research that aims to enhance the
dialogue sounds in movie content [70]. As with other speech-related fields, it
has seen great advances by applying more sophisticated deep-learning-based
approaches (e.g., MTLs, LAMs) [71]. This method can be implemented in the
delivery stage by streaming service providers targeting the consumption stage
where users could customise their listening preferences. More recently, Amazon
Prime’s Dialogue Boost [72] was released as an accessibility tool allowing
users to change the relative speech volume with respect to the background
music and effects. Subjective experiments, designed and informed by Dialogue
Understandability, are needed to understand how Amazon Prime’s Diaologue
Boost, and other dialogue enhancement methods, are affecting Dialogue
Understandability. For instance, reducing the background noise, music, etc.,
in a scene undoubtedly affects the context or tone of a scene at the
consumption stage. With the advent of dialogue boosting, to what level
artistic choices in a scene be influenced in the creation stage?

For the sound mix process, object-based audio proposes an alternative to the
traditional channel-based way of mixing sound and gives an opportunity to
improve broadcast accessibility for hard-of-hearing individuals [63]. This
approach proposes to transmit audio signals as speech components and mix them
at the receiver’s end-point using descriptive metadata. This allows more
flexibility whenever a component needs to be prioritised based on the user’s
demand. From a Dialogue Understandability perspective, this approach would be
beneficial for managing sound elements that are considered critical for the
dialogue and plot understanding. In addition, it would allow a more
personalised solution based on individual needs instead of a unique
enhancement process that tries to cover a wide range of users and consumption
settings (e.g., devices, environments, user traits, etc.). The adoption of
object-based audio formats has been growing in recent years, especially among
streaming service providers (e.g., Netflix, Amazon Prime, Disney+, AppleTV).
This is thanks to the implementation of the delivery systems required to
handle object-based audio transmission. There are, however, some aspects that
need to be explored whenever object-based audio is used, for instance, an
audio-based mixing approach would remove the artistic control from creators
and rely on other stakeholders (e.g., hardware manufacturers, streaming
service providers, or end consumers) to get the ‘right’ sound mix. How would
this affect the choices made by the creators?

Subtitles help individuals interpret and grasp content that might otherwise be
inaccessible to them. Production companies and streaming service providers
delegate the generation of subtitles to in-house production teams, external
subtitling agencies and expert freelancers. Despite being considered a key
tool for dialogue and content understanding, potential negative effects such
as scene blocking, font size and text legibility, and demanding levels of
visual attention are reported by users [73]. Moreover, subtitles may not be
equally effective for everyone. Reading difficulties arising from different
levels of quality of education and socio-economic or neurological factors
(e.g., dyslexia) could impact the use of subtitles, increasing the cognitive
load and reducing content engagement for consumers. One alternative is the
implementation of smart subtitling (or dynamic captioning). This approach aims
to optimise the position of captions through speaker recognition and scene
analysis [74]. Early studies have shown good levels of acceptance among users
reporting more immersive experiences and less missed content during the
viewing sessions [74]. There is, however, the need to explore to what level
the subtitling positioning would impact artistic choices like the scene
composition and the cinematography design. Dialogue Understandability provides
the needed framework to explore and understand such impact. Subjective
experiments, guided by Dialogue Understandability, will help understand and
guide the design of smart subtitles with minimal impact on the intended
content.

###  VI-C Characterising Dialogue Understandability

This study provides a formal categorization of the different aspects that
influence how we consume movies outside cinema theatres. We have presented a
list of potential methods to characterise Dialogue Understandability, and they
were tested over a set of pilot studies to explore their applicability and
maturity to measure or describe Dialogue Understandability factors. We analyse
the potential of these tools to characterise Dialogue Understandability based
on their availability, practicality, and maturity.

From an availability standpoint, tools to automatically measure factors like
equipment and environment, social immersion, and multitasking were not readily
available for testing. Although there are studies focused on measuring these
factors, they are mostly restricted to subjective experiments [75]. In terms
of practicality, mapping the Dialogue Understandability factors in the
streaming pipeline and linking them to the involved stakeholders allowed us to
understand the degree of control they offer. Creation factors like the
artistic and production style, the language accent and dialogue content and
the sound mix cannot be avoided. However, through a Dialogue Understandability
approach we are in the capacity of using the available information (metadata)
and automatic tools to detect potential issues and enforce methods to mitigate
their effect over Dialogue Understandability (e.g., dialogue enhancement,
subtitles). Regarding their maturity, the pilot studies allowed us to explore
the suitability of these methods for movie-type content. Based on those
results, it was concluded that the available tools can’t be used to measure
Dialogue Understandability factors in their current state. Complementary tools
and model adaptation (e.g., ASR components, fine-tuning steps) are needed to
capture movie-like content characteristics. In addition, larger experiments
are necessary to evaluate these adapted tools.

###  VI-D Limitations and Future Experiments

Merely identifying the origin of a factor to a particular stage or stakeholder
will not rectify the Dialogue Understandability. Systematically understanding
the factors contributing to Dialogue Understandability facilitates an engaged
conversation about the extent to which each stakeholder can contribute to
solutions. The potential and limitations of technology-based solutions are
also framed, acknowledging that there are no universal silver bullet
solutions. The list and the mapping presented in this study is an attempt to
cover the entire spectrum of factors that influence Dialogue Understandability
at different stages, but it does not answer the question of what is the share
of the impact that these factors have on the overall Dialogue
Understandability. The set of pilot experiments presented in this study served
to evaluate the level of maturity of currently available measurement tools.
Future experiments, to quantify the impact of these factors, will require
larger content material curated following a Dialogue Understandability
perspective. A promising initiative is the dataset presented at [76],
containing 1600 h of audio tracks from movie and TV shows for speech and music
activity detection. The dataset is manually annotated with time stamps for
speech and music segments in the audio track. Datasets like this could be used
for the implementation and evaluation of Dialogue Understandability
optimization tools.

##  VII Conclusions

In this study, we provided a working definition of Dialogue Understandability.
The term describes the evolving factors that shape the way we consume movies
and streamed-like content. We organised these factors into six sub-categories
based on their root causes. We linked them to existing QoE influential
factors, the common streaming lifecycle stages, and the involved stakeholders
to bring attention to how manageable these factors are and who are the main
responsible actors that they depend on. The results from a set of pilot
studies, that verified the suitability of the current methods in the
literature, highlighted the need for adaptation before using them over movie-
like content. They also evidenced that certain factors, being artistic and
aesthetic aspects decided by content creators, cannot be avoided. But, they
can be spotted at an early stage, and help guide the implementation of
assistance tools to mitigate their effect and improve Dialogue
Understandability (e.g., dialogue enhancement, smart subtitles, or sound mix).
Our study can help the design of relevant datasets, experimental protocols,
and objective tools to characterise the perceived Dialogue Understandability
of movie-like content.

We begin this paper by asking the question, why are we streaming movies with
subtitles? The answer lies in the evolving complexity of processes for movie
creation, transmission and consumption. More and more subtitles are being used
in response to a poor Dialogue Understandability experience. Our work provides
a systematic structure for researchers to explore and implement tools to
improve every viewer’s experience.

## References

  * [1] H. Kim, Y. Tao, C. Liu, Y. Zhang, and Y. Li, “Comparing the impact of professional and automatic closed captions on video-watching experience,” in Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems, pp. 1–6, 2023.
  * [2] E. Ellcessor, “Captions on, off, on tv, online: Accessibility and search engine optimization in online closed captioning,” Television & New Media, vol. 13, no. 4, pp. 329–352, 2012.
  * [3] Ideal Insight, “Netflix and captions?: A new study shows that 85% of Netflix customers in the UK use subtitles while watching.” https://idealinsight.co.uk/infographics/netflix-captions.  Accessed: 21 November 2023.
  * [4] YouGov, “Most American adults under 30 prefer watching TV with subtitles.” https://today.yougov.com/entertainment/articles/45987-american-adults-under-30-watching-tv-subtitles.  Accessed: 21 November 2023.
  * [5] H. Schepker, K. Haeder, J. Rennies, and I. Holube, “Perceived listening effort and speech intelligibility in reverberation and noise for hearing-impaired listeners,” International journal of audiology, vol. 55, no. 12, pp. 738–747, 2016.
  * [6] F. Dobrian, V. Sekar, A. Awan, I. Stoica, D. Joseph, A. Ganjam, J. Zhan, and H. Zhang, “Understanding the impact of video quality on user engagement,” ACM SIGCOMM computer communication review, vol. 41, no. 4, pp. 362–373, 2011.
  * [7] B. M. Gorman, M. Crabb, and M. Armstrong, “Adaptive subtitles: Preferences and trade-offs in real-time media adaption,” in Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, pp. 1–11, 2021.
  * [8] Vox, “Why we all need subtitles now.” https://youtu.be/VYJtb2YXae8.  Accessed: 21 November 2023.
  * [9] B. Shirley and L. Ward, “Intelligibility versus comprehension: understanding quality of accessible next-generation audio broadcast,” Universal Access in the Information Society, vol. 20, no. 4, pp. 691–699, 2021.
  * [10] M. Tapaswi, Y. Zhu, R. Stiefelhagen, A. Torralba, R. Urtasun, and S. Fidler, “Movieqa: Understanding stories in movies through question-answering,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4631–4640, 2016.
  * [11] Y. Zhu, I. Heynderickx, and J. A. Redi, “Understanding the role of social context and user factors in video quality of experience,” Computers in Human Behavior, vol. 49, pp. 412–426, 2015.
  * [12] European Network on Quality of Experience in Multimedia Systems and Services (COST Action IC 1003), “Qualinet White Paper on Definitions of Quality of Experience,” March, 2013.
  * [13] A. Ragano, E. Benetos, and A. Hines, “Automatic quality assessment of digitized and restored sound archives,” Journal of the Audio Engineering Society, 2022.
  * [14] P. W. Sun and A. Hines, “Listening effort informed quality of experience evaluation,” Frontiers in Psychology, vol. 12, 2022.
  * [15] Z. Akhtar, K. Siddique, A. Rattani, S. L. Lutfi, and T. H. Falk, “Why is multimedia quality of experience assessment a challenging problem?,” IEEE Access, vol. 7, pp. 117897–117915, 2019.
  * [16] Q. M. Qadir, A. A. Kist, and Z. Zhang, “A novel traffic rate measurement algorithm for quality of experience-aware video admission control,” IEEE transactions on multimedia, vol. 17, no. 5, pp. 711–722, 2015.
  * [17] U. Reiter, K. Brunnström, K. De Moor, M.-C. Larabi, M. Pereira, A. Pinheiro, J. You, and A. Zgank, Factors Influencing Quality of Experience, pp. 55–72.  Cham: Springer International Publishing, 2014.
  * [18] S. Möller, W.-Y. Chan, N. Côté, T. H. Falk, A. Raake, and M. Wältermann, “Speech quality estimation: Models and trends,” IEEE Signal Processing Magazine, vol. 28, no. 6, pp. 18–28, 2011.
  * [19] A. Raake, Speech quality of VoIP: assessment and prediction.  John Wiley & Sons, 2007.
  * [20] R. Plomp, The intelligent ear: On the nature of sound perception.  Psychology Press, 2001.
  * [21] V. I. Levenshtein, “Binary codes capable of correcting deletions, insertions, and reversals,” in Soviet physics doklady, vol. 10, pp. 707–710, 1966.
  * [22] S. Lev-Ari, “Comprehending non-native speakers: Theory and evidence for adjustment in manner of processing,” Frontiers in psychology, vol. 5, p. 1546, 2015.
  * [23] V. Puhacheuskaya and J. Järvikivi, “I was being sarcastic!: The effect of foreign accent and political ideology on irony (mis) understanding,” Acta Psychologica, vol. 222, p. 103479, 2022.
  * [24] Netflix, “Netflix Sound Mix Specifications & Best Practices v1.4.” https://partnerhelp.netflixstudios.com/hc/en-us/articles/360001794307-Netflix-Sound-Mix-Specifications-Best-Practices-v1-4.  Accessed: 21 November 2023.
  * [25] Medium, “Detecting Speech and Music in Audio Content.” https://netflixtechblog.com/detecting-speech-and-music-in-audio-content-afd64e6a5bf8.  Accessed: 21 November 2023.
  * [26] The Guardian, “Mumbling actors, bad speakers or lazy listeners? Why everyone is watching TV with subtitles on.” https://www.theguardian.com/tv-and-radio/2023/jan/28/mumbling-actors-bad-speakers-or-lazy-listeners-why-everyone-is-watching-tv-with-subtitles-on.  Accessed: 21 November 2023.
  * [27] G. Mittag, B. Naderi, A. Chehadi, and S. Möller, “NISQA: A Deep CNN-Self-Attention Model for Multidimensional Speech Quality Prediction with Crowdsourced Datasets,” in Proc. Interspeech, pp. 2127–2131, 2021.
  * [28] J. Yuan, M. Liberman, and C. Cieri, “Towards an integrated understanding of speaking rate in conversation,” in Ninth International Conference on Spoken Language Processing, 2006.
  * [29] N. H. De Jong and T. Wempe, “Praat script to detect syllable nuclei and measure speech rate automatically,” Behavior Research Methods, vol. 41, pp. 385–390, May 2009.
  * [30] X. Fan and J. H. Hansen, “Acoustic analysis and feature transformation from neutral to whisper for speaker identification within whispered speech audio streams,” Speech communication, vol. 55, no. 1, pp. 119–134, 2013.
  * [31] R. Jahangir, Y. W. Teh, H. F. Nweke, G. Mujtaba, M. A. Al-Garadi, and I. Ali, “Speaker identification through artificial intelligence techniques: A comprehensive review and research challenges,” Expert Systems with Applications, vol. 171, p. 114591, 2021.
  * [32] Preply, “Survey: Why America is obsessed with subtitles.” https://preply.com/en/blog/americas-subtitles-use/.  Accessed: 21 November 2023.
  * [33] N. C. Nilsson, R. Nordahl, and S. Serafin, “Immersion revisited: A review of existing definitions of immersion and their relation to different theories of presence,” Human technology, vol. 12, no. 2, pp. 108–134, 2016.
  * [34] A. Perkis, C. Timmerer, S. Baraković, J. B. Husić, S. Bech, S. Bosse, J. Botev, K. Brunnström, L. Cruz, K. De Moor, et al., “Qualinet white paper on definitions of immersive media experience (imex),” arXiv preprint arXiv:2007.07032, 2020.
  * [35] Z. Yuan, G. Ghinea, and G.-M. Muntean, “Beyond multimedia adaptation: Quality of experience-aware multi-sensorial media delivery,” IEEE Transactions on Multimedia, vol. 17, no. 1, pp. 104–117, 2014.
  * [36] W. H. Nam, T. Lee, S. C. Ko, Y. Son, H. K. Chung, K.-R. Kim, J. Kim, S. Hwang, and K. Lee, “Ai 3d immersive audio codec based on content-adaptive dynamic down-mixing and up-mixing framework,” in Audio Engineering Society Convention 151, Audio Engineering Society, 2021.
  * [37] S. Dupont and J. Luettin, “Audio-visual speech modeling for continuous speech recognition,” IEEE transactions on multimedia, vol. 2, no. 3, pp. 141–151, 2000.
  * [38] Z. Akhtar and T. H. Falk, “Audio-visual multimedia quality assessment: A comprehensive survey,” IEEE Access, vol. 5, pp. 21090–21117, 2017.
  * [39] H. Becerra Martinez, A. Hines, and M. C. Q. Farias, “Perceptual quality of audio-visual content with common video and audio degradations,” Applied Sciences, vol. 11, no. 13, 2021.
  * [40] M. S. J. Lewis, “Television captioning: A vehicle for accessibility and literacy,” Idaho Assistive Technology Project. Disability Information Resources, 1999.
  * [41] C. S. Swartz, Understanding digital cinema: a professional handbook.  Taylor & Francis, 2005.
  * [42] S. Möller and A. Raake, Quality of experience: advanced concepts, applications and methods.  Springer, 2014.
  * [43] Midge Costin, Making Waves: The Art of Cinematic Sound [Film].  Ain’t Heard Nothin’ Yet, GoodMovies Entertainment, 2019.
  * [44] I. Union, “Itu-t recommendation p. 910: Subjective video quality assessmentmethods for multimedia applications,” 2008.
  * [45] Howard Overman, War of the Worlds [TV series].  Urban Myth Films, Fox Networks Group, StudioCanal, 2019.
  * [46] European Commission, “Audiovisual media services: breakthrough in EU negotiations for modern and fairer rules.” https://ec.europa.eu/commission/presscorner/detail/en/IP_18_3567.  Accessed: 21 November 2023.
  * [47] B. Limov, “Click it, binge it, get hooked: Netflix and the growing us audience for foreign content,” International Journal of Communication, vol. 14, p. 20, 2020.
  * [48] S. Sánchez-Mompeán, “Netflix likes it dubbed: Taking on the challenge of dubbing into english,” Language & Communication, vol. 80, pp. 180–190, 2021.
  * [49] J. Zuluaga-Gomez, S. Ahmed, D. Visockas, and C. Subakan, “CommonAccent: Exploring Large Acoustic Pretrained Models for Accent Classification Based on Common Voice,” in Proc. INTERSPEECH 2023, pp. 5291–5295, 2023.
  * [50] The Guardian, “Hard to pardon: why Tenet’s muffled dialogue is a very modern problem.” https://www.theguardian.com/film/2020/sep/03/tenet-dialogue-christopher-nolan-sound-technology.  Accessed: 21 November 2023.
  * [51] M. Torcoli, A. Freke-Morin, J. Paulus, C. Simon, and B. Shirley, “Background ducking to produce esthetically pleasing audio for tv with clear speech,” in Audio Engineering Society Convention 146, Audio Engineering Society, 2019.
  * [52] I. Rec, “J. 148 (2003). requirements for an objective perceptual multimedia quality model,” International Telecommunication Union, Genf.
  * [53] L. Chen, Y. Zhou, and D. M. Chiu, “Smart streaming for online video services,” IEEE transactions on multimedia, vol. 17, no. 4, pp. 485–497, 2015.
  * [54] C. Zhou, C.-W. Lin, and Z. Guo, “mdash: A markov decision-based rate adaptation approach for dynamic http streaming,” IEEE Transactions on Multimedia, vol. 18, no. 4, pp. 738–751, 2016.
  * [55] P. Falkowski-Gilski, “On the consumption of multimedia content using mobile devices: A year to year user case study,” Archives of Acoustics, vol. 45, no. 2, pp. 321–328, 2020.
  * [56] P. Falkowski-Gilski and T. Uhl, “Current trends in consumption of multimedia content using online streaming platforms: a user-centric survey,” Computer Science Review, vol. 37, p. 100268, 2020.
  * [57] A. Shokrpour and M. J. Darnell, “How people multitask while watching tv,” in Proceedings of the 2017 ACM international conference on interactive experiences for TV and online video, pp. 11–19, 2017.
  * [58] J. Aagaard, “Media multitasking, attention, and distraction: A critical discussion,” Phenomenology and the Cognitive Sciences, vol. 14, pp. 885–896, 2015.
  * [59] S. Van Kuyk, W. B. Kleijn, and R. C. Hendriks, “An evaluation of intrusive instrumental intelligibility metrics,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 26, no. 11, pp. 2153–2166, 2018.
  * [60] T. H. Falk, C. Zheng, and W.-Y. Chan, “A non-intrusive quality and intelligibility measure of reverberant and dereverberated speech,” IEEE Transactions on Audio, Speech, and Language Processing, vol. 18, no. 7, pp. 1766–1774, 2010.
  * [61] T. H. Falk, V. Parsa, J. F. Santos, K. Arehart, O. Hazrati, R. Huber, J. M. Kates, and S. Scollie, “Objective quality and intelligibility prediction for users of assistive listening devices: Advantages and limitations of existing tools,” IEEE signal processing magazine, vol. 32, no. 2, pp. 114–124, 2015.
  * [62] U. Kjems, J. B. Boldt, M. S. Pedersen, T. Lunner, and D. Wang, “Role of mask pattern in intelligibility of ideal binary-masked noisy speech,” The Journal of the Acoustical Society of America, vol. 126, no. 3, pp. 1415–1426, 2009.
  * [63] B. Shirley and R. Oldfield, “Clean audio for tv broadcast: an object-based approach for hearing impaired viewers,” Journal of the Audio Engineering Society, vol. 63, no. 4, 2015.
  * [64] G. Li, X. Wang, R. Hu, H. Zhang, and S. Ke, “Intelligibility enhancement via normal-to-lombard speech conversion with long short-term memory network and bayesian gaussian mixture model,” IEEE Transactions on Multimedia, vol. 23, pp. 3035–3047, 2021.
  * [65] L. D. Rosenblum, “Speech perception as a multimodal phenomenon,” Current directions in psychological science, vol. 17, no. 6, pp. 405–409, 2008.
  * [66] W. Kang, M. J. Alam, and A. Fathan, “Deep learning-based end-to-end spoken language identification system for domain-mismatched scenario,” in Proceedings of the Thirteenth Language Resources and Evaluation Conference, pp. 7339–7343, 2022.
  * [67] Y. Zheng, R. Sproat, L. Gu, I. Shafran, H. Zhou, Y. Su, D. J. R. Starr, and S.-Y. Yoon, “Accent detection and speech recognition for shanghai-accented mandarin,” in Interspeech Conference Proceedings 2005, 2005.
  * [68] J. Valk and T. Alumäe, “Voxlingua107: A dataset for spoken language recognition,” 2021 IEEE Spoken Language Technology Workshop (SLT), pp. 652–658, 2020.
  * [69] M. Chinen, J. Skoglund, C. K. A. Reddy, A. Ragano, and A. Hines, “Using Rater and System Metadata to Explain Variance in the VoiceMOS Challenge 2022 Dataset,” in Proc. Interspeech 2022, pp. 4531–4535, 2022.
  * [70] A. Master, L. Lu, J. Samuelsson, H.-M. Lehtonen, S. Norcross, N. Swedlow, and A. Howard, “Deepspace: Dynamic spatial and source cue based source separation for dialog enhancement,” in ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1–5, IEEE, 2023.
  * [71] J. Paulus, M. Torcoli, C. Uhle, J. Herre, S. Disch, and H. Fuchs, “Source separation for enabling dialogue enhancement in object-based broadcast with mpeg-h,” Journal of the Audio Engineering Society, vol. 67, pp. 510–521, 08 2019.
  * [72] Amazon Staff, “Prime Video launches a new accessibility feature that makes it easier to hear dialogue in your favorite movies and series.” https://www.aboutamazon.com/news/entertainment/prime-video-dialogue-boost.  Accessed: 21 November 2023.
  * [73] M. Crabb, R. Jones, M. Armstrong, and C. J. Hughes, “Online news videos: the ux of subtitle position,” in Proceedings of the 17th international ACM SIGACCESS conference on Computers & accessibility, pp. 215–222, 2015.
  * [74] A. Brown, R. Jones, M. Crabb, J. Sandford, M. Brooks, M. Armstrong, and C. Jay, “Dynamic subtitles: the user experience,” in Proceedings of the ACM international conference on interactive experiences for TV and online video, pp. 103–112, 2015.
  * [75] A. Hines, E. Gillen, D. Kelly, J. Skoglund, A. Kokaram, and N. Harte, “Visqolaudio: An objective audio quality metric for low bitrate codecs,” The Journal of the Acoustical Society of America, vol. 137, no. 6, pp. EL449–EL455, 2015.
  * [76] Y.-N. Hung, C.-W. Wu, I. Orife, A. Hipple, W. Wolcott, and A. Lerch, “A large tv dataset for speech and music activity detection,” EURASIP Journal on Audio, Speech, and Music Processing, vol. 2022, no. 1, p. 21, 2022.