# Exploring Holistic HMI Design for Automated Vehicles: Insights from a
Participatory Workshop to Bridge In-Vehicle and External Communication

Haoyu Dong<EMAIL_ADDRESS>Eindhoven University of
TechnologyEindhovenNetherlands 0000-0002-7164-7608 ,  Tram Thi Minh Tran
<EMAIL_ADDRESS>Design Lab, The University of SydneySydneyAustralia
0000-0002-4958-2465 ,  Rutger Verstegen<EMAIL_ADDRESS>Eindhoven
University of TechnologyEindhovenNetherlands 0000-0003-1934-2898 ,  Silvia
Cazacu<EMAIL_ADDRESS>KU LeuvenLeuvenBelgium
0000-0002-7952-0919 ,  Ruolin Gao<EMAIL_ADDRESS>Eindhoven University of
TechnologyEindhovenNetherlands 0009-0003-8898-5093 ,  Marius Hoggenmüller
<EMAIL_ADDRESS>Design Lab, The University of
SydneySydneyAustralia 0000-0002-8893-5729 ,  Debargha Dey
<EMAIL_ADDRESS>Information Sciences, Cornell Tech2 W Loop Rd, NY
10044New YorkUnited States 0000-0001-9266-0126 ,  Mervyn Franssen
<EMAIL_ADDRESS>Eindhoven University of TechnologyEindhovenNetherlands
0009-0009-3095-4829 ,  Markus Sasalovici<EMAIL_ADDRESS>Institute of Media Informatics, Ulm UniversityUlmGermany 0000-0001-9883-2398 ,
Pavlo Bazilinskyy<EMAIL_ADDRESS>Eindhoven University of
TechnologyEindhovenNetherlands 0000-0001-9565-8240 and  Marieke Martens
<EMAIL_ADDRESS>Eindhoven University of Technology EindhovenNetherlands
0000-0002-1661-7019

(2024)

###### Abstract.

Human–Machine Interfaces (HMIs) for automated vehicles (AVs) are typically
divided into two categories: internal HMIs for interactions within the
vehicle, and external HMIs for communication with other road users. In this
work, we examine the prospects of bridging these two seemingly distinct
domains. Through a participatory workshop with automotive user interface
researchers and practitioners, we facilitated a critical exploration of
holistic HMI design by having workshop participants collaboratively develop
interaction scenarios involving AVs, in-vehicle users, and external road
users. The discussion offers insights into the escalation of interface
elements as an HMI design strategy, the direct interactions between different
users, and an expanded understanding of holistic HMI design. This work
reflects a collaborative effort to understand the practical aspects of this
holistic design approach, offering new perspectives and encouraging further
investigation into this underexplored aspect of automotive user interfaces.

††journalyear: 2024††copyright: rightsretained††conference: Extended Abstracts
of the CHI Conference on Human Factors in Computing Systems; May 11–16, 2024;
Honolulu, HI, USA††booktitle: Extended Abstracts of the CHI Conference on
Human Factors in Computing Systems (CHI EA ’24), May 11–16, 2024, Honolulu,
HI, USA††doi: 10.1145/3613905.3651086††isbn: 979-8-4007-0331-7/24/05††ccs:
Human-centered computing Interaction design

##  1\. Introduction

The integration and acceptance of automated vehicles (AVs) into our
transportation systems hinges, amongst other things, upon their ability to
communicate effectively. This communication is crucial not only for the
occupants of the vehicle, such as drivers and passengers, but also for
external road users including pedestrians, cyclists, and drivers of manual
vehicles (Bengler et al., 2020; Hollander et al., 2021; Dey et al., 2020;
Tabone et al., 2021; Colley et al., 2022a; Avsar et al., 2021; Papakostopoulos
et al., 2021). In this context, extensive research has been conducted
regarding the design of human–machine interfaces (HMIs) for AVs, adopting a
reductionist approach (Blythe et al., 2007) that focuses either exclusively on
internal interfaces (iHMIs) or external interfaces (eHMIs).

Bridging this segregation, Bengler et al. (2020) previously proposed an HMI
framework for automated driving. This framework categorises HMIs based on
their orientation towards internal and external communication, aligning with
the standards outlined in ISO/TR 21959 (iso, 2019). Central to this framework
lies the emphasis on synchronisation and consistency across different types of
HMIs, advocating for a holistic HMI design approach to communication in AVs.
While this theoretical work has called for further research on the
coordination of internal and external communication, the limited research on
this approach raises questions: Is it due to a perceived lack of relevant use
cases, or are there inherent challenges in implementing a holistic HMI? This
gap in literature necessitates further investigation into the practical
implementation and its potential impacts in real-world scenarios.

To address this gap, we conducted a participatory workshop with twelve
researchers and practitioners in the field of automotive user interfaces. Our
objective was not to assume the necessity of such integration but to
facilitate an open and critical exploration of potential use cases and
scenarios involving holistic HMIs.

The workshop resulted in three distinct scenarios showcasing the potential
benefits of employing holistic HMI design. It is important to note, however,
that holistic HMIs are not positioned as universal solutions for all contexts.
The initial insights from our workshop suggest potential applications and
opportunities for enhancing user interactions with AVs through holistic HMIs,
and discuss notable challenges in this area.

This late-breaking work breaks new ground in the field of automotive HMI
design and research by showcasing the promise of holistic HMIs in certain
situations. We hope that this can act as a launching pad for discussions
around the strategy of taking holistic HMIs into account from the beginning of
the design process. This paves the way towards an actionable investigation of
an underexplored area of AV interaction.

##  2\. Participatory Workshop

A participatory workshop was held as part of an academic conference
[(conference name redacted for anonymity)]AutomotiveUI conference 2023, in
Ingolstadt, Germany (Dong et al., 2023). Twelve participants attended the
workshop, all of whom were researchers or practitioners in Human-Computer
Interaction (HCI) and human factors, or technology consultancy. They varied in
their experience, ranging from junior researchers/ PhD students, to
experienced professors or industry professionals. Their research focus lay
within automotive user interfaces (iHMI, eHMI, and/or general automotive human
factors), which were represented by coloured badges handed out upon arrival.

The workshop started with an introduction of the objectives, schedule, and
expected outcomes. Two invited keynote speakers, specialised in iHMIs and
eHMIs, then provided an overview of the state of the art in their respective
domains. This was followed by first round of plenary discussion, where
participants equally voiced their ideas, concerns, or visions, to form a
common understanding of holistic HMI design for AVs.

Figure 1. Overview of the workshop.

A diagram illustrating the step-by-step process followed during the workshop,
with a photo capturing the scenario of the workshop, depicting participants
and facilitators engaged in group activities at three tables.

Participants were then divided into three groups for the facilitated group
activity. Each group consisted of four members with mixed research focus,
based on the coloured badges. During the group activity, each group was tasked
to collaboratively develop one interaction scenario involving multiple traffic
participants, thus setting up use cases for holistic HMIs. The workshop
concluded with each group presenting their scenarios, followed by a final
plenary discussion that reflected on the holistic HMI design approach. The
workshop overview is shown in Figure 1.

###  2.1. Group Activity

To assist participants in creating scenarios in which perspectives of multiple
users are considered, we utilised the participatory workshop technique with a
set of toolkits including detailed instructions and physical tokens (Sanders
and Stappers, 2014). These instructions broke down the scenario into four key
components: Users, Vehicle, Environmental Setting, and Interaction. For
defining each of these components, we provided four guiding questions.

##### Users

Two types of users were considered: the in-vehicle user and the pedestrian.
The questions for defining each user were inspired by the Empathy Map (Dam and
Teo, 2023), a commonly used tool in design thinking:

  * •

Who are you? (e.g., age, gender, job)

  * •

What are you doing?

  * •

What are you perceiving/hearing/seeing/smelling?

  * •

What is your state of mind?

##### Vehicle

The questions that define the vehicle were designed to allow the participants
to freely explore, identify, and specify its properties and/or characteristics
by taking the perspective of the non-human traffic participants (Tomitsch et
al., 2021):

  * •

What type of vehicle are you? (e.g., passenger car, bus, truck)

  * •

How is your external appearance?

  * •

Explain how you can support and communicate with your internal users?

  * •

Explain how you can support and communicate with external road users?

##### Environmental Setting

The questions aimed at describing the environment were focused on defining the
spatial and temporal settings. We also included two main aspects (type of road
and weather) to define traffic scenarios based on (Fratini et al., 2023):

  * •

What is the day of the year or season?

  * •

What is the time of the day?

  * •

What is the location and type of road?

  * •

How is the weather at the moment?

##### Interaction

The questions aimed at defining the interactions encompass four key aspects:

  * •

How would the vehicle and the internal user interact, highlighting the
vehicle’s advanced features?

  * •

How would the vehicle and the external user interact, highlighting the
vehicle’s advanced features?

  * •

How could the vehicle enable an interaction between internal and external
users?

  * •

What could be a direct interaction between internal and external users?

Each group defined all four components, with the order: Users, Vehicle,
Environmental Setting, and Interaction. For each component, participants took
turns and each participant answered one question by writing down only keywords
on the token and briefly presenting their answers. All the tokens were laid
out on the table, facilitating an easier overview and rearrangement.

Then, each group was required to review all the tokens and their connections
to resolve conflicts (e.g., ‘winter’ and ‘heatwave’ as weather components of
the proposed scenario, which cannot coexist), thereby enabling the creation of
a consistent narrative of the scenario. Throughout the group activity, a
researcher was present as the facilitator in each group to provide guidance
and clarification on the instructions as needed, and the group collaboratively
came to a consensus regarding the final narrative of the scenario. A meta
facilitator assisted all three groups during the collaborative activity,
ensuring that they are in sync with each other and that they follow the
protocol.

###  2.2. Data Collection and Analysis

Photographs were taken of the three scenarios developed by the three groups.
Additionally, with the participants’ verbal consent, all group activities and
discussions were captured via video and audio recording. Following the
workshop, the facilitators of each group summarised the discussions and the
created scenarios. They accomplished this by reviewing and annotating the
audio recordings, a method influenced by the concept of ‘direct analysis’ in
qualitative research (Point and Baruch, 2023; Parameswaran et al., 2020). To
ensure the reliability of our data, a second reviewer–the facilitator from a
different group–was assigned to verify the annotations. Subsequently, the
scenarios developed were subjected to collaborative analysis and coding by the
authors, leading to the extraction of key insights.

##  3\. Results

This section details the scenarios and highlights key points of the group
discussions. We created sketches to visualise the created scenarios, focusing
on depicting the user and the environment (see Figure 2). The list of keywords
included in creating each scenario is in Appendix A.

Figure 2. Sketches depicting the scenarios: (Left) ‘Rainy Traffic Jam’
scenario, (Middle) ‘Snowy Mountain Road’ scenario, (Right) ‘Summer Night
Roundabout’ scenario.

Sketches depicting the scenarios draw by hand: (Left) ‘Rainy Traffic Jam’
scenario, (Middle) ‘Snowy Mountain Road’ scenario, (Right) ‘Summer Night
Roundabout’ scenario.

###  3.1. Scenario One: Rainy Traffic Jam

####  3.1.1. Scenario Description

The scenario unfolded in a busy city where a heavy rain caused a traffic jam.
Inside the AV, the in-vehicle user remained relaxed, enjoying a YouTube video
with the volume turned up. Meanwhile, a stressed pedestrian navigated through
the rain, while being engaged in a phone call (as illustrated in Figure 2
Left).

The AV displayed an eHMI icon with letters, signalling its still-standing
traffic jam status. As the pedestrian approached, the AV subtly highlighted
the pedestrian’s location with lights and non-intrusive audio cues to the in-
vehicle user. Similarly, the pedestrian perceived various LED lights and icons
through the AV’s external display.

There was no direct interaction between the in-vehicle user and the
pedestrian, as both were preoccupied with their respective activities.

####  3.1.2. Discussion Highlights

Recognising that both users were distracted from the actual situation engaging
in something else, group one contemplated the possibility of intensifying the
AV communication to attract both attention, for example, increasing the volume
of audio cues, or having the LED light blinking. The intensified HMI may
successfully draw both users’ attention, fostering a direct interaction
between the two, at most a shared glance acknowledging each other’s presence.
Later on, the group considered that the still-standing traffic jam situation
might eliminate the necessity of direct interaction and a direct interaction
between the internal and external users should be a fallback option in case
the AV can not handle a situation. In this scenario, with no breakdown in the
AV’s functionality, the absence of direct interaction was deemed acceptable,
with both individuals continuing their activities undisturbed.

###  3.2. Scenario Two: Snowy Mountain Road

####  3.2.1. Scenario Description

In this winter scenario, a mountain road was busy and treacherous due to
seasonal traffic and slippery conditions, posing potential hazards. An AV
carried an older in-vehicle user, who was absorbed in internet browsing. The
AV was equipped with a driver monitoring system, constantly assessing the
state of the in-vehicle user. A woman walked along the road and passed by the
AV, while listening to a podcast and being mindful of her safety in such
challenging conditions. Both users were less alert to their surrounding
environment (as illustrated in Figure 2 Middle).

In the event of danger, the AV employed a transformer-inspired mechanism to
alert the in-vehicle user, adapting the warning methods according to the
severity of the situation and the driver’s current state. This system
escalated its alerts from subtle visual signals to auditory warnings and seat
vibrations. For external communications, the pedestrian initially received
alerts on her smartwatch, functioning as a virtual assistant. If the initial
warning was ignored, the system would temporarily interrupt her podcast,
ensuring the pedestrian becomes fully attentive to her surroundings.

####  3.2.2. Discussion Highlights

In the interaction defining phase, Group Two considered that both the in-
vehicle user and the pedestrian were in a potentially hazardous situation.
This realisation influenced the group’s decision to focus on guaranteeing
safety, by employing uniformity in the escalation of both eHMI and iHMI to
ensure both parties are aware of the situation and receive relevant messages.
This led to a discussion about the potential of simultaneously mirroring
information to both parties facing the same danger. However, concerns were
also raised about the difference in implementation for iHMI and eHMI (e.g.,
monitoring systems). Finally, this group discussed the possibility of the
vehicle reflecting the internal user’s state and emotions, serving as a direct
communication channel between the in-vehicle user and the pedestrian.

###  3.3. Scenario Three: Summer Night Roundabout

####  3.3.1. Scenario Description

This scenario happened on a summer evening with a clear sky. An old woman
strolled on the street while watching YouTube videos on her phone. She
remained alert to the sounds around her yet not looking around. She approached
a roundabout where an autonomous shuttle, reserved for individual use,
approached. The shuttle carried a young man, who was eagerly anticipating a
date. He was immersed in the music playing from the shuttle’s speakers (as
illustrated in Figure 2 Right).

As the shuttle detected the woman, it subtly adjusted the music volume and
activated its virtual avatar to gently notify the in-vehicle user.
Simultaneously, the shuttle changed its exterior colour to yellow, in an
attempt to alert the woman. However, the woman, absorbed in the Youtube video,
remained oblivious. Then, the shuttle extended its outreach beyond its
external interface by sending a message to the woman’s phone. It also lowered
its window, allowing the young man to speak directly to her. In adverse
weather conditions, this interaction could alternatively occur virtually, with
the window remaining closed and the conversation broadcast externally through
a speaker inspired by the Tesla Model 3
Boombox111https://www.tesla.com/ownersmanual/model3/en_us/GUID-79A49D40-A028-435B-A7F6-8E48846AB9E9.html.

####  3.3.2. Discussion Highlights

Group Three employed more gentle communication means (e.g., lowering the
music, changing the exterior colour) and more noticeable measures (e.g., a
talking avatar, text message). They also carefully considered the state and
current activities of both users to suggest suitable communication methods in
this scenario. The discussion focused on the circumstances surrounding both
users—the tranquil evening during which the encounter took place. This context
would allow for a more personal form of communication. The positive and non-
aggressive states of both individuals also influenced the decision for this
interaction. As a result, the final interaction entailed the vehicle opening
its window, enabling direct communication between the two users. This scenario
illustrates a harmonious interaction facilitated by technology, demonstrating
the potential for direct communication between internal and external users in
shared spaces.

##  4\. Discussion

In this section, we discuss key similarities and differences among the three
scenarios, reflecting on the holistic HMI design approach utilised in
developing the scenarios.

###  4.1. HMI Escalation as a Shared Design Strategy

Regarding the environmental setting, two out of three scenarios involved
traffic jams, which is an atypical scenario given that the majority of
existing literature on AV communication focuses on fast-paced, high-risk
situations (Wilbrink et al., 2017; Tran et al., 2021; Cauffman et al., 2022).
In these slow-moving traffic situations, the kinematic cues of the AV
(referred to by Bengler et al. (2020) as dynamic HMIs) become harder to
observe, potentially necessitating the use of more explicit types of HMIs.
Additionally, two out of the three scenarios involved special weather
conditions that posed potential hazards, including a snowy mountain road that
was narrow and slippery, and a rainy situation that affected the AV sensor
performance.

In all scenarios, both the in-vehicle user and the pedestrian were occupied
with their own activities, predominantly consuming media. These scenarios
mirror real-world situations of distracted pedestrians who use their phones
while walking (Holländer et al., 2020). For in-vehicle users, the rise of
automated driving systems (ADS) increasingly allows them to engage in non-
driving related tasks.

The influence of environmental settings and the distracted state of the
involved users prompted all groups to arrive at a similar strategy of
escalation of selective user interface elements in response to non-action of
users. Here, escalation refers to the process of progressively increasing or
intensifying the level of interaction between the AV and the human users. This
escalation is designed to ensure effective communication and response,
especially in critical or complex scenarios. Therefore, it was deemed most
relevant in scenarios which carried the highest potential risks (e.g., the
‘Snowy Mountain Road’ in our case).

In implementing this strategy, all the groups considered various modalities
and technologies, in some cases, leveraged devices that are sources of
engagement or distraction. For example, in the ‘Summer Night Roundabout’
scenario, the shuttle sends a message to the woman’s phone and lowers the
music inside the shuttle. Moreover, we observed the potential for all related
technologies within a traffic scenario to be interconnected, facilitating
easier dissemination and optimisation of information delivery. For instance,
pedestrians could receive notifications on their own devices. With the
increase in connected devices and the development of novel systems connecting
vehicles (i.e., Vehicle-to-Everything or V2X) (Harding et al., 2014; Lu et
al., 2014; Gunther et al., 2015), the integration of different types of HMIs
for a holistic AV communication approach is clearly feasible.

Despite HMI escalation being a shared design strategy that applies to both
internal and external communication, the implementation for iHMIs and eHMIs
could be different. For example, considerations such as the availability and
privacy consent related to monitoring systems, both internally and externally,
were discussed by Group Two and are evident in existing literature (Dey et
al., 2020). In particular, an iHMI can often follow a more standardised
approach since it deals primarily with the functionality of the car itself,
which tends to be more universal. In contrast, eHMIs interact with a broader
environment and various road users. This interaction requires a deeper
understanding of local customs and non-verbal communication cues (Weber et
al., 2019; Ranasinghe et al., 2020). As a result, while HMI escalation can be
applied to both iHMIs and eHMIs, it may not necessarily have to occur
simultaneously or in the same manner for both. This strategy represents a
relatively unexplored area that offers significant opportunities for advancing
HMI research.

###  4.2. Interaction Between In-Vehicle User and Pedestrian

Interaction between internal and external users, either directly or mediated
by the AV, is scarcely considered in the design of HMIs in the context of
automated driving. The design and research on iHMI often focus on either input
or output channels between the in-vehicle user and the AV, through interfaces
with various modalities (Detjen et al., 2021). Meanwhile, eHMI research
typically concerns fully autonomous vehicles without any occupants inside (SAE
Level 5 (SAE, 2021)). However, insightful findings do exist, such as potential
conflicts arising from opposing cues given by drivers or passengers and the
eHMI (Colley et al., 2022b).

In the workshop, we found varying degrees of interaction between the in-
vehicle user and the pedestrian being discussed across the three scenarios.
The range of interaction varies from no interaction needed (or at most a
shared glance) in ‘Rainy Traffic Jam’, to mediated interaction (vehicle
expressing the driver’s emotion) in ‘Snowy Mountain Road’, to a direct
interaction (conversation between the shuttle passengers and pedestrians) in
‘Summer Night’. Regarding the AV expressing the driver’s emotions, this aspect
echoes with an eHMI dimension referred to as Vehicle Occupant State by Dey et
al. (2020), which captures whether the eHMI enables the vehicle to communicate
the state of its occupants to external users (e.g., ‘angst’). Besides, the
direct interaction was not due to a failure of AV communication, as in a study
by Brown et al. (2023) where the passenger had to apologise for the AV
behaviour, saying ‘Sorry, it’s a self-driving car.’ Instead, the direct
interaction was facilitated by the shuttle lowering its window and adding
another layer of interaction, which might aid safety and efficiency.

###  4.3. Towards an Expanded Understanding of Holistic HMI Design

Bengler et al. (2020) refers to a holistic HMI communication approach as
‘considering all HMI types when researching the interaction strategies of AVs
with its passenger or surrounding human road users’. Findings from our
workshop contribute to a more expanded understanding of holistic HMI design.
The holistic perspective could imply either a singular design for all users,
or an integration of various HMI designs into a cohesive set of interactions.
First, it may involve a shared design strategy that could be applicable for
both iHMIs and eHMIs, facilitating information mirroring and unified
interaction strategy (e.g., HMI escalation) for consistent communication among
all involved parties. Second, it also encourages a design process that
considers both internal and external users within the same interaction
scenario, fostering the integration of multiple designs.

Contrary to the traditional separation of iHMI and eHMI under a Design-as-
Engineering approach (Wright et al., 2006) or a reductionist approach (Blythe
et al., 2007), this holistic perspective aligns with HCI’s evolving focus from
usability to experience-focused design (Wright et al., 2006; Bannon, 2011;
Karapanos, 2013). This shift acknowledges that experience design not only
involves the designed system, but also considers user’s internal states and
the context in which interactions occur (Hassenzahl, 2008, 2010).

In the context of AVs, this holistic approach, which serves as a bridge
between iHMI and eHMI, underscores the importance of integrating both the AV
and internal and external users into the same setup. This integration is vital
for creating cohesive user experiences and emphasising how AVs mediate and
alter human users’ activities and perceptions in daily life. We posit that the
design space of HMI for AV shows the potential of expanding to the design of
an ‘interspace’ (proposed by Winograd (1997)) inhabited by multiple people and
AVs, in a traffic environment with complex interactions. This view also aligns
with research focus on scalability in HMI design for AV (Dey et al., 2020;
Tran et al., 2023).

Furthermore, the holistic approach acknowledges the intricate interconnections
among various factors that shape user experience, without sacrificing
complexity for easy measurements of the impact of individual HMI elements
(Wright and Blythe, 2007; Karapanos, 2013; Ackoff, 1973; Blythe et al., 2007).
This perspective underscores the importance of a coherent design language
capable of accommodating the dynamic roles individuals assume in diverse
traffic environments. For instance, users may seamlessly transition between
roles as pedestrians, passengers, or drivers in their daily life, experiencing
either iHMI or eHMI at different time points. This necessitates the
implementation of adaptable interfaces.

##  5\. Limitations and Future Work

Despite effort to mix participants with diverse backgrounds in the group
activities, noticeable similarities emerged in the scenarios developed by all
three groups. This observation raises the possibility of a convergence in
thought process or a general agreement in the research community when
approaching HMI design for AVs. This shared bias could indicate either a
widespread tendency in the domain of automotive HMI, or could be attributed to
the design of the group activities. Hence, while not the primary focus of this
paper, it is crucial to contemplate the methodology’s potential impact on the
final outcomes. Subsequent work will provide a more comprehensive examination
of the methodology, offering detailed insights into the design process of the
group activity and the participatory workshop toolkit. Additionally, given the
exploratory nature of this workshop, the scenarios were constrained to include
only one pedestrian, one in-vehicle user, and one vehicle. Future efforts
should extend to incorporate multiple users, offering a more comprehensive
perspective that mirrors the intricate and diverse nature of real-world
traffic situations.

By showcasing three scenarios developed during the workshop, the early
insights highlight the potential benefits of holistic HMI design, indicating
its positive impact on shaping interactions with AVs and elevating user
experiences in specific scenarios. The findings underscore the viability of
such an approach, highlighting the need for a comprehensive exploration of
scenarios and use cases where holistic HMI approaches could offer significant
value in automotive HMI design. Work is underway to elaborate on the scenarios
and identify opportunities and challenges within the design space of holistic
HMI. This involves multiple brainstorming sessions, and future co-creating
workshops with a wider range of specialists to identify such scenarios. We
posit that such an exhaustive exploration of applicable scenarios also
promises a deeper understanding of the holistic HMI design approach.
Furthermore, we plan to conduct interviews with experts in the field to gain
insights into the multifaceted definition and refine the framework of holistic
HMI design approach. By shedding light on potential limitations and
challenges, we contribute to future implementations and unlock its full
potential in shaping the future of human–vehicle interaction.

##  6\. Conclusion

This paper presents three scenarios created at a workshop implementing
holistic HMI design approach to bridge internal and external communication in
AVs. The initial insights suggest the potential of such an approach in
enriching interactions with AV and enhancing user experience in specific
contexts. Concerns are also raised, highlighting that this is a complex topic,
encompassing both promises and challenges—thereby necessitating further
exploration. Our findings contribute to an expanded understanding of holistic
HMI design approach, emphasising a design process early on focusing on the
intricate dynamics of the ‘interspace’ where interactions unfold among
multiple participants, including in-vehicle users, pedestrians, and AVs. By
sharing these preliminary findings within the HCI community, our goal is to
catalyse meaningful discussions on the applications of holistic HMI design
approach. This serves as a foundation for actionable plans in future work
within the relatively under-explored area of human–vehicle interaction.

###### Acknowledgements.

We would like to thank all the workshop participants for their enthusiastic
and original work, and all the contributors to make this workshop happen. This
research is supported by China Scholarship Council award (No. 202007720018),
the Australian Research Council (ARC) Discovery Project DP200102604 (Trust and
Safety in Autonomous Mobility Systems: A Human-centred Approach), the Dutch
Research Council NWO-NWA Grant (No. NWA.1292.19.298), project ODECO (which has
received funding from the European Union’s Horizon 2020 research and
innovation programme under the Marie Skłodowska-Curie grant agreement No.
955569), China Scholarship Council award (No. 202006030027).

## References

  * (1)
  * iso (2019) 2019\.  _Road Vehicles: Human Performance and State in the Context of Automated Driving: Part 2–Considerations in Designing Experiments to Investigate Transition Processes_.  ISO/PRF TR 21959-2. International Organization for Standardization, Geneva, Switzerland.
  * Ackoff (1973) Russell L. Ackoff. 1973.  Science in the Systems Age: Beyond IE, OR, and MS.  _Oper. Res._ 21 (1973), 661–671.
  * Avsar et al. (2021) Hüseyin Avsar, Fabian Utesch, Marc Wilbrink, Michael Oehl, and Caroline Schießl. 2021.  Efficient communication of automated vehicles and manually driven vehicles through an external Human-Machine Interface (eHMI): Evaluation at T-junctions. In _HCI International 2021-Posters: 23rd HCI International Conference, HCII 2021, Virtual Event, July 24–29, 2021, Proceedings, Part III 23_. Springer, 224–232.
  * Bannon (2011) Liam Bannon. 2011.  Reimagining HCI: Toward a More Human-Centered Perspective.  (2011).  https://doi.org/10.1145/1978822.1978833
  * Bengler et al. (2020) Klaus Bengler, Michael Rettenmaier, Nicole Fritz, and Alexander Feierle. 2020.  From HMI to HMIs: Towards an HMI framework for automated driving.  _Information (Switzerland)_ 11 (2 2020).  Issue 2. https://doi.org/10.3390/info11020061
  * Blythe et al. (2007) Mark Blythe, Marc Hassenzahl, Effie Law, and Arnold Vermeeren. 2007.  An analysis framework for user experience (UX)studies: a green paper.
  * Brown et al. (2023) Barry Brown, Mathias Broth, and Erik Vinkhuyzen. 2023.  The Halting Problem: Video Analysis of Self-Driving Cars in Traffic. In _Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems_ (Hamburg, Germany) _(CHI ’23)_. Association for Computing Machinery, New York, NY, USA, Article 12, 14 pages.  https://doi.org/10.1145/3544548.3581045
  * Cauffman et al. (2022) Stephen J. Cauffman, Mei Lau, Yulin Deng, Christopher Cunningham, David B. Kaber, and Jing Feng. 2022.  Research and Design Considerations for Presentation of Non-Safety Related Information via In-Vehicle Displays during Automated Driving.  _Applied Sciences (Switzerland)_ 12, 20 (10 2022).  https://doi.org/10.3390/app122010538
  * Colley et al. (2022a) Mark Colley, Tim Fabian, and Enrico Rukzio. 2022a.  Investigating the Effects of External Communication and Automation Behavior on Manual Drivers at Intersections.  _Proc. ACM Hum.-Comput. Interact_ 6 (2022), 1–16.
  * Colley et al. (2022b) Mark Colley, Bastian Wankmüller, Tim Mend, Thomas Väth, Enrico Rukzio, and Jan Gugenheimer. 2022b.  User gesticulation inside an automated vehicle with external communication can cause confusion in pedestrians and a lower willingness to cross.  _Transportation Research Part F: Traffic Psychology and Behaviour_ 87 (2022), 120–137.  https://doi.org/10.1016/j.trf.2022.03.011
  * Dam and Teo (2023) R. F. Dam and Y. S. Teo. 2023.  Empathy Map – Why and How to Use It.  _Interaction Design Foundation - IxDF_ (2023).  https://www.interaction-design.org/literature/article/empathy-map-why-and-how-to-use-it
  * Detjen et al. (2021) Henrik Detjen, Sarah Faltaous, Bastian Pfleging, Stefan Geisler, and Stefan Schneegass. 2021.  How to Increase Automated Vehicles’ Acceptance through In-Vehicle Interaction Design: A Review.  _International Journal of Human-Computer Interaction_ 37, 4 (2021), 308–330.  https://doi.org/10.1080/10447318.2020.1860517
  * Dey et al. (2020) Debargha Dey, Azra Habibovic, Andreas Löcken, Philipp Wintersberger, Bastian Pfleging, Andreas Riener, Marieke Martens, and Jacques Terken. 2020.  Taming the eHMI jungle: A classification taxonomy to guide, compare, and assess the design principles of automated vehicles’ external human-machine interfaces.  _Transportation Research Interdisciplinary Perspectives_ 7 (2020), 100174.
  * Dong et al. (2023) Haoyu Dong, Tram Thi Minh Tran, Pavlo Bazilinskyy, Marius Hoggenmüller, Debargha Dey, Silvia Cazacu, Mervyn Franssen, and Ruolin Gao. 2023.  Holistic HMI Design for Automated Vehicles: Bridging In-Vehicle and External Communication. In _Adjunct Proceedings of the 15th International Conference on Automotive User Interfaces and Interactive Vehicular Applications_ (Ingolstadt, Germany) _(AutomotiveUI ’23 Adjunct)_. Association for Computing Machinery, New York, NY, USA, 237–240.  https://doi.org/10.1145/3581961.3609837
  * Fratini et al. (2023) Elena Fratini, Ruth Welsh, and Pete Thomas. 2023.  Ranking Crossing Scenario Complexity for eHMIs Testing: A Virtual Reality Study.  _Multimodal Technologies and Interaction_ 7, 2 (2023).  https://doi.org/10.3390/mti7020016
  * Gunther et al. (2015) Hendrik-jorn Gunther, Oliver Trauer, and Lars Wolf. 2015.  The potential of collective perception in vehicular ad-hoc networks. In _2015 14th International Conference on ITS Telecommunications (ITST)_. IEEE, 1–5.
  * Harding et al. (2014) John Harding, Gregory Powell, Rebecca Yoon, Joshua Fikentscher, Charlene Doyle, Dana Sade, Mike Lukuc, Jim Simons, Jing Wang, et al. 2014\.  _Vehicle-to-vehicle communications: readiness of V2V technology for application._ Technical Report. United States. National Highway Traffic Safety Administration.
  * Hassenzahl (2008) Marc Hassenzahl. 2008.  User Experience (UX): Towards an experiential perspective on product quality.  _IHM 2008_.
  * Hassenzahl (2010) Marc Hassenzahl. 2010.  Experience Design: Technology for All the Right Reasons.  _Synthesis Lectures on Human-Centered Informatics_ 3 (1 2010), 1–95.  Issue 1. https://doi.org/10.2200/s00261ed1v01y201003hci008
  * Hollander et al. (2021) Kai Hollander, Mark Colley, Enrico Rukzio, and Andreas Butz. 2021.  A taxonomy of vulnerable road users for hci based on a systematic literature review. In _Conference on Human Factors in Computing Systems - Proceedings_.  https://doi.org/10.1145/3411764.3445480
  * Holländer et al. (2020) Kai Holländer, Andy Krüger, and Andreas Butz. 2020.  Save the Smombies: App-Assisted Street Crossing. In _22nd International Conference on Human-Computer Interaction with Mobile Devices and Services_ (Oldenburg, Germany) _(MobileHCI ’20)_. Association for Computing Machinery, New York, NY, USA, Article 22, 11 pages.  https://doi.org/10.1145/3379503.3403547
  * Karapanos (2013) Evangelos Karapanos. 2013.  _Modeling Users’ Experiences with Interactive Systems_.  https://doi.org/10.1007/978-3-642-31000-3
  * Lu et al. (2014) Ning Lu, Nan Cheng, Ning Zhang, Xuemin Shen, and Jon W Mark. 2014.  Connected vehicles: Solutions and challenges.  _IEEE internet of things journal_ 1, 4 (2014), 289–299.
  * Papakostopoulos et al. (2021) Vassilis Papakostopoulos, Dimitris Nathanael, Evangelia Portouli, and Angelos Amditis. 2021.  Effect of external HMI for automated vehicles (AVs) on drivers’ ability to infer the AV motion intention: A field experiment.  _Transportation research part F: traffic psychology and behaviour_ 82 (2021), 32–42.
  * Parameswaran et al. (2020) Uma D Parameswaran, Jade L Ozawa-Kirk, and Gwen Latendresse. 2020.  To live (code) or to not: A new method for coding in qualitative research.  _Qualitative Social Work_ 19, 4 (2020), 630–644.  https://doi.org/10.1177/1473325019840394 arXiv:https://doi.org/10.1177/1473325019840394
  * Point and Baruch (2023) Sébastien Point and Yehuda Baruch. 2023.  (Re)thinking transcription strategies: Current challenges and future research directions.  _Scandinavian Journal of Management_ 39, 2 (2023), 101272.  https://doi.org/10.1016/j.scaman.2023.101272
  * Ranasinghe et al. (2020) Champika Ranasinghe, Kai Holländer, Rebecca Currano, David Sirkin, Dylan Moore, Stefan Schneegass, and Wendy Ju. 2020.  Autonomous Vehicle-Pedestrian Interaction Across Cultures: Towards Designing Better External Human Machine Interfaces (EHMIs). In _Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems_ (Honolulu, HI, USA) _(Chi Ea ’20)_. Association for Computing Machinery, New York, NY, USA, 1–8.  https://doi.org/10.1145/3334480.3382957
  * SAE (2021) SAE. 2021.  Taxonomy and definitions for terms related to driving automation systems for on-road motor vehicles.  _J3016, SAE International_ (2021).  https://doi.org/10.4271/J3016_202104
  * Sanders and Stappers (2014) Elizabeth B.-N. Sanders and Pieter Jan Stappers. 2014.  Probes, toolkits and prototypes: three approaches to making in codesigning.  _CoDesign_ 10 (2014), 14 – 5.  https://api.semanticscholar.org/CorpusID:108955372
  * Tabone et al. (2021) Wilbert Tabone, Joost De Winter, Claudia Ackermann, Jonas Bärgman, Martin Baumann, Shuchisnigdha Deb, Colleen Emmenegger, Azra Habibovic, Marjan Hagenzieker, Peter A Hancock, et al. 2021\.  Vulnerable road users and the coming wave of automated vehicles: Expert perspectives.  _Transportation research interdisciplinary perspectives_ 9 (2021), 100293.
  * Tomitsch et al. (2021) Martin Tomitsch, Joel Fredericks, Dan Vo, Jessica Frawley, and Marcus Foth. 2021.  Non-human Personas: Including Nature in the Participatory Design of Smart Cities.  _Interaction Design and Architecture(s)_ (12 2021), 102–130.  Issue 50. https://doi.org/10.55612/s-5002-050-006
  * Tran et al. (2021) Tram Thi Minh Tran, Callum Parker, and Martin Tomitsch. 2021.  A Review of Virtual Reality Studies on Autonomous Vehicle–Pedestrian Interaction.  _IEEE Transactions on Human-Machine Systems_ 51, 6 (2021), 641–652.  https://doi.org/10.1109/thms.2021.3107517
  * Tran et al. (2023) Tram Thi Minh Tran, Callum Parker, and Martin Tomitsch. 2023.  Scoping Out the Scalability Issues of Autonomous Vehicle-Pedestrian Interaction. In _Proceedings of the 15th International Conference on Automotive User Interfaces and Interactive Vehicular Applications_ (Ingolstadt, Germany) _(AutomotiveUI ’23)_. Association for Computing Machinery, New York, NY, USA, 167–177.  https://doi.org/10.1145/3580585.3607167
  * Weber et al. (2019) Florian Weber, Ronee Chadowitz, Kathrin Schmidt, Julia Messerschmidt, and Tanja Fuest. 2019.  Crossing the Street Across the Globe: A Study on the Effects of eHMI on Pedestrians in the US, Germany and China. In _International Conference on Human-Computer Interaction_. Springer, 515–530.
  * Wilbrink et al. (2017) Marc Wilbrink et al. 2017\.  D1.1 Definition of interACT use cases and scenarios.  https://elib.dlr.de/116445
  * Winograd (1997) Terry Winograd. 1997.  _The design of interaction_.  Copernicus, USA, 149–161.
  * Wright and Blythe (2007) Peter Wright and Mark Blythe. 2007.  User Experience Research as an Inter-discipline: Towards a UX Manifesto, E.Law, A. Vermeeren, M. Hassenzahl, and M. Blythe (Eds.).  _Towards a UX Manifesto - Proceedings of a cost294-affiliated workshop on BHCI 2007_ , 65–70.
  * Wright et al. (2006) Peter Wright, Mark Blythe, and John McCarthy. 2006.  User experience and the idea of design in HCI.  _Lecture Notes in Computer Science_ 3941 LNCS, 1–14.  https://doi.org/10.1007/11752707_1

##  Appendix A Set of Keywords

Figure 3. Set of keywords for ‘Rainy Traffic Jam’ scenario (Group One)

A figure showing the set of keywords for ‘Rainy Traffic Jam’ scenario (Group
One): The first column ‘Setting’ includes two sticky notes: ‘Big city (traffic
jam)’ and ‘Heavy rain, sensors are not working well’. The second column
‘Pedestrian’ includes two sticky notes: ‘Using my smartphone (listening to
music)’ and ‘Feeling stressed, engaged in phone calls’. The third column ‘In-
vehicle user’ includes three sticky notes: ‘Watching YouTube in the vehicle’,
‘Listening to loud music via smartphone app’ and ‘Feeling relaxed and focused
in a fresh interior environment’. The fourth column ‘Vehicle’ includes
descriptors such as ‘Unintrusive audio cues and ambient light’ and ‘Icons with
letters’. The fifth column ‘Interactions’ includes: ‘Highlighting pedestrians
with light and audio’, ‘Re-triggering internal user actions due to failed
visible cues’, ‘Utilizing various lights and movement cues’, ‘Enhancing
existing features for clarity’, ‘Providing intuitive messages on the
windshield’, ‘No need for interaction, ensuring no interruption to users’.

Figure 4. Set of keywords for ‘Snowy Mountain Road’ scenario (Group Two)

A figure showing set of keywords for ‘Snowy Mountain Road’ scenario (Group
Two): The first column ‘Setting’ includes three sticky notes: ‘Winter’, ‘1:00
pm’ and ‘Curved mountain road’. The second column ‘Pedestrian’ includes four
sticky notes: ‘Woman’, ‘Podcast listening’, ‘(Strike-through: Urban) traffic
jam’, and ‘Safe?’. The third column ‘In-vehicle user’ includes four sticky
notes: ‘Elderly’,‘Internet surfing’,‘A hazard situation, e.g. animal’,
‘Distracted’. The fourth column ‘Vehicle’ includes three sticky notes:
‘Transformer AV’, ‘Levels ¡¿ Modality escalating’, ‘eHMI’. The fifth column
‘Interactions’ includes three notes: ‘Internal HMI ¡¿ Driver
monitoring’,‘Message understood ¿ modality intensity’, ‘Virtual assistance’.

Figure 5. Set of keywords for ‘Summer Night Roundabout’ scenario (Group Three)

A figure showing set of keywords for ‘Summer Night Roundabout’ scenario (Group
Three): The first column ‘Setting’ includes four sticky notes: ‘2023, today,
sunny’, ‘Almost midnight’, ‘Roundabout’, ‘Clear sky’. The second column
‘Pedestrian’ includes four sticky notes: ‘Woman 80 years old’,‘Watching
Youtube via phone’,‘Focused on the phone but can hear the environment’, ‘Happy
because of the weather’. The third column ‘In-vehicle user’ includes three
sticky notes: ‘20 years old man, university student’,‘Listening to music’,
‘Excited for a date’. The fourth column ‘Vehicle’ includes four sticky notes:
‘Shuttle/Individual use’, ‘Futuristic’, ‘Sound, visual, avatar’, ‘Change
vehicle colour’. Lastly, the fifth column ‘Interactions’ includes eight notes:
‘Change the volume of music (lower)’, ‘Avatar alert verbally’, ‘Change the
colour to yellow’, ‘Send a message’, ‘Look at each other’, ‘Optical/tint mode
+ Loud mic/speaker, pedestrian comm’, ‘Open the window’,‘Passenger talks to
pedestrians to clear the crosswalk’.