# Stringesthesia: Dynamically Shifting Musical Agency Between Audience and
Performer Based on Trust in an Interactive and Improvised Performance

Torin Hopkins*‡, Emily Doherty‡†, Netta Ofer*, Suibi Che Chuan Weng*, Peter
Gyory*, Chad Tobin*, Leanne Hirshfield‡†, Ellen Yi-Luen Do  ATLAS Institute*,
Institute of Cognitive Science‡, Computer Science†University of Colorado
BoulderBoulder, CO, USA<EMAIL_ADDRESS>
(2023)

###### Abstract.

This paper introduces Stringesthesia, an interactive and improvised
performance paradigm. Stringesthesia uses real-time neuroimaging to connect
performers and audiences, enabling direct access to the performer’s mental
state and determining audience participation during the performance.
Functional near-infrared spectroscopy (fNIRS), a noninvasive neuroimaging
tool, was used to assess metabolic activity of brain areas collectively
associated with a metric we call “trust”. A visualization representing the
real-time measurement of the performer’s level of trust was projected behind
the performer and used to dynamically restrict or promote audience
participation: e.g., as the performer’s trust in the audience grew, more
participatory stations for playing drums and selecting the performer’s chords
were activated. Throughout the paper we discuss prior work that heavily
influenced our design, conceptual and methodological issues with using fNIRS
technology, and our system architecture. We then describe feedback from the
audience and performer in a performance setting with a solo guitar player.

Performance paradigms, Neuroimaging, fNIRS, trust, musical agency,
improvisation

††copyright: acmcopyright††journalyear: 2023††doi:
10.1145/3616195.3616209††conference: Audio Mostly; August 30 – September 01,
2023; Edinburgh, UK††price: 15.00††isbn: 979-8-4007-0818-3††ccs: Applied
computing Performing arts††ccs: Applied computing Sound and music computing

##  1\. Introduction

Trust is a crucial component of promoting performer-audience engagement, as
both parties must express some level of confidence to carry out a successful
performance (McCormack et al., 2020). While the interplay of trust has been
explored in some interactive musical performances (McCormack et al., 2019,
2020), much of the relevant trust literature comes from human-automation
interaction studies (Eloy et al., 2022; Bobko et al., 2022). These studies
evaluate trust often through the manipulation of an artificial agent’s
characteristics such as transparency and reliability. In particular, when
agents provide unreliable or incomplete information, their human collaborators
are likely to ignore the information and rely solely on their own experience,
resulting in a lack of trust in the agent. Conversely, when agents provide
information that is overly reliable, humans may become complacent and over-
rely on the agent, resulting in a lack of vigilance and degraded performance
(Hoff and Bashir, 2015; Glikson and Woolley, 2020; Eloy et al., 2022).
Communication or lack thereof between team members can reveal the level of
trust, which can greatly impact the effectiveness of collaboration.

Similarly, in audience-interactive musical improvisation, trust is built
through communication between musicians and audience members, as well as
openness to the creative process (Eaton et al., 2014; Leslie et al., 2014).
However, establishing trust can be a delicate balance. Unreliable (e.g. non-
interpretable rhythms) or overly reliable (e.g. static and unchanging rhythms)
musical performance can negatively impact the outcome of the musical
collaboration (McCormack et al., 2019, 2020).

To explore trust and interactivity in a musical performance setting, some
musicians have included the audience in interactive and improvised performance
paradigms, where the audience contributes music notation or influences the
direction of the music with semiotic information (Lee and Freeman, 2013a; Lee
et al., 2014; Xambó and Goudarzi, 2022; Xambó and Roma, 2020; Fan and Sciotto,
2013; Lee and Freeman, 2013b; Hindle, 2013; Weitzner et al., 2012). For
example, an audience could control the speed or volume of an improvising
musician’s music based on physical controllers positioned in the audience.
However, there is a trade-off between how much musical agency, defined as
control over the musical content, to give the audience vs. the performing
musician/s. Determining musical agency between audience and performer
significantly influences the outcome of the musical performance in interactive
and collaborative performance paradigms (Mudd, 2019; Brown and Paine, 2019;
Hein and Srinivasan, 2019).

Bio-sensing techniques have been used to enhance collaboration through
transparency in both human-AI interaction studies and interactive musical
performance. For example, bio-sensing has been used to display an autonomous
driving system’s level of certainty, which can inform the driver to take over
control before a collision event (Kunze et al., 2019). In musical performance,
bio-sensing has been used to approximate trust and inform performers of an AI-
musician’s confidence in their musical output (McCormack et al., 2019).
Additionally, bio-sensing techniques have been used to involve the audience in
the performance, creating a unique sense of intimacy and engagement between
the performer and the audience (Fan and Sciotto, 2013; Mullen et al., 2015;
Eaton et al., 2014; Leslie et al., 2014).

A commonly used technique to analyze neural signals in performances is
electroencephalography (EEG) (Lucier and Simon, 2012; Fan and Sciotto, 2013;
Leslie et al., 2014; Mullen et al., 2015). However, fNIRS has greater spatial
resolution than EEG, offering more information about the localization of
neural activity associated with cognitive states like trust (Perello-March et
al., 2022). The benefits of fNIRS have also been underexplored in creative
musical settings and further exploration will uncover more about its benefits
and drawbacks (Yuksel et al., 2019).

In this paper, we present Stringesthesia, a collaborative musical
improvisation that explores intimacy between the audience and performer that
is not as prevalent in traditional performance paradigms (see Figure 1).
Stringesthesia measures the performer’s trust level in real-time using fNIRS,
which feeds back to affect the aspects of the performance the audience can
control, such as selecting chords the musician has to play and how many
audience members can play drums. This feedback is designed to dynamically tune
the level of agency the audience or performer have during the performance,
such that when trust is high audience members have more agency, and when trust
is low the performer has more agency. We present the development,
implementation, design process, and performance of Stringesthesia, as well as
reflect on feedback from the audience and performer for improvements in future
performances.

Figure 1. A) Shows the full performance area in Stringesthesia. B) Location of
the 115 audience members during performance. C) Shows audience controllers and
associated instruments at each controller. D) Location of performer and
visualization of trust. E) How the color/chord controllers affected colors of
immersive content and trust visualization.

##  2\. Related Works

The following sections outline prior work that heavily influenced the
components of Stringesthesia.

###  2.1. Neural Signals in Interactive Musical Performance

Biologically-inspired performance paradigms include using electroencephalogram
(EEG) and pulse oximetry to measure neural and cardiac signals that were used
to guide the tempo and bidirectional communication between audience and
performer (Fan and Sciotto, 2013). For example, the installation Ringing
Minds, explores concepts in multi-musician performance in active and
imaginative listening. This promotes the creation of a unique performance
whereby collective brain responses of four audience members interact to inform
a spontaneously generated musical piece (Mullen et al., 2015).

Another example of musician-audience interaction using EEG is the installation
Spukhafte Fernwirkung (Spooky Action at a Distance) (Mullen et al., 2015).
This installation focuses on providing musical ideas for an improvising piano
player by reading EEG signals from an audience member. These examples and more
inform the way that players, audience members, and installation participants
can co-create music using the information provided by the brain of either the
performing artist/s or audience members.

An example of a live collaborative improvisation using neurofeedback is
Eaton’s performance called, The Space Between Us (Eaton et al., 2014). In this
performance EEG signals are read from an audience member and a performer.
Emotional correlates in the EEG signal are converted into musical phrases
played by a computer and a musical score that is presented to a vocalist and
piano performer. During the performance, both the performing vocalist’s and
the audience member’s EEG signals inform the selection of musical passages and
seek to illuminate the similarities and differences in their respective
emotional states during a live performance.

###  2.2. Musical Agency

Musical agency is a critical concept in interactive music systems, referring
to an individual’s ability to control and influence musical outcomes through
their actions, decisions, and intentions. It involves a sense of empowerment
and responsibility over the music-making process, as well as the ability to
shape the musical expression and meaning. In the context of interactive dance
systems, for example, dancers with high musical agency are able to create and
manipulate musical sounds through their movements and gestures, and can
understand how their actions contribute to the overall musical outcome (Brown
and Paine, 2019).

Musical agency has been explored in contexts such as digital musical
instrument design, collaborative music making, and interactive performance. We
focus on musical agency given to audiences in performance in its most commonly
used form, music notation.

####  2.2.1. Music Notation in Interactive Performance for Giving Audience
Members Agency

Audience-performer interactivity has a rich history beginning largely with
audience contributed music notations, as well as mobile and web applications
(Lee and Freeman, 2013a; Lee et al., 2014; Xambó and Goudarzi, 2022; Xambó and
Roma, 2020; Fan and Sciotto, 2013; Lee and Freeman, 2013b; Hindle, 2013;
Weitzner et al., 2012). These systems use audience-contributed classical music
notation, interfaces for audio-based snippets, or translate more accessible
media such as text into classical or other forms of musical notation for the
performer (Baird, 2005; Lee et al., 2012; Dahl et al., 2011; Eaton et al.,
2014). Musical notation is a powerful tool for the audience to provide
information to the performing musician because of its direct relevance to the
content of the performed music. Whether the notation is informed by motion
(Bouchard and Browne, 2019) or another technology, music notation can inspire
new musical directions or directly translate into performed music.

Inspired by these examples, Stringesthesia utilizes color as a mechanism of
delivering musical information to the performer. The performer has a trained
lexical understanding of the color-chord relationship and therefore utilizes
the information provided in the audience designed visualization provided for
the performing artist.

###  2.3. Trust and Affective State Modeling

Trust is a complex, multidimensional construct that has been studied
extensively in contexts such as interpersonal interactions, relationships, and
in interactions with automation (Fett et al., 2014). Recent research on trust
in robotic, virtual, and embedded artificial intelligent systems highlights
the importance of reliability in building trust (Glikson and Woolley, 2020;
Hoff and Bashir, 2015). Reliability is the extent to which a behavior is
accurate and consistent, which can be seen as a situational factor rather than
a design or experimental factor. Changes in reliability are perceived as an
inherent consequence of imperfect AI (Glikson and Woolley, 2020). Trust has
been described as one’s willingness to rely on an external source, and often
demonstrates positive expectations of the trusting party (Rousseau et al.,
1998).

In the context of musical performance, reliability can be seen as a key
component of trust-building, as unpredictable or unreliable musical
performance can negatively impact the outcome of musical collaborations.
Therefore, musicians must strive to establish a high level of reliability in
their musical performance to build trust with their fellow musicians and
audience members (McCormack et al., 2019, 2020). This can be achieved through
consistent and accurate execution of musical elements such as rhythm, melody,
and harmony, as well as through transparent communication and openness to
feedback from collaborators and audiences.

If the performing audience members are too unreliable, for example, if their
rhythms are non-interpretable or their musical contributions are disjointed
and disconnected, the performer may become disengaged or lose trust in the
performance. On the other hand, if the performing audience members are too
rigid and unchanging, the performer may become bored or feel that the
performance lacks creativity and spontaneity. Therefore, trust in musical
performance requires a balance between reliability and unpredictability, as
well as effective communication and openness to the creative process between
performers and audience members.

####  2.3.1. fNIRS Data

Functional Near-Infrared Spectroscopy (fNIRS) is a non-invasive and
lightweight neuroimaging modality that measures changes in hemoglobin
concentration within the brain using infrared light (Ayaz et al., 2022). It
offers several advantages over similar modalities such as EEG and functional
magnetic resonance imaging (fMRI). It is non-invasive which enables
examination in real-life, naturalistic settings. Compared to other forms of
neuroimaging, it is resistant to motion and light artifacts. fNIRS is also
more safe and cost-effective compared to fMRI, and has greater spatial
resolution than EEG (Hirshfield et al., 2019). Following neuronal activation
within a region of the brain, oxygenated hemoglobin travels to that brain
area. Infrared light is used to detect changes in hemoglobin concentration
(oxygenated and deoxygenated), which thus can be used to infer brain activity
(Liu et al., 2015). fNIRS has been shown to be suitable for measurement during
complex cognitive and physical tasks such as yoga (Dybvik and Steinert, 2021),
dance (Noah et al., 2015), and classical music performance (Vanzella et al.,
2019), therefore we used fNIRS during the Stringethesia performance for live
neural recording.

####  2.3.2. Brain Regions Associated with Trust

As fNIRS is a newer neuroimaging technique than comparable technologies like
fMRI and EEG, the literature using fNIRS to measure correlates of trust is
limited (Hirshfield et al., 2019; Eloy et al., 2022; Bobko et al., 2022).
However, since fNIRS and fMRI both measure the hemodynamics of the brain and
have been proven to be correlated (Liu et al., 2015), we can draw from the
extensive literature studying trust using fMRI (as summarized by (Hirshfield
et al., 2019)). The main regions of interest (ROIs) implicated in decision-
making and trust include the frontopolar area (FPA), medial prefrontal cortex
(MPFC), dorsolateral prefrontal cortex (DLPFC), and the bilateral
temporoparietal junction (TPJ) (Hirshfield et al., 2019; Eloy et al., 2022)
(see Figure 2). The regions have often been implicated in Theory of Mind
reasoning, a research paradigm evaluating how one attributes thoughts,
intentions and beliefs to others (Sebastian et al., 2012). These areas are
often activated during social interactions, evaluation of self and others,
decision-making, and determining when to trust (Sebastian et al., 2012; Mahy
et al., 2014; Eloy et al., 2022).

Figure 2. Top-down view of the sensor and detector arrangement for the
measurement of trust (S = sensor, D = detector). LPA and RPA show left and
right side of the head respectively. Iz and Nz represent the back and front of
the head. Colored lines discriminate brain regions.

Trust was further quantified using pilot data collected prior to the
performance in which the performer played both cooperatively and competitively
with another musician. We found that tuning the signal present in brain
regions of the trust network yielded more reliable readings of trust. The
pilot data was used to inform the weighting of the respective brain regions
(TPJ, FPA, MPFC, DLPFC). TPJ was given a weight of 2, meaning that these
bilateral regions had greater weight in our quantification of trust. FPA was
given a weight of 0.5 because FPA is heavily associated with mental workload
and has been implicated in less transparent situations (Eloy et al., 2022).
The other ROIs, DLPFC and MPFC, were given a weight of 1 as we hypothesize
they plan a similar role in trust.

Stringesthesia draws from these paradigms by using the fNIRS neuroimaging
technique to further connect the audience and performer. fNIRS is well-suited
as compared to other technologies such as EEG to accurately measure affective
states such as trust and complex task environments, but to the best of our
knowledge there are no examples of fNIRS used for trust measurement in musical
performance. Our use of fNIRS extends this concept by conveying the internal
state of performers’ trust to the audience and tuning the musical agency given
to the audience based on the performer’s sense of trust–further connecting the
actions of the audience to the outcome of the performance. Because the
performer has no rehearsed material the sense of trust between the performer
and audience is essential for a positive musical outcome.

##  3\. Methods

The performance took place on the campus of the University of Colorado,
Boulder’s black box theater. Equipped with six projectors, stage lights, scrum
curtains, and a large rectangular floor space (approx. 2700 square feet). The
concert lasted for 2 hours and 15 minutes divided by a 15 minute intermission
with a total of 115 audience members in attendance.

The timeline of events was constructed so that audience members had a chance
to move about the space, switch instruments, choose colors, and interact in
different ways throughout the performance. Limitations of the fNIRS readings
required us to divide the performance into sets, allowing the oxygenated
hemoglobin signal from the fNIRS device to return to baseline and enabling a
more effective reading of trust (see Figure 3).

Figure 3. Demonstrates the sequence of events during performance. Cycle is
repeated until performance is over. As jam sessions become longer, blood-
oxygen levels can saturate. Thus, steps 7 and 8 must account for blood-oxygen
levels to normalize.

###  3.1. Measuring and Reporting Trust Levels

Blood-oxygen levels measured by the fNIRS device (NIRSport 2) were live-
streamed via Lab Streaming Layer to MATLAB. The device had 50 channels
spanning across the four aforementioned ROIs (TPJ, FPA, DLPFC, and MPFC). Lab
Streaming Layer parsed the data stream and calculated the relative oxygenated
blood density using the modified Beer-Lambert law equation (Jacques, 2013). It
was then sent to MATLAB on a local device which ran further processing on the
incoming data stream. For the performance, we employed a sliding window
average with a window size of 200 data points and subtracted this figure from
a baseline calculated at the beginning of each jam session. The average
oxygenated blood density levels were calculated at a rate of about 0.5-2Hz.
Jam sessions were variable in length (2-9 minutes) with a 1-3 minute resting
period in between to ensure return of signal to baseline.

###  3.2. Color/Chord Selection Controllers for Musical Agency of the Audience

Chord selection controllers were provided to the audience to give the audience
more agency over the performance. When trust levels were high, more
controllers to select chords that the performer had to use in the jam became
active for the audience.

In order to bridge the gap in musical knowledge between the audience and the
performer, color was used to communicate musical information. We placed 6
controllers in the performance space. Each had six different colors that the
audience members could select (see Figure 4, 1, and 5). The performance
consisted of 14 different “sets” ranging from 2-9 minutes and had 1-3 minutes
in between. Before each set the audience was invited to select a color.
Audience members were free to select any color they wished. For the audience
member, this may have been an aesthetic choice, an exploration of how the
color affects the music, or any other reason the audience member may have had.
Selecting a color caused the controller to change its LEDs to match the
selection, and sent the selected color to the visualization system which
reflected their choice on-screen.

Figure 4. Picture demonstrates a controller that has been reset and one that
has been assigned a color. When reset, six buttons show 6 different colors
from which the audience can select a color. All colors correspond to chords
the performer has memorized.

Six colors were interpreted as chords by the performer who was trained to
establish a connection between the colors and chords present in a major key.
The colors were presented in a spectral order (red, orange, yellow, green,
blue, purple) and represented major scale chords (I, ii, iii, IV, V, vi)
respectively to inform the extemporaneous improvisation (see Figure 4). Color
choices were inspired by common mappings used in color notations between
colors and notes in a major scale (Hopkins et al., 2021; Hopkins and Do, 2020;
HookTheory, 2022).

###  3.3. Interaction Design

We made experience design decisions to encourage audience participation by
ensuring easy access to color controllers and instruments. Four seating areas,
including one in the center of the room, aimed to foster closeness between the
audience and performer. The instruments were rhythmic for easy accessibility,
with limited seating nearby to promote movement and sharing (see Figure 1 and
5).

The performer, guitars, and fNIRS were positioned at the front, with
researchers and computers visible to the audience. A short introduction
explained the performance structure, and color controllers reset in between
sets to invite the audience to choose again.

Figure 5. Shows a top-down view of the performance.

###  3.4. Aesthetic Decisions

Throughout the performance, we made aesthetic decisions to highlight metaphors
within the system’s design, focusing on visualization components and chord
selection controllers for the audience.

Visualizations consisted of connection lines, representing diffusion tensor
imaging, and particles resembling cell nucleus fluorescent staining
techniques. We chose an orb animation displaying complex hive or flocking
dynamics to signify the connection between the audience and performer,
emphasizing the collaborative and improvisational nature of the performance.

Chord selection controllers, placed on music stands, symbolized the audience’s
role in notating music for the performer, even though the music wasn’t
traditional. The controllers resembled modern MIDI button pads to provide a
familiar yet non-traditional feeling, indicating the performance’s
unconventional nature.

##  4\. System Architecture

In this section we describe the various components of the system and their use
in the performance.

Figure 6. System connectivity diagram demonstrating inputs and outputs

###  4.1. Live-Stream Pipeline

The Stringesthesia pipeline is configured to live-stream data between the
musician, audience, and a central processor that visualizes the information
using projection mapping (see Figure 6). All components are connected via WiFi
to a local network transmitting messages using the Universal Datagram Protocol
(UDP).

###  4.2. Audio System

Six microphones were placed near the six rhythmic instruments to create
reactive visualizations for the audience. The microphones and PA amplified the
sound of the instrument, as well as provided a signal to a MAX for Live 8
patch running in Ableton Live 11. The signal was passed to the backline for
amplification, as well as converted to OSC messages in MAX. The volume of the
audio from the receiving microphone was translated into OSC messages which
encoded qualities of a visualization located near the respective drum. The
audio system was used to both control the volume of the drums and provide
interactive visual feedback for the audience.

###  4.3. Unity Engine-Controlled Projection

We used five projectors to create the visualizations during the performance.
To connect the projectors, we used the hardware video wall controller DataPath
(Fx4). After integration of the visual signals into the computer, MadMapper
software (v5.1.5) was used to control the precise location of the projection.

The real-time audio-visual system was made using Unity (2021.1.7f1). The
visual was made with the unity-driven plugin Visual Effect Graph (VFX Graph).
The resulting visuals reacted to the fNIRS and controller signals; both sent
via UDP.

####  4.3.1. Visualization of Trust and Musical Action

The level of trust corresponded to the diameter of the orb behind the
performer (when the orb was small trust was low, and when the orb was large
trust was high) (see Figure 1). Particles appeared to be flowing to and from
the audience members via a stream of diffuse particles and colored lines,
connecting the performer to the audience member. The diffuse stream of colored
particles converged on the orb creating a multicolored representation of
trust. We chose an animation for the orb of particles that appeared to be a
complex motion resembling a hive or flocking dynamics. This choice was meant
to represent the connection between the audience and the performer as having a
life of its own–further drawing the audience toward the shared experience of
collaboration and improvisation.

##  5\. Feedback from Audience and Performer

In this section we report on feedback about the experience from both the
audience and performer about trust, musical agency, unexpected occurrences,
and overall comments on performance.

###  5.1. Audience Feedback

Audience feedback was collected through anonymous written statements about the
performance immediately after the show as well as by email followup. In our
proposed paradigm, audience participation is essential to assess a sense of
the performer’s trust and to create a meaningful performance. Given this was
not a traditional or controlled research setting, we set up a comment box
outside of the auditorium along with pens and paper, prompting audience
members who were interested in telling us about their experience to leave an
anonymous comment.

We specifically were interested in ways to improve the performance, thoughts
on perceived musical agency, and whether the audience members felt a sense of
connection between each other and the performer. Some audience members
reported feeling connected to each other. Three comments in particular
illustrated this: 1) “This was a really cool and immersive, multisensory
experience. I felt like I was really able to be a part of it and felt involved
and connected to the other audience members”, 2) “I’m feeling so inspired to
jam with any and all objects and with everyone I know. Thank you!”, and 3) “I
was discussing the performance with people around me, sharing our
understandings of the visualization. I was also encouraging my friend to
participate by bringing her up to one of the button stations, and when she
said she wanted to play an instrument but she wasn’t sure if any were
available. In other performances, I don’t usually have as much of these
discussions and interactions.”

Some audience members noted that they did not participate musically at all
during the show, stating that they would much rather watch others play music.
This was observed as a continuum between those who preferred to interact and
those who did not. Those that did interact described excitement to play and
choose colors rather than an explicit sense of musical agency. We interpret
this to mean that the ability to playfully interact without being entirely in
the spotlight was exciting and interesting for many. The number of open
stations was a factor that limited the number of people that could
participate, but did not diminish the experience of any individual with
respect to musical agency.

Some members felt the visualization of trust to be too abstract to understand
what was being read. One listener stated: “Because the visualization was
abstract, I wasn’t very focused on interpreting the performer’s trust state.
The main thing I noticed was that the visualization behind the performer was
usually small in the beginning of each piece, and would grow in volume over
the course of the performance..” Another listener suggested that the
“visualizations were striking”, but wanted to know more about its theoretical
underpinnings.

Others felt the visualizations were an exciting component of their experience:
“I felt highly connected [to the performer] because of the immersive visuals
and audio, and the visualization of the trust level gave me more of a glimpse
into the performer’s state.” Another audience member said: “I was fixated on
the graphics, like the orb.”

Overall many audience members indicated that their experience was in line with
our interest in making a performance with over 100 people in the audience feel
very intimate. One audience member reported: “The performance felt like a
combination of the production quality of larger concerts I’ve been to
(particularly due to the visual effects), with the intimacy of small
performances, due to the audience participation. It was similar in the sense
that despite everything going on, for me, the focus was still on the
incredible music; the technology and visualization of trust levels just helped
to enhance that experience. I think overall, these elements made the
experience impactful.”

###  5.2. Performer Feedback

The performer’s perspective offers invaluable insights into the dynamics of
the interactive and improvised performance. The performer noted the impact of
the audience’s emotions on the musical experience, stating, ”The audience’s
excitement and nervousness to participate added an extra dimension to the
performance”. This highlights the importance of considering audience emotions
when designing interactive performances, as these emotions can directly impact
the overall atmosphere (Van Zijl and Sloboda, 2011; Fazekas et al., 2014).

In instances where there were fewer controllers (low trust levels), the
musician reported, ”I had more musical freedom”. This observation suggests a
trade-off between audience interaction and artistic control. Despite some
audience members playing rhythmic instruments even when their station was
deactivated, the performer appreciated the spontaneous nature of these
interactions, saying, ”It added an interesting element to the performance”.

The performer also noted that at times, the audience reclaimed agency by
stomping feet or prolonging beats beyond the musician’s intent. Reflecting on
these actions, the performer said, ”These interactions fostered a high level
of engagement between the audience and me, making the performance more
immersive”.

Overall, the performer described the concert as ”a fun, engaging, and
immersive experience”. They acknowledged that there was sufficient space
between sets for brain levels to desaturate, allowing for dramatic musical
changes, and remarked, ”This balance contributed to the success of the
interactive and improvised performance”.

The performer’s feedback provides a comprehensive understanding of the various
aspects of the interactive and improvised musical performance. It highlights
the importance of balancing audience interaction with artistic control, as
well as the need to consider audience emotions when designing interactive
performances. By considering the performer’s perspective, future academic
musical performance papers can better evaluate and enhance the concert
experience for both the audience and the musicians involved.

##  6\. Discussion

The interactive and improvised performance of Stringesthesia explored the
intricate balance of trust between the performer and audience. The audience’s
enthusiasm and apprehension when participating added a unique dynamic to the
overall experience, emphasizing the importance of considering audience
emotions in the design of interactive performances. The audience’s involvement
in the performance, along with the use of fNIRS to measure the performer’s
trust levels in real-time, provided a rich context for understanding
interactions between the performer and audience.

The feedback from both the audience and the performer reveals some valuable
insights. The audience felt a sense of connection with each other and the
performer, facilitated by the immersive visuals and interactive nature of the
performance. However, some participants found the trust visualization abstract
and difficult to interpret, suggesting a need for a more comprehensible
representation or additional explanation before the performance. This could be
addressed by providing a more in-depth program that outlines the role of trust
and its visualization in the performance.

The performer’s feedback confirms the delicate balance between audience
interaction and artistic control. They appreciated the musical freedom they
experienced during moments of low trust levels, as well as the spontaneous
nature of abundant audience involvement. The audience’s active participation
in the performance, such as stomping feet and prolonging beats, fostered a
high level of engagement between the performer and audience members.

The diverse preferences and experiences of the audience highlight the
importance of accommodating different levels of engagement. Some audience
members preferred not to participate musically, enjoying watching others play
instead, while others found excitement in playfully interacting without being
completely in the spotlight. The limited number of open stations constrained
the number of participants but didn’t diminish any individual’s experience
concerning musical agency.

Stringesthesia presents a novel performance paradigm that promotes audience
participation and fosters a sense of intimacy and connection between the
performer and the audience. The insights gained from the audience and
performer feedback highlight areas for improvement, such as refining the trust
visualization and providing a better explanation of its significance in the
performance. By addressing these issues and taking into account the lessons
learned from this performance, future interactive and improvised musical
performances can strike the right balance between audience engagement and
artistic control, resulting in a more meaningful and fulfilling experience for
all involved.

##  7\. Limitations and Future Work

When interpreting fNIRS data, it is important to note that hemodynamic signal
lags behind neuronal activation. The lag (approx. 3-5 sec (West et al., 2019))
imparted by the technology influenced the performance structure. Both real-
time measurements of trust (using a sliding window average) and an overall
average of trust at the end of every set were assessed to account for both
hemodynamic signal saturation and the signal delay. EEG has better temporal
resolution in this sense, but less spatial specificity offered by fNIRS. A
future study could take advantage of the complimentary features of these
modalities and use concurrent fNIRS-EEG recording during a performance to
measure performer-audience trust.

Additionally, reading neural data from the crowd would add a level of
complexity that may inspire new musical action. The cost of fNIRS machines
hinders this desire, but there are other biological sensors that can (though
less accurately) approximate trust if we explore this option.

A musical limitation was imposed by only having 6 colors used to represent
chords in a Western music derived major key. In future performances this can
be extended to use a larger range of chords, chord qualities, musical style,
as well as color palettes. Additional investigations as to how much influence
the audience has over the performance is also of interest to us in the future.

##  8\. Conclusion

Stringesthesia offers an innovative performance paradigm that encourages
audience participation and cultivates a sense of connection between the
performer and audience members. The use of fNIRS to measure the performer’s
trust levels in real-time provided valuable insights into the delicate balance
between audience interaction and artistic control. Feedback from both the
performer and audience highlighted the importance of considering audience
emotions and fostering engagement while maintaining artistic freedom. The need
for clearer communication regarding the trust visualization was identified,
which can be addressed in future performances. By incorporating these lessons,
future interactive and improvised musical performances can create a more
enriching and immersive experience for everyone involved.

###### Acknowledgements.

We would like to acknowledge the ATLAS Institute’s B2 Center for Media, Arts &
Performance at the University of Colorado Boulder for facilitating the
performance and assisting in the production of the show. We thank Sean Winters
and Chris Petillo for support with audio, as well as Rishi Vanukuru for aiding
with the performance. We would also like to thank the audience that came to
experience the performance with us and the reviewers for their insights.

## References

  * (1)
  * Ayaz et al. (2022) Hasan Ayaz, Wesley B Baker, Giles Blaney, David A Boas, Heather Bortfeld, Kenneth Brady, Joshua Brake, Sabrina Brigadoi, Erin M Buckley, Stefan A Carp, et al. 2022\.  Optical imaging and spectroscopy for the study of the human brain: status report.  _Neurophotonics_ 9, S2 (2022), S24001.
  * Baird (2005) Kevin C. Baird. 2005\.  Real-Time Generation of Music Notation via Audience Interaction Using Python and GNU Lilypond. In _Proceedings of the International Conference on New Interfaces for Musical Expression_. Vancouver, BC, Canada, 240–241.  https://doi.org/10.5281/zenodo.1176695
  * Bobko et al. (2022) Philip Bobko, Leanne Hirshfield, Lucca Eloy, Cara Spencer, Emily Doherty, Jack Driscoll, and Hannah Obolsky. 2022.  Human-agent teaming and trust calibration: a theoretical framework, configurable testbed, empirical illustration, and implications for the development of adaptive systems.  _Theoretical Issues in Ergonomics Science_ (2022), 1–25.
  * Bouchard and Browne (2019) Linda Bouchard and Joseph Browne. 2019.  Live Structures. In _Proceedings of the International Conference on Technologies for Music Notation and Representation – TENOR’19_ , Cat Hope, Lindsay Vickery, and Nat Grant (Eds.). Monash University, Melbourne, Australia, 26–32.
  * Brown and Paine (2019) Courtney Brown and Garth Paine. 2019.  A case study in collaborative learning via participatory music interactive systems: Interactive Tango Milonga.  _New Directions in Music and Human-Computer Interaction_ (2019), 285–306.
  * Dahl et al. (2011) Luke Dahl, Jorge Herrera, and Carr Wilkerson. 2011.  TweetDreams : Making Music with the Audience and the World using Real-time Twitter Data. In _Proceedings of the International Conference on New Interfaces for Musical Expression_. Oslo, Norway, 272–275.  https://doi.org/10.5281/zenodo.1177991
  * Dybvik and Steinert (2021) Henrikke Dybvik and Martin Steinert. 2021.  Real-World fNIRS Brain Activity Measurements during Ashtanga Vinyasa Yoga.  _Brain Sciences_ 11, 6 (2021), 742.
  * Eaton et al. (2014) Joel Eaton, Weiwei Jin, and Eduardo Miranda. 2014.  The Space Between Us. A Live Performance with Musical Score Generated via Emotional Levels Measured in EEG of One Performer and an Audience Member. In _Proceedings of the International Conference on New Interfaces for Musical Expression_. Goldsmiths, University of London, London, United Kingdom, 593–596.  https://doi.org/10.5281/zenodo.1178756
  * Eloy et al. (2022) Lucca Eloy, Emily J Doherty, Cara Ann Spencer, Philip Bobko, and Leanne Hirshfield. 2022.  Using fNIRS to identify transparency-and reliability-sensitive markers of trust across multiple timescales in collaborative human-human-agent triads.  _Frontiers in Neuroergonomics_ (2022), 10.
  * Fan and Sciotto (2013) Yuan-Yi Fan and Myles Sciotto. 2013.  BioSync: An Informed Participatory Interface for Audience Dynamics and Audiovisual Content Co-creation using Mobile PPG and EEG. In _Proceedings of the International Conference on New Interfaces for Musical Expression_. Graduate School of Culture Technology, KAIST, Daejeon, Republic of Korea, 248–251.  https://doi.org/10.5281/zenodo.1178514
  * Fazekas et al. (2014) György Fazekas, Mathieu Barthet, and Mark B Sandler. 2014\.  Novel methods in facilitating audience and performer interaction using the mood conductor framework. In _Sound, Music, and Motion: 10th International Symposium, CMMR 2013, Marseille, France, October 15-18, 2013. Revised Selected Papers 10_. Springer, 122–147.
  * Fett et al. (2014) Anne-Kathrin J Fett, Paula M Gromann, Vincent Giampietro, Sukhi S Shergill, and Lydia Krabbendam. 2014.  Default distrust? An fMRI investigation of the neural development of trust and cooperation.  _Social cognitive and affective neuroscience_ 9, 4 (2014), 395–402.
  * Glikson and Woolley (2020) Ella Glikson and Anita Williams Woolley. 2020.  Human trust in artificial intelligence: Review of empirical research.  _Academy of Management Annals_ 14, 2 (2020), 627–660.
  * Hein and Srinivasan (2019) Ethan Hein and Sumanth Srinivasan. 2019.  The Groove Pizza.  _New directions in music and human-computer interaction_ (2019), 71–94.
  * Hindle (2013) Abram Hindle. 2013\.  SWARMED: Captive Portals, Mobile Devices, and Audience Participation in Multi-User Music Performance. In _Proceedings of the International Conference on New Interfaces for Musical Expression_. Graduate School of Culture Technology, KAIST, Daejeon, Republic of Korea, 174–179.  https://doi.org/10.5281/zenodo.1178550
  * Hirshfield et al. (2019) Leanne Hirshfield, Phil Bobko, Alex Barelka, Natalie Sommer, and Senem Velipasalar. 2019.  Toward interfaces that help users identify misinformation online: using fnirs to measure suspicion.  _Augmented Human Research_ 4, 1 (2019), 1–13.
  * Hoff and Bashir (2015) Kevin Anthony Hoff and Masooda Bashir. 2015.  Trust in automation: Integrating empirical evidence on factors that influence trust.  _Human factors_ 57, 3 (2015), 407–434.
  * HookTheory (2022) HookTheory. 2022\.  Create Amazing Music with HookTheory.  https://www.hooktheory.com.  (2022).
  * Hopkins and Do (2020) Torin Hopkins and Ellen Do. 2020.  Jam Tabs: A Color Based Notation System for Novice Improvisation. In _Proceedings of the International Conference on Technologies for Music Notation and Representation – TENOR’20/21_ , Rama Gottfried, Georg Hajdu, Jacob Sello, Alessandro Anatrini, and John MacCallum (Eds.). Hamburg University for Music and Theater, Hamburg, Germany, 56–62.
  * Hopkins et al. (2021) Torin Hopkins, Peter Pascente, Wayne Seltzer, Kellie Masterson, and Ellen Yi-Luen Do. 2021\.  The Jam Station: Gamifying Collaborative Musical Experiences Through Algorithmic Assessment. In _Proceedings of the Fifteenth International Conference on Tangible, Embedded, and Embodied Interaction_ (Salzburg, Austria) _(TEI ’21)_. Association for Computing Machinery, New York, NY, USA, Article 58, 6 pages.  https://doi.org/10.1145/3430524.3442466
  * Jacques (2013) Steven L Jacques. 2013\.  Optical properties of biological tissues: a review.  _Physics in Medicine & Biology_ 58, 11 (2013), R37.
  * Kunze et al. (2019) Alexander Kunze, Stephen J Summerskill, Russell Marshall, and Ashleigh J Filtness. 2019.  Automation transparency: implications of uncertainty communication for human-automation interaction and interfaces.  _Ergonomics_ 62, 3 (2019), 345–360.
  * Lee et al. (2014) Sang Won Lee, Georg Essl, and Z. Morley Mao. 2014.  Distributing Mobile Music Applications for Audience Participation Using Mobile Ad-hoc Network (MANET). In _Proceedings of the International Conference on New Interfaces for Musical Expression_. Goldsmiths, University of London, London, United Kingdom, 533–536.  https://doi.org/10.5281/zenodo.1178849
  * Lee and Freeman (2013a) Sang Won Lee and Jason Freeman. 2013a.  echobo : Audience Participation Using The Mobile Music Instrument. In _Proceedings of the International Conference on New Interfaces for Musical Expression_. Graduate School of Culture Technology, KAIST, Daejeon, Republic of Korea, 450–455.  https://doi.org/10.5281/zenodo.1178594
  * Lee and Freeman (2013b) Sang Won Lee and Jason Freeman. 2013b.  echobo : Audience Participation Using The Mobile Music Instrument. In _Proceedings of the International Conference on New Interfaces for Musical Expression_. Graduate School of Culture Technology, KAIST, Daejeon, Republic of Korea, 450–455.  https://doi.org/10.5281/zenodo.1178594
  * Lee et al. (2012) Sang Won Lee, Jason Freeman, and Andrew Collela. 2012\.  Real-Time Music Notation, Collaborative Improvisation, and Laptop Ensembles. In _Proceedings of the International Conference on New Interfaces for Musical Expression_. University of Michigan, Ann Arbor, Michigan.  https://doi.org/10.5281/zenodo.1178315
  * Leslie et al. (2014) Grace Leslie, Alejandro Ojeda, and Scott Makeig. 2014\.  Measuring musical engagement using expressive movement and EEG brain dynamics.  _Psychomusicology: Music, Mind, and Brain_ 24, 1 (2014), 75\.
  * Liu et al. (2015) Ning Liu, Xu Cui, Daniel M Bryant, Gary H Glover, and Allan L Reiss. 2015.  Inferring deep-brain activity from cortical activity using functional near-infrared spectroscopy.  _Biomedical optics express_ 6, 3 (2015), 1074–1089.
  * Lucier and Simon (2012) Alvin Lucier and Douglas Simon. 2012.  _Chambers: Scores by Alvin Lucier_.  Wesleyan University Press.
  * Mahy et al. (2014) Caitlin EV Mahy, Louis J Moses, and Jennifer H Pfeifer. 2014\.  How and where: Theory-of-mind in the brain.  _Developmental cognitive neuroscience_ 9 (2014), 68–81.
  * McCormack et al. (2019) Jon McCormack, Toby Gifford, Patrick Hutchings, Maria Teresa Llano Rodriguez, Matthew Yee-King, and Mark d’Inverno. 2019.  In a silent way: Communication between ai and improvising musicians beyond sound. In _Proceedings of the 2019 chi conference on human factors in computing systems_. 1–11.
  * McCormack et al. (2020) Jon McCormack, Patrick Hutchings, Toby Gifford, Matthew Yee-King, Maria Teresa Llano, and Mark D’inverno. 2020.  Design Considerations for Real-Time Collaboration with Creative Artificial Intelligence.  _Organised Sound_ 25, 1 (2020), 41–52.  https://doi.org/10.1017/S1355771819000451
  * Mudd (2019) Tom Mudd. 2019.  Material-oriented musical interactions.  _New Directions in Music and Human-Computer Interaction_ (2019), 123–133.
  * Mullen et al. (2015) Tim Mullen, Alexander Khalil, Tomas Ward, John Iversen, Grace Leslie, Richard Warp, Matt Whitman, Victor Minces, Aaron McCoy, Alejandro Ojeda, et al. 2015\.  MindMusic: playful and social installations at the interface between music and the brain.  _More Playful User Interfaces: Interfaces that Invite Social and Physical Interaction_ (2015), 197–229.
  * Noah et al. (2015) J Adam Noah, Yumie Ono, Yasunori Nomoto, Sotaro Shimada, Atsumichi Tachibana, Xian Zhang, Shaw Bronner, and Joy Hirsch. 2015\.  fMRI validation of fNIRS measurements during a naturalistic task.  _JoVE (Journal of Visualized Experiments)_ 100 (2015), e52116.
  * Perello-March et al. (2022) Jaume R Perello-March, Christopher G Burns, Roger Woodman, Mark T Elliott, and Stewart A Birrell. 2022\.  Using fNIRS to verify trust in highly automated driving.  _IEEE Transactions on Intelligent Transportation Systems_ (2022).
  * Rousseau et al. (1998) Denise M Rousseau, Sim B Sitkin, Ronald S Burt, and Colin Camerer. 1998.  Not so different after all: A cross-discipline view of trust.  _Academy of management review_ 23, 3 (1998), 393–404.
  * Sebastian et al. (2012) Catherine L Sebastian, Nathalie MG Fontaine, Geoffrey Bird, Sarah-Jayne Blakemore, Stephane A De Brito, Eamon JP McCrory, and Essi Viding. 2012.  Neural processing associated with cognitive and affective Theory of Mind in adolescents and adults.  _Social cognitive and affective neuroscience_ 7, 1 (2012), 53–63.
  * Van Zijl and Sloboda (2011) Anemone GW Van Zijl and John Sloboda. 2011.  Performers’ experienced emotions in the construction of expressive musical performance: An exploratory investigation.  _Psychology of Music_ 39, 2 (2011), 196–219.
  * Vanzella et al. (2019) Patricia Vanzella, Joana B Balardin, Rogério A Furucho, Guilherme Augusto Zimeo Morais, Thenille Braun Janzen, Daniela Sammler, and João R Sato. 2019.  fNIRS responses in professional violinists while playing duets: evidence for distinct leader and follower roles at the brain level.  _Frontiers in psychology_ 10 (2019), 164.
  * Weitzner et al. (2012) Nathan Weitzner, Jason Freeman, Stephen Garrett, and Yan-Ling Chen. 2012.  massMobile -an Audience Participation Framework. In _Proceedings of the International Conference on New Interfaces for Musical Expression_. University of Michigan, Ann Arbor, Michigan.  https://doi.org/10.5281/zenodo.1178449
  * West et al. (2019) Kathryn L West, Mark D Zuppichini, Monroe P Turner, Dinesh K Sivakolundu, Yuguang Zhao, Dema Abdelkarim, Jeffrey S Spence, and Bart Rypma. 2019.  BOLD hemodynamic response function changes significantly with healthy aging.  _NeuroImage_ 188 (2019), 198–207.
  * Xambó and Goudarzi (2022) Anna Xambó and Visda Goudarzi. 2022.  The Mobile Audience as a Digital Musical Persona in Telematic Performance. In _Proceedings of the International Conference on New Interfaces for Musical Expression_. The University of Auckland, New Zealand, Article 3\.  https://doi.org/10.21428/92fbeb44.706b549e
  * Xambó and Roma (2020) Anna Xambó and Gerard Roma. 2020.  Performing Audiences: Composition Strategies for Network Music using Mobile Phones. In _Proceedings of the International Conference on New Interfaces for Musical Expression_ , Romain Michon and Franziska Schroeder (Eds.). Birmingham City University, Birmingham, UK, 55–60.  https://doi.org/10.5281/zenodo.4813192
  * Yuksel et al. (2019) Beste F. Yuksel, Kurt B. Oleson, Remco Chang, and Robert J. K. Jacob. 2019.  _Detecting and Adapting to Users’ Cognitive and Affective State to Develop Intelligent Musical Interfaces_.  Springer International Publishing, Cham, 163–177.  https://doi.org/10.1007/978-3-319-92069-6_11