# Fields: Toward Socially Intelligent Spatial Computing

Leonardo Giusti<EMAIL_ADDRESS>Google1600 Amphitheatre ParkwayMountain
ViewCalifornia94043 ,  Lauren Bedal<EMAIL_ADDRESS>Google1600 Amphitheatre
ParkwayMountain ViewCalifornia94043 ,  Eiji Hayashi<EMAIL_ADDRESS>Google1600 Amphitheatre ParkwayMountain ViewCalifornia94043 ,  Timi Oyedeji
<EMAIL_ADDRESS>Google1600 Amphitheatre ParkwayMountain ViewCalifornia94043
,  Jin Yamanaka<EMAIL_ADDRESS>Google1600 Amphitheatre ParkwayMountain
ViewCalifornia94043 ,  Colin Bay<EMAIL_ADDRESS>Concrete UX2189 NW
Wilson StPortlandOregon97210 and  Ivan Poupyrev<EMAIL_ADDRESS>Google1600 Amphitheatre ParkwayMountain ViewCalifornia94043

(2022)

###### Abstract.

In our everyday life, we intuitively use space to regulate our social
interactions. When we want to talk to someone, we approach them; if someone
joins the conversation, we adjust our bodies to make space for them. In
contrast, devices are not as considerate: they interrupt us, require us to
input commands, and compete for our attention. In this paper, we introduce
Fields, a design framework for ubiquitous computing that informs the design of
connected products with social grace. Inspired by interactionist theories on
social interaction, Fields builds on the idea that the physical space we share
with computers can be an interface to mediate interactions. It defines a
generalized approach to spatial interactions, and a set of interaction
patterns that can be adapted to different ubiquitous computing systems. We
investigated its value by implementing it in a set of prototypes and
evaluating it in a lab setting.

ubiquitous computing; ambient computing; radar sensing; implicit interaction;
proxemics

††copyright: acmcopyright††journalyear: 2023††doi:
10.1145/1122445.1122456††conference: ; ; ††price: 15.00††ccs: Human-centered
computing Implicit input Figure 1. We introduce Fields, a design framework for
ubiquitous computing systems. This approach defines interaction fields as
regions of space around users and devices that capture a metric of Potential
Interest (PI). This measures the amount of engagement between a user and
device based on the overlap between interaction fields.

##  1\. Introduction

Ubiquitous computing (Weiser, 1991; Abowd and Mynatt, 2000; Gershenfeld, 1999)
is here. Thermostats, doorbells, and speakers all have sensing capabilities,
process information, and run machine learning algorithms. The average U.S.
household has 25 connected devices, more than double the amount from 2019
(Koetsier, 2022). Computers have proliferated our spaces —however, they are
far from being invisible.

Pioneer Mark Weiser’s vision for ubiquitous computing described a future where
devices support us while we are engaged in the physical world without spending
our attention (Weiser and Brown, 1997). Instead, connected products seem to do
the opposite. First, they often interrupt our everyday activities (Anderson et
al., 2018; Gibbs, 2005), diverting our attention with notifications and
alerts. The effect of this information being communicated irrespective of a
user’s current context is negative, in terms of task completion (Bailey and
Konstan, 2006), performance (Leroy, 2009), and emotional state (Mark et al.,
2008). Furthermore, devices today frequently demand input (Gershenfeld, 1999;
Ju and Leifer, 2008) and rely on explicit input methods, predominately voice
or touch. For example, products today require using a wake word —e.g. “OK
Google” to begin a conversation with a voice assistant; a double-tap is often
required to activate a smart display. The repetition of these commands is
tiresome (Ahire and Rohs, 2020), and hinder the natural flow of a conversation
(Clark et al., 2019). These demands of user’s input and attention prevent the
seamless integration of products into our everyday life in a way that feels
considerate and effortless.

At the core of these concerns lies two key challenges for ubiquitous
computing: context-awareness (Lyytinen et al., 2004; Hong and Suh, [n.d.];
Dey, 2001; Lyytinen and Yoo, 2002), and interaction techniques (Abowd et al.,
2002; Ju and Leifer, 2008; Greenberg et al., 2011).

Context-awareness is often approached as a technical challenge requiring
applications, as Gregory Abowd and Elizabeth Mynatt describe, ”to adapt their
behavior based on information sensed from the physical and computational
environment. Many applications have leveraged simple context, primarily
location and identity, but numerous challenges remain in creating reusable
representations of context, and in creating more complex context from sensor
fusion and activity recognition.” (Abowd and Mynatt, 2000). The research on
interaction technique represents another view: it suggests that interactions
can be designed to be far less jarring or disruptive than what we experience
now. In the context of ubiquitous computing, many complimentary schools of
thought have emerged regarding the need of interaction techniques that
facilitate interactions including natural interfaces (Abowd and Mynatt, 2000),
considerate computing (Gibbs, 2005), attentive user interfaces (Vertegaal et
al., 2003), and calm technology (Weiser and Brown, 1996).

This paper focuses on the latter, suggesting new paradigms for interaction as
computers move ”off-desktop” and into the physical world. We focus our
investigation on the idea of natural interfaces, defined here as interaction
methods that understand our implicit, nonverbal cues in order to unlock a rich
method of communication between people and devices. As Wendy Ju eloquently
describes, ”’Implicit interactions’ - those that occur without the explicit
behest or awareness of the user- will become increasingly important as human-
computer interactions extend beyond the desktop computer into new arenas” (Ju
and Leifer, 2008).

Implicit communication is natural to us as humans. We understand each other
intuitively, often without an explicit command (Clark, 1996; Goffman, 1967):
we hold the door open for others when we see that their arms are full, we keep
certain distances to others depending on familiarity; when we want to talk to
someone, we approach them; if someone else joins the conversation, we adjust
our bodies to make space for them. Nonverbal cues such as changes of spatial
relationships are an implicit form of communication that we understand
intuitively, and allows us to seamlessly initiate, maintain or end a
conversation with others –what if computers could understand us this way?

In this paper, we introduce ”Fields” a framework to design ubiquitous
computing systems with social grace, which supports the design of implicit
interactions that feel considerate, welcoming, and reciprocal (Ju and Leifer,
2008). Inspired by social behaviors and interactionist theories by Adam Kendon
(Kendon, 1990) and Erving Goffman (Goffman, 1967) this approach focuses on one
core idea: the physical space we share with computers can be used as an
interface to mediate interactions.

  * •

First, we introduce two new concepts, Interaction Fields and Potential
Interest, as key components of the Field framework. An Interaction Field is a
region of space surrounding a user or a device that carries a particular
significance for interactions; the Potential Interest tracks the overlap
between Interaction Field to continuously estimate the level of engagement
between people and devices. These concepts are defined independently of a
specific technological implementation, can be applied to a variety of
connected products and devices.

  * •

Second, we demonstrate how the Potential Interest can be used to define a set
of interaction patterns that solve typical interaction problems for ubiquitous
computing systems, such as initiate and terminate interactions, turn-taking,
etc. These interaction patterns include Greeting, Turn-taking, and Revealing.

  * •

Third, we implemented four prototypes using these interaction patterns to
explore how the Fields approach could be deploy in real-world scenarios.

  * •

Fourth, we conducted a study with 14 participants to evaluate a set of 4 use
cases. Through self-reported measures on effort and focus as well as
qualitative findings, our evaluation demonstrates receptiveness and
desirability for using the proposed interaction technique as opposed to
existing techniques.

##  2\. Related Work

The majority of works to understand spatial relationships for ubiquitous
computing originate from the idea of proxemics, a term coined by cultural
anthropologist, Edward Hall (Hall and Hall, 1966). Proxemics addresses how
people perceive, understand, and use space with an emphasis on varying
distances between people. His book The Hidden Dimension, published in 1966,
presented foundational ideas of how surrounding distances of people correlated
with interpersonal relationships. Hall identified four types of informal
space: (1) intimate, (2) casual—personal, (3) social—consultative, and (4)
public.

Hall’s work is commonly seen within HCI. For example, Hello.Wall from 2003
(Figure 2) leverages ideas of Hall’s proxemic theory by introducing the idea
of ‘distance dependent semantics’, discrete zones around the display which
allow for a progression from ambient information, to notifications, to direct
interactions (Ballendat et al., 2010). Vogel et al. (Vogel and Balakrishnan,
2004) and Ju (Ju et al., 2008) also show applications of proxemic frameworks,
showing how notifications and personal and private information can surface
based on a user’s proximity to the device. (Roussel et al., 2004) The Audience
Funnel Framework introduced by Michelis and Miiller (Muller et al., 2010) maps
interaction states based on discrete distances and correlating user
motivations.

Figure 2. Hello.Wall (Prante et al., 2003) leverages ideas of Hall’s proxemic
theory by introducing the idea of ‘distance dependent semantics’, discrete
zones around the display which allow for a progression from ambient
information, to notifications, to direct interactions

Although proximity is a valuable tool to understand spatial relationships,
this approach provides only a course understanding of spatial relationships.
The majority of proxemic interfaces works described above define distance
through fixed, binary values (Ju et al., 2008; Prante et al., 2003; Vogel and
Balakrishnan, 2004). A binary understanding of proximity (in or out of a given
threshold) between a user and device does not capture a multitude of natural
movements that occur around a device such as orientation and trajectories
which, when added together with proximity, contain valuable and relevant
information for designers. For example, if I’m close to a device, but just
passing by, close proximity to a display should not elicit a device response.

In 2010, The Proxemic Interaction Framework introduced by Greenberg et al.
addresses this issue. It used Hall’s ideas as a foundation to operationalize
proxemics for ubiquitous computing with one key difference: recognizing
spatial relationships as a continuous, rather than discrete value (Marquardt
and Greenberg, 2015). The authors recognized that using only binary, proxemic
values is limited, and instead introduced different proximal dimensions
(position, orientation, movement and identity) to help facilitate transitions
between people and large displays from awareness to direct interaction. Many
works leverage this framework and resulting Proximity Toolkit (Marquardt et
al., 2011) for interactive media players (Ballendat et al., 2010) , smart
devices (Rädle et al., 2014), public displays (Wang et al., 2012) and even
device to device interactions (Marquardt et al., 2012). For example, the
Proxemic Presenter (Greenberg et al., 2011) by Miaosen Wang tracks distance,
orientation and identification around a device specifically for presentations
on large displays. This work leverages cues such as orientation and proximity
towards the display to show presenter notes and quick shortcuts (e.g. ability
to skip slides).

The majority of work on spatial interactions takes inspiration from Hall’s
work on proxemics (Hall and Hall, 1966), which references the space
surrounding individuals as territories to defend from intruders; he describes
the ways people arrange themselves in space to optimize their own comfort.
This means that as you step too close to someone, they may step away and give
up their territorial space. However, outside this conceptual view, when we
look broadly at the way people use space in their everyday life, a more
complex and rich picture emerges. Take for instance, how one might initiate a
conversation. When we want to talk to someone, we approach them; in response,
they might take a couple of steps toward us, or turn around if they have no
intention to talk to us. If someone else joins the conversation, we all adjust
our bodies to make space for them. In the interaction with others, we don’t
only respond to variations in distance, orientation and movement, we
proactively use the space around us to establish different social
relationships and negotiate different level of engagements.

Several researchers within the interactionism perspective (Stryker and Vryan,
2006; Blumer, 1986) in sociology including Adam Kendon and Erving Goffman
(Persson, 2018) have tried to identify the underlying principles that regulate
spatial arrangements in social encounters. For example, Goffman noticed that
when people enter into a “focused interaction” they place themselves in a
spatial-orientation arrangement such that each is facing inward around a
shared space (Goffman, 1967); they create a common domain or arena where
glances, gestures and words can be exchanged.

In his book, Conducting Interaction (Kendon, 1990), Kendon describes the space
for focused interaction as “transactional segments”, a space where individuals
direct their attention and carry out activities. The size of this space can
vary depending upon the activity in which they are engaged, e.g., watching
television versus reading a book. It is also important to notice that a
transactional segment is defined in relation to individuals’ lower body; thus
by turning their head or shoulders, they can direct their gaze out of it.
Kendon calls spatial arrangements, F-formations (or facing formation [18]). An
F-formation is formed whenever two or more people arrange themselves such that
their transactional segments overlap, creating a joint transactional space
termed an o-space to which they have equal and exclusive access (Figure 3).

The interactionism perspective we believe provides a valuable foundation for
the design of ubiquitous computing systems as it provides the conceptual tools
to describe how people use space to initiate, maintain and terminate
interactions with others. We can take inspiration from this sociological
perspective to design connected products that are able to take initiative, and
engage in interaction with users in a way that feels natural and considerate.

Figure 3. F-formations consist of 3 concentric spaces; the overlapping space
between two people is called the O-space, a transactional segment that’s
reserved for the main activity the group is pursuing. Other spatial areas
which include the p-space, the ring of space surrounding a group of people,
and the r-space which is the surrounding region, the outside world.

##  3\. Fields Framework

The goal of the Fields framework is to inform the design and development of
ubiquitous computing systems that recognize and respect the user’s intent to
initiate, maintain and end interactions.

The Field framework attempts to translate theories addressing the spatial
relationships between people, particularly core concepts of transactional
segements and social formations as described by Adam Kendon, into the domain
of human computer interaction. We derived three principles to ground the
creation of this framework, and the design of spatial interactions.

  * •

Coequal. Both the user and the device play an equally important role in
establishing the level of engagement within an interactive system, and their
respective social spaces should be taken into consideration.

  * •

Active. The space surrounding an actor (either a device or a user) is not
passively experienced as a defensive territory; rather, it is a social
construct that can be used actively, to initiate, maintain, and end
interactions.

  * •

Emergent. Interaction opportunities between an actor and a device emerge from
the intersection of their individual spaces, and is continuously adjusted and
negotiated.

To facilitate the implementation of these principles in a variety of
ubiquitous computing systems, the Fields framework defines two operative
concepts: interaction fields, and potential interest.

###  3.1. Interaction Field

In our framework, we define an interaction field as the region of space
surrounding an actor, either a user or a device, that carries particular
significance for interaction (Figure 4).

The notion of “fields” has been used in other disciplines to define the
peripersonal space (Teneggi et al., 2013; Bufacchi and Iannetti, 2018). The
peripersonal space is the space surrounding the body where we can reach or be
reached by external entities, including objects or other individuals
(Rizzolatti et al., 1997). It represents a mediation zone between the body and
the environment, that controls our interaction with the physical world as well
as our social interactions. (Coello and Cartaud, 2021; Lloyd, 2009).

The shape of an interaction field can be specified by designers and developers
to capture the unique characteristics of the involved actors, as well as the
physical, cultural and social context where the interaction occurs. While
Interaction Fields can assume a variety of shapes, we can identify two typical
instantiations: (1) user’s interaction field; (2) the device’s interaction
field.

Figure 4. Interaction fields are regions of space around any ‘actor’ (person
or device) within a computing system.

####  3.1.1. User interaction field

. Fields conceptualizes an interaction field around a user, similar to the
peripersonal space. In this paper, we started from using a circular shape for
the ease of initial understanding and experimentation. Furthermore, as John
Fruin mentions in his book, Designing for Pedestrians, the shape of personal
space extends forward in the direction of the motion (Fruin, 1992). This
change of shape, especially when a user is in motion, contains valuable
information about the actor’s intent and level of interest in their
surrounding space. Based on these considerations, we define the the
interaction field as an ellipsis, with the actor occupying one of the foci;
the ellipses’ major axis aligns with the actor direction of motion, and its
eccentricity (distance between the foci) increases proportionally to the actor
speed, while the overall area remains constant. When the actor is not moving,
the shape of the interaction field is circular: since the actor velocity is
equal to zero, the distance between the foci will be 0 as well. Based on
existing literature around size of social spaces, we can approximate the
radius of the interaction field (defining the size of the circle, when
velocity is 0) to 1.2 meters (Hall and Hall, 1966) (Figure 5).

Figure 5. When the actor starts moving, the field morphs into an ellipsis with
the eccentricity increasing as the speed of the actor increases. The overall
area of the interaction field remains constant; as result, while the major
axis of the ellipses increases as the speed increases, the minor axis of the
ellipses decreases in length, projecting the interaction field forward.

###  3.2. Device interaction field.

The Fields framework also considers placing interaction fields around devices
to give a sense of agency to both users and devices. Interaction fields around
devices can vary in size; their configuration largely depends on technological
constraints as well as the device functionality, context, input and form
factors. For example, a TV will have very different interaction field than a
smart speaker, since interactions with a TV will occur when a user is located
in front of the device, while interactions with a smart speaker are
independent of a user’s angle relative to the speaker. Additionally, the
device interaction field can support directional or non-directional sensing.
The non-directional orientation defines an interaction field that surrounds
the device; while a directional orientation extends forward from one side of
the device. Non-directional orientation can be used when the angle of approach
is not relevant for the interaction; directional orientation can be used to
respond to users when they are in front of the device, rather than behind or
to the sides. (Figure 6)

(a) Directional interaction field

(b) Non-directional interaction field

Figure 6. The non-directional orientation defines an interaction field that
surrounds the device; while a directional orientation extends forward from one
side of the device.

###  3.3. Potential Interest

The Potential Interest operationalizes the notion of social formations
proposed by Kendon: as described in the previous section, when two or more
people engage in a social interaction, they orient their transactional
segments so that they overlap, creating a shared, communal space. Based on
this consideration, we believe that the intersection between interaction
fields contains valuable information about a user’s potential interest to
engage in interaction. We define Potential Interest as a function of the
overlap between two intersecting fields around a person and a device, which
can be calculated as an IOU (intersection over union) value between the two
fields (Eq 1) where $F_{1}$ and $F_{2}$ denote fields for the person and the
device.

(1) $\textnormal{Potential Interest}=\frac{F_{1}\cap F_{2}}{F_{1}\cup F_{2}}$

For example, when there is no overlap, the Potential Interest is equal to 0;
when the user’s interaction field completely overlaps with the device’s
interaction field, the Potential Interest is equal to 1. The potential
interest allows to capture the complexities of the spatial relationship
between people and devices, enabling ubiquitous computing system to respond to
implicit interactions: for example, a user approaching a device will produce
an increase of the potential interest that could trigger a change of state
-e.g., the device shift from a sleep to an active state, revealing the content
of a notification; however, if the user is just passing by, the device might
refrain to interrupt the user. This simple example demonstrate how the
potential interest captures important aspects of spatial relationships: in
addition to distance, it takes into consideration orientation and movement
(Figure 7).

(a) Potential Interest = 0.20

(b) Potential Interest = .60

Figure 7. Additionally, even though a user is at a similar distance from the
device, it could refrain from interrupting the user with a notification when a
user is close but just walking by, and only reveal a notification once you’re
engaged.

.

##  4\. Interaction Patterns

###  4.1. Interaction Patterns

Fields opens up the opportunity to define implicit interaction patterns, as
reusable templates to address some of the common problems in the current
generation of ubiquitous computing products (see Introduction). Each of these
interaction patterns use the Potential Interest value as an input, and specify
engagement thresholds to sequence interactions between users and devices.
Thresholds are flexible parameters that specify the level of mutual focus
(Potential Interest) required, for example, to trigger a certain device
response, change device status (e.g. from sleep to active), or transition
between implicit to explicit interactions. In this paper, we present three
interaction patterns modeled around implicit behaviors we typically use in
social interactions –Greeting, Turn-taking, and Revealing.

###  4.2. Greeting

The Greeting pattern allows a user to wake-up or initiate a process on a
device by intentionally overlapping its interaction field with the one of the
device and engage in interaction without the use of explicit commands, such as
hot-words or special gestures (e.g. double-tap on touchscreen). Previous work
demonstrates that wake words and direct commands are frustrating to users
(Ahire and Rohs, 2020). Moreover, as more devices are permeated around us, the
disambiguation of a user’s focus to activate the right device is a critical
problem (Fujinami, 2010). Particularly in our homes, where interactions with
devices such as smart speakers and display are short and frequent, and often
part of a larger set of activities, the repetition of commands every time we
want to interact can be tedious, and result in the fragmentation of our daily
tasks.

The Greeting pattern defines an engagement threshold [t1], and changes its
value based on the history of interaction, to seamlessly initiate, or end an
interaction session. For example, when the Potential Interest value is above a
defined threshold [t1], the device gets ready to interact with the user,
switching from sleep to an active state: e.g. waking up the screen, turning on
sensors such as touch-screen or microphone. Once the interaction with a device
has been initiated, the engagement threshold can be lower [t2], making the
dialogue with the device easier to maintain: e.g. even if the user moves a bit
more outside the device interaction field, the device keeps an active state
(Figure 8) . Finally, when the Potential Interest value is lower than the
engagement threshold [t2], the device goes back to a sleep state: it turns-off
sensing systems which save power and preserve the user’s privacy.

Figure 8. Greeting. Left: a user approaches a device and the potential
interest value ¿ t1. Middle: A user stand facing a device and potential
interest value ¿ t2. Right: A user walks away from a device and the potential
interest value ¡ t2.

This pattern can be used to accelerate frequent interactions with smart
speakers and displays. In particular for voice-based interaction, it can also
be used to disambiguate among a set of devices, which one the user is
referring to when issuing a verbal command. It can be used also with laptops
to increase security: for example, and as soon as the user leaves the area
nearby the laptop, they are immediately logged off without the need of an
explicit action.

###  4.3. Turn-taking

The Turn-taking pattern allows a device to pause and resume an ongoing
activity by understanding the amount of overlap between interaction fields,
which is a byproduct of a user’s movements and body language.

Particularly in our homes, the repetition of commands every time we want to
initiate an interaction can be tedious, and result in the fragmentation of our
daily tasks. This design pattern helps reduce repetitive tasks such as
reaching out to play / pause videos and creates seamless transitions while a
user is multitasking.

The Turn-taking pattern specifies a threshold [t1]. When an ongoing activity
begins (such as playing a video) via touch, a user is engaged by default. Once
the Potential Interest is lower than the engagement threshold [t1] the ongoing
activity can pause. It only resumes once the Potential Interest is higher than
the engagement threshold [t1] again (Figure 9).

Figure 9. T

urn-taking. Left: a user is near device and the potential interest value ¿ t1.
Middle: A user turns away to grab an item and potential interest value ¡ t1.
Right: A user walks away from a device and the potential interest value ¿ t1.

This pattern can be useful in a variety of tasks. One example is playing /
pausing a recipe video. By understanding the potential interest to engage, a
device can play and pause the video in the right moments. This pattern can
also be used for more passive activities, like watching a video while someone
is sitting on the couch. When a user gets up and leaves the device’s
interaction field, the video can pause, and then resume when the Potential
Interest is above a given threshold.

###  4.4. Revealing

The Revealing allows a device to progressively reveal information based on a
users’ awareness or engagement towards a device. Today, the majority of our
interfaces are designed for touch interaction. As a result, many GUIs contain
multiple pieces of information on a screen at once, and are primarily read
from a close distance (within arm’s reach). This pattern provides more
flexibility and customization for user interfaces to progressively reveal
content based on different levels of awareness or engagement of a person
relative to a device. The Progressive Reveal pattern typically specifies
multiple thresholds [t1, t2, tx] that are corresponding to varying levels of
information. When the Potential Interest is higher than the first threshold
[t1], glanceable or minimal information can appear on an interface. When the
Potential Interest is higher than the second threshold [t2], more detailed
information can appear. This provides a situation in which two thresholds are
used; theoretically, more thresholds can be added based on the context and
information required (Figure 10). For example, a notification can show a
preview while the device is in their periphery, showing just enough detail for
them to notice it. When they are engaged with the device, a notification
expands and shows more details and a possible call to action.

Figure 10. Revealing. Left: a user is near device and the potential interest
is ¡ t1. Middle: A user approaches and interest is ¿ t1 but ¡ 2. Right: A user
stands near the device and the potential interest value ¿ t2.

This pattern can be used in a variety of contexts where a computing system
needs to progressively disclose information. For example, this pattern can be
used to create various levels of information density for a weather app. When a
device is aware of a user’s presence, it can provide glanceable information
such as the current temperature. As the user is engaged with the device, it
can reveal a detailed view of the weekly weather forecast.

##  5\. Prototypes

We built four prototypes to investigate potential applications of the Fields
design framework. These prototypes allowed us to explore the implementation of
specific interaction patterns as defined in the previous section, and learn
from user feedback about their desirability as well as emerging interaction
design challenges.

Play and Pause a Movie: In this prototype, a video plays on a television: when
the participant walks away for brief tasks, the video pauses. When the
participant returns to the couch, the video resumes. This prototype
implemented the turn-taking interaction pattern (Figure 11).

Email Notification: On a tablet the system displays three levels of
notification for a priority email. When the user is at a distance and
orientation (say, walking away) connoting lack of engagement, no notification
is displayed. At an intermediate level, a condensed notification with a
subject appears. At a closer approach, the notification displays more detail,
including a preview of the first part of the message. This use case
investigates the use of the Revealing interaction pattern (Figure 14).

Scroll by Voice: This use case prototype investigates how the Greeting
interaction pattern can be used to follow a recipe step-by-step with a voice
assistant, without repeating the wake word for every single interaction. When
the user is engaged, the voice assistant wakes up (and the microphone is
turned on) so that the user can ask to go to the next step (Figure 13).

Play and Pause Recipe: This prototypes also investigates the use of turn-
taking interaction pattern, in the more dynamic context of the kitchen. A
tablet playing a cooking how-to video pauses when the user moves away from the
screen to grab a tool or an ingredient, then continues playback when the user
moves back towards the device to continue cooking (Figure 12).

Figure 11. Play and Pause a Movie. A video plays while participant is oriented
in front of the screen, engaged with the video. The video pauses while
participant leaves the area and is no longer engaged with the screen.  Figure
12. Email Notification. When the user is present, a small notification
appears. When the user is engaged, the notification expands.  Figure 13.
Scroll by Voice. When a user is engaged, the mic opens which option to advance
to the next step of the recipe using a voice command. When the user is not
engaged, the mic is not ‘open’, advancing to the next step requires a hotword.
Figure 14. Play and Pause a Recipe A video plays while the user is engaged
with the recipe. The video pauses while the user turns away from the device
and is no longer engaged with the screen.

###  5.1. Implementation

The Field framework we proposed in this paper can be supported by any
technologies providing users’ and devices’ locations in physical spaces. We
used Soli, a frequency modulated continuous wave (FMCW) radar, connected to a
Raspberry Pi (Figure 15). In the Raspberry Pi, we executed DNN based algorithm
to track the user’s two dimensional position and velocity. The system allowed
us to detect a user’s location accurately in 5 meter x 5 meter space. The mean
absolute error of its range and angle estimates was 0.05 (SD=0.14) meters and
5.5 (SD=7.4) degree. The output from the DNN algorithm was processed with a
heuristic algorithm that calculated the potential interest (Eq 1).

As discussed in Section 3, we implemented a user’s field as a ellipse changing
its shape as a function the user’s velocity as shown in (Eq 2) and (Eq 3),
where $R_{1}$ and $R_{2}$ denote the major and minor axes of the ellipse, $v$
is the speed, and $k$ is a dynamics coefficient. The major axis is in the
direction of $\boldsymbol{v}$.

Following the existing literature (Hall and Hall, 1966) we used the radius of
$R_{1}andR_{2}=1.2$ meters when $v=0$ (i.e., $c=1.2^{2}$). Because all devices
used in the prototype were display devices, we defined the devices fields as a
semicircle with a radius varying based on different prototypes (Table 1). In
the first two prototypes, we empirically chose $k=0.25$, and for the latter
two prototypes we used $k=0.5$ For the .25 k value, we found this responds
better to courser changes of body position, while the .5 k value is more
sensitive, and is more reactive to subtler changes of orientation, such as
turning towards and away from the device. Based on the layout of the room and
the task at hand, we chose which movements to optimize for (turning, or
changes in 2D position). Finally we set a Potential Interest thresholds as
shown in Table (Table 1).For the Revealing interaction pattern used in the
Email notification we set the two as a basic starting point for a first
evaluation.

(2) $\displaystyle\frac{R_{1}}{R_{2}}$ $\displaystyle=kv+1$ (3) $\displaystyle
R_{1}R_{2}$ $\displaystyle=c$

Table 1. Summary of the values defined for each prototype. UIF = User Interaction Field; DIF = Device Interaction Field; Thresholds Use Case | UIF Radius | UIF K Coefficient | DIF Radius | Thresholds
---|---|---|---|---
Play/pause entertainment video | 1.2m | .25 | 3m | .14
Email notification | 1.2m | .25 | 4m | .04, .08
Scroll by voice | 1.2m | .5 | 1m | .6
Play/pause recipe video | 1.2m | .5 | 1m | .6
Figure 15. Left: We used a radar sensor connected to a Raspberry Pi 4 running
a long-range presence algorithm that interprets sensor data and outputs
inferred data about user movement. Right: We displayed the user interface for
each use case on a tablet.

##  6\. Evaluation

We conducted a lab study of the four prototypes to evaluate their
desirability, and study their impact on everyday tasks, such as cooking
following recipes on smart displays or watching TV, in terms of percieved
effort and ease of focus. We tested the prototypes with 14 participants who
were already daily users of smart home devices. We combined the advantages of
lab study consistency with the advantages of a realistic, in-home usage
environment by conducting all sessions in the kitchen and living room of a
single fully-furnished, rented home.

###  6.1. Participants

Recruited participants were daily users of smart home devices such as a home
assistant, smart thermostat, smart camera doorbell, smart door lock, Wi-Fi-
enabled light bulb, or smart home security system, with a mix of gender
identity, age, and ethnicity. Of the 14 participants who completed study
sessions, 43 percent were female and 57 percent were male. Age ranges were 7
percent 18-24, 43 percent 25-34, 43 percent 35-44, and 7 perfect 45-54.
Ethnicities were 21 percent Asian or Pacific islander, 7 percent Hispanic, 7
percent Middle Eastern, 7 percent Native American, 42 percent White, and 14
percent other.

###  6.2. Protocol

During recruiting, participants completed a questionnaire about demographics
and smart home and mobile device usage.

Test sessions were held in a rented home, where the living room served as a
lobby and testing of use cases took place in the kitchen and living room. Each
session began with the moderator giving a short description of the radar
technology and what it would do. Then participants were invited to spend
several minutes playing with a “use case 0” prototype that showed a real-time
visualization of the level of engagement computed by an algorithm. We had some
concern about participants having too much of an “oh, wow” experience and thus
focusing more on the novelty of the technology than on the use case
experiences. To mitigate that effect, we wanted to demystify the technology,
so we created a use case 00 prototype for participants to try before
experiencing any of the main use cases.

Next, participants experienced the four main use cases—play/pause
entertainment video, email notification, scroll recipe instructions by voice,
and play/pause recipe video—in one of 8 different sequences to reduce order
effects. Each use case was designed to exercise the functionality several
times, giving participants a solid opportunity to experience the technology.

Following each use case, the moderator asked the participant to rate their
experience on two 5-point scales:

  * •

Effort: This rating captures the level of effort required to carry out the
task compared to a traditional device, where 1 means a standard smartphone or
tablet requires less effort and 5 means the prototype they just used requires
less effort.

  * •

Ease of focus: This rating captures how easy it was to focus on their task or
goal compared to traditional devices, essentially a distraction measurement,
where 1 means a standard smartphone or tablet allows them to focus more on
their task, and a 5 means the prototype allows them to focus more on their
task.

The moderator also probed for participant rationales for their ratings and
gathered free-form comments on the attraction of the use case. The moderator
then collected a rating on the value of this interaction capability overall,
encouraging them to think of the capability generally, e.g., not limited to
cooking or TV-watching. A final set of open-ended questions was designed to
give participants multiple ways of articulating their reaction to the
technology. The moderator asked participants to talk about how the use case
experiences compared to a traditional smart device, what the experience felt
like, how they would describe the technology to a friend, what impact these
kinds of interactions had on the way they think about technology, and so on.

###  6.3. Quantitative Results

All quantitative ratings are on a -2 to +2 scale, where a positive score is
better for the prototype technology. Ratings from the 1 to 5 scale were
converted to a zero-center scale by subtracting 3 from each score. Ratings for
low effort (i.e., less need for tapping, swiping, speaking) compared to
traditional devices and focus (i.e., freedom from distraction) compared to
traditional devices for each use case are shown in the charts below, with a
maximum mean of +2.0 for each and error bars showing the standard error. A
score of 0.0 would mean this technology was considered equally effortless and
allowed easy focus on the task vs. a traditional smart device. A +2.0 score
would mean this new technology was definitely more effortless and easier to
focus with. -2.0 would mean a traditional smart device was definitely better.
Mean scores for all four use cases were on the positive side for the
prototype, even the least popular ones (Table 2).

Table 2. Ratings for low effort (i.e., less need for tapping, swiping, speaking) compared to traditional devices and focus (i.e., freedom from distraction) compared to traditional devices for each use case are shown in the charts below, with a maximum mean of +2.0 for each and error bars showing the standard error. Use Case | Effortless | Easy to Focus
---|---|---
Play/pause entertainment video | +1.4 | +1.3
Email notification | +1.0 | +0.5
Scroll by voice | +1.2 | +1.5
Play/pause recipe video | +0.9 | +0.5
Mean for all use cases | +1.1 | +1.0

Figure 16. R

atings on Effort and Focus across all four use cases. Mean scores for all four
use cases were on the positive side for the prototype, even the least popular
ones.

These ratings prefigure the top-2 rankings that show the play/pause
entertainment video and scroll by voice use cases well ahead of the other two
use cases. They also show that what is difficult about the less popular use
cases is something related to focus, which the verbal explanations given by
participants make clear. These four use cases are quite different in some
respects: for example, the recipe scroll prompt use case merely displays a
message that makes a voice control invocation unnecessary, rather than
directly taking an action on behalf of the user. Nonetheless, it may be
worthwhile to average ratings across use cases to get a sense of the overall
value of this technology, shown in figure 6. The mean rating overall was +1.1
for effort and +1.0 for focus, a positive result even when including the least
attractive use cases (Figure 16).

Finally, as a measure of overall value, participants rated how much they would
want this general capability in the next smart home device they purchase on a
5-point Likert scale, where 1 is “I’m not interested in having that
capability” and 5 is “I’m very interested in having that capability.” Figure 8
shows those results. No participants gave the technology a rating of less than
3, and 12 of 14 participants chose the highest ratings (4 or 5), for a mean
rating of 4.3 out of 5.

###  6.4. Qualitative Results

In our analysis we looked especially closely at rationales given for effort
and focus ratings, as well as freeform comments made by participants as they
tried out the prototypes and later described their experiences in the final
debrief discussion.

  * •

Play/pause entertainment video: In quantitative ratings, this use case
received high scores for both effort and focus. Why? When asked for rationales
for their scores, participants gave similar reasons. Typical answers about the
positive value focused on obviating the need to pause using a remote (which
requires three or four steps each time). Comments included these: “I didn’t
have to do anything special.” “I think it’s fascinating, that idea of not
having to search for a remote.” “If you’re watching something and you need to
get out, you don’t need to look for the remote.” “The device I just used is
definitely easier.” “I don’t have to do anything, right? I can still like grab
whatever I want to grab and go back and [it plays] for me. I’m not distracted
by the remote.”

  * •

Email notification: Participants explained that modest ratings for email
notification were a result not of the technology but of the nature of
notifications. First, people already get too many notifications of emails,
messages, etc., so having another trigger was considered too much. Second,
they worried about other household members walking by and seeing their message
previews. One representative response summed it up: “It may be easier, but
it’s not preferable.” Indeed, only 2 of the 14 participant scores for effort
favored traditional smart devices, but for ability to focus on their task, 5
of 14 did.

  * •

Scroll by voice: The scroll by voice use case was second highest in the top-2
exercise and had a higher focus score than the most popular use case
(play/pause entertainment video). Verbal comments showed that what led to high
scores was the practical value. People said that being able to advance recipe
instructions when their hands have food on them was highly convenient.
Furthermore, they had full control over when to scroll the screen, rather than
having it happen automatically when they approached the device, which might
not match their need in the moment. Most people assumed the prompt was shown
all the time, and very few noticed they had skipped the invocation. The
moderator’s task introduction did explain how the software for this use case
worked, but this information was quickly forgotten.

  * •

Play/pause recipe video: This use case has an interesting overlap with two
others: it has identical functionality with the play/pause entertainment video
use case, i.e., engagement triggers play or pause of a video; and it overlaps
with the scroll by voice use case in that both allowed advancing through how-
to content for cooking. The critical difference was not the device
functionality; it was the task context. As one participant said, “My only
concern with that was if I’m standing in front of it and it’s going, I’m
starting to fall behind or something like that. I didn’t try [and see] if
there’s a way to pause.” A secondary issue was that because cooking requires
mental focus and frequent movement in a constrained space, sudden starting and
stopping of the video became distracting. These issues led to to middling and
lower ratings for focus.

Study participants summarized the potential of the technology in mainly
positive descriptions. “The fact I didn’t have to prompt something to be a
smart device was really convenient. I could just go about whatever I’m doing.”
“It’s exciting. It’s effortless.” “The convenience factor feels like night and
day to me [compared to] a voice-activated thing, which is most of my smart
devices.” These comments align with the overall ratings, where participants’
desire to have the technology in their next smart home device was high, a mean
of 4.3 out of 5.

The few negative terms tended toward hedging, with positives mixed in: “good
but limited,” “choppy at times,” “nice but not necessary,” “takes getting used
to.” One participant commented, “I definitely have a lot of concerns but that
doesn’t mean that I wouldn’t be totally gung ho about trying it.” Another
said, “I think it could be very useful…unless it gets it wrong.”

###  6.5. Discussion

As shown by the variation in reactions to the four use case experiences, it is
difficult to get user feedback on an interaction framework without having
participants chiefly evaluate the particular implementation. We mitigated this
effect by testing the prototypes in a variety of contexts: we tested them in
two of the rooms mostly likely to contain smart home devices, the kitchen and
the living room, covering a variety of use cases: passive entertainment
(play/pause entertainment video), instructional text (scroll by voice), how-to
video (play/pause recipe video), and communications (email notification). In
spite of the different responses to the prototypes, we can extrapolate some
important learning regarding the Field design framework.

Overall, the user feedback across use cases came through positively. Ratings
on the effort savings compared to traditional smart devices were averaging
+1.1 (on a -2 to +2 scale) across all use cases; similarly the ease of focus
scored +1.0 on average. This seems to suggest that proposed interaction design
framework and its implementation was successfully in helping user accomplish a
variety of tasks with less effort and more focus than traditional devices. It
is interesting to notice how the common denominator of the least attractive
use cases, Email Notification and Play and Pause Recipe was the misalignment
with how people normally handle these tasks. While the Fields framework can
help reduce effort and increase focus for certain tasks, it is important to
situate its implementation in the specific context of use, for example taking
into consideration privacy concerns for notifications, or the natural flow of
everyday activities, such as cooking.

It is interesting to observe how people describe the whole experience. For
example, one participant said: “It’s only when I’m paying attention to it that
it’s paying attention to me.” Another one, said “It’s nice that you don’t have
to initiate it yourself, that it’s basically the one initiating it”. While
this requires more investigation, these comments seem to refer to a user’s
mental model of the system, where both the user and the device play an active
role in establishing the conditions for the interaction to occur. And overall,
participants felt that the prototypes were participating in the task execution
in a way that felt more considerate. As one participant put it: “I think it
requires a lot less intentional effort. […] It’s nice to be able to just move
around and it kind of responds to you like an ebb and flow, in a natural way.”
In the same vein: ”It adapts to what you are doing to make it easier for you
to do certain things without having to be really connected to your devices”.

However, while people responded positively to the reduction of explicit input,
they were concerned that the device wouldn’t recognize their specific needs. A
common theme for participants, mentioned across the different use cases, was
finding themselves pulled between the urge to customize and the urge to step
back: they wanted to make the technology do just what they wanted when they
wanted, but they also wanted to relax and let it help them. One put it this
way: “Obviously I would want it to work exactly how I want it to, but I think
in the end what I would prefer is having something that’s more automated and I
don’t have to think about what I’m doing.” Another looked ahead to having more
smart home technology: “I think maybe in the future it would be more necessary
when we do have more technology around us—you don’t want to have to fiddle
with every single little [device]. So if it is able to do things on its own
for us, that would be nice.” In the implementation of the Fields framework is
therefore important to consider the level of customization left to the end-
users. Design and developers can make specific choices in relation to the
shape of the interaction field, and specific Potential Interest thresholds,
given a certain application and context of use; however, there might be
certain situation where the user might want to control certain aspects of the
implementation to fine-tune the overall system behaviors to their unique
needs.

In conclusion, both numerical subjective ratings and verbal comments by
participants suggest that the Fields framework is a potential approach for the
design of ubiquitous systems that reduce the need of user’s input and
attention, while supporting the execution of everyday tasks. More in general,
the qualitative results seem to suggest that this approach can lead to the
implementation of devices that are able to participate in our everyday life in
a way that feels more considerate and natural.

##  7\. Limitations and Future work

In this paper, we introduced Fields, a framework to inform the design of
connected products with social grace. Inspired by interactionist theories, the
Fields framework builds on the idea that the physical space we share with
computers can be an interface to mediate interactions. It defines a
generalized approach to design for spatial interactions along with a set of
interaction patterns that can be adapted to a variety of computing systems. We
investigated its value by implementing it in a set of prototypes, which were
evaluated with end users in a lab setting, simulating a living spaces (e.g.
kitchen, living room).

The results of our evaluation demonstrate the use case prototypes came through
positively. This suggests the proposed interaction design framework and its
implementation was successful in helping user accomplish a variety of tasks
with less effort and more focus than traditional devices. Additionally, our
qualitative analysis and general remarks on the technology provides early
hints at a shift of mental model that points to devices being helpful and
considerate. As one participant put it: “I think it requires a lot less
intentional effort. […] It’s nice to be able to just move around and it kind
of responds to you like an ebb and flow, in a natural way.” The remarks behind
these prototypes also suggest receptiveness to a new paradigm where devices
are equal participants in the interaction cycle.

There are a few limitations worth noting for this work. First, we defined the
Fields framework with a basic representation of space, and more sophisticated
parameters can be provided to allow designer and developers to explore
different shapes and configurations. Second, the algorithm used in this study
is still very early, and not ready yet to be deployed in a real world
scenario. This naturally affected the evaluation, constraining it to a lab
setting. Furthermore, since the prototypes and interaction concepts presented
were novel to users, and participants only had a limited amount of time to try
out the use cases, it is reasonable to think that ratings and comments would
change if the participant had more time to experience the technology.

These limitations suggest a few areas for future work. The first is to deepen
our work on the framework itself. One area includes adding additional
flexibility in how we consider interaction fields. For example, In the IF
framework we can add flexibility to adopt the notion of “field” to be more
than a traditional in-out zone. Interaction fields can have an homogenous
magnitude, with a clear definition of their boundaries; or they can be defined
as gradient or density function.

Another area of future work expand this framework to accommodate several
interaction typologies, including how multiple people and multiple devices.
For example, what is the significance of multiple fields overlapping? What if
multiple people’s interaction fields overlap? What if someone is wearing a
device? This opens up new areas of spatial and social context understanding.
For example, if the interaction fields of two people overlap while in front of
a TV, perhaps suggestions appear that are relevant for group settings.
Additionally, if two people are watching a movie and one person leaves,
perhaps the TV bookmarks the place where a user left off. In addition, we’d
like to build additional prototypes for different use cases that test a range
of different parameters (such as changes in the size of the device interaction
field) and prototypes that do not contain screens. Since ubiquitous computing
is grounded in multi-user and multi-device systems, these contexts seems to be
a valuable next step.

We’d also like to develop an expanded set of prototypes that help refine our
interaction pattern library, our use of thresholds, and even serve as a
vehicle to develop new interaction patterns. With these prototypes, we would
like to add additional rounds of evaluation on a wider variety of use cases
that, taken together, give participants a broader sense of potential
applicability, as befits a test seeking to evaluate a technology. We’d also
like to have more robust prototypes that can support longitudinal studies. We
do see benefit in participants having more exploration time and a longer
experience with each use case to see the nuances of interaction.

While this is just a first step, we believe that an approach like Fields has
the potential to become a new standard within interaction design for
ubiquitous computing systems. In the same way interface guidelines specify
appropriate ways to use inputs (mouse and keyboard) and interaction patterns
to standardize user experience (cursors, hover states, window managers), we
believe that a new set of interaction patterns and standards will have to
emerge as we move computing ”off-desktop” and into the physical world. Fields,
we believe, pushes the design practice towards this direction as a starting
point. We hope this work facilitates continued discussion toward spatial
interaction standards for ubiquitous computing.

## References

  * (1)
  * Abowd and Mynatt (2000) Gregory D Abowd and Elizabeth D Mynatt. 2000.  Charting past, present, and future research in ubiquitous computing.  _ACM Transactions on Computer-Human Interaction (TOCHI)_ 7, 1 (2000), 29–58.
  * Abowd et al. (2002) Gregory D Abowd, Elizabeth D Mynatt, and Tom Rodden. 2002\.  The human experience [of ubiquitous computing].  _IEEE pervasive computing_ 1, 1 (2002), 48–57.
  * Ahire and Rohs (2020) Shashank Ahire and Michael Rohs. 2020.  Tired of wake words? Moving towards seamless conversations with intelligent personal assistants. In _Proceedings of the 2nd Conference on Conversational User Interfaces_. 1–3.
  * Anderson et al. (2018) Christoph Anderson, Isabel Hübener, Ann-Kathrin Seipp, Sandra Ohly, Klaus David, and Veljko Pejovic. 2018.  A survey of attention management systems in ubiquitous computing environments.  _Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies_ 2, 2 (2018), 1–27.
  * Bailey and Konstan (2006) Brian P Bailey and Joseph A Konstan. 2006.  On the need for attention-aware systems: Measuring effects of interruption on task performance, error rate, and affective state.  _Computers in human behavior_ 22, 4 (2006), 685–708.
  * Ballendat et al. (2010) Till Ballendat, Nicolai Marquardt, and Saul Greenberg. 2010\.  Proxemic interaction: designing for a proximity and orientation-aware environment. In _ACM International Conference on Interactive Tabletops and Surfaces_. 121–130.
  * Blumer (1986) Herbert Blumer. 1986\.  _Symbolic interactionism: Perspective and method_.  Univ of California Press.
  * Bufacchi and Iannetti (2018) Rory J Bufacchi and Gian Domenico Iannetti. 2018.  An action field theory of peripersonal space.  _Trends in cognitive sciences_ 22, 12 (2018), 1076–1090.
  * Clark (1996) Herbert H Clark. 1996\.  Arranging to do things with others. In _Conference Companion on Human Factors in Computing Systems_. 165–167.
  * Clark et al. (2019) Leigh Clark, Nadia Pantidi, Orla Cooney, Philip Doyle, Diego Garaialde, Justin Edwards, Brendan Spillane, Emer Gilmartin, Christine Murad, Cosmin Munteanu, et al. 2019\.  What makes a good conversation? Challenges in designing truly conversational agents. In _Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems_. 1–12.
  * Coello and Cartaud (2021) Yann Coello and Alice Cartaud. 2021.  The interrelation between peripersonal action space and interpersonal social space: psychophysiological evidence and clinical implications.  _Frontiers in Human Neuroscience_ 15 (2021), 636124.
  * Dey (2001) Anind K Dey. 2001\.  Understanding and using context.  _Personal and ubiquitous computing_ 5, 1 (2001), 4–7.
  * Fruin (1992) John Fruin. 1992\.  Designing for pedestrians.  _Public Transportation United States_ (1992).
  * Fujinami (2010) Kaori Fujinami. 2010\.  Interaction design issues in smart home environments. In _2010 5th International Conference on Future Information Technology_. IEEE, 1–8.
  * Gershenfeld (1999) Neil Gershenfeld. 1999\.  _When things start to think_.  Macmillan.
  * Gibbs (2005) W Wayt Gibbs. 2005\.  Considerate computing.  _Scientific American_ 292, 1 (2005), 54–61.
  * Goffman (1967) Erving Goffman. 1967\.  Interaction ritual: Essays on face-to-face interaction.  (1967).
  * Greenberg et al. (2011) Saul Greenberg, Nicolai Marquardt, Till Ballendat, Rob Diaz-Marino, and Miaosen Wang. 2011\.  Proxemic interactions: the new ubicomp?  _interactions_ 18, 1 (2011), 42–50.
  * Hall and Hall (1966) Edmund T Hall and Edward Twitchell Hall. 1966.  _The hidden dimension_. Vol. 609.  Anchor.
  * Hong and Suh ([n.d.]) Jong-yi Hong and Eui-ho Suh. [n.d.].  Kim. S.(2009). Context-aware systems.  _Expert Syst. Appl_ 36, 4 ([n. d.]), 8509–8522.
  * Ju et al. (2008) Wendy Ju, Brian A Lee, and Scott R Klemmer. 2008.  Range: exploring implicit interaction through electronic whiteboard design. In _Proceedings of the 2008 ACM conference on Computer supported cooperative work_. 17–26.
  * Ju and Leifer (2008) Wendy Ju and Larry Leifer. 2008.  The design of implicit interactions: Making interactive systems less obnoxious.  _Design Issues_ 24, 3 (2008), 72–84.
  * Kendon (1990) Adam Kendon. 1990\.  _Conducting interaction: Patterns of behavior in focused encounters_. Vol. 7.  CUP Archive.
  * Koetsier (2022) John Koetsier. 2022\.  Smart home: Apple is the fastest-growing connected device company.  https://www.forbes.com/sites/johnkoetsier/2022/08/31/smart-home-apple-is-the-fastest-growing-connected-device-company/?sh=38920817dd48
  * Leroy (2009) Sophie Leroy. 2009\.  Why is it so hard to do my work? The challenge of attention residue when switching between work tasks.  _Organizational Behavior and Human Decision Processes_ 109, 2 (2009), 168–181.
  * Lloyd (2009) Donna M Lloyd. 2009\.  The space between us: A neurophilosophical framework for the investigation of human interpersonal space.  _Neuroscience & Biobehavioral Reviews_ 33, 3 (2009), 297–304.
  * Lyytinen and Yoo (2002) Kalle Lyytinen and Youngjin Yoo. 2002.  Ubiquitous computing.  _Commun. ACM_ 45, 12 (2002), 63–96.
  * Lyytinen et al. (2004) Kalle J Lyytinen, Youngjin Yoo, Upkar Varshney, Mark Ackerman, Gordon Davis, Michel Avital, Daniel Robey, Steve Sawyer, and Carsten Sorensen. 2004.  Surfing the next wave: design and implementation challenges of ubiquitous computing.  _Communications of the Association for information systems_ 13, 1 (2004), 40.
  * Mark et al. (2008) Gloria Mark, Daniela Gudith, and Ulrich Klocke. 2008\.  The cost of interrupted work: more speed and stress. In _Proceedings of the SIGCHI conference on Human Factors in Computing Systems_. 107–110.
  * Marquardt et al. (2011) Nicolai Marquardt, Robert Diaz-Marino, Sebastian Boring, and Saul Greenberg. 2011.  The proximity toolkit: prototyping proxemic interactions in ubiquitous computing ecologies. In _Proceedings of the 24th annual ACM symposium on User interface software and technology_. 315–326.
  * Marquardt and Greenberg (2015) Nicolai Marquardt and Saul Greenberg. 2015.  Proxemic interactions: From theory to practice.  _Synthesis Lectures on Human-Centered Informatics_ 8, 1 (2015), 1–199.
  * Marquardt et al. (2012) Nicolai Marquardt, Ken Hinckley, and Saul Greenberg. 2012\.  Cross-device interaction via micro-mobility and f-formations. In _Proceedings of the 25th annual ACM symposium on User interface software and technology_. 13–22.
  * Muller et al. (2010) Jorg Muller, Florian Alt, Daniel Michelis, and Albrecht Schmidt. 2010\.  Requirements and design space for interactive public displays. In _Proceedings of the 18th ACM international conference on Multimedia_. 1285–1294.
  * Persson (2018) Anders Persson. 2018\.  _Framing social interaction: Continuities and cracks in Goffman’s Frame Analysis_.  Taylor and Francis.
  * Prante et al. (2003) Thorsten Prante, Carsten Röcker, Norbert Streitz, Richard Stenzel, Carsten Magerkurth, Daniel Van Alphen, and Daniela Plewe. 2003\.  Hello. wall–beyond ambient displays. In _Adjunct Proceedings of Ubicomp_ , Vol. 2003. 277–278.
  * Rädle et al. (2014) Roman Rädle, Hans-Christian Jetter, Nicolai Marquardt, Harald Reiterer, and Yvonne Rogers. 2014\.  Huddlelamp: Spatially-aware mobile displays for ad-hoc around-the-table collaboration. In _Proceedings of the Ninth ACM International Conference on Interactive Tabletops and Surfaces_. 45–54.
  * Rizzolatti et al. (1997) Giacomo Rizzolatti, Luciano Fadiga, Leonardo Fogassi, and Vittorio Gallese. 1997.  The space around us.  _Science_ 277, 5323 (1997), 190–191.
  * Roussel et al. (2004) Nicolas Roussel, Helen Evans, and Heiko Hansen. 2004\.  Proximity as an interface for video communication.  _IEEE MultiMedia_ 11, 3 (2004), 12–16.
  * Stryker and Vryan (2006) Sheldon Stryker and Kevin D Vryan. 2006.  The symbolic interactionist frame.  _Handbook of social psychology_ (2006), 3–28.
  * Teneggi et al. (2013) Chiara Teneggi, Elisa Canzoneri, Giuseppe di Pellegrino, and Andrea Serino. 2013.  Social modulation of peripersonal space boundaries.  _Current biology_ 23, 5 (2013), 406–411.
  * Vertegaal et al. (2003) Roel Vertegaal et al. 2003\.  Attentive user interfaces.  _Commun. ACM_ 46, 3 (2003), 30–33.
  * Vogel and Balakrishnan (2004) Daniel Vogel and Ravin Balakrishnan. 2004.  Interactive public ambient displays: transitioning from implicit to explicit, public to personal, interaction with multiple users. In _Proceedings of the 17th annual ACM symposium on User interface software and technology_. 137–146.
  * Wang et al. (2012) Miaosen Wang, Sebastian Boring, and Saul Greenberg. 2012\.  Proxemic peddler: a public advertising display that captures and preserves the attention of a passerby. In _Proceedings of the 2012 international symposium on pervasive displays_. 1–6.
  * Weiser (1991) Mark Weiser. 1991\.  The Computer for the 21st Century.  _Scientific american_ 265, 3 (1991), 94–105.
  * Weiser and Brown (1996) Mark Weiser and John Seely Brown. 1996.  Designing calm technology.  _PowerGrid Journal_ 1, 1 (1996), 75–85.
  * Weiser and Brown (1997) Mark Weiser and John Seely Brown. 1997.  The coming age of calm technology.  In _Beyond calculation_. Springer, 75–85.