∎

11institutetext: Dave Murray-Rust 22institutetext: Human Centred Design
Industrial Design Engineering
TU Delft, Netherlands
22email<EMAIL_ADDRESS>33institutetext: Maria Luce Lupetti
44institutetext: 44email<EMAIL_ADDRESS>55institutetext: Iohanna
Nicenboim 66institutetext: 66email<EMAIL_ADDRESS>77institutetext:
Wouter van der Hoog 88institutetext: 88email<EMAIL_ADDRESS>
# Grasping AI: experiential exercises for designers

Dave Murray-Rust  Maria Luce Lupetti  Iohanna Nicenboim  Wouter van der Hoog

(Received: date / Accepted: date)

###### Abstract

Artificial intelligence (AI) and machine learning (ML) are increasingly
integrated into the functioning of physical and digital products, creating
unprecedented opportunities for interaction and functionality. However, there
is a challenge for designers to ideate within this creative landscape,
balancing the possibilities of technology with human interactional concerns.
We investigate techniques for exploring and reflecting on the interactional
affordances, the unique relational possibilities, and the wider social
implications of AI systems. We introduced into an interaction design course
(n=100) nine ‘AI exercises’ that draw on more than human design, responsible
AI, and speculative enactment to create experiential engagements around AI
interaction design. We find that exercises around metaphors and enactments
make questions of training and learning, privacy and consent, autonomy and
agency more tangible, and thereby help students be more reflective and
responsible on how to design with AI and its complex properties in both their
design process and outcomes.

###### Keywords:

Artificial Intelligence design prototyping design education experiential
methods AI exercises more-than-human design

##  1 Introduction

Designers increasingly need to develop a facility with artificial
intelligence, as it becomes part of the way that products and services
function and appears in an increasing number of the contexts in which
designers work Benjamin et al (2021); Dove et al (2017). However, there are
several challenges for design students in engaging with AI, from the broadness
of the term AI and the fuzziness with which it is applied Littman (2021), to
the difficulty of getting to grips with the technical and computational
complexities of these systems Yang et al (2020); Nicenboim et al (2022). These
challenges around understanding and making sense of the new capabilities of AI
become urgent as the technology emerges from it’s latest winter into a new
spring, developing at a fast pace Littman (2021); Samoli et al (2020); Floridi
(2020).

The range of techniques for making creative use of AI has been rapidly
growing: Runway offered easy access to generative spaces and now video Runway
(2020); EdgeImpulse offers sound and gesture classification for
microcontrollers with training through a web interface EdgeImpulse (2019); the
current sets of generative image models such as DALL-E, Midjourney and
StableDiffusion and language models (ChatGPT etc.) allow a natural language
interaction through the use of prompts. Along with learning materials for more
traditional toolkits TensorFlow (2015); OpenCV (1999) and model development
and exchange initiatives (e.g. HuggingFace, 2016) these form a downward
pressure on the technical barrier to entry, even as the complexity of the
underlying models increases. The conceptual barrier can remain high, though,
reducing the possibility for designerly engagement and appropriation. There is
a large jump from “my first ML model” to understanding the implications of ML
technology, and designers often want – and need – to engage with these
implications. Creating models in practice helps, but this needs conceptual
framing to help direct and contextualise the activities - for example, in
related fields, courses such as Creative Applications of Deep Learning Mital
(2016) and the more provocative follow up Cultural Appropriation with Deep
Learning Mital (2021) look at visual practice, or Machine Learning for
Musicians and Artists Fiebrink (2022) unpack these systems for creative
practitioners.

However, simply thinking about why it is hard for designers to appropriate AI
technologies into their practices also misses a key question: what can design
practice bring to the development and understanding of AI systems Benjamin et
al (2021), especially as the technologies become more pervasive and more
collaborative Wang et al (2020). Designers have their own strategies for
making use of, critiquing and appropriating new technologies Westerlund and
Wetter-Edman (2017), so there is an interest in understanding what designerly
methods could reveal about human AI relations, particularly where it involves
interactions between humans and technological systems - considering the
“social, political, ethical, cultural, and environmental factors of
implementing AI into daily human-to-computer interactions” Wong (2018). Design
research methods, speculations Auger (2013); Kirman et al (2022), fictioning
Forlano and Mathew (2014); Wong et al (2017); Troiano et al (2021); Benjamin
et al (2023), probes and toolkits Sanders and Stappers (2014), more than human
design Coulton and Lindley (2019) and the general practices of Research
through Design (RtD) Giaccardi (2019); Stappers and Giaccardi (2017), are all
well suited to thinking into the socio-technical aspects Holton and Boyd
(2021); Sartori and Theodorou (2022); Theodorou and Dignum (2020),
possibilities, and implication of AI in everyday life, just as they have been
applied to understanding digital sensing technologies Pierce (2021),
blockchains Murray-Rust et al (2022a), the future of automation Cavalcante
Siebert et al (2022) and so on.

Our aim is to help students to design products and services that make use of
AI technologies, while developing a critical understanding of its
implications. This means articulating both the technical and relational
aspects of AI so that meaningfully shape the development of products, services
and systems even if they are not intimately familiar with the technical
details of its operation. As such we are looking for ways to sensitize
interaction designers to AI, to create experiences rather than explanations.
In relation to the typology developed by Yang et al (2020) of ways to aid
designers around AI, our work contributes to the early stages of ‘creating AI-
specific design processes’ by probing with concrete exercises ways in which
educators can support designers in ideating in a space mediated by the
capabilities and implications of AI systems.

In order to explore this space, we created a set of methods for designing AI
driven products and services (Section 3.2) that draw on theories about how
people relate to technology and AI in particular. These methods take the form
of short, autonomous, experiential exercises that can be used to develop and
enrich the design of interactive technological products and services. We
introduced these exercises partway through an interaction design course
(Section 3.1), where, students (n=100) in small groups (n=28) are asked to
design future products and services through iterative prototyping and testing.
We collected an immediate written reaction from each group as to what the
students had done with the materials, and the aspects they found useful or
resonant. We interviewed a self-selecting subset of the students (n=12) and
their coaches (n=7) to dig deeper into questions of how the methods had
changed their understandings and relations to AI.

To explore the potential of these methods, we investigate the following
research questions:

  * •

RQ1: How do the exercises stimulate and modulate changes to the students’
design process to accommodate AI, in particular the way that they are
conceptualising and prototyping their projects?

  * •

RQ2: How do these experiential exercises affect student’s grasp of AI and ML,
in particular in relation to interactional, relational and contextual
qualities which are key points in the recent theoretical developments in AI
within HCI?

  * •

RQ3: How do the exercises help to develop a critical design perspective while
engaging with AI technology as a socio-technical system?

Through investigating and discussing these research questions, the
contributions of this work are:

  1. 1.

A set of exercises that translate theoretical developments in design and AI,
into experiential exercises for designers that can be carried out
autonomously, with reflection on the experiential, pragmatic and reflective
qualities that made the exercises effective. These exercises are available at
[redacted] for future use and development.

  2. 2.

Insights into how and for what to apply these exercises in a pedagogical
context to support design processes for creating AI enabled products and
services.

  3. 3.

Insights about how these exercises affected student’s reasoning and design
activities, bringing agency, relationality and criticality alongside
development of technical facility.

  4. 4.

Methodological reflections around the possibilities afforded by the methods
and how these contribute to nurturing a uniquely designerly AI culture that
supports future design education.

##  2 Background

Working with AI presents particular challenges for designers. One of them is
around engaging with emerging and complex technologies, with different
behaviours from traditional design materials. Yang et al (2020) point at two
key challenges: the uncertainty about the capability of AI systems, and the
complexity of their outputs. The second challenge is around understanding AI,
given that the metaphors and imaginaries around it, obscure the real processes
that are needed for maintaining such a technology Murray-Rust et al (2022b).
Contrary to competing terms like ‘complex information processing systems’ or
‘machine intelligence,’ the term AI fires the mind with ideas of human-like
reasoning. While these imaginaries seem to be better for marketing, they are
certainly no good for developing a grounded sense of the capabilities of AI as
a technology. Hildebrandt (2020)

###  2.1 Designing AI

Despite the challenges for designers to engage with AI, there are currently
many areas where design and AI touch on each other.

At a low level, there is growing attention to the meeting of AI and user
experience (UX), as the new possibilities offered by the technology allow new
kinds of interaction, and are susceptible to new pitfalls. Techniques are
emerging that help to create user interfaces that work with AI systems
Subramonyam et al (2021b, a), or support innovating AI-powered services and
systems within enterprises Yildirim et al (2022). This can be seen in
Microsoft’s guidelines for human-AI interaction Amershi et al (2019), or
Google PAIR’s Guidebook PAIR (2020), as well as efforts to bring HCI together
with AI Inkpen et al (2019). Recently, the identification of ‘AI capabilities’
Yildirim et al (2023) provides a concrete way to think about design spaces for
the interactional aspects of computational system. Negotiation between AI and
HCI can be deep and subtle: interactional affordances help to calibrate trust
and reliance between humans and AI; conceptual metaphors sculpt the relations
formed with conversational agents Jung et al (2022); Khadpe et al (2020); and
appropriate abstractions make AI qualities at hand for creative practitioners
Fiebrink (2019); Tremblay et al (2021). User experience, in its broader sense,
goes beyond designing the immediate experiences, with work starting to
consider how to develop frameworks for creating more or less personal,
dependent and discretionary interfaces Kliman-Silver et al (2020), or at how
to generate heuristic models of meaningful engagement with AI artworks Hemment
et al (2022b).

Zooming out slightly, a collection of theoretical issues around AI relate to
emerging fields in the third (and fourth) wave HCI and the philosophy of
design communities. Scholars in those areas have grappled with how the
concepts used in design and HCI practices might be tied to the industrial era,
and how they might have to change and adapt to the new kinds of products and
materials which are enabled by AI. This includes lines of research such as
post-industrial design, more than human practices Giaccardi and Redström
(2020), entanglement thinking Frauenberger (2020); Murray-Rust et al (2019);
Hodder (2016), fluid assemblages and multi-intentionality Redström and Wiltse
(2018); Wiltse (2020). All these offer vibrant pictures of a new set of
relationships between humans and the material world in which both entities
‘co-constitute’ each other. Along with reorienting the relationships between
humans and non-humans, scholars within those fields have been rethinking what
it is to ‘do design’, breaking with traditions focused on the subject-object
dichotomy Giaccardi and Redström (2020), where design goes beyond a mere
problem solving enterprise and becomes an ongoing and more inclusive practice.
Although these theoretical developments seem to be gathering momentum, they
are still not fully translated into practical tools for designers – the jump
from Barad’s agential realism Barad (2007) to configurations of bits and
programs takes careful work Scurto et al (2021); Seymour et al (2022); Sanches
et al (2022).

At a broader scale, beyond the immediate interactions, some of the theories
and practices at play are oriented towards engineering particular system
qualities and properties: value sensitive design can help to make sense of
fluid and evolving systems de Reuver et al (2020), where many different human
values may be at play Yurrita et al (2022); Fish and Stark (2021); Shen et al
(2021); questions of meaningful human control modulate the relations of
responsibility between humans and automated systems Cavalcante Siebert et al
(2022), as does responsible AI design Benjamins et al (2019). Here, design is
an instrumental part of making systems behave in certain ways. AI ethics is a
broad field Hagendorff (2020), and well as directly affecting system
properties, work from the Fairness Accountability and Transparency (FAccT)
community looks to support documentation that helps maintain these properties
in communities, such as documentation for models and datasets Mitchell et al
(2019); Gebru et al (2018), and the ethical aspects of system development
Mohammad (2021); Murray-Rust and Tsiakas (2022).

###  2.2 AI and Design Education

The specificities and challenges of AI and ML technologies add up to an
ongoing discourse at the intersection of design education and technological
progress. Traditional formats and scopes for carrying out design are being
questioned and revised, with canonical, linear, causal, and instrumental
approaches being criticized in favor of novel models inspired by complexity
theory, system science, and practical philosophy. This moves towards an aim of
reconceptualizing design as a moral act Findeli (2001); Lin (2014). Designers
and design researchers, in fact, are increasingly recognized as actors whose
decisions have ethical as well as political implications Lloyd (2019).
Parallel to this, the societal implications of AI and ML are more clearly
pervasive and unpredictable. However, when introduced in design education,
these technologies are typically either approached as the ultimate tools to
learn or used as ‘context’ for grounding alternative and critical design
explorations. On the one hand, ML courses are increasingly offered to design
students for promoting ML/AI literacy but remain an addition to the main
curricula, rather than being integrated into project courses (as in Jonsson
and Tholander (2022); van der Vlist et al (2008)). These technologies are
still rarely integrated in design education Dove et al (2017), and often
approached with the believe that even just exposing students to cutting edge
technology can stimulate the emergence of innovative and technologically
advanced design solutions to real problems McCardle (2002). In other cases,
however, the approach is diametrically different: AI and ML are conceptualised
as phenomena to be understood and questioned, because of its potential impact
in society. For instance, Auger (2014) used the theoretical lens of
domestication for challenging students to ideate future domestic robots and
reflect on their implications in everyday settings. The two perspectives on AI
and ML, the focus of which we summarize as technical competency vs critique,
tend to remain distinct approaches with apparently opposite scopes. Even when
there is an explicit commitment to bridge the two approaches, existing
pedagogy struggles to combine the ambition to build AI literacy while also
fostering a critical mindset around AI/ML projects, and reflections do not
lead to rich critiques about situated and contextual implications of AI and ML
unless they are integrated into project development. For some counterexamples,
Jonsson and Tholander (2022) purposefully crafted a course for students to
approach and appreciate AI tools as creative partners, and learned that AI
qualities, such as uncertainty, imperfection, and under-determination, can be
a rich source of inspiration for generating creative expressions as well as
powerful triggers of reflection. Mital’s Cultural Appropriation with Deep
Learning course Mital (2021) weaves together learning about the operation of
deep networks with recognising their role in society. Fiebrink’s work Fiebrink
(2019) distinctively looks at ML as a design material and situates it within
project development. Perhaps most similar to the work outlined here is
’Graspable AI’ Ghajargar et al (2021, 2022); Ghajargar and Bardzell (2023),
which brings together tangibility and AI, using explanation as a path to
understanding and form as a language for communicating AI affordances. Even in
these cases, however, the emphasis is on one side of the spectrum, that is on
how to teach ML effectively to any population and enable the emergence of new
creative outputs.

The disciplinary call for exposing the design questions involved in making AI
and ML systems – as well as the complexity and trade-offs that implementing
these in the world implies Bilstrup et al (2022) – remains largely unanswered.
Our work sits at the intersections of these experiences and aims to fill the
gap between technical efforts and critical explorations. Specifically, we set
out to integrate AI and ML explorations within the development of design
projects, in a way that both enable students to build AI literacy, as well as
to empower them to take a critical stand towards these technologies in
society.

###  2.3 Summary and research direction

Part of the work of design as a discipline is to mediate between these
philosophies and actionable practices that can be brought to bear on
particular situations. That is the starting point for the work presented in
this paper: we are interested in how to bring conceptual developments from
design theory and AI into something that is at hand to design students, that
can make a difference to how they go about conceptualising and prototyping
interactions.

In order to bridge the gap between the practical and technical engagement with
AI, we propose that three levels of engagement between AI and design are all
potentially at play within design projects creating AI powered systems:

Interactional Affordances of AI

that allow new means of interaction between systems and people. At a low
level, AI brings new possibilities for sensing, responding, recognising and
classifying from which to build interactions. These interactional affordances
and possibilities for action Stoffregen (2003) offered by machine intelligence
can take the form of capabilities offered by the technology (see Yildirim et
al (2023) for a comprehensive overview), but also of modulations of existing
capabilities with AI specific qualities such as probabilistic outcomes.

AI Relationality

as it is brought into constellations and forms new relations between people
and things. Beyond the immediate interaction, design with AI intervenes
conceptually and materially in constellations Coulton and Lindley (2019) of
humans and objects. Designers must navigate the increased agency and depth of
interaction that intelligent systems bring, and the changes in the way that we
understand and relate to technological systems.

Wider Implications of AI:

as it affects social structures and people’s lives outside the immediate
interactions. Concerns about the implications of systems are not new, but AI
and data driven systems that are built through processing large amounts of
data about people bring new and subtle ways in which they can be unfair or
unjust, more blurring of responsibility, and more potential unintended
consequences at scale.

For the work at hand, we are interested in how these levels relate to design
education, in particular how students start to engage with AI as a design
material. To create a broad coverage, we looked at creating methods that could
create engagement with the specifics of working with AI systems on these three
levels, as well as balancing the educational concerns of developing a better
understanding of and facility with the technology, and encouraging critique.
Based on this, we created a set of ‘design exercises’ - rapid, experiential
engagements that draw on the theoretical developments above but can be carried
out productively in the context of conceptual development and prototyping
interactions with AI systems (Figure 1).

Figure 1: Situation of the methods underdiscussion across two axes: i) the
level of consideration, from direct interactional afforances, through building
relationality out to wider implications and ii) the balance between developing
facility with the technology and critiquing it’s uses.

##  3 Study

###  3.1 Course and context

The context of this study is a one-semester (20 week) design and prototyping
course for first year Masters students in the ‘Design for Interaction’
programme at TU Delft. All students in the course (n=100) have a design
background with a mixed range of computational skills, from no technical
knowledge to beginner level in software engineering. The students were grouped
by the course coordinators into 28 teams, and coached by 7 experienced coaches
from the Industrial Design Engineering faculty.

The course is structured in three stages, with student-teams of 3 or 4
students working 13 per week on their design project (Figure 2). They worked
on design briefs that asked them to speculate about near future interactions
supported by technology. Many of these briefs were provided by client
companies, for example new forms of human-vehicle interactions, possibilities
for more sustainable cooking through smart kitchen appliances and pervasive
computing in hotel rooms. The students had little to no pre-course familiarity
with Machine Learning and AI methods, theories and tools, however most of the
coaches had at least some experience with these technologies.

The main learning objective of the course is to introduce students to various
ways of prototyping with interactive technology. Students were asked to design
within the context of the client company, creating and testing a new iteration
of their prototype each week in discussion with their coach. They were
prompted to draw on some form of AI or ML, although technical capabilities
could be acted out rather than implemented in code. The course ran in three
phases: a “First Shot” familiarisation with AI and technology, “Iterating
Forward” to develop concepts and “Polishing Up” the final ideas and prototypes
(Figure 2), with an exhibition at the end of each stage. Client companies were
invited at the end of each stage to provide feedback to the student teams,
organised in the form of an exhibition with interactive prototypes.

Figure 2: Course Structure; In the first stage (4 weeks) students were given
context on AI and ML, and hands-on engagements with AI technology were
provided through a series of workshops with existing tools (Edge Impulse,
Teachable Machine and Voiceflow). At the end of the first stage, teams
presented multiple ideas demonstrated in multiple early prototypes. The second
stage introduced lectures covering AI capabilities, Human-Agent partnerships
and the conceptual shifts mentioned above as the students developed their core
concept, leading to a second exhibition of interactive prototypes. The third
stage introduced the exercises discussed in this paper, as the students
refined their projects towards a highly immersive final exhibition with one or
more interactive prototypes.

###  3.2 Exercises

The intervention involved a set of 9 exercises (Table LABEL:table:METHODS).
Each design exercise was introduced on a single page, containing a title, a
short description and instructions on how to execute the exercise, a
background section describing the intent, usefulness and ideas behind the
exercise and references to papers and related projects (Figure 3, see
supplementary material for full set). The choice of this set of exercises was
exploratory: we derived them from a combination of existing design practices,
emerging work from the researchers and the theories mentioned above, through
extensive discussion between the researchers. We aimed to have a spread of
exercises across immediate interactional affordances of AI, mid-level human-
machine relations and concerns about the wider implications of AI, as well as
across developing fluency and supporting critique (Figure 1, description in
Table LABEL:table:METHODS, details in Supplementary Material). Some of the
methods were pre-existing explorations, some had been used extensively, and
some were adaptations of existing techniques to fit the AI context or the
autonomous format. There was a strong focus on activities that could be
performed relatively simply by students, that were experiential, and that
would work across a range of topics and levels of technical accomplishment.
Each exercise was intended for application to an existing project, i.e. not a
brainstorming or early ideation tool, but a way to develop existing work.

Figure 3: Example of the exercises, showing 1) title, 2) expected time 3)
suitable project types 4) process 5) custom illustration 6) background, 7)
references and example projects (full set of exercises in Supplementary
Material)

###  3.3 Execution and Data Collection

Towards the start of the third stage, after feedback from the second
exhibition (Figure 2), a half-day workshop was organised in which all student
teams were introduced to the 9 design exercises with the aim of refining their
project. This timing was chosen primarily for educational reasons – the
methods here were designed to help develop and refine existing ideas, rather
than generate new ones, so we waited until the final stage of the course. This
timing was the subject of some discussion – see Section 4.1.2

This half day workshop allowed each group to execute one or two of the design
exercises for their design project. Output of the half-day workshop was
captured on A3 templates, including a questionnaire with some first prompts on
the usefulness and effectiveness of the exercises applied. The half-day
workshop was setup to be executed autonomously by the student teams, selecting
the exercises themselves, with lecturers present to observe and assist when
necessary. All output materials of the design exercises during the workshops
were collected afterwards.

Two weeks after the workshop we invited each team to select a representative
to take part in a one-to-one semi-structured interview to discuss their
experience and the effect it had on their project. In order to minimise
educational disruption at a busy time and limit the possibility of coercion,
we did not attempt to get full coverage, but allowed self selection by the
students, in return for a €20 contribution to the teams prototyping budget.
This led to 12 out of 28 teams participating in the interview. We interviewed
all coaches the week before the end of the course ($n=7$) to see what effect
they had perceived on the students work. Interview questions and structure can
be found in the Supplementary Material.

###  3.4 Analysis and evaluation

The interviews were audio recorded and both the interviews as well as the
output materials from the design exercises were transcribed and analysed by a
team of 7 researchers. We inductively coded both the written materials and
interviews with students and interviews with coaches. We conducted
collaborative thematic analysis: the coding team collectively familiarized
themselves with the data and defined a shared coding scheme. At least two
members of the team coded each of the transcribed materials using this scheme.
Finally, coded materials were collectively discussed to synthesize insights
into key themes, framed by the three levels of engagement with AI discussed
earlier.

##  4 Findings

Our findings are structured in two parts which build on both the A3 worksheets
($n=28$) and the student and coach interviews ($n=12$, $n=7$ respectively).
While the analysis of the A3 sheets revealed recurring topics and common
themes, the interviews revealed in-depth insights about what the students took
from the exercises. The first part (Section 4.1) covers the execution of the
methods: which ones were chosen and how they were perceived and valued by the
students. The second part (Section 4.2) describes the links made to AI and
machine learning at the interactional and relational levels as well as wider
implications. In all cases, comments from student interviews are marked as
[p⟨id⟩] and those from coaches as [C⟨id⟩]; extra context about the project
that the quote relates to is given in square brackets. The students who
participated in interviews were working on projects around: comfort and
behavioural encouragement while driving, as well as behaviour modelling and
matchmaking (for Ford); collection of data while surfing and intelligent ski
clothes (for O’Neill); smart objects and energy manifestation in hotel rooms
(for Citizen M Hotels); photography for reconstructive surgery (for Erasmus
Medical Centre); and speculating on spirituality and life coaching with AI
(for the DCODE project).

###  4.1 Method Execution

To build context about the way the exercises were carried out, we give a
quantitative summary of students’ opinions of their project before the
workshop, and their evaluation of the clarity of the exercises. We then look
more qualitatively at two themes: students’ sense of the relevance and overall
evaluation of the methods, and an analysis of the ways in which they found the
methods useful. Table LABEL:table:USAGE summarises the number of times each
method was and provides key quotes for their use in these four areas: concept
development, detailing interactions, understanding AI and supporting
reflection.

####  4.1.1 Quantitative self-assessment and Clarity of the Methods

Analysing the worksheets, we looked into how many times each method was used
as well as the perceived clarity of the instructions (Table
LABEL:table:USAGE). 20 of 28 groups carried out two exercises, with the
remaining 8 carrying out only one. Counting responses to questions about their
projects where a value greater than 0 was given (Figure 4), 16 groups (0.57)
felt their project critically investigated technology; 12 (0.43) were solving
real-world problems; 15 (0.54) made use of AI qualities; 22 (0.79) engaged
with complex relationships and 16 (0.57) intended to consider the wider
implications of their work.

All of the methods were rated as clear (Likert scale $-3=$”very unclear”,
$3=$”very clear”, $m=1.5,sd=1.0,m_{\\_}{min}=1.0$), with only a single
instance being rated negatively. This indicates that students felt they
understood the purpose and structure of each method.

Figure 4: Student response (per group, n=28) to questions about their
orientation and their project scope. Answers are on likert scales from ‘Not at
all’ (-3) to ‘It’s the core of our project’ (+3).

####  4.1.2 Relevance and situation in the course

Most groups chose exercises to address what they considered unexplored in
their projects, or in some cases, even limitations of their concepts. For
instance, Metaphor Shifts was picked to “look for something else that better
describes [their project]” [p1] or “finding a nice metaphor [to make]
interaction with the AI more empathetic to the user” [p15]. Roleplaying AI
Networks offered the hope to be “really precise and defined in the personality
that we were gonna give the AI [that was deciding people’s futures]” [p27],
and Uncertain Interactions was chosen to help “map out all the responses and
interactions that we were not considering before” [p19] in a multi-object
interaction. Other groups saw the exercises as a more general way to “check if
we were in the right direction” [p1], “plan ahead like the possible problems”
[p6], “have a discussion point […] instead of just everybody thinking in
different directions” [p15] or more radically “just start again and we go
somewhere else” [p6]. Some methods were explicitly avoided as the groups felt
they had experienced the methods before, e.g. “role playing” [p19], or because
they didn’t fully understand what a method entailed [p16]. Although the
students reported that the methods were relatively clear, they suggested that
the individual differences might have impacted to some level how students
interpreted the exercises: “we’re all from different cultures. So we all
interpreted some questions differently” [p1].

There was a common response that this activity would have been more useful
earlier in the course [p2,p5,p16,p19,p27, …] and it could have helped
generating more prototype ideas [p23]. Part of this was due to the sense that
the activities felt like “an ideation – like an inspiration activity” [p2]. As
students saw the exercises as tools for divergent thinking, they would have
liked to use them for ideation in close connection to the prototyping
experimentations in the first period of the course [p5]. Others were concerned
that the moment when they did the exercises was their time to “optimize the
prototype for the exhibition” [p16] and wanted to spend all of their time in
making. In contrast, feedback from coaches on the timing was more positive:
“To have sort of a zoomed out exercise at that point is I think a very
powerful thing to do.[…] if you don’t know where you’re heading, then all
these things, I don’t think they will help you. […] So I wouldn’t move it. ”
[CIa]. Several students echoed this perspective emphasising how it helped
their process, e.g., “because we were kind of stuck with our idea in general”
[p7], that it helped “think of more details” [p20] around a developed idea.

Overall, there was a positive attitude towards the activities, even from
groups that were initially suspicious: “we were quite surprised because we
were thinking ‘Ohh workshop again. […] What’s going [to come] out of it?’ […]
And then in the end there was actually some things that really helped us.”
[p1]. Some negative responses (4) revealed student’s concerns on carrying out
the exercise properly [p5], or spending too long on one interaction [p19].
Some (2), had a hard time finding usefulness in the experience [p15,p16] , as
they were already familiar with the methods, as “wouldn’t say it brought me a
new understanding because the metaphor is something we [already] had” [p15].

####  4.1.3 Perceived Utility of the Exercises

Many students saw the exercises as a form of “ideation, like an inspiration
activity” [p2],“ kind of a brainstorm” [p7] that can help “to get a better
idea” [p7]. Several groups noted that they came up with different and more
interesting ideas [p5] and that they “could use this new inspirations” [p23].
The methods were seen as useful for sharpening projects and defining practical
next steps, such as planning for when things went wrong, or checklists of
common concerns. Metaphor Shifts was particularly generative of new ideas
[p5,p16]. Beyond this, the methods were seen to help in the following four
areas:

##### Conceptual development:

The methods supported articulation and “helped us get our story right, like
the overall purpose of the concepts” [p2], to develop “a better detailed new
metaphor” [p5] and to “make [a] choice in what we wanted” [p7]. The benefit of
gaining more conceptual clarity was also mentioned by some coaches [CLu].
Uncertain Interactions was seen as useful for mapping out the edges of a
concept, so that students could easily get into details and next steps [p6].
The methods also helped grounding ideas, asking whether the concept “can also
work on the AI or do we need some future technologies that are not there yet
to make it real” [p1]. In some specific instances, the exercise helped to
“start thinking about time” [p15], or to “find new ways of taking the same
idea and spreading it” [p5]. The exercises helped to get an overview of things
that students should think about [p1], making implications concrete and
graspabale, in a way that is “so in your face that you don’t even think about
the fact that it will be in the future” [p7].

##### Refining Interactions:

Many groups came out with a more refined idea about how their conceptual
interaction should play out, as the exercise “asks you to go into parts that
maybe you don’t want to explore” [p6] and make projects more well rounded. The
activities also helped students define interaction contexts better. Groups
felt invited to “draw [AI] already in the context” [p15], and to “think about
the interaction with some of the objects in the [interactive hotel room]
scenario” [p19]. Refinements also pushed them to account for the potential
meaningfulness of the projects, to “clarify intentions” [p20], and anticipate
outcomes, e.g., “what happens if the user doesn’t understand what [the smart
objects are] talking about” [p19]. The experiential nature of the exercises
helped to “translate something abstract as ”being challenged” or ”supporting”
[good behaviour while driving] to something actually tangible” [p2], to think
into the “aesthetic experience” [p15] of AI where the “metaphor [of ritual
cleansing for data collection] helped to think about materials as well” [p15].

##### Reflection:

The workshops were seen as a moment of reflection, a break from the “many
layers in such a project” [p25] to focus on particular aspects. This could be
on a technical level for the groups who “never really took the time to think
about AI” [p2] or more interactional when they “stopped to think about this
character sort of thing” [p27]. There was developing a “critical lens, in
terms of moral responsibility” [p25] and “seeing how important this is, to
acknowledge the mistakes, to be trustworthy” [p6]. Beyond the initial
designerly sense of responsibility, they engaged with broader factors
contributing to “moral responsibility for an AI system [that encouraged
spirituality]” [p25]. Overall, the moment for reflection was seen positively,
developing aspects of their work that were not thought through, and a sense
that “confidence comes once you […] manage the critical points” [p6] of the
interactions.

##### Understanding AI:

The workshop improved the confidence of students about working with AI, as “
before the course it was just like ‘I don’t know how to use an algorithm to do
something cool’ […] and this makes it kind of [makes] everything just specific
in one one workshop” [p2]. This was often not based on a deeper technical
understanding of algorithmic operation, but on a thinking about how the AI
would relate to things around it. Some groups ended up “actually using more AI
because of this [workshop]” [p7], with confidence coming from “now that we
know what’s going wrong, and we know how to respond to that” [p6].

Figure 5: Conceptual Map of students’ reflections on the benefits of the
methods grouped across the three levels of AI engagement: interactional
affordances, relationality and wider implications.

###  4.2 Key themes for engaging with AI

Now we discuss findings in relation to broader theoretical developments in
HCI, according to the three levels we have identified earlier: interactional
affordances, relational questions and wider implications. An overview of these
findings can be seen in Figure 5

####  4.2.1 AI Interactional Affordances

Students found that the workshops illustrated that “there are actually a lot
of possibilities with AI” [p7], beyond the tutorials at the start of the
course, and that working through the experiences left them with a “whole list
of things that [AI] could say or do” [p2]. They already had some experience
with particular topics, but this opened up a greater sense of how these
possibilities could be deployed in relation to their work. This did not always
change the concept of the interaction, but did provide a confidence that many
interactional designs could potentially be realised.

##### Data and meaning:

Role playing helped with sensitising the students to the role of data in AI
driven systems, questioning “where is the AI getting the information?” [p27]
both generally and through very detailed questions of “where we’re gonna put
[the camera that understands human-vehicle interactions]” [p5]. The coaches
noticed the attention to physical detail as well, seeing development of a “way
of bringing the data and looking at it and experiencing it” [CIa]. There were
moves to think about how to work with people in wheelchairs, and what it would
mean to “recognise these things and build the dataset” [CLu], as well as the
broader question of how “how [the collected data] can be meaningful for you as
a person” [CIa].

##### Character and expression:

The experiential nature of the exercises was, unsurprisingly, suited to
engaging with designerly questions of the character and expression of the
autonomous parts of the system. Students notice the possibility that they
could be “really precise and defined in the personality that we were gonna
give the AI” [p27], questioning default assumptions about how the system might
respond. With conversational agents, it was noted that “there is a lot of
space between the yes or no” [p19], but also that working probabilistically
could smooth out interactional challenges, so that humans “don’t have to
become machines ourselves” [p6]. The possibility arose to create pluralistic
engagements that gave “different answers based on different characters and
based on different situations [for patients undergoing reconstructive
surgery]” [p23]. This opened the possibility of making stronger bonds with
users, and working on an emotional level, which we will return to in the next
session.

##### Interactional Limitations:

In general, the coaches were more sensitive than the students to the potential
limitations of technology, for example noticing when “the way they acted out
looks good on screen but it doesn’t reflect the deeper issues with
understanding […] Whereas if you use a conversational AI model I think you
will run into a lot of problems that are hard to act out” [CGi]. It was clear
to them that some of the enactments would require sophisticated behaviour that
could easily be glossed over with WoZ techniques, and they questioned whether
the exercises could also point to these moments of glossing, or help notice
points of complexity. For the more technically realised groups, the coaches
noticed students working around limitations of the technology, where “ it was
not very good at detecting facial expressions, but you made a hand gesture”
[CGi] that conveys emotion purposefully, leading to a rethinking of the
interaction schema.

####  4.2.2 AI relationality

Students felt that “[t]here are so many layers in such a project, where you
are constantly building” [p25] and noted that the workshops took them into
some of the complex, multilayered aspects of working with interactive AI
systems.

##### Deeper relationships:

Following the theme of character above, the workshops prompted students to
think about the ways that humans related to the things being designed, giving
an impetus to “think more in an empathetic way” [p15] about the end users and
what AI mediation would “mean for a human to human relationships” [p25].
Roleplaying the situation with the device helped to look across some of the
other people around the interaction, for example working with an system that
was helping to take medical photos for reconstructive surgery and seeing “the
relationships between the AI [and] doctor, assistant to friends or to your
family members” [p23]. This was partly driven by a sense that the AI systems
could interact in increasingly human-like ways, with metaphors like “a friend
in your car” [p5], or a pet. There was a move to look at some of the longer-
term relationships formed and the bonds that people made with AI systems.
Students developed increasingly anthropic concerns from whether “ people feel
at a loss after they need to give [their smart mirror] back” [p23] at the end
of a process, to questions of developing care and love relations to the
objects.

##### Creepiness and agency:

Interestingly, some of the more than human metaphors helped students to about
when agency was troubling, and were “open to more scenarios that we didn’t
see” [p20]. Manifesting home energy use using a metaphor of ‘fireflies’ caused
a concern that it “will follow you through your room as a dog follows you.
This might be kinda creepy. So what if they [users] don’t want to be
followed?” [p20]. The potential intimacy of relations with a vehicle raised
concerns about “how intimate your interaction with your decentralized car be”
[p5], and how “if you’re driving and you’re stressed and you somehow just get
like this random unexpected hug from your car” [p5] it would cause emotional
discomfort. Even when autonomous behaviour was not emotionally invasive, there
were concerns that “sometimes the [smart hotel room] objects want to speak for
themselves but at the same time you don’t want to scare the human that is the
guest in this room” [p19].

##### More-than-human relations:

Going beyond metaphors of caring for cars as one might care for a dog, coaches
noticed that students would “use design as a medium to amplify the voice of
nature” [CIo] or “activate […] energy consumption in a different way than just
a tool” [CLu] in their AI mediated interactions, making a shift to both non-
human perspectives and the idea of technological mediation rather than tools
for particular outcomes. The students looked into new relationships that might
emerge, e.g designing “clothes to learn from every person that wears them to
[and] grow its own personality” [p16]. The coaches noticed the roleplaying
aspect of the exercises prompted critical reflection into the scenarios and
relationships at hand, including noticing “that the setting that they were
imagining and the role of the AI within that setting was not a very good fit”
[CGi]. Students found the practice useful for articulating what their vision
for the future of human-AI relationships at individual and societal levels
ought to be, including questions of governance and democracy.

####  4.2.3 AI and wider implications

##### Responsibility:

While methods targeted at interrogating control (Meaningful Human Control)
explored agency and control, other methods (Metaphor Shifts) still gave space
for these questions to arise. Students reflected on “considering moral
responsibility for an AI system” [p25] within the creation process; and the
coaches noticed that the workshops provided “a way to create distance and look
at the project from a different perspective” [CIa], to re-evaluate the project
beyond the immediate concerns of development, with a sense that it was the
designer’s responsibility to make sure that purposes and potential issues were
clear upfront. Some students found that the workshops made the idea that
people might misuse their system concrete, so for a friendly car system they
“gave ourselves some guidance for the next steps, [not] for concepts […], but
more like OK, this is now a checklist that we need to put next through concept
every time to make sure we think about this” [p1]. Responsibility often came
through thinking through what might go wrong, with evidence of ‘zooming out’
through the exercises, to think about what would happen if these systems were
widespread, and their failure modes constantly present for users.

##### Consent and Privacy:

Several students mentioned issues around consent; while some felt this was a
core part of their existing work, others found that discussion around the
workshops was what they needed really understand the implications, and “a
solution for something that [is] difficult to think about” [p1]. Groups
managed to “dig deeper in that space” [CLu] and better manifest the issues
that they were already dealing with, and in some cases this meant that
“[consent] was actually a very explicit part of their final concept and that
was not at their departure, I think, was driven in part by going a bit more
speculative than they were imagining at first” [CGi].

##### Vision and Criticality:

A common point from the students was that these workshops helped to think
beyond the initial concerns of prototyping and into the multi-layered nature
of the projects, not just the around AI responsibility but that “it asks you
to go into parts that maybe you don’t want to explore” [p6] and rethink the
purpose and shape of the project itself. Coaches were mixed about whether they
saw changes in the level of critical thinking around the workshop, with some
noticing no change, some a progression, and some seeing a strong difference
where critical thought was brought in. Some of these were tradeoffs: “They
became more critical. They were focussing more on the experience, but I’m not
sure they were more engaged with the AI” [CIo]. However, others noticed
engaged with the human AI relationships, questions of datasets and the role of
the project as critique, and “really thought about it, how you negotiate with
the machine and how much freedom you should have and how much agency you just
have” [CMa].

##  5 Discussion

In this discussion, we address some potentials for developing the exercises,
and reflect on our initial research questions. We discuss how the AI exercises
address the current methodological gaps and, more broadly, how this work
contributes to a larger program around design, HCI and AI, nurture a
distinctively designerly AI culture.

###  5.1 Effectiveness and Future Work

The exercises were seen as effective overall, although they could further be
improved through use, observation and iteration. They produced thoughtful,
socially engaged responses, but to a large extent remained far from the rapid
and technically grounded results generated at the beginning of the course,
when students were provided with tutorials focused on learning a particular AI
technologies. As an example, despite the deep technical grounding of Uncertain
Interactions, most student responses did not get deep into the specifics of
model output and how to make use of it. Future versions of the exercises could
look to bridge this gap, as could their use in more technical contexts, were
models were really being trained and deployed. There could also be support to
help students to decide which concerns to prioritise – for example, worries
about people falling in love with their AI devices might not be the key
problematics of the technology as created. While this prioritisation is
arguably a part of general design practice, having concrete examples to
contextualise the discoveries would be helpful. Practically, most students
were relying on pre-built models and ‘Wizard of Oz’ setups Browne (2019);
Dahlbäck et al (1993) that used human action to simulate complex behaviour.
This limited the utility of data driven exercises (e.g. Poor Datasets) and fed
into a focus on the anthropomorphic possibilities of AI. This also led to less
engagement with the possibilities of new forms of human-machine interaction
than we might have hoped for.

####  5.1.1 Timing and Situation

The time that the students had to execute the exercises was short, which may
have limited the potential for deep reflection and thoughtful practice. While
the students still had access to the methods, few groups chose to make use of
them, so there is space to explore more prolonged engagement. The positioning
in the course was somewhat contentious, with many students feeling the methods
had been introduced too late (Section 4.1.2) - this is coupled to their
assumption that the methods were there for concept development and ideation.
However, the overall feeling from the coaches was that the timing was sensible
– it provided a way to zoom out around existing concepts and add richness.
Part of this divergence of opinion is part and parcel of process based
education – there are often different views from within the process than
outside it. However, it does point to the need for a stronger sense of what
one can expect from the methods, and an indication of when and how they could
be productively deployed.

####  5.1.2 Choice and Range of Methods

This initial set of exercises was based on a particular set of theoretical
ideas; it is clear that other theories and concepts could prompt additional
methods, and other methods could be derived from the theories used. There is
certainly no shortage of candidates, whether agential cuts Shotter (2014)
provide techniques to divide up complex systems and consider multiple
boundaries through more or less embodied encounters Vagg (2022), or ideas of
cyborg intentionality Verbeek (2008) lead us to enact parings with composite
possibilities Rapp (2021), introspection provides a lens to think about
relations between AI and lived experience Brand et al (2021). Methods with a
clear technical genesis would offer immediate experiences that are deeply
embedded in and shaped by the technology, for example deliberately misusing
vision algorithms van der Burg et al (2022) or using computer vision as a site
of enquiry Malsattar et al (2019). We see this as the start of a collection of
ways to engage in this area, which will grow over time. Additional exercises
might emphasize different parts of the design process and different modalities
of experience as well as introducing new theories or grappling with particular
qualities of AI.

####  5.1.3 Applicability

In terms of subject matter, the exercises were applicable to a range of
projects across autonomous cars, robots, Internet of Things, hospitality and
so on. They also helped with a range of issues, from shaping overall concepts
to detailing important parts of the interaction. The application here was
somewhat particular: the middle stages of an exploratory, creative prototyping
brief. We would expect that the methods can be used in other processes and
different levels of technical fidelity. In fact several of the methods, such
as roleplaying AI networks and Thing Ethnography of AI systems are likely to
give better results as the project is more developed and the context is
stronger. Others, such as Poor Datasets are likely to be more useful with a
developed technical implementation, while Uncertain Interactions could help
with ways to create interfaces around probabilistic models in deployment.

###  5.2 RQ1: Conceptualising and prototyping practices with AI

The exercises illustrated some of the issues that students have when carrying
out prototyping and conceptualisation with AI: the need to deal with
uncertainty, the possibilities of more human-like interactions but less
clearly defined capabilities, the need to hold multiple levels together. This
clearly asks a lot from designers, especially in this case, where many of them
did not have strong electronics and coding skills before the course. The
experiential Hemment et al (2022a) and enacted Elsden et al (2017) aspects of
the workshops were helpful to navigate this terrain, as the subjects of
discussion could be played out in the group, adding to the sense of
tangibility and refining how interactions should unfold. The interactional
focus of this work makes it distinct from ideation tools such as AIxDesigns
ideation cards AIxDesign (2022) which focus on conceptual innovation, or work
on developing user experiences Subramonyam et al (2021b, a) which makes the
interface the primary subject of design. In line with open ended, critical and
speculative prototyping methods Malsattar et al (2019); van der Burg et al
(2022); Nicenboim et al (2020) the exercises took the students into the
relational and interactional possibilities of AI.

From the feedback, it was important to give students exercises that were
concrete enough that they could follow the steps. Several of the exercises
were close to relatively standard design practices - Uncertain Interactions
drew on the creation of state diagrams as a design articulation tool and the
idea of acting out interactions as a form of prototyping is well established
Van Der Helm and Stappers (2020). However, they were adapted to bring AI
qualities into the familiar interaction design practices, emphasizing aspects
like uncertainty, interface capabilities, distributed responsibilities and so
on. It is clear that for some of the students, the simple forms of the
exercises would have been enough – simply asking ‘what might go wrong’ and
drawing a state machine to deal with it, rather than getting into the idea
that machine learning systems produce probabilistic outputs produced useful
results. None of the students chose to work directly with datasets; this may
be a feature of their projects, as there was not much training and learning
happening, or simply a lack of attraction to the particular exercise.

There was a tendency with many of the groups to drift into
anthropomorphisation, to imagine relations as overly human Marenko and van
Allen (2016), and be diffuse about the capabilities of the technology. This
relates to thinking into some of the particular AI characteristics that we
will discuss in the next section, it is clear that prototyping will start to
take different forms. The evolution of prompt engineering as a discipline Liu
and Chilton (2022) and the potential to generate working systems from prompts
(e.g. Aptly aptly, 2022) indicates that new forms of prototyping are emerging.
Here, the constraints are less well defined than working with code on Arduino,
but no less present - training a TeachableMachine Carney et al (2020) to
detect a gesture has just as many concerns as using the electronic gesture
sensor built into the Arduino BLE Sense the students were using, but the
failure modes play out differently, and a different set of prototyping
practices are brought to bear. The multiple viewpoints contained in the
exercises here – people, things, datastreams, algorithms, networks – help to
tease out the parts of interactions to prototype. Enacting these possibilities
makes it easy to fall into broad, fuzzy, anthropomorphic thinking about what
systems might do; the challenge for developing new forms of prototyping is to
temper this with a grounding in the capacities of the systems being designed,
and to engage with the human-like affordances of technology, without missing
the new machinic possibilities. The exercises here helped students to clarify
their concepts, move forward with their prototyping, and develop ideas about
the responsibilities of creating AI systems, while maintaining designerly
concerns of materials, aesthetics, function, fit to context and engagements
with multiple actors.

###  5.3 RQ2: Grasping interactional, relational and contextual qualities of
AI

Our analysis of the student responses in relation to the current paradigm
shifts in HCI, shows that interactional, relational and contextual qualities
of AI could be important elements of design and AI educational programs. To
unpack this, we look at our findings in relation with wider notions of agency,
human-machine relations, and understandings of AI.

####  5.3.1 Agency

As noted above, AI is a tricky term, but a lot is contained in the ideas of
agency which it can develop, in particular around ‘non-humanesque agencies’
Hildebrandt (2020). Some of these agencies were clear prompts for our work: a
failure to recognise certain kinds of people as being human Buolamwini and
Gebru (2018) shapes the inter-agences between vision systems and people; the
ability to make decisions rapidly and constantly gives a sense of autonomy,
but one which differs both in character and meaningfulness from the one of
humans. Several of the exercises were aimed at interrogating these questions:
‘Be the AI’ prompted a reflection on exactly what the machines were doing, the
interviewing and roleplaying exercises asked the participants to feel into
what the agential possibilities were, and the more conceptual exercises
questioned what agencies and responsibilities humans had around the systems.
This helped participants to think about “what are the actual choices that we
are gonna make or what part of the interaction are we gonna do ourselves and
what part is the machine gonna do?” [p16] – a key part of getting past the
myths about AI capabilities Natale and Ballatore (2020).

Much of the thinking reported around issues of agency had to do with “how it’s
gonna be alive for people” [p20] \- the clear, animate, characterful side of
agency. This surely has roots in some of the roleplaying methods. The shape of
the collective roleplaying exercises was informed by an increased emphasis on
co-performance, as humans were brought in to act out what the smart
technologies would do, and notice possibilities for more shared agency,
whether co-learning with the AI or finding ways for objects to speak for
themselves.

####  5.3.2 Human Machine Relations

Some of the exercises prompted students to position AI in relation to humans
and non-humans. Thing Ethnography of AI systems instructed the students to map
the ecosystem of the thing and its touchpoints, and reflect on who and what
interacts with the concept. In Conversations with AI, the students were asked
to enact an AI agent. Metaphor Shifts asked students to design systems based
on a particular metaphor and then compare it to others. These exercises
highlighted the relations of humans and non-humans within AI systems Coskun et
al (2022); Nicenboim et al (2020). They did that by decentering the designers
perspective, to consider more actors and interactions that go beyond one user
and one device Verbeek (2020). They also invited students to relate to AI not
only as a tool, but as a social agent that shapes people’s lives. In students’
prototypes, intelligence, as well as responsibility, were not seen as a
properties of machines alone, but shared between humans and artificial
partners. Similarly, uncertainty and unpredictability were ‘collaboratively
curated’ to ‘imagine forms of digital interaction’ Marenko and van Allen
(2016).

The findings show that the exercises helped students to expand their concepts
to account for the ecologies around them, especially when their projects were
centred on particular embodiments, for example extending from a plant pot to a
community of plants, and looking at the plant-plant relations as well as the
plant-human ones. The exercises also helped them acknowledge other people
beyond the immediate users, thinking into how would they relate to the system,
and what are the responsibilities that the user, system and designer have
towards them. Furthermore, thinking of their concepts in relation to humans
and non-humans, created awareness within the students on the human labour that
is implicated in sustaining AI systems Sinders and Ahmad (2021). The kind of
metaphorical social relationships the AI had with others ultimately influenced
the designs: when the system was cast as a friend, it was seen, designed and
conceptualised differently from when it was cast as a pet.

####  5.3.3 From Explanations to Understandings of AI

One of the current challenges in the design of AI systems is how to support
people in understanding them, especially when used to make autonomous
decisions or create knowledge. AI explainability is especially challenging
when based on deep learning models, given that some of the paths that AI
systems use to give recommendations are not interpretable Ehsan and Riedl
(2020), and the source of many generative outputs is complex (e.g. Kovaleva et
al, 2019). While understanding ML in its technical sense is important, recent
approaches in the explainability of AI have pointed at other ways of
understandings which are not based on technical explanations and instead,
promote experimentation, challenging boundaries, or promoting respect
Nicenboim et al (2022); Hemment et al (2022a); Seymour et al (2022). The
findings expand the agenda of Explainability of AI by illustrating and
unpacking particular design engagements with AI that go beyond mastering ML
technical capabilities. This points at particular aspects that are important
in the kind of understandings that designers might need to gain of AI. Some
particular aspects that helped designers understand AI are exercises that
could prompt reflection into the affordances, relations and wider implications
that those systems might have. Those engagements were not based on learning
how to code ML models, but on experimenting with changing perspectives,
provoking failures, enacting behaviours, and drawing schemas. These tactics
could become part of a new agenda for supporting designers in understanding
AI, especially one that is aligned with theoretical developments in HCI such
as the posthuman turn Lindgren and Holmström (2020) as well as practical
developments in design (such as methods used in critical, speculative and
adversarial design DiSalvo (2015); Irani and Silberman (2014); Bozic Yams and
Aranda Muñoz (2021)).

###  5.4 RQ3: Critical design perspectives while engaging with AI as a socio-
technical system

While the exercises helped the students to develop their projects (from
ideation to conceptualisation and detailing) they especially illuminated and
modulated changes in the students design processes in relation to the
sociotechnical aspects of AI systems Crawford (2021). The exercises supported
students in reflecting on the role of AI within their concepts, in being more
specific in what kind of aspects of AI are present, and developing a critical
design perspective on AI around values of responsibility and agency.

Designing with AI as a socio-technical system, means acknowledging that it is
not only a technical domain, but also entangled with social practices,
institutions and infrastructures, politics and culture. AI “is both embodied
and material, made from natural resources, fuel, human labor, infrastructures,
logistics, histories, and classifications” Crawford (2021). This is not an
entirely new perspective – AI has long been considered a material practice
Agre (1997), but there is a need to consider the interaction between humans
and machines as part of broader societal contexts, and the broader discursive
settings in which AI is socially constructed as a phenomenon with related
hopes and fears Lindgren and Holmström (2020).

From the findings, it seems the exercises provided a space for students to go
beyond the immediate concerns of a rapid prototyping session and engage in
reflective practices that can position AI within its broader societal
contexts. It is clear from many of the responses that the workshop carried out
here provided a moment to reflect. Some of this can be simply ascribed to the
sole fact of having the intervention – a space that prompted further thought.
However, some of the exercises were more specific triggers for reflective
engagement, with Meaningful Human Control and Resisting/Subverting AI asking
students to explicitly consider critical perspectives. There was evidence of
‘reflection-in-action’ Yanow and Tsoukas (2009), moments where in the midst of
carrying out experiential exercises around the prototypes they ‘ma[de]
previously implicit assumptions about the work explicit’ Wegener et al (2019).
Where it had previously seemed that an insect swarm would give a warm sense of
companionship, looking at the technological sense of surveillance and
following revealed a darker possibility for the user; the idea that a
supportive hug came from technical rather than human agency was found on
reflection to be disturbing.

Some of the exercises prompted a sense of ‘zooming out’ Nicolini (2009), to
consider wider networks of things and people, and this zooming out was part of
the students move towards more empathic design. This touched on the temporal
Pschetz and Bastian (2018) aspects of their design, as they thought not just
about interactional moments, but the slower unfolding of relationships over
time. Overall, there were many moves to develop a sense of criticality within
their design: the AI oriented methods created viewpoints for considering the
role of technology and its possible overreaches from pragmatic and whimsical
perspective, in line with the ‘ongoing practical, critical, and generative
acts of engagement’ Suchman (2020) that build a responsibility for the things
being designed.

###  5.5 Beyond Education: Nurturing designerly AI cultures

We started with the aim of helping students to grasp enough qualities of AI to
adapt their processes and conceptualisations accordingly. The study
highlighted that there are distinct ways to engage with AI that are
appropriate to our setting, where the culture and practices of designers
centre particular ways of working. In this section we discuss the aspects of
our work that nurtures this designerly AI culture.

The use of AI technologies varies by field. If we look at AI in terms of the
enabling technology and the culture that surrounds it Caramiaux and Alaoui
(2022), some of the differences and parallels with the use of AI in design
become clear. There are common moves to cast AI as a creative partner Llano et
al (2022); McCormack et al (2020) within music, as a solution for optimisation
Noor (2017) within engineering, as a formalisation and purification of human
thought Chiusi (2020); Singh et al (2019) in decision making organisations and
so on. Within design, the places that AI might sit are being negotiated. Do we
bring it into the process as a sparring partner for ideation Simeone et al
(2022) or a source of creative inspiration Yun et al (2022)? Do we use it to
re-understand the world through divergent practices Malsattar et al (2019)? Is
it a new computational capacity for which we have to develop new UX practices
Subramonyam et al (2021b)? Or a boundary object whose politics need critique
Crawford and Paglen (2019); Lyons (2020). All of these are within the remit of
design. What we are interested in accenting here is the possibility for a
designerly culture around the use of AI technologies, whether in processes,
outcomes or critique. Just as a shift from explanation to shared understanding
Nicenboim et al (2022) speaks to a relational, experiential mode of
engagement, the exercises here create those experiences, and give ways to pick
up, tangle and hold those relations. We suggest there are three key features
of the methods that support this: experientiality, pragmatism and reflection.

The experiential nature of the methods appeared to be key in bringing in
different perspectives on existing work, from noticing potential implications
to uncovering new actors and interrogating positive ideas of agency. In this
prototyping oriented style of working, enacting and dramatising possibilities
helped to grasp concepts. This was particularly relevant to working with AI
systems, where the level of agency expected of the technology is high, so
vitalising it makes intuitive sense.

Secondly, the pragmatic nature of the exercises, distilling complex ideas down
to a set of steps to explore supported critical discussion. Rather than
starting from the theory, students were able to develop grounded experiences
and respond to them. This led to practices such as developing their own
checklists for responsibility as well as rethinking interactions based on new
metaphors for the relations between technologies and humans.

Finally, the exercises all point to building the skills that a reflective
designer in AI might need – “Perhaps the thing that they have in common is
that they make you reconsider what your intention was and how that intention
has manifested itself into the concept” [p16]. As such, they are distinct from
technical support, even technical support tailored to creative practitioners
AIxDesignComm (2020), but look to build bridges from more than human thinking
Coskun et al (2022); Coulton and Lindley (2019); Giaccardi and Redström
(2020); Nicenboim et al (2020) towards technical practice.

By providing this kind of multiple toolbox, we contribute to shaping the
emerging AI-Design culture as something distinct from the technical,
scientific, artistic and socio-legal cultures that are relatively well
established. Further, we believe that this practice of grasping AI can be
useful beyond the classroom, a powerful and versatile support for design
professionals to meaningfully engage with the development of intelligent
systems.

##  6 Conclusions

There is a growing need for designers to engage with artificial intelligence
and machine learning in their practice as it becomes integrated into the
functioning of the physical and digital systems that they design. A particular
challenge here is how to carry out ideation and early stage prototyping around
AI/ML, when the exploratory nature of the work makes it impossible to invest
much time in detailed technical understanding of particular algorithms or
systems. At the same time, the technical possibilities of emerging algorithms
can exert an overly large pull on designs, artificially narrowing the solution
space and drawing away from the needs and qualities of the interaction.

To develop the potential for designers to engage meaningfully in this space,
working from an educational perspective, this paper introduced a series of ‘AI
exercises’ informed by recent theoretical developments in third wave HCI to
help students grasp AI as a socio-technical system. We developed three levels
of consideration for designing AI systems: interactional affordances,
relational possibilities, and the wider social implications of AI systems; and
provided methods for working at each level. Through qualitative analysis of
these exercises with a group of students, we build up a picture of what kind
of impact the interventions had on their understanding of AI and their project
development. Through the exercises, the students refined their designs and
clarified their concepts, and were able to move forwards with their
prototyping with a greater sense of confidence in their designs and
responsibility around the process. The experiential, pragmatic aspects of the
exercises helped to make theoretical ideas concrete and generative of new
possibilities, while keeping a sense of materiality and interaction with
humans. The space for reflection provided by the exercises helped the students
to develop a wider perspective on their work within the bounds of a rapid
prototyping project.

The study findings highlight ways in which experimental design exercises could
support students in understanding AI, especially considering that such
understanding needs to go beyond mastering ML technical qualities. The
exercises here helped illuminate and modulate changes to the students design
processes in relation to the interactional, relational and contextual
qualities of AI, helping students develop a reflective and critical design
perspective while responding to the key theoretical developments that are
discussed in the AI community within HCI. Through the discussion, we raise
questions of how a socio-technical view of AI, through ideas of agency and
relationality can support a designerly culture around the development of AI.

##  7 Acknowledgements

We would like to thank our course collaborators for assistance in both running
and analysing the course, in particular Ianus Keller, Aadjan van der Helm,
Tomasz Jaskeiwicz, Dieter Vandoren, Gijs Huisman, Nazli Cila and Martin
Havranek, as well as Seowoo Nam for graphic design and data collection around
the methods. Thanks to the others in the Human Centred Design department who
helped us to think about and frame this work, in particular Elisa Giaccardi.
This work was partly supported by the Microsoft Research PhD fellowship
awarded to Iohanna Nicenboim. Finally, as an piece of educational research we
would like to thank the students on the course for their hard work,
thoughtfulness, creativity and boldness.

##  8 Additional Information

On behalf of all authors, the corresponding author states that there is no
conflict of interest. The datasets generated during and/or analysed during the
current study are available from the corresponding author on reasonable
request.

## References

  * Agre (1997) Agre P (1997) Computation and Human Experience. Learning in Doing, Cambridge University Press, Cambridge ; New York
  * AIxDesign (2022) AIxDesign (2022) AI Ideation Cards. URL https://www.aixdesign.co/shop
  * AIxDesignComm (2020) AIxDesignComm (2020) AIxDesign Community. URL https://www.aixdesign.co
  * Alves-Oliveira et al (2021) Alves-Oliveira P, Lupetti ML, Luria M, Löffler D, Gamboa M, Albaugh L, Kamino W, K Ostrowski A, Puljiz D, Reynolds-Cuéllar P, Scheunemann M, Suguitan M, Lockton D (2021) Collection of Metaphors for Human-Robot Interaction. In: Designing Interactive Systems Conference 2021, ACM, Virtual Event USA, pp 1366–1379, https://doi.org/10.1145/3461778.3462060, URL https://dl.acm.org/doi/10.1145/3461778.3462060
  * Amershi et al (2019) Amershi S, Weld D, Vorvoreanu M, Fourney A, Nushi B, Collisson P, Suh J, Iqbal S, Bennett P, Inkpen K, Teevan J, Kikin-Gil R, Horvitz E (2019) Guidelines for Human-AI Interaction. In: CHI 2019, URL https://www.microsoft.com/en-us/research/publication/guidelines-for-human-ai-interaction/
  * aptly (2022) aptly (2022) Introducing Aptly that Turns Ideas into Working Apps. https://appinventor.mit.edu/blogs/hal/2022/03/21/Aptly
  * Auger (2013) Auger J (2013) Speculative design: Crafting the speculation. Digital Creativity 24(1):11–35, https://doi.org/10.1080/14626268.2013.767276, URL https://doi.org/10.1080/14626268.2013.767276
  * Auger (2014) Auger J (2014) Living with robots: A speculative design approach. Journal of Human-Robot Interaction 3(1):20–42, https://doi.org/10.5898/JHRI.3.1.Auger, URL https://doi.org/10.5898/JHRI.3.1.Auger
  * Barad (2007) Barad KM (2007) Meeting the Universe Halfway Quantum Physics and the Entanglement of Matter and Meaning. Duke University Press, Durham
  * Benjamin et al (2021) Benjamin JJ, Berger A, Merrill N, Pierce J (2021) Machine Learning Uncertainty as a Design Material: A Post-Phenomenological Inquiry. In: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, ACM, Yokohama Japan, pp 1–14, https://doi.org/10.1145/3411764.3445481
  * Benjamin et al (2023) Benjamin JJ, Biggs H, Berger A, Rukanskaitė J, Heidt MB, Merrill N, Pierce J, Lindley J (2023) The Entoptic Field Camera as Metaphor-Driven Research-through-Design with AI Technologies. In: Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, Association for Computing Machinery, New York, NY, USA, CHI ’23, pp 1–19, https://doi.org/10.1145/3544548.3581175
  * Benjamins et al (2019) Benjamins R, Barbado A, Sierra D (2019) Responsible AI by Design in Practice. arXiv:190912838 [cs] 1909.12838
  * Bilstrup et al (2022) Bilstrup KEK, Kaspersen MH, Assent I, Enni S, Petersen MG (2022) From Demo to Design in Teaching Machine Learning. In: 2022 ACM Conference on Fairness, Accountability, and Transparency, Association for Computing Machinery, New York, NY, USA, FAccT ’22, pp 2168–2178, https://doi.org/10.1145/3531146.3534634, URL https://doi.org/10.1145/3531146.3534634
  * Bowler et al (2022) Bowler RD, Bach B, Pschetz L (2022) Exploring Uncertainty in Digital Scheduling, and The Wider Implications of Unrepresented Temporalities in HCI. In: CHI Conference on Human Factors in Computing Systems, ACM, New Orleans LA USA, pp 1–12, https://doi.org/10.1145/3491102.3502107
  * Bozic Yams and Aranda Muñoz (2021) Bozic Yams N, Aranda Muñoz Á (2021) Poetics of Future Work: Blending Speculative Design with Artistic Methodology. In: Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems, Association for Computing Machinery, New York, NY, USA, CHI EA ’21, pp 1–8, https://doi.org/10.1145/3411763.3443451, URL https://doi.org/10.1145/3411763.3443451
  * Brand et al (2021) Brand N, Odom W, Barnett S (2021) A Design Inquiry into Introspective AI: Surfacing Opportunities, Issues, and Paradoxes. In: Designing Interactive Systems Conference 2021, Association for Computing Machinery, New York, NY, USA, DIS ’21, pp 1603–1618, https://doi.org/10.1145/3461778.3462000
  * Browne (2019) Browne JT (2019) Wizard of Oz Prototyping for Machine Learning Experiences. In: Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems, Association for Computing Machinery, New York, NY, USA, CHI EA ’19, pp 1–6, https://doi.org/10.1145/3290607.3312877, URL https://doi.org/10.1145/3290607.3312877
  * Buolamwini and Gebru (2018) Buolamwini J, Gebru T (2018) Gender shades: Intersectional accuracy disparities in commercial gender classification. In: Conference on Fairness, Accountability and Transparency, PMLR, pp 77–91
  * Caramiaux and Alaoui (2022) Caramiaux B, Alaoui SF (2022) ”Explorers of Unknown Planets”: Practices and Politics of Artificial Intelligence in Visual Arts. Proceedings of the ACM on Human-Computer Interaction
  * Carney et al (2020) Carney M, Webster B, Alvarado I, Phillips K, Howell N, Griffith J, Jongejan J, Pitaru A, Chen A (2020) Teachable machine: Approachable Web-based tool for exploring machine learning classification. In: Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems, pp 1–8
  * Cavalcante Siebert et al (2022) Cavalcante Siebert L, Lupetti ML, Aizenberg E, Beckers N, Zgonnikov A, Veluwenkamp H, Abbink D, Giaccardi E, Houben GJ, Jonker CM, van den Hoven J, Forster D, Lagendijk RL (2022) Meaningful human control: Actionable properties for AI system development. AI and Ethics https://doi.org/10.1007/s43681-022-00167-3
  * Chiusi (2020) Chiusi F (2020) Automating Society Report 2020. Tech. rep., Algorithm Watch, URL https://automatingsociety.algorithmwatch.org
  * Coskun et al (2022) Coskun A, Cila N, Nicenboim I, Frauenberger C, Wakkary R, Hassenzahl M, Mancini C, Giaccardi E, Forlano L (2022) More-than-human Concepts, Methodologies, and Practices in HCI. In: Extended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems, Association for Computing Machinery, New York, NY, USA, CHI EA ’22, pp 1–5, https://doi.org/10.1145/3491101.3516503, URL https://doi.org/10.1145/3491101.3516503
  * Coulton and Lindley (2019) Coulton P, Lindley JG (2019) More-Than Human Centred Design: Considering Other Things. The Design Journal 22(4):463–481, https://doi.org/10.1080/14606925.2019.1614320, URL https://doi.org/10.1080/14606925.2019.1614320
  * Crawford (2021) Crawford K (2021) The Atlas of AI: Power, Politics, and the Planetary Costs of Artificial Intelligence. Yale University Press, https://doi.org/10.2307/j.ctv1ghv45t, URL http://www.jstor.org/stable/10.2307/j.ctv1ghv45t
  * Crawford and Paglen (2019) Crawford K, Paglen T (2019) Excavating AI: The politics of images in machine learning training sets. Excavating AI
  * Dahlbäck et al (1993) Dahlbäck N, Jönsson A, Ahrenberg L (1993) Wizard of Oz studies — why and how. Knowledge-Based Systems 6(4):258–266, https://doi.org/10.1016/0950-7051(93)90017-N, URL https://www.sciencedirect.com/science/article/pii/095070519390017N
  * de Reuver et al (2020) de Reuver M, van Wynsberghe A, Janssen M, van de Poel I (2020) Digital platforms and responsible innovation: Expanding value sensitive design to overcome ontological uncertainty. Ethics and Information Technology 22(3):257–267, https://doi.org/10.1007/s10676-020-09537-z
  * Devendorf and Ryokai (2015) Devendorf L, Ryokai K (2015) Being the machine: Reconfiguring agency and control in hybrid fabrication. In: Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, pp 2477–2486
  * DiSalvo (2015) DiSalvo C (2015) Adversarial Design. Mit Press
  * Dove et al (2017) Dove G, Halskov K, Forlizzi J, Zimmerman J (2017) UX Design Innovation: Challenges for Working with Machine Learning as a Design Material. In: Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, Association for Computing Machinery, New York, NY, USA, CHI ’17, pp 278–288, https://doi.org/10.1145/3025453.3025739, URL https://doi.org/10.1145/3025453.3025739
  * EdgeImpulse (2019) EdgeImpulse (2019) Edge Impulse. URL https://www.edgeimpulse.com/
  * Ehsan and Riedl (2020) Ehsan U, Riedl MO (2020) Human-Centered Explainable AI: Towards a Reflective Sociotechnical Approach. In: Stephanidis C, Kurosu M, Degen H, Reinerman-Jones L (eds) HCI International 2020 - Late Breaking Papers: Multimodality and Intelligence, Springer International Publishing, Cham, Lecture Notes in Computer Science, pp 449–466, https://doi.org/10.1007/978-3-030-60117-133
  * Elsden et al (2017) Elsden C, Chatting D, Durrant AC, Garbett A, Nissen B, Vines J, Kirk DS (2017) On Speculative Enactments. In: Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, Association for Computing Machinery, New York, NY, USA, pp 5386–5399
  * Elwes (2019) Elwes J (2019) Zizi. URL https://www.jakeelwes.com/project-zizi-2019.html
  * Fiebrink (2019) Fiebrink R (2019) Machine Learning Education for Artists, Musicians, and Other Creative Practitioners. ACM Transactions on Computing Education 19(4):31:1–31:32, https://doi.org/10.1145/3294008, URL https://doi.org/10.1145/3294008
  * Fiebrink (2022) Fiebrink R (2022) Machine Learning for Musicians and Artists. URL https://www.kadenze.com/courses/machine-learning-for-musicians-and-artists-v/info
  * Findeli (2001) Findeli A (2001) Rethinking design education for the 21st century: Theoretical, methodological, and ethical discussion. Design issues 17(1):5–17
  * Fish and Stark (2021) Fish B, Stark L (2021) Reflexive Design for Fairness and Other Human Values in Formal Models. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, Association for Computing Machinery, New York, NY, USA, pp 89–99
  * Floridi (2020) Floridi L (2020) AI and Its New Winter: From Myths to Realities. Philosophy & Technology 33(1):1–3, https://doi.org/10.1007/s13347-020-00396-6, URL https://link.springer.com/article/10.1007/s13347-020-00396-6
  * Forlano and Mathew (2014) Forlano L, Mathew A (2014) From design fiction to design friction: Speculative and participatory design of values-embedded urban technology. Journal of Urban Technology 21(4):7–24
  * Frauenberger (2020) Frauenberger C (2020) Entanglement HCI The Next Wave? ACM Transactions on Computer-Human Interaction 27(1):1–27, https://doi.org/10.1145/3364998
  * Gebru et al (2018) Gebru T, Morgenstern J, Vecchione B, Vaughan JW, Wallach H, Daumé III H, Crawford K (2018) Datasheets for datasets. arXiv preprint arXiv:180309010 1803.09010
  * Ghajargar and Bardzell (2023) Ghajargar M, Bardzell J (2023) Making AI Understandable by Making it Tangible: Exploring the Design Space with Ten Concept Cards. In: Proceedings of the 34th Australian Conference on Human-Computer Interaction, Association for Computing Machinery, New York, NY, USA, OzCHI ’22, pp 74–80, https://doi.org/10.1145/3572921.3572942
  * Ghajargar et al (2021) Ghajargar M, Bardzell J, Renner AS, Krogh PG, Höök K, Cuartielles D, Boer L, Wiberg M (2021) From ”Explainable AI” to ”Graspable AI”. In: Proceedings of the Fifteenth International Conference on Tangible, Embedded, and Embodied Interaction, Association for Computing Machinery, New York, NY, USA, TEI ’21, pp 1–4, https://doi.org/10.1145/3430524.3442704
  * Ghajargar et al (2022) Ghajargar M, Bardzell J, Smith-Renner AM, Höök K, Krogh PG (2022) Graspable AI: Physical Forms as Explanation Modality for Explainable AI. In: Sixteenth International Conference on Tangible, Embedded, and Embodied Interaction, Association for Computing Machinery, New York, NY, USA, TEI ’22, pp 1–4, https://doi.org/10.1145/3490149.3503666
  * Giaccardi (2019) Giaccardi E (2019) Histories and futures of research through design: From prototypes to connected things. International Journal of Design 13(3):139–155
  * Giaccardi and Redström (2020) Giaccardi E, Redström J (2020) Technology and More-Than-Human Design. Design Issues 36(4):33–44, https://doi.org/10.1162/desia00612
  * Giaccardi et al (2016) Giaccardi E, Cila N, Speed C, Caldwell M (2016) Thing Ethnography: Doing Design Research with Non-Humans. In: Proceedings of the 2016 ACM Conference on Designing Interactive Systems, Association for Computing Machinery, New York, NY, USA, DIS ’16, pp 377–387, https://doi.org/10.1145/2901790.2901905
  * Hagendorff (2020) Hagendorff T (2020) The Ethics of AI Ethics: An Evaluation of Guidelines. Minds and Machines 30(1):99–120, https://doi.org/10.1007/s11023-020-09517-8, URL https://doi.org/10.1007/s11023-020-09517-8
  * Hemment et al (2022a) Hemment D, Murray-Rust D, Belle V, Aylett R, Vidmar M, Broz F (2022a) Experiential AI: Enhancing explainability in artificial intelligence through artistic practice
  * Hemment et al (2022b) Hemment D, Zeilinger M, Vidmar M, Elwes J, Warner H, Sarmiento D, Hill R (2022b) Towards a heuristic model for experiential AI: Analysing the Zizi show in the new real. In: DRS Biennial Conference Series, URL https://dl.designresearchsociety.org/drs-conference-papers/drs2022/researchpapers/311
  * Hildebrandt (2020) Hildebrandt M (2020) Smart technologies. Internet Policy Review 9(4)
  * Hodder (2016) Hodder I (2016) Studies in Human-Thing Entanglement. Online
  * Holton and Boyd (2021) Holton R, Boyd R (2021) ‘Where are the people? What are they doing? Why are they doing it?’(Mindell) Situating artificial intelligence within a socio-technical framework. Journal of Sociology 57(2):179–195, https://doi.org/10.1177/1440783319873046, URL https://doi.org/10.1177/1440783319873046
  * HuggingFace (2016) HuggingFace (2016) Hugging Face – The AI community building the future. URL https://huggingface.co/
  * Inkpen et al (2019) Inkpen K, Chancellor S, De Choudhury M, Veale M, Baumer EPS (2019) Where is the Human? Bridging the Gap Between AI and HCI. In: Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems, Association for Computing Machinery, New York, NY, USA, CHI EA ’19, pp 1–9, https://doi.org/10.1145/3290607.3299002
  * Irani and Silberman (2014) Irani L, Silberman MS (2014) From critical design to critical infrastructure: Lessons from turkopticon. Interactions 21(4):32–35, https://doi.org/10.1145/2627392, URL https://doi.org/10.1145/2627392
  * Jonsson and Tholander (2022) Jonsson M, Tholander J (2022) Cracking the code: Co-coding with AI in creative programming education. In: Creativity and Cognition, Association for Computing Machinery, New York, NY, USA, C&C ’22, pp 5–14, https://doi.org/10.1145/3527927.3532801, URL https://doi.org/10.1145/3527927.3532801
  * Jung et al (2022) Jung JY, Qiu S, Bozzon A, Gadiraju U (2022) Great Chain of Agents: The Role of Metaphorical Representation of Agents in Conversational Crowdsourcing. In: CHI Conference on Human Factors in Computing Systems, ACM, New Orleans LA USA, pp 1–22, https://doi.org/10.1145/3491102.3517653
  * Khadpe et al (2020) Khadpe P, Krishna R, Fei-Fei, Li, Hancock T Jeffrey, Bernstein Michael S (2020) Conceptual Metaphors Impact Perceptions of Human-AI Collaboration. Proceedings of the ACM on Human-Computer Interaction https://doi.org/10.1145/3415234
  * Kirman et al (2022) Kirman B, Bates O, Lord C, Alter H (2022) Thinking outside the bag: Worker-led speculation and the future of gig economy delivery platforms. In: DRS Biennial Conference Series, URL https://dl.designresearchsociety.org/drs-conference-papers/drs2022/researchpapers/107
  * Kliman-Silver et al (2020) Kliman-Silver C, Siy O, Awadalla K, Lentz A, Convertino G, Churchill E (2020) Adapting user experience research methods for AI-driven experiences. In: Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems, pp 1–8
  * Kovaleva et al (2019) Kovaleva O, Romanov A, Rogers A, Rumshisky A (2019) Revealing the Dark Secrets of BERT. https://doi.org/10.48550/arXiv.1908.08593, URL http://arxiv.org/abs/1908.08593, 1908.08593
  * Lin (2014) Lin CH (2014) Rethinking Design Education-A Study for the Importance of General Education in Design. International Journal of Liberal Arts and Social Science 2(9):88–97
  * Lindgren and Holmström (2020) Lindgren S, Holmström J (2020) A Social Science Perspective on Artificial Intelligence: Building Blocks for a Research Agenda. Journal of Digital Social Research 2(3):1–15, https://doi.org/10.33621/jdsr.v2i3.65, URL https://jdsr.se/ojs/index.php/jdsr/article/view/65
  * Littman (2021) Littman ML (2021) Gathering Strength, Gathering Storms: The One Hundred Year Study on Artificial Intelligence (AI100). Tech. rep., Stanford University, URL https://ai100.stanford.edu/2021-report/gathering-strength-gathering-storms-one-hundred-year-study-artificial-intelligence
  * Liu and Chilton (2022) Liu V, Chilton LB (2022) Design Guidelines for Prompt Engineering Text-to-Image Generative Models. In: Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, Association for Computing Machinery, New York, NY, USA, CHI ’22, pp 1–23, https://doi.org/10.1145/3491102.3501825
  * Llano et al (2022) Llano MT, d’Inverno M, Yee-King M, McCormack J, Ilsar A, Pease A, Colton S (2022) Explainable Computational Creativity. https://doi.org/10.48550/arXiv.2205.05682, URL http://arxiv.org/abs/2205.05682, 2205.05682
  * Lloyd (2019) Lloyd P (2019) You make it and you try it out: Seeds of design discipline futures. Design Studies 65:167–181, https://doi.org/10.1016/j.destud.2019.10.008, URL https://www.sciencedirect.com/science/article/pii/S0142694X19300675
  * Lockton et al (2019) Lockton D, Singh D, Sabnis S, Chou M, Foley S, Pantoja A (2019) New metaphors: A workshop method for generating ideas and reframing problems in design and beyond. In: Proceedings of the 2019 on Creativity and Cognition, Association for Computing Machinery, New York, NY, USA, C&C ’19, p 319–332, https://doi.org/10.1145/3325480.3326570, URL https://doi.org/10.1145/3325480.3326570
  * Lupetti et al (2020) Lupetti ML, Smit I, Nicenboim I, Huisman G, Villaronga EF, Bedo V, Jongeling M (2020) Subversive Citizen Manual for the More-Than-Human City
  * Lyons (2020) Lyons MJ (2020) Excavating ”Excavating AI”: The Elephant in the Gallery. https://doi.org/10.5281/zenodo.4037538, URL http://arxiv.org/abs/2009.01215, 2009.01215
  * Malsattar et al (2019) Malsattar N, Kihara T, Giaccardi E (2019) Designing and Prototyping from the Perspective of AI in the Wild. In: Proceedings of the 2019 on Designing Interactive Systems Conference, Association for Computing Machinery, New York, NY, USA, DIS ’19, pp 1083–1088, https://doi.org/10.1145/3322276.3322351
  * Marenko and van Allen (2016) Marenko B, van Allen P (2016) Animistic design: How to reimagine digital interaction between the human and the nonhuman. Digital Creativity 27(1):52–70, https://doi.org/10.1080/14626268.2016.1145127
  * McCardle (2002) McCardle JR (2002) The Challenge of Integrating AI & Smart Technology in Design Education. International Journal of Technology and Design Education 12(1):59–76, https://doi.org/10.1023/A:1013089404168, URL https://doi.org/10.1023/A:1013089404168
  * McCormack et al (2020) McCormack J, Hutchings P, Gifford T, Yee-King M, Llano MT, D’inverno M (2020) Design Considerations for Real-Time Collaboration with Creative Artificial Intelligence. Organised Sound 25(1):41–52, https://doi.org/10.1017/S1355771819000451, URL https://www.cambridge.org/core/journals/organised-sound/article/abs/design-considerations-for-realtime-collaboration-with-creative-artificial-intelligence/D14EFAAB865FF356E475353C75245CE2
  * Mital (2016) Mital P (2016) Creative Applications of Deep Learning. URL https://www.kadenze.com/courses/creative-applications-of-deep-learning-with-tensorflow-iv/info
  * Mital (2021) Mital P (2021) Cultural Appropriation with Machine Learning. URL http://pkmital.com/home/2021/09/01/ucla-course-on-cultural-appropriation-with-machine-learning/
  * Mitchell et al (2019) Mitchell M, Wu S, Zaldivar A, Barnes P, Vasserman L, Hutchinson B, Spitzer E, Raji ID, Gebru T (2019) Model Cards for Model Reporting. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, Association for Computing Machinery, New York, NY, USA, FAT* ’19, pp 220–229, https://doi.org/10.1145/3287560.3287596
  * Mohammad (2021) Mohammad SM (2021) Ethics Sheets for AI Tasks. arXiv:210701183 [cs] 2107.01183
  * Murray-Rust and Tsiakas (2022) Murray-Rust D, Tsiakas K (2022) EthicAmanuensis: Supporting machine learning practitioners making and recording ethical decisions. In: International Conference on Tool with Artificial Intelligence
  * Murray-Rust et al (2019) Murray-Rust D, Gorkovenko K, Burnett D, Richards D (2019) Entangled Ethnography: Towards a collective future understanding. In: Proceedings of the Halfway to the Future Symposium 2019, Association for Computing Machinery, New York, NY, USA, HTTF 2019, pp 1–10, https://doi.org/10.1145/3363384.3363405
  * Murray-Rust et al (2022a) Murray-Rust D, Elsden C, Nissen B, Tallyn E, Pschetz L, Speed C (2022a) Blockchain and Beyond: Understanding Blockchains through Prototypes and Public Engagement. Transactions on Computer-Human Interaction 2112.11891
  * Murray-Rust et al (2022b) Murray-Rust D, Nicenboim I, Lockton D (2022b) Metaphors for designers working with AI. In: DRS Biennial Conference Series, https://doi.org/10.21606/drs.2022.667
  * Natale and Ballatore (2020) Natale S, Ballatore A (2020) Imagining the thinking machine: Technological myths and the rise of artificial intelligence. Convergence 26(1):3–18, https://doi.org/10.1177/1354856517715164
  * Nicenboim et al (2020) Nicenboim I, Giaccardi E, Søndergaard MLJ, Reddy AV, Strengers Y, Pierce J, Redström J (2020) More-Than-Human Design and AI: In Conversation with Agents. In: Companion Publication of the 2020 ACM Designing Interactive Systems Conference, ACM, Eindhoven Netherlands, pp 397–400, https://doi.org/10.1145/3393914.3395912
  * Nicenboim et al (2022) Nicenboim I, Giaccardi E, Redström J (2022) From explanations to shared understandings of AI. DRS Biennial Conference Series
  * Nicolini (2009) Nicolini D (2009) Zooming in and zooming out: A package of method and theory to study work practices. In: Organizational Ethnography: Studying the Complexities of Everyday Life, Ed. M. Worline, SAGE Publications Ltd., pp 120–138
  * Noor (2017) Noor AK (2017) AI and the Future of the Machine Design. Mechanical Engineering 139(10):38–43, https://doi.org/10.1115/1.2017-Oct-2, URL https://doi.org/10.1115/1.2017-Oct-2
  * OpenCV (1999) OpenCV (1999) OpenCV. URL https://opencv.org/
  * PAIR (2020) PAIR (2020) People + AI Guidebook. URL https://pair.withgoogle.com/guidebook
  * Pierce (2021) Pierce J (2021) Eccentric Sensing Devices: Using Conceptual Design Notes to Understand Design Opportunities, Limitations, and Concerns Connected to Digital Sensing. In: Creativity and Cognition, 37, Association for Computing Machinery, New York, NY, USA, p 1
  * Pschetz and Bastian (2018) Pschetz L, Bastian M (2018) Temporal Design: Rethinking time in design. Design Studies 56:169–184, https://doi.org/10.1016/j.destud.2017.10.007, URL https://www.sciencedirect.com/science/article/pii/S0142694X17300765
  * Pschetz et al (2019) Pschetz L, Pothong K, Speed C (2019) Autonomous Distributed Energy Systems: Problematising the Invisible through Design, Drama and Deliberation. In: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, ACM, Glasgow Scotland Uk, pp 1–14, https://doi.org/10.1145/3290605.3300617
  * Rapp (2021) Rapp A (2021) Wearable technologies as extensions: A postphenomenological framework and its design implications. Human–Computer Interaction 0(0):1–39, https://doi.org/10.1080/07370024.2021.1927039
  * Reddy et al (2020) Reddy A, Nicenboim I, Pierce J, Giaccardi E (2020) Encountering ethics through design: A workshop with nonhuman participants. AI & SOCIETY pp 1–9, https://doi.org/10.1007/s00146-020-01088-7
  * Reddy et al (2021) Reddy A, Kocaballi AB, Nicenboim I, Søndergaard MLJ, Lupetti ML, Key C, Speed C, Lockton D, Giaccardi E, Grommé F, Robbins H, Primlani N, Yurman P, Sumartojo S, Phan T, Bedö V, Strengers Y (2021) Making Everyday Things Talk: Speculative Conversations into the Future of Voice Interfaces at Home. In: Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems, Association for Computing Machinery, New York, NY, USA, CHI EA ’21, pp 1–16, https://doi.org/10.1145/3411763.3450390
  * Redström and Wiltse (2018) Redström J, Wiltse H (2018) Changing Things : The Future of Objects in a Digital World. Bloomsbury Academic
  * Runway (2020) Runway (2020) Runway - Online Video Editor — Everything you need to edit video, fast. URL https://runwayml.com
  * Samoli et al (2020) Samoli S, López-Cobo M, Gómez E, De Prato G, Martínez-Plumed F, Delipetrev B (2020) AI watch: Defining Artificial Intelligence: Towards an operational definition and taxonomy of artificial intelligence. Tech. rep., Joint Research Centre, European Commission, URL https://data.europa.eu/doi/10.2760/382730
  * Sanches et al (2022) Sanches P, Howell N, Tsaknaki V, Jenkins T, Helms K (2022) Diffraction-in-action: Designerly Explorations of Agential Realism Through Lived Data. In: Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, Association for Computing Machinery, New York, NY, USA, CHI ’22, pp 1–18, https://doi.org/10.1145/3491102.3502029
  * Sanders and Stappers (2014) Sanders EBN, Stappers PJ (2014) Probes, toolkits and prototypes: Three approaches to making in codesigning. CoDesign 10(1):5–14, https://doi.org/10.1080/15710882.2014.888183, URL https://doi.org/10.1080/15710882.2014.888183
  * Sartori and Theodorou (2022) Sartori L, Theodorou A (2022) A sociotechnical perspective for the future of AI: Narratives, inequalities, and human control. Ethics and Information Technology 24(1):4, https://doi.org/10.1007/s10676-022-09624-3, URL https://doi.org/10.1007/s10676-022-09624-3
  * Scurto et al (2021) Scurto H, Caramiaux B, Bevilacqua F (2021) Prototyping Machine Learning Through Diffractive Art Practice. In: Designing Interactive Systems Conference 2021, Association for Computing Machinery, New York, NY, USA, DIS ’21, pp 2013–2025, https://doi.org/10.1145/3461778.3462163
  * Seymour et al (2022) Seymour W, Van Kleek M, Binns R, Murray-Rust D (2022) Respect as a Lens for the Design of AI Systems. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp 641–652, https://doi.org/10.1145/3514094.3534186, 2206.07555
  * Shen et al (2021) Shen H, Deng WH, Chattopadhyay A, Wu ZS, Wang X, Zhu H (2021) Value Cards: An Educational Toolkit for Teaching Social Impacts of Machine Learning through Deliberation. In: Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, ACM, Virtual Event Canada, pp 850–861, https://doi.org/10.1145/3442188.3445971, URL https://dl.acm.org/doi/10.1145/3442188.3445971
  * Shotter (2014) Shotter J (2014) Agential realism, social constructionism, and our living relations to our surroundings: Sensing similarities rather than seeing patterns. Theory & Psychology 24(3):305–325, https://doi.org/10.1177/0959354313514144
  * Siebert et al (2022) Siebert LC, Lupetti ML, Aizenberg E, Beckers N, Zgonnikov A, Veluwenkamp H, Abbink D, Giaccardi E, Houben GJ, Jonker CM, van den Hoven J, Forster D, Lagendijk RL (2022) Meaningful human control: Actionable properties for AI system development. AI and Ethics https://doi.org/10.1007/s43681-022-00167-3, 2112.01298
  * Simeone et al (2022) Simeone L, Mantelli R, Adamo A (2022) Pushing divergence and promoting convergence in a speculative design process: Considerations on the role of AI as a co-creation partner. DRS Biennial Conference Series
  * Sinders and Ahmad (2021) Sinders C, Ahmad S (2021) The labor behind the tools: Using design thinking methods to examine content moderation software. Interactions 28(4):6–8, https://doi.org/10.1145/3470492, URL https://doi.org/10.1145/3470492
  * Singh et al (2019) Singh J, Cobbe J, Norval C (2019) Decision Provenance: Harnessing data flow for accountable systems. IEEE Access 7:6562–6574, https://doi.org/10.1109/ACCESS.2018.2887201, URL http://arxiv.org/abs/1804.05741, 1804.05741
  * Stappers and Giaccardi (2017) Stappers PJ, Giaccardi E (2017) Research through Design. In: The Encyclopedia of Human-Computer Interaction, The Interaction Design Foundation, URL https://www.interaction-design.org/literature/book/the-encyclopedia-of-human-computer-interaction-2nd-ed/research-through-design
  * Stoffregen (2003) Stoffregen TA (2003) Affordances as Properties of the Animal-Environment System. In: How Shall Affordances Be Refined? Four Perspectives, Routledge
  * Subramonyam et al (2021a) Subramonyam H, Seifert C, Adar E (2021a) Protoai: Model-informed prototyping for ai-powered interfaces. In: 26th International Conference on Intelligent User Interfaces, pp 48–58
  * Subramonyam et al (2021b) Subramonyam H, Seifert C, Adar E (2021b) Towards a process model for co-creating AI experiences. In: Designing Interactive Systems Conference 2021, pp 1529–1543, https://doi.org/10.1145/3461778.3462012
  * Suchman (2020) Suchman L (2020) Agencies in technology design: Feminist reconfigurations. In: Machine Ethics and Robot Ethics, Routledge, pp 361–375
  * TensorFlow (2015) TensorFlow (2015) TensorFlow. URL https://www.tensorflow.org/
  * Theodorou and Dignum (2020) Theodorou A, Dignum V (2020) Towards ethical and socio-legal governance in AI. Nature Machine Intelligence 2(1):10–12, https://doi.org/10.1038/s42256-019-0136-y, URL https://www.nature.com/articles/s42256-019-0136-y
  * Tremblay et al (2021) Tremblay PA, Roma G, Green O (2021) Enabling Programmatic Data Mining as Musicking: The Fluid Corpus Manipulation Toolkit. Computer Music Journal 45(2):9–23, https://doi.org/10.1162/comja00600, URL https://doi.org/10.1162/comja00600
  * Troiano et al (2021) Troiano GM, Wood M, Sonbudak MF, Padte RC, Harteveld C (2021) ”Are We Now Post-COVID?”: Exploring Post-COVID Futures Through a Gamified Story Completion Method. In: Designing Interactive Systems Conference 2021, Association for Computing Machinery, New York, NY, USA, pp 48–63, URL https://doi.org/10.1145/3461778.3462069
  * Vagg (2022) Vagg J (2022) Experiencing-With Data: Exploring Posthuman Creativity Through Rhizomatic Empathy. Qualitative Inquiry 28(5):541–551, https://doi.org/10.1177/10778004211069696
  * van der Burg et al (2022) van der Burg V, Salah AA, Chandrasegaran S, Lloyd P (2022) Ceci n’est pas une chaise: Emerging practices in designer-AI collaboration. DRS Biennial Conference Series
  * Van Der Helm and Stappers (2020) Van Der Helm A, Stappers PJ (2020) Prototyping with A Team: Acting Machines Support Shared Understanding. In: DS 104: Proceedings of the 22nd International Conference on Engineering and Product Design Education (E&PDE 2020), VIA Design, VIA University in Herning, Denmark. 10th-11th September 2020
  * van der Vlist et al (2008) van der Vlist B, van de Westelaken R, Bartneck C, Hu J, Ahn R, Barakova E, Delbressine F, Feijs L (2008) Teaching Machine Learning to Design Students. In: Pan Z, Zhang X, El Rhalibi A, Woo W, Li Y (eds) Technologies for E-Learning and Digital Entertainment, Springer, Berlin, Heidelberg, Lecture Notes in Computer Science, pp 206–217, https://doi.org/10.1007/978-3-540-69736-723
  * Verbeek (2008) Verbeek PP (2008) Cyborg intentionality: Rethinking the phenomenology of human–technology relations. Phenomenology and the Cognitive Sciences 7(3):387–395, https://doi.org/10.1007/s11097-008-9099-x
  * Verbeek (2020) Verbeek PP (2020) Politicizing Postphenomenology. In: Miller G, Shew A (eds) Reimagining Philosophy and Technology, Reinventing Ihde, Philosophy of Engineering and Technology, Springer International Publishing, Cham, pp 141–155, https://doi.org/10.1007/978-3-030-35967-69, URL https://doi.org/10.1007/978-3-030-35967-69
  * Wang et al (2020) Wang D, Churchill E, Maes P, Fan X, Shneiderman B, Shi Y, Wang Q (2020) From human-human collaboration to Human-AI collaboration: Designing AI systems that can work together with people. In: Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems, pp 1–6
  * Wegener et al (2019) Wegener FE, Gonçalves MG, Dankfort Z (2019) Reflection-in-Action When Designing Organizational Processes: Prototyping Workshops for Collective Reflection-in-Action. Proceedings of the Design Society: International Conference on Engineering Design 1(1):1255–1264, https://doi.org/10.1017/dsi.2019.131, URL https://www.cambridge.org/core/journals/proceedings-of-the-international-conference-on-engineering-design/article/reflectioninaction-when-designing-organizational-processes-prototyping-workshops-for-collective-reflectioninaction/E7E5E6EB5924415E90A585472389FDA8
  * Westerlund and Wetter-Edman (2017) Westerlund B, Wetter-Edman K (2017) Dealing with wicked problems, in messy contexts, through prototyping. The Design Journal 20(sup1):S886–S899, https://doi.org/10.1080/14606925.2017.1353034, URL https://doi.org/10.1080/14606925.2017.1353034
  * Wiltse (2020) Wiltse H (ed) (2020) Relating to Things: Design, Technology and the Artificial. Bloomsbury Publishing Plc, https://doi.org/10.5040/9781350124288
  * Wong (2018) Wong JS (2018) Design and fiction: Imagining civic AI. Interactions 25(6):42–45, https://doi.org/10.1145/3274568
  * Wong et al (2017) Wong RY, Van Wyk E, Pierce J (2017) Real-fictional entanglements: Using science fiction and design fiction to interrogate sensing technologies. In: Proceedings of the 2017 Conference on Designing Interactive Systems, pp 567–579
  * Yang et al (2020) Yang Q, Steinfeld A, Rosé C, Zimmerman J (2020) Re-examining whether, why, and how human-AI interaction is uniquely difficult to design. In: Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, pp 1–13
  * Yanow and Tsoukas (2009) Yanow D, Tsoukas H (2009) What is Reflection-In-Action? A Phenomenological Account. Journal of Management Studies 46(8):1339–1364, https://doi.org/10.1111/j.1467-6486.2009.00859.x, URL https://onlinelibrary.wiley.com/doi/abs/10.1111/j.1467-6486.2009.00859.x
  * Yildirim et al (2022) Yildirim N, Kass A, Tung T, Upton C, Costello D, Giusti R, Lacin S, Lovic S, O’Neill JM, Meehan RO (2022) How Experienced Designers of Enterprise Applications Engage AI as a Design Material. In: CHI Conference on Human Factors in Computing Systems, pp 1–13
  * Yildirim et al (2023) Yildirim N, Oh C, Sayar D, Brand K, Challa S, Turri V, Crosby Walton N, Wong AE, Forlizzi J, McCann J, Zimmerman J (2023) Creating Design Resources to Scaffold the Ideation of AI Concepts. In: Proceedings of the 2023 ACM Designing Interactive Systems Conference, Association for Computing Machinery, New York, NY, USA, DIS ’23, pp 2326–2346, https://doi.org/10.1145/3563657.3596058
  * Yun et al (2022) Yun G, Cho K, Jeong Y, Nam TJ (2022) Ideasquares: Utilizing generative text as a source of design inspiration. DRS Biennial Conference Series
  * Yurrita et al (2022) Yurrita M, Agate B, Murray-Rust D, Bozzon A (2022) Towards a multi-stakeholder value-based assessment framework for algorithmic systems. In: FAccT2022