# Exploring the Role of Social Support when Integrating Generative AI in Small
Business Workflows

Quentin Romero Lauro  0009-0000-0620-2146 University of
PittsburghPittsburghPennsylvaniaUSA<EMAIL_ADDRESS>,  Jeffrey P. Bigham
0000-0002-2072-0625 Carnegie Mellon UniversityPittsburghPennsylvaniaUSA
<EMAIL_ADDRESS>and  Yasmine Kotturi  0000-0001-6201-7922 University of
Maryland, Baltimore CountyBaltimoreMarylandUSA<EMAIL_ADDRESS>
(2024)

###### Abstract.

Small business owners stand to benefit from generative AI technologies due to
limited resources, yet they must navigate increasing legal and ethical risks.
In this paper, we interview 11 entrepreneurs and support personnel to
investigate existing practices of how entrepreneurs integrate generative AI
technologies into their business workflows. Specifically, we build on
scholarship in HCI which emphasizes the role of small, offline networks in
supporting entrepreneurs’ technology maintenance. We detail how entrepreneurs
resourcefully leveraged their local networks to discover new use-cases of
generative AI (e.g., by sharing accounts), assuage heightened techno-anxieties
(e.g., by recruiting trusted confidants), overcome barriers to sustained use
(e.g., by receiving wrap-around support), and establish boundaries of use.
Further, we suggest how generative AI platforms may be redesigned to better
support entrepreneurs, such as by taking into account the benefits and
tensions of use in a social context.

entrepreneurship, generative AI, social support, lean economies

††journalyear: 2024††copyright: rightsretained††conference: Companion of the
2024 Computer-Supported Cooperative Work and Social Computing; November 9–13,
2024; San Jose, Costa Rica††booktitle: Companion of the 2024 Computer-
Supported Cooperative Work and Social Computing (CSCW Companion ’24), November
9–13, 2024, San Jose, Costa Rica††doi: 10.1145/3678884.3681895††isbn:
979-8-4007-1114-5/24/11††ccs: Human-centered computing Empirical studies in
HCI

Figure 1.  Through an interview study, we detailed how entrepreneurs used
their social support networks to integrate and maintain generative AI
technologies in their business workflows (e.g., shared accounts with co-
owners, side-by-side tutoring sessions, proxy use through trusted
intermediaries); we then probed the benefits and tensions of social support.

Forms, Benefits, and Tensions of Social Support in Using Generative AI.
Different configurations of social support when using generative AI include
shared accounts, side-by-side use, and use via intermediaries. While, benefits
of social support include discovering new use cases, developing self-efficacy,
easing techno-anxieties, and receiving wrap-around support, social support
when using generative AI comes with tensions including social comparison,
feeling left behind, reputational concerns, and unclear social norms.

##  1\. Introduction

Small business owners who effectively leverage generative AI for their
business can boost their efficiency, creativity, and ultimately increase their
bottom line (Otis et al., 2023). But integrating these novel technolog-
ies—machine learning models that can generate high-quality text, images, and
other content from training data (Staff, 2023)—into existing workflows might
not be as straightforward as generative AI platforms tout (OpenAI, 2024). For
instance, entrepreneurs must consider the implications of providing
intellectual property to generative AI platforms, and the ill-defined legality
and ethics of using output generated from mining the web (Michael M. Grynbaum
and Ryan Mac, 2023). In addition, prompt engineering, an essential skill to
use generative AI platforms such as ChatGPT and DALL-E, is not as intuitive as
it seems and non-expert users struggle to iterate on prompts to generate
desired output (Zamfirescu-Pereira et al., 2023). These challenges are further
heightened for entrepreneurs from resource-constrained communities, or lean
economies (Dillahunt et al., 2018), who must deploy resourceful tactics to
overcome limited access to infrastructural support such as education and
equipment (Avle et al., 2019).

In response to entrepreneurs’ difficulties with keeping pace with technology
advancements, human-computer interaction (HCI) scholars have detailed at
length the role that small, offline networks play in helping small business
owners plan and implement a strategy for tech maintenance (Hui et al., 2020;
Dillahunt et al., 2018; Kotturi et al., 2022). These networks, or
“collectives,” comprise peers, mentors, friends, and even loyal customers, and
play an especially important role for entrepreneurs from lean economies
because of sparse access to infrastructural support for technology adoption
(Hui et al., 2020). For instance, local community groups can provide vetted
information on which platforms and technologies entrepreneurs should adopt
(Kotturi et al., 2024a; Dillahunt et al., 2018), skill training opportunities
(Kotturi et al., 2022), as well as general encouragement (critical, as many
entrepreneurs work in isolation) (Dillahunt et al., 2018; Hui et al., 2020).
Yet, it remains unclear how these offline networks may facilitate
entrepreneurs’ adoption of generative AI technologies into their business.
While recent work suggests that local community centers may play an essential
role in entrepreneurs’ introduction to generative AI through guided, communal
introduction (Kotturi et al., 2024a), not all entrepreneurs have reliable
access to such spaces. In addition, the rapid pace of change—with new
generative AI platforms and features deployed every day—makes it hard for even
the most advanced business incubators to update curricula. Finally,
entrepreneurs may have different preferences of use of generative AI given the
multitude of unfolding ethical, legal, and inaccuracy concerns as use relates
to their business (and therefore their livelihood) (Kotturi et al., 2024a; MIT
Sloan, 2023).

This paper, therefore, investigates how small business owners use various
forms of social support from their offline, informal networks to determine how
to integrate generative AI into their business workflows. We ask: how does
social support facilitate entrepreneurs’ use (and non-use) of generative AI
technologies? We interviewed 11 entrepreneurs and entrepreneurship support
personnel (e.g., tech support staff at local community centers) to better
understand existing practices for how entrepreneurs integrated generative AI,
as well as how they used their offline networks to guide these decisions,
support maintenance and repair, and refusal of use (e.g. deciding to not use
generative AI (Garcia et al., 2020)). While not “high-tech entrepreneurs”
(Roberts, 1991), the entrepreneurs in our study used myriad computing tools to
market, create, and sell their goods (Kotturi et al., 2022), and primarily
targeted their local economy with their product or service-based businesses
(i.e., local entrepreneurs (Hui et al., 2018)). For instance, the types
entrepreneurship represented in our study included running a streetwear
clothing brand, offering gifts and party planning, providing choreography and
dance classes, running letterpress and educational programming, and selling
paintings; see Table 1. To engage entrepreneurs who were not yet embedded in a
network with expertise on generative AI, interviews included a series of
probes (in the form of paper storyboards (Truong et al., 2006)) to enable
deeper discussions for how to design generative AI systems with such a social
context in mind. Combining semi-structured interviews and probes, therefore,
enabled a deeper discussion of both current and desired states for how social
support may—and may not—be helpful when integrating generative AI technologies
into business workflows.

We found that offline support networks’ role ranged from providing supportive
nudges to overcome the associated anxieties with getting started, to a
collaborative partner in formulating prompts and finding out new ideas for how
to apply these nascent technologies to small business. For example, despite
tools’ single-user assumption (e.g., standard accounts on ChatGPT presume only
one user), entrepreneurs appropriated existing systems for shared use (i.e.,
by creating one account and sharing passwords with multiple users) to overcome
operational barriers, like prompt engineering. These networks—comprising
business incubators and accelerators, community centers, close friends,
spouses, and business partners—served as sources of discovery over time,
showcasing the diverse applications of generative AI, and as mediators,
helping to refine requests and ensure useful outputs. We observed that this
shared use was critical to keep up with the pace of innovation: entrepreneurs
who sought continuous support through their local networks reflected on how
they observed how others used the technology, specifically those whom they
trusted with their business affairs. Additionally, participants in our study
reflected on their boundaries when adopting generative AI, which were
sometimes informed by their networks’ attitudes towards the nascent technology
(e.g., refusing to use generative AI when it might incur a reputational cost
attributed to duplicity). Further, we observed tensions that entrepreneurs
experienced when collaboratively using generative AI platforms, and detailed
how entrepreneurs’ considered the emergent social norms when sharing
generative AI technologies.

Taken together, this paper contributes empirical data on how small business
owners integrated generative AI technologies into their workflows and how
their supportive networks facilitated this process. We consider how generative
AI technologies may be redesigned to support the social context of use among
small business owners, such as by taking into account the unique nature of
entrepreneur relationships. For instance, we posit how systems may take into
account the mix of cooperation and competition among entrepreneurs (i.e.,
“coopetition” (Kuhn and Galloway, 2015)), when fostering critical use in a
social context.

##  2\. Related Work

###  2.1. Entrepreneurship in the Digital Age

Entrepreneurship in the digital age can be a driver for innovation, economic
mobility, and social change (von Hippel, 2005), especially due to reduced
startup costs and widening customer reach resulting from online, e-commerce,
and mobile platforms (Kraus et al., 2019). Yet, scholars have questioned
whether digital entrepreneurship is truly a “democratizing” force (von Hippel,
2005), surfacing how the constant self-upgrading and technical maintenance
required for entrepreneurs to stay relevant disproportionately disadvantages
under-resourced and non-technical entrepreneurs (Avle et al., 2019; Hui et
al., 2018).

In particular, prior scholarship in HCI has outlined an array of technology
challenges that local entrepreneurs—entrepreneurs who target their local
economy to overcome limited job opportunities (Hui et al., 2018)—face, such as
heightened risk given resource constraints, lower social capital, and
infrastructural barriers to access (Dillahunt et al., 2018; Avle et al., 2019;
Hui et al., 2020, 2018; Pei and Crooks, 2020; Kotturi et al., 2024b). In light
of these challenges, scholars have highlighted the crucial role of small,
local networks for entrepreneurs in resource-constrained communities to
overcome barriers in technology adoption and economic mobility (Dillahunt et
al., 2022; Hui et al., 2020; Kotturi et al., 2022). For instance, Hui et al.
found how local entrepreneurs prefer to stay digitally engaged by leveraging
low-tech social support such as in-person meetings, paper planning tools, and
resource-connecting organizations (Hui et al., 2020). Dillahunt et al.
presented “the Village” model of mentorship, and found that in-person
interaction and trust, which technologically-mediated relationships may not
facilitate, were required to sustain economic mobility. This importance of
social support extends to the context of generative AI. For instance,
introducing generative AI in a communal setting among local entrepreneurs
helped to assuage the anxieties which these nascent and powerful technologies
can elicit upon introduction (Kotturi et al., 2024a), sometimes referred to as
“techno-anxieties” (Marcoulides, 1989). However, not all entrepreneurs have
reliable access to community spaces, and must instead rely on more informal
support networks such as close friends and family for social support. We,
therefore, build on this scholarship to investigate how small business owners
leverage their informal, social support networks to determine how to integrate
generative AI into business workflows, and the benefits and tensions with such
an approach.

###  2.2. Generative AI in HCI

Recent advancement in large-language and diffusion models have stimulated many
recent end-user developments of generative AI technologies. Interaction with
generative AI platforms has mainly been centered around prompt engineering, in
which a user enters text to prime the system to generate an output (text,
image, video, speech) (Company, 2023). Despite its advertised simplicity,
recent work with non-experts, even those from technical backgrounds, highlight
the challenges when using prompting as an interaction technique with large-
language models (LLMs) (Zamfirescu-Pereira et al., 2023; Kotturi et al.,
2024a). Given the limitations of prompting, HCI researchers and practitioners
have tried to address these shortcomings in a number of ways. For instance,
researchers have created and published guidelines and practices for prompting
generative systems (Liu and Chilton, 2022), investigated how to aid in the
prompt writing process by automatically generating variations of prompts
(Strobelt et al., 2023), and enabled users to repeatedly assess and iterate on
prompts (Wu et al., 2023). Further, recent work has shown how socially-
situated generative AI can aid co-creation and collaboration between designers
(Verheijden and Funk, 2023) and evidenced the benefit of generative AI in
stimulating collaborative ideation processes (Shin et al., 2023). While prior
scholarship focused on settings where users are working towards shared goals,
our work considers the use of generative AI in a context where users seeking
social support exhibit behavior as both collaborators and competitors (Kuhn
and Galloway, 2015), and additionally must navigate a complex environment for
information sharing to safeguard intellectual property and reputation.

##  3\. Methodology

We conducted 11 semi-structured interview comprising both open-ended interview
(Section 3.1.1) and probe-based questions (Section 3.1.2) with seven
entrepreneurs (E1-E7) and four entrepreneur support staff (P1-P4); see Table
1. We recruited participants from two local entrepreneurial hubs based in
Pittsburgh focused on racial and gender equity in entrepreneurship and
technology. Three of four support personnel held or were pursuing graduate
degrees, and provided technical support within the entrepreneurial hubs (e.g.,
as one-on-one tutors (Kotturi et al., 2022)). The research team had multi-year
collaborations with both entrepreneurial hubs, where the last author started
an on-going technology support program and entrepreneurial programming in both
hubs (Kotturi et al., 2022, 2024b), providing the needed foundation of mutual
understanding and trust, and historical context to engage in the research
study (Le Dantec and Fox, 2015; Harrington et al., 2019). Entrepreneurs had a
range of product and service-based businesses such as apparel, dance, gift
baskets, painting, and education. Recruitment prioritized those who had some
exposure or experience with generative AI platforms such as: Canva text-to-
image, ChatGPT, DALL-E, or Gemini. Details on entrepreneurs’ businesses and
uses of generative AI can be found in Table 1. Participants were compensated
$20 per hour.

###  3.1. Interviews

Interviews included two parts: open-ended questions about participants’
backgrounds and businesses, followed by probing potential futures with paper
storyboards.

####  3.1.1. Part 1: Background questions

First, we asked a series of questions about participants’ businesses and/or
experiences mentoring business owners, their technology background or how they
have integrated technology within their business (e.g., “How comfortable are
you with technology in your daily operations?”), and what their support
networks look like (e.g., “What kind of support do you find helpful when
navigating problems in your business?”). We also asked questions specifically
about generative AI technologies: which platforms participants used and their
concerns. As interviews were rooted in past experiences, we also wanted engage
participants in deeper discussion about potential and preferred futures, as it
related to integrating generative AI technologies into their business
workflows, and how their support networks (existing or desired) might play a
role in this next technological frontier. Therefore, in the second part of
interviews, we used probes in the form of paper storyboards, detailed in the
next section.

####  3.1.2. Part 2: Storyboards as Probes

Storyboards—or short scenarios of possible futures—can be an effective way to
engage users in grounded discussions of preferred futures (Truong et al.,
2006), especially in a workplace context undergoing rapid technological
transformation (Kawakami et al., 2023). In our study, we created a set of
paper storyboards to probe how entrepreneurs may (and may not) integrate
generative AI into their business workflows, and how their offline social
networks may facilitate this process. To derive storyboard scenarios, we used
Ideation Decks, a brainstorming tool that helps define specific design
problems within a broader problem space (Golembewski and Selby, 2010). To
ensure storyboards represented adequate coverage of the relevant design space
and was informed by prior work, we generated five category suits, with 5-11
instances in each category: (1) which local network the entrepreneur sought
support from such as local community centers (Dillahunt et al., 2022), peers
(Kotturi et al., 2024b), technical providers (Kotturi et al., 2022), family
(Hui et al., 2020) (2) the form factor for delivering social support when not
co-located such as chat-based, email-based, browser-based, or SMS
interventions (to meet entrepreneurs where they are, what tools they currently
use (Kotturi et al., 2024a), rather than introducing an entirely new system
(Grevet and Gilbert, 2015)) (3) the stages of the entrepreneurs’ workflows
where generative AI may be more or less helpful, such as early on when
brainstorming new product or service ideas (Gero et al., 2022) versus when
refining materials such as business plans (4) various pain points for
entrepreneurs as detailed in prior work (Kotturi et al., 2022), such as
generating content (for social media, websites, etc) (Kotturi et al., 2022),
or navigating legal issues when using AI-generated content for business
purposes (Kotturi et al., 2024a), and finally, (5) various configurations of
blended human-machine support given that entrepreneur support networks may not
always be available (especially busy peers (Kotturi et al., 2024b)). Note:
each storyboard did not contain all category suits, but rather a compact
subset in order for the scenario to fit within a short, 6-panel layout. See
the Appendix Exploring the Role of Social Support when Integrating Generative
AI in Small Business Workflows for a subset of storyboards created.

Throughout the duration of the study and based on participants’ feedback, we
removed, refined, and added storyboards to maximize discussion, resulting in
26 storyboards total. Language in the presented storyboards avoided technical
jargon for non-experts. Following background questions (Part 1), the
interviewer presented a subset of the storyboards that best responded to the
emergent needs that arose from participants’ responses to background
questions. For example, if an entrepreneur was not interested in using image-
based generative AI tools, we did not show them storyboards which featured a
scenario of an entrepreneur wanting to use image-generation technology. After
storyboards were presented, the interviewer asked participants to reflect upon
and compare them.

PID |  Role |  Job/Business Description |  Uses of Generative AI |  Frequency
---|---|---|---|---
E1 |  Local Entrepreneur |  Streetwear clothing brand |  Creating animation code, podcast editing, captioning |  Often
E2 |  Local Entrepreneur |  Gift Basket Business |  Image & text generation for social media |  Often
E3 |  Local Entrepreneur |  Choreography, dance classes and fitness lessons & podcast |  Background & financial information |  Minimal
E4 |  Local Entrepreneur |  Wholesale letterpress & educational programming |  Organizing lesson plans |  Minimal
E5 |  Local Entrepreneur |  Bolly-fusion dance classes |  Writing emails |  Often
E6 |  Local Entrepreneur |  Wholesale letterpress & educational programming |  Rewording copy, Generating social media captions |  Minimal
E7 |  Local Entrepreneur |  Artist; Painting |  Creating Instagram Captions |  Minimal
P1 |  Support Personnel |  Tech-support volunteer |  — |  N/A
P2 |  Support Personnel |  Tech-support volunteer |  Cooking recipes |  Minimal
P3 |  Support Personnel |  Co-founder of local entrepreneurship hub |  Coding |  Minimal
P4 |  Support Personnel |  Co-founder of local entrepreneurship hub |  Writing emails |  Often

Table 1. Participants included both entrepreneurs (“E_”) and support personnel
(“P_”) who had diverse businesses, used generative AI for a range of tasks and
at various frequencies; together, this provided well-rounded perspectives in
this exploratory study.

Summary of Participants’ Roles and AI Usage. This table summarizes
participants, including local entrepreneurs and support personnel, detailing
their roles, business descriptions, uses of generative AI, and the frequency
of AI usage. Entrepreneurs used generative AI for tasks such as animation code
creation, podcast editing, captioning, and social media content generation.
Support personnel primarily provided tech support and engaged in activities
like coding and email writing.

###  3.2. Data Analysis

On average, interviews lasted 47.6 minutes and 5-10 storyboards were presented
in each interview. 5 out of 11 interviews were conducted remotely via Zoom,
based on participants’ preferences. In-person interviews took place at
Carnegie Mellon’s laboratory or a local community center. Both the first and
last authors took turns interviewing participants while the other took field
notes; all interviews were audio recorded. Audio recordings were transcribed
automatically with Temi.com, which the first author then checked for errors
and fixed by hand before the team engaged in thematic analysis. We adhered to
quote editing conventions commonplace in applied social science research
methods (Corden and Sainsbury, 2006). The first author created an affinity
diagram, clustering key themes which emerged from open-coding the interview
transcripts such as tensions to use and social support (operational barriers,
privacy concerns) and benefits of social support (staying in the loop,
reducing costs).

The first author then wrote analytic memos for each participant with an
average word count of 1,060 words (Charmaz, 2006, Page 72). Analytic memos—or
short narratives used to translate rich qualitative data into narrative
format—were critical to triangulate participants’ responses to interview
questions, alongside open-ended discussions about the storyboards they were
presented (Charmaz, 2006). Memos were structured around sections which
expanded upon the benefits and tensions of social support: what AI
technologies participants used, desired to use, or refused to use; social
support preferences; workflows. Each analytic memo was then shared with the
research team for review, and the last author provided detailed comments to
engage in back and forth discussion.

##  4\. Findings

The entrepreneurs in our study leveraged their offline networks to appropriate
existing single-user generative AI platforms for shared use by sharing
accounts and working together to use generative AI collaboratively in-person
and online. Entrepreneurs reflected on the value of these small networks, by
either reflecting on their own experiences or in response to brainstorming
with the probes if they had not yet had the opportunity to build up such a
network. For instance, in these small, trusted networks, entrepreneurs
overcame technology-driven anxiety, collaborated on formulating prompts, and
discovered new ways to use the technology. However, entrepreneurs had to
navigate unclear standards for behavior, the potential for dampening self-
efficacy, and embarrassment that could occur when collaborating with their
network on generative AI platforms. To begin to unpack the dynamics the social
networks facilitating entrepreneurs’ use of generative AI in our study, we
detail them here.

###  4.1. Forms of Social Support, and Benefits

Often, entrepreneurs’ introduction to generative AI technologies were driven
by a fear of being “left behind” (P4). Entrepreneurs’ offline networks were
critical in turning this anxiety into action, by providing the supportive
nudge to learn more about the technology. For example, E3, who ran a dance
business and was also a full-time writer, described a conversation she had
with a her spouse where she voiced her concern: “[If] Gmail autofills and
completes my emails at this point, [what’s] stopping it from auto-filling all
of my documents that I’m spending …three months writing?” She recalled her
spouse’s response, “ ‘You should just learn it and then be more informed and
be less afraid.’ ”. E3 shared that this conversation with her spouse motivated
her to join a Udemy class on prompting and start exploring ChatGPT more. E4
and E6, who owned a letterpress business together, encountered ChatGPT for the
first time at a recent small business incubator. E4 described how her and her
business partner (E6)—who were in their 30s—felt like “boomers,” having not
encountered the technology while everyone else seemed to be using it. Amidst
this anxiety, E6 noted the helpfulness of another entrepreneur there who went
on to teach her how to use ChatGPT.

Support personnel also took on an active role to critically engage
entrepreneurs with the technology. For instance, P4, the co-founder of a local
entrepreneurial hub focused on equitable community development, felt the
responsibility to be the person in entrepreneurs’ networks that provided the
supportive nudge for entrepreneurs to get started. P4 referred to the
developments in the last year the “GPT Revolution” where “It felt like [an]
earthquake and a divide in the ground, and I was slowly moving away from the
rest of society.” They felt that if they did not “start integrating AI into
[their] daily practices” they would be “left behind”. Because of this feeling,
they took on responsibility to encourage the entrepreneurs that they serve to
follow suit: “I tell everyone I work with that [they] really need to get on
board because they will be left behind…Even if I don’t necessarily feel that
I’m on the other side of the divide yet, I certainly encourage the
entrepreneurs and the other contractors and freelancers [and] business folks
that I work with.” To P4, the “divide” represented the digital divide between
those who are learning about generative AI and those who are not. In fact, E2
later learned about generative AI technologies from a workshop at the
community hub that P4 co-founded. Whether through peers, spouses, business
partners or support personnel, entrepreneurs’ local networks were key in
engaging them in critical use.

Despite predominant generative AI platforms being designed for single-user
use, E1, E2, and E5 leveraged their local networks to collaboratively use
generative AI technologies by sharing accounts and working together in-person.
By working together while using generative AI technologies, entrepreneurs were
able to observe prompting strategies, build self-efficacy, and discover new
ways to apply the technology. For instance, E1, who ran a streetwear clothing
brand with six other business partners, shared access to a ChatGPT Premium
account with everyone in the business. E1 noted that by sharing access to the
account he gets the benefit of learning from his peers mistakes, “I only have
to [fail] maybe once or twice because I’ve seen six other people try and fail
in different ways.”. “Failing” was when the prompt he entered did not produce
the output that he had in mind. By seeing the prompts and the results of his
peers, E1 could passively learn about what did and did not work well when
prompting ChatGPT without the added time to figure it out for himself. He
called this “Group Work for AI”. E1 also pointed out that by sharing an
account, it opened a new way for him and his business partners to keep tabs on
how certain projects were progressing.

E2, who ran a gift basket and party planning business, described how she
recently took part in a workshop on how to use generative AI. At the event she
described how she was paired with a technical provider for side-by-side
support while using ChatGPT. This format helped E2 overcome the apprehension
she had about using generative AI, “When we were able to sit around the table
and you got to sit with a [provider] … [who] makes you feel like you can do it
yourself, and it makes you feel at ease.” For E2, who described her first
experience with text and image generating technologies as “horrifying”, using
the technology alongside a technical provider helped her build self-efficacy
and overcome her initial anxiety in getting started. When the workshop ended,
E2 replicated side-by-side support while using generative AI with her spouse,
who sometimes created prompts for her business tasks. When her spouse or tech
providers were not available to meet side-by-side, E2 shared that she texts
them “three or four times a day” to get support. E5, who ran a dance business
and shared an account with her spouse, described using ChatGPT together: he
brainstormed a new use case for ChatGPT and then created an initial prompt,
acting as a translator of her needs and an intermediary to using the
technology while she focused on daily business operations. While E5 described
using ChatGPT very frequently, she has only used it on her own twice; the rest
of the time, her spouse was there to translate her needs and mediate her use.

While not all entrepreneurs in our study collaborated with others in their
network to use generative AI platforms (because they did not have the network
to do so, or preferred to work independent of others), when responding to
storyboards that featured different forms of social support with generative AI
(e.g. multi-user interfaces), entrepreneurs and support personnel pointed out
the potential benefits. For instance, in response to S17 (see Appendix Figure
3), which featured a SMS prompt sharing service for familiar entrepreneurs, E3
shared how she could work towards “developing [her] own skill set by emulating
or using other folks as a starting point” —similar to how E1 learned from his
business partners prompts through his shared ChatGPT Premium account. Reacting
to S9 (see Appendix Figure 2), which featured entrepreneurs with a shared
community context pairing up to use a generative AI system, P2 and P3
reflected on how such pairings could “bridge the gap [of] comfort around these
tools” (P2)—similar to how E2 experienced in her use of ChatGPT at a recent
community workshop. Overall, entrepreneurs shared interest in systems that
would further enable them to use generative AI technologies with a trusted
confidant, with the hope of experiencing the support E1, E2, and E5 described
to build self-efficacy and to discover new use-cases.

###  4.2. Tensions of Social Support

Participants’ expressed detailed concerns when it came to using generative AI
for their small business, and how social use, even alongside trusted
confidants, may not always be desirable, specifically pointing to reputational
concerns, embarrassment and unclear social norms. To start, in response to S9
(see Appendix Figure 2), which featured two entrepreneurs from the same
community sharing an interface into ChatGPT, P2 highlighted how shared use
with another entrepreneur could intensify feelings of being left behind,
especially if there was a large disparity in how comfortable the entrepreneurs
felt using generative AI. This sentiment was echoed by E3 who said she would
be scared to use a “stupid” prompt that others might see, fretting her
reputation as a competent business owner would be at stake. While E3 was not
open to sharing prompts with others, she indicated that she would be
interested to observe others’ prompts. To E2, simply observing would not be
acceptable, saying “You don’t want it being one person is doing all the work
and the other person [is not]. [Then], you don’t get anything out of it!”.

While entrepreneurs wrestled with these social norms of collaboratively using
generative AI technologies (such as whether to share prompts with others), E1,
who shared a ChatGPT Premium account with business partners, had to figure out
when to share prompts with others. He discussed how he actively chose to use
his personal, free account, despite the lower performing model, when working
on side projects to not “get anybody mad” by adding “clutter” to the account
that he and business partners shared. Similarly who to collaborate on a
generative AI system with added another layer of tension. For instance, E2
reflected on how she would not want to share with a “techie person” because it
was important to her that she had similar “core values” with the person that
she would be sharing with. Entrepreneurs like E4 and E7 illustrated how social
interactions within their networks influenced the formation of their
perspectives and usage patterns of generative AI in their business practices.
For example, E4 acknowledged that her decisions about using ChatGPT was
heavily influenced by her anticipation of her peers’ opinions. When reflecting
on creating an artist statement, she shared how she would feel embarrassed if
a peer asked her if she used ChatGPT to generate it, because she would want
them to think she was “smart enough already to be able to write it from [her]
own intellect and brain rather than [needing] a tool to help [her]”. On the
other hand, E7 (also an artist entrepreneur) shared that, while she was eager
to talk about image-generation technologies with other artists, her
uncertainty around others’ opinions made it difficult to have those
conversations. She believed that, for art, as long as people were transparent
about their use of generative AI technologies (specifically image-generation
platforms), they should be allowed to use them as they wish. E7 indicated that
talking about these technologies with peers is “a conversation that depending
on [who] you’re talking to [can] be a bit of a touchy subject” and that unless
E7 found other entrepreneurs excited to talk about generative AI, she held
back. In this way, her uncertainty around other entrepreneurs’ stance in using
generative AI, fueled by myriad controversies related to workforce roles,
copyright, and data privacy, among others, made it difficult for her to
console her network’s opinions on the ethical aspects of different use cases.

##  5\. Limitations

One limitation of this work was that participants tended to be the most
engaged entrepreneurs at the entrepreneurial hubs we recruited from,
potentially skewing the results to reflect more opportunistic relationships
with generative AI (i.e., those who had less interest in generative AI may
have been less likely to respond to our recruitment efforts). While
entrepreneurs and support personnel were keen to discuss the pitfalls of using
generative AI for small business, future work can prioritize more critical
stances and take up longitudinal approaches to observe use (and non-use) over
the long term. Further, the small sample size of 11 participants limits the
robustness of our findings.

##  6\. Discussion and Future Work

From our exploratory study, we provided a snapshot for how small, offline
networks facilitated entrepreneurs’ adoption, continued use, and refusal of
use of generative AI systems, as well as the tensions of this social support.
As the chat-based interaction paradigms of predominant generative AI
technologies inevitably evolve in attempts to better support workers (Morris
et al., 2023), we consider how current trajectories of interface design may
overlook the needs of workers with less traditional forms of employment, such
as the entrepreneurs in our study. For instance, while a growing body of
scholarship aims to address social use of generative AI tools in an
organizational context (Shin et al., 2023; Hauptman et al., 2022; Brachman et
al., 2024)—where workers have shared goals, teams and incentive structures—the
entrepreneurs in our study pursued distinct goals, and worried about their
reputations and intellectual property when seeking support from their offline
networks (even among vetted networks). And yet, the entrepreneurs in our study
resourcefully leveraged social support to overcome isolated work conditions
and the difficulty of keeping pace with technological advancements, such as by
repurposing single-user accounts for collaborative use (not unlike how crowd
workers overcome isolation and complete tasks collaboratively (Gray et al.,
2016).

Looking towards future work, we consider theoretical frameworks that may be
helpful when informing how generative AI systems could account for the
benefits and tensions of social support, due to the nuanced social
relationships between entrepreneurs and their support networks. For instance,
one relevant theory may be “coopetition” (Kuhn and Galloway, 2015)—or how
social relationships between entrepreneurs and their networks comprise a
combination of competitive and cooperative dynamics. Concretely, generative AI
systems for collaborative use among entrepreneurs could include more dynamic
controls for information sharing, where a user could select which parts of
their inputs and outputs are included or excluded from what other users see
(also known as “broadcast levels” (Oeldorf-Hirsch and Gergle, 2020)). By
providing flexible visibility options, designers can aid entrepreneurs in
navigating disparate preferences when engaging social support to use
generative AI technology outside of often infrequent in-person support. This
could be particularly important in the case of ethics-based conversations
between entrepreneurs, where one entrepreneurs’ use of generative AI
technology could be used to harm their reputation by another entrepreneur
looking to gain a competitive edge.

##  7\. Conclusion

This paper investigated the role of social support—among vetted, offline
networks—for small business owners integrating generative AI into their
workflows. To do so, we probed the benefits and tensions of entrepreneurs’
current and preferred sources of social support. Despite being designed for
single-user use, entrepreneurs shared the benefits of collaboratively using
generative AI: discovering new use cases, developing self-efficacy, easing
techno-anxieties, and wrap-around support. Yet, even among the trusted and
vetted networks they built, these configurations also facilitated social
comparison, feelings of being left behind, reputational concerns, and surfaced
unclear social norms during shared use of generative AI. We therefore consider
how generative AI systems can be designed with these nuanced relationships—a
combination of cooperation and competition—to support both use and non-use.

## References

  * (1)
  * Avle et al. (2019) Seyram Avle, Julie Hui, Silvia Lindtner, and Tawanna Dillahunt. 2019.  Additional Labors of the Entrepreneurial Self.  _Proc. ACM Hum.-Comput. Interact._ 3, CSCW, Article 218 (nov 2019), 24 pages.  https://doi.org/10.1145/3359320
  * Brachman et al. (2024) Michelle Brachman, Amina El-Ashry, Casey Dugan, and Werner Geyer. 2024.  How Knowledge Workers Use and Want to Use LLMs in an Enterprise Context. In _Extended Abstracts of the CHI Conference on Human Factors in Computing Systems_. 1–8.
  * Charmaz (2006) Kathy Charmaz. 2006.  _Constructing Grounded Theory: A Practical Guide Through Qualitative Analysis_. Vol. 1.  72–95.
  * Company (2023) McKinsey & Company. 2023.  What is prompt engineering?  _McKinsey Explainers_ (2023).  https://www.mckinsey.com/featured-insights/mckinsey-explainers/what-is-prompt-engineering
  * Corden and Sainsbury (2006) Anne Corden and Roy Sainsbury. 2006.  Using Verbatim Quotations in Reporting Qualitative Social Research: The views of research users.  (01 2006).
  * Dillahunt et al. (2018) Tawanna R Dillahunt, Vaishnav Kameswaran, Desiree McLain, Minnie Lester, Delores Orr, and Kentaro Toyama. 2018.  Entrepreneurship and the socio-technical chasm in a lean economy. In _Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems_. 1–14.
  * Dillahunt et al. (2022) Tawanna R Dillahunt, Alex Jiahong Lu, Aarti Israni, Ruchita Lodha, Savana Brewer, Tiera S Robinson, Angela Brown Wilson, and Earnest Wheeler. 2022.  The Village: Infrastructuring Community-based Mentoring to Support Adults Experiencing Poverty. In _Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems_ (New Orleans, LA, USA) _(CHI ’22)_. Association for Computing Machinery, New York, NY, USA, Article 574, 17 pages.  https://doi.org/10.1145/3491102.3501949
  * Garcia et al. (2020) Patricia Garcia, Tonia Sutherland, Marika Cifor, Anita Say Chan, Lauren Klein, Catherine D’Ignazio, and Niloufar Salehi. 2020.  No: Critical refusal as feminist data practice. In _conference companion publication of the 2020 on computer supported cooperative work and social computing_. 199–202.
  * Gero et al. (2022) Katy Ilonka Gero, Vivian Liu, and Lydia Chilton. 2022.  Sparks: Inspiration for science writing using language models. In _Proceedings of the 2022 ACM Designing Interactive Systems Conference_. 1002–1019.
  * Golembewski and Selby (2010) Michael Golembewski and Mark Selby. 2010.  Ideation decks: a card-based design ideation tool. In _Proceedings of the 8th ACM Conference on Designing Interactive Systems_. 89–92.
  * Gray et al. (2016) Mary L. Gray, Siddharth Suri, Syed Shoaib Ali, and Deepti Kulkarni. 2016.  The Crowd is a Collaborative Network. In _Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing_ (San Francisco, California, USA) _(CSCW ’16)_. Association for Computing Machinery, New York, NY, USA, 134–147.  https://doi.org/10.1145/2818048.2819942
  * Grevet and Gilbert (2015) Catherine Grevet and Eric Gilbert. 2015.  Piggyback Prototyping: Using Existing, Large-Scale Social Computing Systems to Prototype New Ones. In _Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems_ (Seoul, Republic of Korea) _(CHI ’15)_. Association for Computing Machinery, New York, NY, USA, 4047–4056.  https://doi.org/10.1145/2702123.2702395
  * Harrington et al. (2019) Christina Harrington, Sheena Erete, and Anne Marie Piper. 2019.  Deconstructing community-based collaborative design: Towards more equitable participatory design engagements.  _Proceedings of the ACM on Human-Computer Interaction_ 3, CSCW (2019), 1–25.
  * Hauptman et al. (2022) Allyson I. Hauptman, Wen Duan, and Nathan J. Mcneese. 2022.  The Components of Trust for Collaborating With AI Colleagues. In _Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing_ (Virtual Event, Taiwan) _(CSCW’22 Companion)_. Association for Computing Machinery, New York, NY, USA, 72–75.  https://doi.org/10.1145/3500868.3559450
  * Hui et al. (2020) Julie Hui, Nefer Ra Barber, Wendy Casey, Suzanne Cleage, Danny C. Dolley, Frances Worthy, Kentaro Toyama, and Tawanna R. Dillahunt. 2020.  Community Collectives: Low-tech Social Support for Digitally-Engaged Entrepreneurship. In _Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems_ (, Honolulu, HI, USA,) _(CHI ’20)_. Association for Computing Machinery, New York, NY, USA, 1–15.  https://doi.org/10.1145/3313831.3376363
  * Hui et al. (2018) Julie Hui, Kentaro Toyama, Joyojeet Pal, and Tawanna Dillahunt. 2018.  Making a living my way: Necessity-driven entrepreneurship in resource-constrained communities.  _Proceedings of the ACM on Human-Computer Interaction_ 2, CSCW (2018), 1–24.
  * Kawakami et al. (2023) Anna Kawakami, Shreya Chowdhary, Shamsi T Iqbal, Q Vera Liao, Alexandra Olteanu, Jina Suh, and Koustuv Saha. 2023.  Sensing wellbeing in the workplace, why and for whom? envisioning impacts with organizational stakeholders.  _Proceedings of the ACM on Human-Computer Interaction_ 7, CSCW2 (2023), 1–33.
  * Kotturi et al. (2024a) Yasmine Kotturi, Angel Anderson, Glenn Ford, Michael Skirpan, and Jeffrey P Bigham. 2024a.  Deconstructing the Veneer of Simplicity: Co-Designing Introductory Generative AI Workshops with Local Entrepreneurs. In _Proceedings of the CHI Conference on Human Factors in Computing Systems_. 1–16.
  * Kotturi et al. (2022) Yasmine Kotturi, Herman T Johnson, Michael Skirpan, Sarah E Fox, Jeffrey P Bigham, and Amy Pavel. 2022.  Tech Help Desk: Support for Local Entrepreneurs Addressing the Long Tail of Computing Challenges. In _Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems_ (New Orleans, LA, USA,) _(CHI ’22)_. Association for Computing Machinery, New York, NY, USA, Article 15, 15 pages.  https://doi.org/10.1145/3491102.3517708
  * Kotturi et al. (2024b) Yasmine Kotturi, Jenny Yu, Pranav Khadpe, Erin Gatz, Harvey Zheng, Sarah E Fox, and Chinmay Kulkarni. 2024b.  Peerdea: Co-Designing a Peer Support Platform with Creative Entrepreneurs.  _Proceedings of the ACM on Human-Computer Interaction_ 8, CSCW1 (2024), 1–24.
  * Kraus et al. (2019) Sascha Kraus, Carolin Palmer, Norbert Kailer, Lukas K. Friedrich, and Jonathan Spitzer. 2019.  Digital entrepreneurship: A research agenda on new business models for the twenty-first century.  _International Journal of Entrepreneurial Behaviour & Research_ 25, 2 (2019), 353–375.  https://www.proquest.com/scholarly-journals/digital-entrepreneurship/docview/2182418213/se-2 Copyright - © Emerald Publishing Limited 2018; Last updated - 2023-11-26; SubjectsTermNotLitGenreText - Balearic Islands; United States–US; Finland; Spain.
  * Kuhn and Galloway (2015) Kristine M Kuhn and Tera L Galloway. 2015.  With a little help from my competitors: Peer networking among artisan entrepreneurs.  _Entrepreneurship Theory and Practice_ 39, 3 (2015), 571–600.
  * Le Dantec and Fox (2015) Christopher A Le Dantec and Sarah Fox. 2015.  Strangers at the gate: Gaining access, building rapport, and co-constructing community-based research. In _Proceedings of the 18th ACM conference on computer supported cooperative work & social computing_. 1348–1358.
  * Liu and Chilton (2022) Vivian Liu and Lydia B Chilton. 2022.  Design Guidelines for Prompt Engineering Text-to-Image Generative Models. In _Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems_ (, New Orleans, LA, USA,) _(CHI ’22)_. Association for Computing Machinery, New York, NY, USA, Article 384, 23 pages.  https://doi.org/10.1145/3491102.3501825
  * Marcoulides (1989) George A. Marcoulides. 1989.  Measuring Computer Anxiety: The Computer Anxiety Scale.  _Educational and Psychological Measurement_ 49, 3 (1989), 733–739.  https://doi.org/10.1177/001316448904900328 arXiv:https://doi.org/10.1177/001316448904900328
  * Michael M. Grynbaum and Ryan Mac (2023) Michael M. Grynbaum and Ryan Mac. 2023.  New York Times sues OpenAI and Microsoft over A.I. Use of Copyrighted Work.  _The New York Times_ (2023).  https://www.nytimes.com/2023/12/27/business/media/new-york-times-open-ai-microsoft-lawsuit.html
  * MIT Sloan (2023) MIT Sloan. 2023.  Legal Issues Presented by Generative AI.  https://mitsloan.mit.edu/ideas-made-to-matter/legal-issues-presented-generative-ai.  Accessed: 2024-15-01.
  * Morris et al. (2023) Meredith Ringel Morris, Carrie J. Cai, Jess Holbrook, Chinmay Kulkarni, and Michael Terry. 2023.  The Design Space of Generative Models.  arXiv:2304.10547 [cs.AI]
  * Oeldorf-Hirsch and Gergle (2020) Anne Oeldorf-Hirsch and Darren Gergle. 2020.  ’Who Knows What’: Audience Targeting for Question Asking on Facebook.  _Proc. ACM Hum.-Comput. Interact._ 4, GROUP, Article 11 (jan 2020), 20 pages.  https://doi.org/10.1145/3375191
  * OpenAI (2024) OpenAI. 2024.  ChatGPT.  https://openai.com/chatgpt/.  Accessed: 2024-05-22.
  * Otis et al. (2023) Nicholas G. Otis, Rowan P. Clarke, Solene Delecourt, David Holtz, and Rembrand Koning. 2023.  The Uneven Impact of Generative AI on Entrepreneurial Performance.  OSF Preprints.  https://doi.org/10.31219/osf.io/hdjpk
  * Pei and Crooks (2020) Lucy Pei and Roderic Crooks. 2020.  Attenuated Access: Accounting for Startup, Maintenance, and Affective Costs in Resource-Constrained Communities. In _Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems_ (, Honolulu, HI, USA,) _(CHI ’20)_. Association for Computing Machinery, New York, NY, USA, 1–15.  https://doi.org/10.1145/3313831.3376587
  * Roberts (1991) Edward B Roberts. 1991.  _Entrepreneurs in high technology: Lessons from MIT and beyond_.  Oxford University Press.
  * Shin et al. (2023) Joon Gi Shin, Janin Koch, Andrés Lucero, Peter Dalsgaard, and Wendy E. Mackay. 2023.  Integrating AI in Human-Human Collaborative Ideation. In _Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems_ (Hamburg, Germany) _(CHI EA ’23)_. Association for Computing Machinery, New York, NY, USA, Article 355, 5 pages.  https://doi.org/10.1145/3544549.3573802
  * Staff (2023) IBM Research Staff. 2023.  What is Generative AI?  _IBM Research Blog_ (2023).  https://research.ibm.com/blog/what-is-generative-AI
  * Strobelt et al. (2023) Hendrik Strobelt, Albert Webson, Victor Sanh, Benjamin Hoover, Johanna Beyer, Hanspeter Pfister, and Alexander M. Rush. 2023.  Interactive and Visual Prompt Engineering for Ad-hoc Task Adaptation with Large Language Models.  _IEEE Transactions on Visualization and Computer Graphics_ 29, 1 (2023), 1146–1156.  https://doi.org/10.1109/TVCG.2022.3209479
  * to Visual (2023) Verbal to Visual. 2023.  How to Draw Star People.  https://youtu.be/zq00icc13I4?si=09FoEogESJg6O9OE.  Accessed: 2024-01-23.
  * Tohidi et al. (2006) Maryam Tohidi, William Buxton, Ronald Baecker, and Abigail Sellen. 2006.  Getting the right design and the design right. In _Proceedings of the SIGCHI conference on Human Factors in computing systems_. 1243–1252.
  * Truong et al. (2006) Khai Truong, Gillian Hayes, and Gregory Abowd. 2006.  Storyboarding: an empirical determination of best practices and effective guidelines. 12–21.  https://doi.org/10.1145/1142405.1142410
  * Verheijden and Funk (2023) Mathias Peter Verheijden and Mathias Funk. 2023.  Collaborative Diffusion: Boosting Designerly Co-Creation with Generative AI. In _Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems_ (, Hamburg, Germany,) _(CHI EA ’23)_. Association for Computing Machinery, New York, NY, USA, Article 73, 8 pages.  https://doi.org/10.1145/3544549.3585680
  * von Hippel (2005) Eric von Hippel. 2005.  _Democratizing Innovation_.  The MIT Press.  https://doi.org/10.7551/mitpress/2333.001.0001 arXiv:https://direct.mit.edu/book-pdf/2092624/book_9780262285636.pdf
  * Wu et al. (2023) Sherry Wu, Hua Shen, Daniel S Weld, Jeffrey Heer, and Marco Tulio Ribeiro. 2023.  ScatterShot: Interactive In-context Example Curation for Text Transformation. In _Proceedings of the 28th International Conference on Intelligent User Interfaces_ (Sydney, NSW, Australia) _(IUI ’23)_. Association for Computing Machinery, New York, NY, USA, 353–367.  https://doi.org/10.1145/3581641.3584059
  * Zamfirescu-Pereira et al. (2023) J.D. Zamfirescu-Pereira, Richmond Y. Wong, Bjoern Hartmann, and Qian Yang. 2023.  Why Johnny Can’t Prompt: How Non-AI Experts Try (and Fail) to Design LLM Prompts. In _Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems_ (, Hamburg, Germany,) _(CHI ’23)_. Association for Computing Machinery, New York, NY, USA, Article 437, 21 pages.  https://doi.org/10.1145/3544548.3581388

##  Appendix A Example Storyboards

We include a subset of storyboards from our study which probed how offline
networks—such as from local community centers (Kotturi et al., 2022; Hui et
al., 2020)—among entrepreneurs may provide social support online when in-
person meetings were not possible. To improve readability, we overlay text on
the storyboards’ panels. Characters featured in the storyboards were drawn as
“star people” (to Visual, 2023) in attempts to avoid reinforcing gender and
cultural stereotypes, and technical jargon was kept to a minimum. The paper
format of storyboards helped to convey malleability of the ideas and foster
critical conversations with participants (Tohidi et al., 2006).

Figure 2. Storyboard 9 probed how entrepreneurs may provide each other with
informal social support alongside more formal resources (Kotturi et al.,
2024a). In this storyboard, vetted peers who met at a local community center
use a browser and chat-based generative AI platform collaboratively—when they
cannot make it to the center—by seeing and building off of each others’
prompts, and holding each other accountable to continued support.

Figure 3. Storyboard 17 probed how a SMS-based system for social use of
generative AI may meet entrepreneurs where are by leveraging an existing
technology (Grevet and Gilbert, 2015; Kotturi et al., 2022) to more readily
share helpful prompts with other entrepreneurs. Similarly to Storyboard 9,
this storyboard considers how entrepreneurs can support each other when they
are unable to make it in person to their community center.

Figure 4. Storyboard 26 probed how entrepreneurs can learn about integrating
generative AI into their business workflows by learning from peers in their
local community center, specifically through watching demonstrative videos by
those in their community.


###### Abstract

Humans usually perceive the world in a multimodal way that vision, touch,
sound are utilised to understand surroundings from various dimensions. These
senses are combined together to achieve a synergistic effect where the
learning is more effectively than using each sense separately. For robotics,
vision and touch are two key senses for the dexterous manipulation. Vision
usually gives us apparent features like shape, color, and the touch provides
local information such as friction, texture, etc. Due to the complementary
properties between visual and tactile senses, it is desirable for us to
combine vision and touch for a synergistic perception and manipulation. Many
researches have been investigated about multimodal perception such as cross-
modal learning, 3D reconstruction, multimodal translation with vision and
touch. Specifically, we propose a cross-modal sensory data generation
framework for the translation between vision and touch, which is able to
generate realistic pseudo data. By using this cross-modal translation method,
it is desirable for us to make up inaccessible data, helping us to learn the
object’s properties from different views. Recently, the attention mechanism
becomes a popular method either in visual perception or in tactile perception.
We propose a spatio-temporal attention model for tactile texture recognition,
which takes both spatial features and time dimension into consideration. Our
proposed method not only pays attention to the salient features in each
spatial feature, but also models the temporal correlation in the through the
time. The obvious improvement proves the efficiency of our selective attention
mechanism. The spatio-temporal attention method has potential in many
applications such as grasping, recognition, and multimodal perception.

Multimodal Perception,

Vision,

Touch,

Multimodal Translation,

Attention Mechanism,

###### keywords:

##  Chapter 0 Multimodal perception for dexterous manipulation

###  1 Introduction

As in humans, even in the daily task such as cleaning teeth, both visual and
tactile senses are implemented subconsciously. People use eyes to recognise
and locate the toothbrush. When the toothbrush is in the mouth, people use
tactile feedback to complete the blind spot cleaning due to the occluded
views. Humans interact and perceive the world with various feelings including
the vision, touch, sound, taste, and smell. Multiple feelings can be combined
together to have a synergistic effect where the results can not be
accomplished by using each sense separately.

For robotic manipulation, vision and touch are two main sensory modalities for
perception. In robotics, vision is often obtained with a camera by capturing
the image from a distance, and it can provide us the appearance, shape, and
color of the object. The visual modality has been extensively implemented in
many applications such as recognition [1], object detection [2], and pose
estimation [3]. However, it is still challenging to complete the manipulation
task using a single visual modality. For example, it is difficult for robots
to recognise the transparent objects that the color of which is same to
surroundings. In addition to vision, the tactile information is also used by
robotic, enabling the understanding of the world from another dimension. The
tactile information is achieved by direct interaction between tactile sensors
and contacted objects, which can give us the object’s surface properties
including the textures, local geometry, friction, hardness even under
occlusions. The transparent object, that can not be detected by vision, is
able to be easily recognised through touching. Recently, researchers have
achieved considerable progress on the tactile sensing [4, 5, 6, 7], such as
object recognition [8, 9, 10, 11, 12, 13, 14], slip detection [15, 16], shape
perception [17, 18], object exploration [19] and pose estimation [20, 21]. Due
to the complementary characteristics between vision and touch, the combination
of vision and touch for multimodal perception has become a popular topic [22,
23, 24].

In addition, work [25] shows that the human brain employs a multisensory model
to perceive the world in daily experience. It indicates that the same features
from various sensory modalities share a common subspace. Through the common
subspace, it is possible to perform a unitary presentation from multiple
modalities or transfer the feature from one modality to another. For instance,
human is able to imagine the taste of food by smelling it or tell the smell by
the taste, which is called synaesthesia. Inspired by this, progressive
researches have been carried out such as cross model learning especially for
recognition [22, 23, 26, 27, 28], and multimodal translation [29, 30]. We
propose a cross-modal sensory data generation framework for the translation
between vision and touch, through which we can make up realistic pseudo data
to replace the inaccessible real data.

Moreover, the attention mechanism adapted from Nature Language Processing
field comes to a hot topic in robotic manipulation and perception. As the
attention mechanism emphasises the salient features, it is able to learn the
features more effectively and efficiently. We propose a spatio-temporal
attention model for tactile texture recognition. It not only pays focus on
salient spatial features but also models the correlation of each location
through the time. The recognition results show the robustness and efficiency
of our selective attention model.

The rest of paper is presented as follows: An overview of multimodal
perception is discussed including the cross-modal learning in Section 2, 3D
reconstruction in Section 3, multimodal translation in Section 4 and attention
mechanism in Section 5; conclusions and future work are summarised in Section
6.

###  2 Cross-modal Learning

Vision and tactile sensing have been investigated to perceive the objects
respectively. Currently, many researchers try to match the visual information
and the corresponding tactile signals. In [25], it is proved that the
knowledge in different modalities can be transferred. It also means different
modalities shared a common latent space. Lin et al. [31] demonstrate the
correlation between the vision and touch. The tactile features and visual
features are given to recognise if or not they belong to the same object.

Apart from matching the vision and touch, the cross-modal learning has been
explored by many researchers [32, 33, 22, 23, 26, 27, 28]. In [32], a cross-
modal framework is developed in the visual retrieval problem. The tactile
modality is used as the query to search the image in the visual modality. By
using a structured dictionary and a classifier, the modal-invariant
representation can be learned to generate corresponding visual results. From
the opposite perspective, the visual modality is also applied to estimate the
tactile properties [31]. In this work, a shared continuous latent space is
learnt between visual and tactile modalities by using an encoder-decoder
network. In [22, 23, 26, 27, 28], they focus on the task of recognition of
materials. The visual modality and corresponding tactile information are fused
together, which improves the performance of recognition compared with using a
single modality. Most recently, Zheng et al. [34] propose a lifelong learning
framework for material perception with vision and touch. The knowledge is
accumulated in previous learning and works in later tasks, solving the
inefficiency of isolated learning paradigm.

###  3 3D Reconstruction

The vision provides us apparent characteristics such as shape, and tactile
sense gives us the local geometry and precise location. In the object 3D
reconstruction, it is difficult to use the vision to reconstruct an accurate
model due to the occlusion and noise of the measurement. The use of the
tactile sensing for reconstruction is too slow due to the small contact area
and the alignment is challenging. To this end, the tactile sense, used as a
complementary modality, can be applied to refine the shape of the object
combined with vision.

In [24], a 3D shape of the object is firstly generated by using a single view
based on point cloud and a symmetry assumption to determine the grasp point,
and then the tactile sense together with vision are used to estimate the 3D
shape with an Iterated Extended Kalman Filter, minimising the uncertainty of
the measurement. The work [35] implements the Gaussian Process regression for
estimating the shape and the variance is used to measuring the uncertainty for
contact. Subsequently, the shape is updated with the local contact
information. In [36], the Gaussian Process is also used for shape prediction.
Differently, the learned shape priors is applied to reconstruct the shape
globally and effectively rather than using a local update. To determine where
to touch to refine the shape with tactile sensor, the network’s confidence
score of voxel, defined by the output after sigmoid function, is proposed to
measure the uncertainty of the prediction.

###  4 Multimodal Translation

Multimodal translation is applied to transfer the data from one modality to
another, which is able to provide data that are not easy to access. Thanks to
Generative Adversarial Network (GAN) [37], which is proposed to generate the
data with realistic features as training data, that has been widely used in
the multimodal translation [29, 30]. It consists of a generator and a
discriminator, where the generator tries to generate new data that close to
the target distribution, and the discriminator is used to classify if the
input belongs to real data or generated results.

The multimodal translation based on GAN is also applied between vision and
touch. In [38], a generative partial visual-tactile fused framework is
proposed for the object clustering task. The translated results are used to
compensate for the missing of the real data because of occlusion and noises,
which helps us to obtain inaccessible data to improve the performance of
clustering. Li et al. [39] also propose a framework based on GAN to connect
the vision and touch in robotic manipulation. The scene of how the robots
interact with object is deduced using the tactile inputs as well as the
tactile data of object can be inferred with the visual scene. What is more,
the work [40] applies a tactile display system to test the translated tactile
signal by human. Similar to previous methods, the tactile signal is generated
with the visual input, and then the signal is simulated on a vibration device
so that human can test the feeling of generated signal compared with a real
signal.

####  1 “Touching to See” and “Seeing to Feel”

Visual-to-Tactile | Colour-SSIM
---|---
Cloth Types | 0.89717
Network Parameters | 0.90896
ROI, with noise | 0.92059
ROI, no noise | 0.91188
Paired Dataset | 0.90471
Table 1:  The similarity between visual-to-tactile images and real tactile images are measured by Colour-SSIM. The metric is implemented on different conditions and the resulting average is calculated.  Tactile-to-Visual | Colour-SSIM
---|---
Cloth Types | 0.77279
Network Parameters | 0.89595
Paired Dataset | 0.91338
Table 2:  The similarity between tactile-to-visual images and real visual images are measured by Colour-SSIM. The metric is implemented on different conditions and the resulting average is calculated.  | Visual Images | Tactile Images
---|---|---
Iteration | Real Images | Real/Gen. | Real Images | Real/Gen.
1 | 0.9623 | 0.9710 | 0.7776 | 0.8506
2 | 0.9789 | 0.9770 | 0.8249 | 0.8465
3 | 0.9802 | 0.9853 | 0.8902 | 0.8699
4 | 0.9830 | 0.9839 | 0.8952 | 0.8736
5 | 0.9858 | 0.9835 | 0.9118 | 0.8911
6 | 0.9821 | 0.9867 | 0.9007 | 0.8989
7 | 0.9871 | 0.9867 | 0.9127 | 0.9067
8 | 0.9858 | 0.9867 | 0.9141 | 0.9044
9 | 0.9876 | 0.9894 | 0.9099 | 0.8989
10 | 0.9881 | 0.9894 | 0.9131 | 0.9058
Table 3: Classification accuracy results by using real data, and using both
real data and translated data, in visual classification and tactile
classification. Figure 1:  Cross-modal sensory data generation between vision
and touch  Figure 2:  Architecture of the proposed method including visual to
tactile network and tactile to visual network.

Specifically, as is shown in Fig.1, a framework of cross-modal sensory data
generation for visual tactile perception is proposed in [41], visual images
and tactile textures from the fabric are translated to each other by using the
Conditional-GAN. Therefore, it is possible to generate a visualisation by
using tactile sense, or tactile feelings by seeing the objects, which is taken
as ”touching to see” and ”seeing to feel”. The generated results are
photorealistic compared with the real data, and the results can be used to
expand the dataset to improve the performance of recognition.

####  2 The Framework of Cross-modal Sensory Generation

As the overall architecture is shown in Fig.2, the auxiliary information $Y$
denotes data of input modality while the $X$ represents the target modality.
The generator takes the $Y$ to generate the realistic pseudo data that close
to the distribution of $X$, which is used to fool the fixed discriminator. The
discriminator is trained to determine if the input image is from real data or
generated images with a fixed generator. The framework includes the
translation from vision to touch and touch to vision as well. As a result, the
mapping between vision and tactile modality can be learnt through this
framework. The objective function can be represented as follows:

$\begin{array}[]{r}\min_{G}\max_{D}V(D,G)=\mathbb{E}_{x\sim p_{\text{data
}}(x)}[\log D(x\mid y)]+\\\ \mathbb{E}_{z\sim p_{z}(z)}[\log(1-D(G(z\mid
y)))]\end{array}$ (1)

where the $D$ and $G$ represent the generator and discriminator respectively,
$x$ and $y$ denote the data from target domain and input domain, and $z$
represents the noise used in the generation.

####  3 Experimental Results

This work applies the ViTac Cloth dataset [28], which consists of the visual
images and corresponding tactile images collected by a digital camera and a
GelSight sensor [42] respectively from 100 pieces of fabrics. More detailed
information is referred to [28]. Several experiments have been conducted to
measure the performance of the cross-modal translation model under different
conditions. (1) Cloth properties: Different cloth types own unique properties
including the weaving patterns, colours, and textures. To this end, selected
materials are used to test the performance of the model with different
materials. (2) Network parameters: The internal parameters in network have
different impact on the results of generated data. Therefore, the internal
parameters are altered such as increasing the batch size, adding L1 loss into
objective function, increasing the number of iterations, training images and
resolution. (3) Region of Interest: As the tactile data is collected from
different locations of fabric, the Region of Interest (ROI) is selected to
reduce the views of tactile modality that allows the training image more
consistent. (4) Paired Dataset: The generated images are projected back to the
original input domain, enforcing a cycle consist training.

Moreover, a Colour Structural Similarity index (Colour-SSIM) is proposed to
measure the similarity between real image and generated images, which can be
represented as:

$\begin{array}[]{l}\text{ Colour-SSIM }(x,y)=\\\
\frac{\left(2\mu_{x}^{(i)}\mu_{y}^{(i)}+C_{1}\right)\left(2\sigma_{xy}^{(i)}+C_{2}\right)}{\left(\left(\mu_{x}^{(i)}\right)^{2}+\left(\mu_{y}^{(i)}\right)^{2}+C_{1}\right)\left(\left(\sigma_{x}^{(i)}\right)^{2}+\left(\sigma_{y}^{(i)}\right)^{2}+C_{2}\right)}\end{array}$
(2)

where $\mu_{x}$ and $\mu_{y}$ represent the means of pixels value in image $x$
and $y$ respectively, $\sigma_{x}$ and $\sigma_{y}$ denotes the variance,
$\sigma_{xy}$ is the covariance of $x$ and $y$, $i$ represents the index of
channels in image. $C_{1}$ and $C_{2}$ are scalars with a small value.

Figure 3:  The visualisation of generated results while altering the internal
parameters in the network. (a) Input Images (b) Outputs with default
parameters (c) Increase batch size in each iteration (d) Add L1 loss to
objective function (e) Increase iteration number (f) Increase amount of
training images (g) Ground Truth (h) Image resolution changes from 256×256 to
512×512 (i) Select ROI (j) The output is projected back to original domain.

From Table 1, it can be found that most methods give a result around 0.9 of
Colour-SSIM as the generated tactile results own the key characteristic of the
real tactile data. However, a more realistic data can be achieved with further
adjusting the parameters or training methods. From Table 2, it demonstrates a
lower score compared with the results of Table 1. One reason is that the
pattern and colour of visual image is difficult to be inferred from the
tactile information due to different sensing principles. In general, as can be
seen from Fig.3, the proposed methods output realistic results from both
visual to tactile and tactile to visual perspective.

The generated images are also evaluated on the classification tasks. The
generated results together with real data are applied on 11 categories
classification compared with using real data alone. Table 3 shows that the
generated data can effectively improve the performance in the early
iterations. The generated results have great potential to expand the limited
dataset and enable the learning more efficiently.

###  5 Attention Mechanism

The attention mechanism is firstly proposed in the Nature Language Processing
(NLP) field [43]. It is soon applied in the visual perception, such as [44,
45, 46]. The attention mechanism assigns higher weights to address the salient
features and lower weights to suppress the redundant features, which enables
the learning effectively.

Currently, attention mechanism becomes popular in the robotic perception and
manipulation tasks. Gu et al. [47] propose an attention grasping network to
learn the grasp points in the robotic grasping. The salient features are paid
more attention which mitigates the details lose in fully convolutional neural
networks. In the multimodal perception of vision and touch, the vision and
touch are usually fused directly that results in some redundant and in
inefficient features for perception. In [48], a new fusion method based on
self-attention is used to learn representation from both modalities more
effectively to predict the success of grasping.

####  1 Spatio-temporal Attention Model

Recently, a spatio-temporal attention model (STAM) is proposed for the tactile
texture recognition [49]. The textures are shown in Fig.5. The tactile
textures are collected by GelSight sensor (shown in Fig.4) by different
interaction including pressing, sipping, twisting with 100 categories fabrics.
In the spatio-temporal attention model, the spatial attention module is used
to emphasise the crucial area in each frame of texture sequence, whereas
temporal attention is used to select the important features considering the
whole sequence regardless of the distance through the time dimension.

Figure 4: GelSight Sensor. Left: a GelSight Sensor. Right: a GelSight consists
of a piece elastomer coated with reflective membrane, a camera at base, LEDs
and light guiding plate. When the sensor contacts the object, the elastomer is
deformed due to the contact pressure. The geometry of the object’s surface is
taken by the deformation, which is recorded by the camera.  Figure 5: Sample
Tactile Sequences. Leftmost column: visual images recorded by a digital camera
from fabrics; Right four columns: tactile sequences collected by the GelSight
sensor with different interaction with fabric including pressing (first row),
slipping (second row) and twisting (third row).  Figure 6: The proposed STAM
framework for tactile texture recognition. The proposed methods consists of
three parts: (1) a CNN module to extract high dimensional feature map, (2) a
spatial attention module to emphasise informative features in each frame, (3)
a temporal attention module to model the temporal correlation in a whole
sequence. Finally, the model outputs a predicted category by using fully
connection layers.

The overall architecture of spatio-temporal attention model is illustrated in
Fig. 6. The texture sequence is firstly fed into a pretrained CNN to obtain a
higher dimension feature map $\boldsymbol{F}\in\mathbb{R}^{h\times w\times
c}$, then the spatial attention and temporal attention are used to highlight
the salient features from space dimension and time dimension respectively. At
last, the network output label $y$ which is the predicted category of the
texture.

####  2 Spatial Attention

Figure 7: Spatial Attention Module. The average-pooling and max-pooling are
used to learn the feature efficiently. A convolutional layer is used to
calculate the attention map.

The spatial attention aims to assign higher weights to the salient features in
each frame. As is shown in Fig. 7, a max-pooling and average-pooling are
applied on the feature map $\boldsymbol{F}$ to describe the features
effectively. Then a convolutional layer with a kernel of $7\times 7$ with a
sigmoid function are used to calculate the attention map, which describes
which location is more informative. The attention map $\boldsymbol{A_{S}(F)}$
can be calculated as:

$\begin{split}A_{S}(F)&=\sigma(f^{7\times 7}([MaxPool(F);AvgPool(F)]))\\\
&=\sigma(f^{7\times 7}([F_{max}^{S};F_{avg}^{S}])),\end{split}$ (3)

where $\sigma$ represents the activation function. The output feature map can
be calculated as:

$\begin{split}F^{S}=A_{S}(F)\otimes F,\end{split}$ (4)

where $\otimes$ denotes element-wise multiplication.

Figure 8: Temporal Attention Module. $1\times 1\times 1$ convolutions is used
to turn $\boldsymbol{F^{S(n)}}$ into different feature spaces. $\oplus$
denotes element-wise addition.

####  3 Temporal Attention

Different from spatial attention considering a single frame, the temporal
attention is used to compute the relevance of all locations in a sequence. As
is shown in Fig.8, each frame in the sequence is concatenated and denoted as
$\boldsymbol{F^{S(n)}}$ firstly , then $\boldsymbol{F^{S(n)}}$ is turned into
to two different feature spaces by using $1\times 1\times 1$ convolutional
layers, which can be represented as :

$\begin{split}q(F^{S(n)})=W_{q}F^{S(n)}\end{split}$ (5)
$\begin{split}k(F^{S(n)})=W_{k}F^{S(n)}\end{split}$ (6)

After, the $q(F^{S(n)})\;\text{and}\;k(F^{S(n)})\in\mathbb{R}^{m\times c}$,
where $m={n}\times{h}\times{w}$, are used to calculate the temporal attention
map that describe the correlation of all the regions regardless of time. The
attention map can be computed as follows:

$A_{T}(F^{S(n)})_{j,i}=\frac{\exp\left(s_{ij}\right)}{\sum_{i=1}^{m}\exp\left(s_{ij}\right)},$
(7) $\text{where}\quad
s_{ij}=q\left(F^{S(n)}_{i}\right)k\left(F^{S(n)}_{j}\right)^{T}$ (8)

The $A_{T}(F^{S(n)})_{j,i}$ illustrates the extent of $F^{S(n)}_{i}$
correlating with $F^{S(n)}_{j}$. The temporal attention outputs the feature
map:

$F^{T}={\boldsymbol{(}F^{T}_{1},F^{T}_{2},...,F^{T}_{j},...,F^{T}_{m})}$ (9)
$\text{where}\quad
F^{T}_{j}=\sum_{i=1}^{m}A_{T}(F^{S(n)})_{j,i}v\left(F^{S(n)}_{i}\right)+F^{S(n)}_{j}$
(10) $v\left(F^{S(n)}\right)=W_{v}F^{S(n)}$ (11)

where $W_{v}$ denotes a trainable matrix. Moreover, as illustrated in Fig.6,
multiple temporal attention are applied which allows the model a joint
learning from various feature spaces. At last, the outputs of the temporal
attention module are connected with a fully connected layer to predict which
category the perceived fabric is.

Table 4: Texture recognition results with different models and different
lengths of input sequences.

Models | $n=2$ | $n=3$ | $n=4$ | $n=5$ | $n=6$ | $n=7$
---|---|---|---|---|---|---
CNNs | 67.23% | 72.04% | 75.26% | 78.06% | 79.56% | 81.29%
CNNs+Spatial Attention | 72.12% | 73.97% | 78.60% | 80.43% | 80.43% | 80.86%
STAM | 76.50% | 79.35% | 80.00% | 80.64% | 81.72% | 81.93%

Table 5: Texture recognition results while the dataset includes some noisy
data collected before contact.

Models | $n=2$ | $n=3$ | $n=4$ | $n=5$ | $n=6$ | $n=7$
---|---|---|---|---|---|---
CNNs | 53.20% | 58.20% | 59.60% | 61.23% | 64.60% | 69.40%
CNNs+Spatial Attention | 55.40% | 60.80% | 62.60% | 62.80% | 65.40% | 71.00%
STAM | 72.00% | 72.20% | 75.80% | 76.61% | 80.80% | 80.20%

####  4 Experimental Results

The experiments mainly consist of three parts. 1) An ablation study is
performed that the spatial attention module is used separately at first, and
then spatial attention and temporal attention module are applied jointly to
understand the effectiveness of the designed method step by step. 2) Short and
long length of the sequence have a different effect on the the recognition
performance and processing time. Therefore different length of input sequences
is explored in our experiment. 3) Due to the uncertainty in the dynamic
environment, some tactile images are collected before the contact happened.
These tactile images are introduced as noisy data to test the robustness of
the model.

As is shown in Table 4 and Table 5, the proposed method has the best
recognition results in all conditions with different lengths of sequences
used. Compared with using CNNs only, there is a $4.45\%$ average improvement
in recognition accuracy by using proposed method. In a further step, after
introducing the noisy data, the accuracy increases by $15.23\%$ on average
compared with baseline. What is more, with a longer length of sequences
applied, the accuracy has an upward trend.

From Table 4, it can be learnt that the lengths of sequences lead to differing
impacts on performance for different models. For example, when the lengths of
sequences change from 7 to 2, the recognition accuracy of proposed methods
only decrease $5.43\%$, the CNNs method and spatial attention method decrease
$14.06\%$ and $8.74\%$ respectively. It can be found that, by using the
spatial attention methods, the accuracy improves slightly in most cases
compared with baseline. By using both spatial attention and temporal
attention, the performance has a further improvement, achieving the highest
accuracy in all cases. The results indicate that the proposed method has the
ability to extract the obvious features effectively, especially with limited
information.

From Table 5, it can be found that the introduction of noisy data leads to a
lower accuracy on a whole for the texture recognition task. However, the
proposed method sustains a similar recognition results with a small decrease
compared with Table 4 while the other two methods have an obvious drop on the
performance. Specifically, when the length of sequence is 6, accuracy of
baseline drop $14.96\%$ and the recognition accuracy of proposed method falls
only $0.92\%$. The results demonstrate that the proposed spatio-temporal
attention method is robust to the noise and has the ability to select salient
features effectively.

To summarise, two strengths of proposed model can be listed: 1) A recognition
model with strong robustness is proposed. By using the attention mechanism,
the informative information can be selected and noise can be restrained. 2)
The learning of the feature is more efficient and effective. It shows that the
proposed method has a better performance under limited information when the
length of sequence is short.

####  5 Attention Distribution Visualisation

In this section, the attention distributions are visualised for spatial
attention and temporal attention respectively to illustrate how they work in
the recognition tasks.

Spatial Attention Distribution. To visualise the spatial attention
distribution, the gradient class activation maps (Grad-CAM) [50] is performed
on both spatial attention based methods and the methods using CNNs only. In
the Grad-CAM, the gradients are used to express the saliency of region
according to the classification results. The Grad-CAM allows us to understand
the informative location in different methods. As is demonstrated in Fig.9, a
larger contact regions are highlighted by the spatial attention method
compared with baseline, which indicates that more informative locations are
utilized by spatial attention.

Temporal Attention Distribution. To understand the correlation between each
locations regardless of time, the attention map of temporal attention is
visualised. Due to the application of multiple temporal attention, the value
of attention maps are averaged. As is shown in Fig.10, a green dot is randomly
selected in the first frame, then 3 most related locations which is pointed by
arrow are highlighted according to the attention map in the other frames. As
most highlighted regions are located in latest contact area, it shows that the
temporal attention is able to make use of latest information in the time
dimension for recognition task.

Figure 9: Spatial Attention Distribution. The first row and second row
represent a same tactile sequence, but activated by non-attention method and
spatial attention method with Grad-CAMs. It can be found that more contact
regions are activated while using spatial attention method. Figure 10:
Temporal Attention Distribution. Upper row: the sequence of tactile textures
in order. Lower row: the temporal attention distribution where the green dot
refers to a randomly selected regions, and the highlighted regions pointed by
arrows are top 3 related regions in each texture. It can be found that most
highlighted regions refer to latest contacted locations

###  6 Conclusion and Discussions

In this chapter, we discuss the multimodal perception for dexterous perception
including the cross-modal learning, 3D reconstruction, multimodal translation
as well as the attention mechanism. Due to the complementary characteristics
between different modalities, it seems a more natural way to enable the robots
multiple modalities to perceive the world as humans. As a result, the robots
are able to observe and interact with the objects from different dimensions,
multiplying the understanding of the physical world. Specifically, we discuss
a cross-modal sensory data generation framework to make up inaccessible data,
and a spatio-temporal attention model for tactile texture recognition, which
can learn the informative feature efficiently. In future work, the selective
attention mechanism can be used to facilitate the multimodal perception and
manipulation. There also exists many promising applications to be explored: 1)
Instead of fusing different modalities directly, the algorithms that can fuse
the vision and tactile sense more effectively is worth to be investigated. 2)
Currently, most of the tasks are isolated learning paradigms. Hence, how to
develop a lifelong learning framework to leverage the prior knowledge,
improving the efficiency of the learning? 3) The robots always pays attention
equally to the surroundings. How to enable the robots to act more like humans
with selective attention, which is necessary to be explored.

## References

## References

  * [1] K. He, X. Zhang, S. Ren, J. Sun, Deep residual learning for image recognition, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770–778.
  * [2] J. Redmon, A. Farhadi, Yolo9000: better, faster, stronger, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 7263–7271.
  * [3] E. Brachmann, A. Krull, F. Michel, S. Gumhold, J. Shotton, C. Rother, Learning 6d object pose estimation using 3d object coordinates, in: European conference on computer vision, Springer, 2014, pp. 536–551.
  * [4] S. Luo, J. Bimbo, R. Dahiya, H. Liu, Robotic tactile perception of object properties: A review, Mechatronics 48 (2017) 54–67.
  * [5] D. F. Gomes, P. Paoletti, S. Luo, Generation of gelsight tactile images for sim2real learning, arXiv preprint arXiv:2101.07169 (2021).
  * [6] D. F. Gomes, Z. Lin, S. Luo, Blocks world of touch: Exploiting the advantages of all-around finger sensing in robot grasping, Frontiers in Robotics and AI 7 (2020).
  * [7] D. F. Gomes, Z. Lin, S. Luo, Geltip: A finger-shaped optical tactile sensor for robotic manipulation, in: 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), IEEE, 2020, pp. 9903–9909.
  * [8] N. Gorges, S. E. Navarro, D. Göger, H. Wörn, Haptic object recognition using passive joints and haptic key features, in: 2010 IEEE International Conference on Robotics and Automation, IEEE, 2010, pp. 2349–2355.
  * [9] S.-H. Kim, J. Engel, C. Liu, D. L. Jones, Texture classification using a polymer-based mems tactile sensor, Journal of micromechanics and Microengineering 15 (5) (2005) 912.
  * [10] R. Li, E. H. Adelson, Sensing and recognizing surface textures using a gelsight sensor, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2013, pp. 1241–1247.
  * [11] S. Luo, W. Mou, K. Althoefer, H. Liu, Novel tactile-sift descriptor for object shape recognition, IEEE Sensors Journal (2015).
  * [12] S. Luo, W. Mou, K. Althoefer, H. Liu, Iterative closest labeled point for tactile object shape recognition, in: IROS, 2016.
  * [13] S. Luo, X. Liu, K. Althoefer, H. Liu, Tactile object recognition with semi-supervised learning, in: ICIRA, 2015.
  * [14] S. Luo, W. Mou, K. Althoefer, H. Liu, iCLAP: Shape recognition by combining proprioception and touch sensing, Autonomous Robots (2019).
  * [15] J. Li, S. Dong, E. Adelson, Slip detection with combined tactile and visual information, in: 2018 IEEE International Conference on Robotics and Automation (ICRA), IEEE, 2018, pp. 7772–7777.
  * [16] J. W. James, N. Pestell, N. F. Lepora, Slip detection with a biomimetic tactile sensor, IEEE Robotics and Automation Letters 3 (4) (2018) 3340–3346.
  * [17] A. Aggarwal, P. Kampmann, J. Lemburg, F. Kirchner, Haptic object recognition in underwater and deep-sea environments, Journal of field robotics 32 (1) (2015) 167–185.
  * [18] M. Meier, M. Schopfer, R. Haschke, H. Ritter, A probabilistic approach to tactile shape reconstruction, IEEE Transactions on Robotics 27 (3) (2011) 630–635.
  * [19] S. Luo, W. Mou, K. Althoefer, H. Liu, Localizing the object contact through matching tactile features with visual map, in: ICRA, 2015.
  * [20] A. Petrovskaya, O. Khatib, S. Thrun, A. Y. Ng, Bayesian estimation for autonomous object manipulation based on tactile sensors, in: Proceedings 2006 IEEE International Conference on Robotics and Automation, 2006. ICRA 2006., IEEE, 2006, pp. 707–714.
  * [21] J. Bimbo, S. Luo, K. Althoefer, H. Liu, In-hand object pose estimation using covariance-based tactile to geometry matching, IEEE Robotics and Automation Letters 1 (1) (2016) 570–577.
  * [22] O. Kroemer, C. H. Lampert, J. Peters, Learning dynamic tactile sensing with robust vision-based training, IEEE transactions on robotics 27 (3) (2011) 545–557.
  * [23] T. Taunyazov, W. Sng, H. H. See, B. Lim, J. Kuan, A. F. Ansari, B. C. Tee, H. Soh, Event-driven visual-tactile sensing and learning for robots, perception 4 5.
  * [24] J. Ilonen, J. Bohg, V. Kyrki, Fusing visual and tactile sensing for 3-d object reconstruction while grasping, in: 2013 IEEE International Conference on Robotics and Automation, IEEE, 2013, pp. 3547–3554.
  * [25] S. Lacey, C. Campbell, K. Sathian, Vision and touch: multiple or multisensory representations of objects?, Perception 36 (10) (2007) 1513–1521.
  * [26] P. Falco, S. Lu, A. Cirillo, C. Natale, S. Pirozzi, D. Lee, Cross-modal visuo-tactile object recognition using robotic active exploration, in: 2017 IEEE International Conference on Robotics and Automation (ICRA), IEEE, 2017, pp. 5273–5280.
  * [27] H. Liu, Y. Yu, F. Sun, J. Gu, Visual–tactile fusion for object recognition, IEEE Transactions on Automation Science and Engineering 14 (2) (2016) 996–1008.
  * [28] S. Luo, W. Yuan, E. Adelson, A. G. Cohn, R. Fuentes, Vitac: Feature sharing between vision and tactile sensing for cloth texture recognition, in: 2018 IEEE International Conference on Robotics and Automation (ICRA), IEEE, 2018, pp. 2722–2727.
  * [29] J. Johnson, A. Gupta, L. Fei-Fei, Image generation from scene graphs, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 1219–1228.
  * [30] T.-H. Oh, T. Dekel, C. Kim, I. Mosseri, W. T. Freeman, M. Rubinstein, W. Matusik, Speech2face: Learning the face behind a voice, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 7539–7548.
  * [31] J. Lin, R. Calandra, S. Levine, Learning to identify object instances by touch: Tactile recognition via multimodal matching, in: 2019 International Conference on Robotics and Automation (ICRA), IEEE, 2019, pp. 3644–3650.
  * [32] H. Liu, F. Wang, F. Sun, B. Fang, Surface material retrieval using weakly paired cross-modal learning, IEEE Transactions on Automation Science and Engineering 16 (2) (2018) 781–791.
  * [33] K. Takahashi, J. Tan, Deep visuo-tactile learning: Estimation of tactile properties from images, in: 2019 International Conference on Robotics and Automation (ICRA), IEEE, 2019, pp. 8951–8957.
  * [34] W. Zheng, H. Liu, F. Sun, Lifelong visual-tactile cross-modal learning for robotic material perception, IEEE transactions on neural networks and learning systems (2020).
  * [35] M. Björkman, Y. Bekiroglu, V. Högman, D. Kragic, Enhancing visual perception of shape through tactile glances, in: 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems, IEEE, 2013, pp. 3180–3186.
  * [36] S. Wang, J. Wu, X. Sun, W. Yuan, W. T. Freeman, J. B. Tenenbaum, E. H. Adelson, 3d shape perception from monocular vision, touch, and shape priors, in: 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), IEEE, 2018, pp. 1606–1613.
  * [37] I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, Y. Bengio, Generative adversarial networks, arXiv preprint arXiv:1406.2661 (2014).
  * [38] T. Zhang, Y. Cong, G. Sun, J. Dong, Y. Liu, Z. Ding, Generative partial visual-tactile fused object clustering, arXiv preprint arXiv:2012.14070 (2020).
  * [39] Y. Li, J.-Y. Zhu, R. Tedrake, A. Torralba, Connecting touch and vision via cross-modal prediction, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 10609–10618.
  * [40] X. Li, H. Liu, J. Zhou, F. Sun, Learning cross-modal visual-tactile representation using ensembled generative adversarial networks, Cognitive Computation and Systems 1 (2) (2019) 40–44.
  * [41] J.-T. Lee, D. Bollegala, S. Luo, “touching to see” and “seeing to feel”: Robotic cross-modal sensory data generation for visual-tactile perception, in: 2019 International Conference on Robotics and Automation (ICRA), IEEE, 2019, pp. 4276–4282.
  * [42] W. Yuan, R. Li, M. A. Srinivasan, E. H. Adelson, Measurement of shear and slip with a gelsight tactile sensor, in: 2015 IEEE International Conference on Robotics and Automation (ICRA), IEEE, 2015, pp. 304–311.
  * [43] D. Bahdanau, K. Cho, Y. Bengio, Neural machine translation by jointly learning to align and translate, arXiv preprint arXiv:1409.0473 (2014).
  * [44] V. Mnih, N. Heess, A. Graves, K. Kavukcuoglu, Recurrent models of visual attention, arXiv preprint arXiv:1406.6247 (2014).
  * [45] J. Hu, L. Shen, G. Sun, Squeeze-and-excitation networks, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 7132–7141.
  * [46] S. Woo, J. Park, J.-Y. Lee, I. S. Kweon, Cbam: Convolutional block attention module, in: Proceedings of the European conference on computer vision (ECCV), 2018, pp. 3–19.
  * [47] Q. Gu, J. Su, X. Bi, Attention grasping network: A real-time approach to generating grasp synthesis, in: 2019 IEEE International Conference on Robotics and Biomimetics (ROBIO), IEEE, 2019, pp. 3036–3041.
  * [48] S. Cui, R. Wang, J. Wei, J. Hu, S. Wang, Self-attention based visual-tactile fusion learning for predicting grasp outcomes, IEEE Robotics and Automation Letters 5 (4) (2020) 5827–5834.
  * [49] G. Cao, Y. Zhou, D. Bollegala, S. Luo, Spatio-temporal attention model for tactile texture recognition, in: 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2020, pp. 9896–9902.
  * [50] R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, D. Batra, Grad-cam: Visual explanations from deep networks via gradient-based localization, in: ICCV, 2017.