# Deceptive, Disruptive, No Big Deal: Japanese People React to Simulated Dark
Commercial Patterns

Katie Seaborn<EMAIL_ADDRESS>0000-0002-7812-9096 Tokyo Institute
of TechnologyTokyoJapan ,  Tatsuya Itagaki  0009-0006-0455-5611 Tokyo
Institute of TechnologyTokyoJapan<EMAIL_ADDRESS>,  Mizuki
Watanabe  0000-0003-3036-7041 Tokyo Institute of TechnologyTokyoJapan
<EMAIL_ADDRESS>,  Yijia Wang<EMAIL_ADDRESS>0009-0004-2250-9163 Tokyo Institute of TechnologyTokyoJapan ,  Ping Geng
<EMAIL_ADDRESS>0009-0001-8227-7497 Tokyo Institute of
TechnologyTokyoJapan ,  Takao Fujii<EMAIL_ADDRESS>0009-0004-5059-3323 Tokyo Institute of TechnologyTokyoJapan ,  Yuto Mandai
<EMAIL_ADDRESS>0009-0002-5866-155X Tokyo Institute of
TechnologyTokyoJapan ,  Miu Kojima<EMAIL_ADDRESS>0009-0006-6122-6750 Tokyo Institute of TechnologyTokyoJapan and  Suzuka
Yoshida  0009-0004-8694-3073<EMAIL_ADDRESS>Tokyo Institute of
TechnologyTokyoJapan

(2024)

###### Abstract.

Dark patterns and deceptive designs (DPs) are user interface elements that
trick people into taking actions that benefit the purveyor. Such designs are
widely deployed, with special varieties found in certain nations like Japan
that can be traced to global power hierarchies and the local socio-linguistic
context of use. In this breaking work, we report on the first user study
involving Japanese people (n=30) experiencing a mock shopping website injected
with simulated DPs. We found that Alphabet Soup and Misleading Reference
Pricing were the most deceptive and least noticeable. Social Proofs, Sneaking
in Items, and Untranslation were the least deceptive but Untranslation
prevented most from cancelling their account. Mood significantly worsened
after experiencing the website. We contribute the first empirical findings on
a Japanese consumer base alongside a scalable approach to evaluating user
attitudes, perceptions, and behaviours towards DPs in an interactive context.
We urge for more human participant research and ideally collaborations with
industry to assess real designs in the wild.

Dark Patterns, Deceptive Design, Deceptive Design Pattern, Manipulative
Design, Persuasive Design, User Interface Design, User Study, Japan

††journalyear: 2024††copyright: rightsretained††conference: Extended Abstracts
of the CHI Conference on Human Factors in Computing Systems; May 11–16, 2024;
Honolulu, HI, USA††booktitle: Extended Abstracts of the CHI Conference on
Human Factors in Computing Systems (CHI EA ’24), May 11–16, 2024, Honolulu,
HI, USA††doi: 10.1145/3613905.3651099††isbn: 979-8-4007-0331-7/24/05††ccs:
Human-centered computing Empirical studies in HCI††ccs: Human-centered
computing User studies††ccs: Human-centered computing User models††ccs:
Information systems Web applications††ccs: Information systems Web
interfaces††ccs: Human-centered computing Web-based interaction††ccs: Human-
centered computing User interface design††ccs: Applied computing Psychology

Figure 1. User flow of the website representing a typical online shopping
experience involving all DPs. Illustration © shigureni.

A six-part user flow including browsing the homepage, browsing for a set of
items, checking out and signing up, and canceling membership. Also includes
DPs experienced throughout the website.

##  1\. Introduction and Background

Dark patterns and deceptive designs (DPs) are aspects of user interfaces (UI)
that have been specifically designed to manipulate user actions and choices
(Gray et al., 2023; Harry, 2023). Much work has gone into describing (Gray et
al., 2018, 2021b), finding (Di Geronimo et al., 2020; Hidaka et al., 2023; Soe
et al., 2020; Gunawan et al., 2021; Mathur et al., 2019; Kyi et al., 2023;
Yada et al., 2022), and regulating (OECD, 2022, 2023) DPs. However, relatively
less work has explored the _user side_ of deceptive UIs: how everyday
consumers, end-users, and netizens feel about the DPs that they encounter.
Notably, few have explored real digital offerings using participant research
methods (Borberg et al., 2022; Nazarov and Baimukhambetov, 2022; Chaudhary et
al., 2022; M. Bhoot et al., 2021; Gray et al., 2021a). This gap can be traced
to the sensitive and even controversial nature of DPs. When portrayed as
“deceptive,” the implication for companies and other purveyors who employ DPs
is negative. Also, whether a particular interface pattern is “deceptive” and
for whom remains a matter of debate (Fansher et al., 2018; Mathur et al.,
2021; Gray et al., 2018, 2021b; Feng et al., 2023; Kyi et al., 2023).

While ideal, the sensitivity of the topic, ethical restrictions, and practical
issues have hampered work with real users on real products. Some researchers
have not even named the systems evaluated, e.g., (Borberg et al., 2022; Hidaka
et al., 2023; Nazarov and Baimukhambetov, 2022). One alternative is the use of
realistic _simulations_ , which include simulated DPs injected into real
systems (Monge Roffarello and De Russis, 2022; Utz et al., 2019; Berens et
al., 2022; Bermejo Fernandez et al., 2021), imagined designs of complete apps
(Bongard-Blanchy et al., 2023; Schäfer et al., 2023), and fully-fledged
systems created from scratch (Voigt et al., 2021; van Nimwegen and de Wit,
2022; Cranor, 2022; Graßl et al., 2021; Habib et al., 2022). Still, much of
this work focuses on specific contexts or DPs in isolation, e.g., app
permission requests (Bongard-Blanchy et al., 2023), cookie consent banners
(Cranor, 2022; Bermejo Fernandez et al., 2021; Berens et al., 2022; Habib et
al., 2022), the Sneak into Basket/Toying with Emotion DP classes (van Nimwegen
and de Wit, 2022). Moreover, the simulations are often _non-interactive_
—screen shots, mockups, and videos, e.g., (Bongard-Blanchy et al., 2023;
Schäfer et al., 2023; Di Geronimo et al., 2020)—which limits ecological
validity in terms of use case realism and assessment of DPs that require
interaction (Carter et al., 2008). Much more work is needed—with a larger
volume of potential and real end-users across an array of digital offerings
and varieties of DPs—before consensus can be reached.

One emerging user base to study is Japanese consumers. Deception in commercial
UIs has become a key topic in Japan, especially as the number of online
shoppers continues to grow. Nearly 53% of households went online to shop in
2022, a sharp incline from 34.3% in
2018111https://www.statista.com/statistics/1182675/japan-online-shopping-
penetration-households/. Last CHI, Hidaka et al. (2023) reported on a
heuristic analysis of DPs in the Japanese mobile app market. They found that
93.5% of the top 200 Google Play Store apps contained an average of 3.9 DPs
(Hidaka et al., 2023). Moreover, they also discovered the presence of special
Japanese DPs linked to global power structures, i.e., the influence of foreign
corporations and their local presence in Japan, and the Japanese socio-
linguistic context (Hidaka et al., 2023). Even so, to date, no research in
Japan has involved human participants directly experiencing DPs. Japan has
also started to take legal action on DPs (OECD, 2022). The Consumer Affairs
Agency of the Government of Japan also led the 2023 OECD follow-up report on
vulnerability and consumer policy (OECD, 2023). Even so, the degree to which
Japanese netizens are aware of DPs, how they feel about their “deceptive”
properties, and the implications in typical use cases remain unknown.

To this end, we conducted the first human participants user study on DPs with
a diversity of Japanese people. We asked two entwined research questions
(RQs): _RQ1: How deceptive are DPs to the average Japanese person?_ and _RQ2:
How does the average Japanese person feel about each form of DP?_ Due to
ethics restrictions, we developed a simulation (refer to 2.3.1) containing a
range of DPs based on those typically found in real e-commerce websites (refer
to Figure 1 and 2.3 for details). We found that the deceptibility of people
and interfaces varied significantly, and mood was negatively affected by the
experience. We contribute:

  * •

the first empirical findings on how Japanese consumers perceive a variety of
DPs, including the Japanese varieties; and

  * •

a methodology for conducting interactive human participant research on DPs
using a simulated website that can be translated to in-situ studies.

We hope to inspire further interactive, participant-based research on DPs in
Japan and elsewhere. We also aim to create a sensitivity-free artefact and
training tool for engaging consumers and purveyors alike in discussions on
deception in design.

##  2\. Methods

We conducted a human factors user study (Stanton et al., 2017) featuring a
concurrent think-aloud protocol (van den Haak et al., 2003; Alshammari et al.,
2015), questionnaires, and a semi-structured interview (Blandford et al.,
2016). We received ethics approval (#2023265) and conducted the study in
December 2023.

###  2.1. Participants

We recruited 30 Japanese people with the aid of NHK, a public broadcaster and
third-party recruiter, aiming to capture a diverse sample from the larger
population. Fourteen women and sixteen men (none of another gender identity)
participated. Two were aged 18–24, three 25–34, two 35–44, eight 45–54, nine
55–64, four 65–74, and two 75+. Most had gone to university but not completed
studies (n=11), while eight had a bachelor’s degree, two a graduate degree,
one a degree from a preparatory school, and seven the equivalent of a high
school diploma; one had not completed high school yet. Most did not know about
DPs (only four of 28; for two, this data was missed). The recruiter used
several different participant pools, each with a different rate of
compensation; hence, participants were compensated in accordance with each
pool at $\sim$5000-8000 yen (USD $\sim$$30-50) each. Participants were aware
and gave consent in advance for their anonymous data and screen footage to be
used for media broadcast as well as academic research.

###  2.2. Procedure

Participants were asked to evaluate the interactive features of a new
e-commerce website. As in prior work, e.g., (Bermejo Fernandez et al., 2021),
the true focus on DPs was hidden to avoid priming (Head et al., 1988).
Participants were informed about the procedure (without mentioning DPs) and
provided consent. They then sat before a laptop and filled out the pre-
questionnaire (refer to 2.4.1).

Next, participants watched a training video about the concurrent think aloud
protocol procedure. A staple of HCI research, this protocol involves
participants speaking their thoughts out loud while performing tasks with a
system (van den Haak et al., 2003; Alshammari et al., 2015). In the video, a
researcher performed the protocol while reading a cook book. Then,
participants practiced with the same cook book for $\sim$2 minutes.

Next, they experienced the shopping website. They were guided by a researcher
through a typical shopping user flow (refer to Figure 1 and 2.3.3). They were
not forced to carry out the tasks in a certain way or order. This meant that
not all participants viewed all pages on the website or experienced all DPs.
An observer watched and filled in the observation checklist (refer to 2.4.2).
After the task, participants filled out the post-questionnaire (refer to
2.4.1).

Participants were then interviewed about their experience (refer to 2.4.3).
They were told about DPs and shown a PDF containing screen shots of each page
in the website. They were asked to point out and describe any notable parts.
The observer noted whether they noticed any DPs this second time (refer to
2.4.2).

Finally, the host researcher revealed the true nature of the study: to
evaluate DPs. Participants were asked to keep the true focus of the study on
DPs a secret. Each session took $\sim$50-60 minutes.

###  2.3. Materials and Activity

####  2.3.1. Website Simulation

A small team of developers created a mock online shopping website for a fake
electronics retailer called “CyberStore,” (Figure 2), which was modelled on
e-commerce websites used by Japanese people. A WordPress WooCommerce
theme222Powered by the WooCommerce WordPress plugin with a customized WP
Cirqle theme as the base. was modified to embed DPs within the shopping
experience.

Figure 2. Screen shots of the website: the homepage (left), check-out page
(middle), and acceptable use policy page (right).

Three screen shots representing key pages in the “CyberStore” website. The
first shows the homepage with a leading menu and sale call-outs. The second
depicts the check-out page, featuring a sneaked-in product, subtotals, and an
urgency timer. The third features the acceptable use policy page, where
participants must go to cancel their account.

####  2.3.2. DPs

We used the seven categories of DPs compiled in 2022 by the Organisation for
Economic Co-operation and Development (OECD) (OECD, 2022), which was based on
the taxonomical work of Conti and Sobiesk (2010), Bösch et al. (2016), Gray et
al. (2018), Mathur et al. (2019), and Luguri and Strahilevitz (2021), plus the
Japanese-only DPs found in 2023 by Hidaka et al. (2023): Forced Action,
Interface Interference, Nagging, Obstruction, Sneaking, Social Proof, Urgency,
and Linguistic Dead-Ends. For each, we created DP _classes_ —subtypes of DPs
within these categories—and DP _cases_ —a point of interaction wherein one or
more DPs can be experienced as a gestalt (Di Geronimo et al., 2020). These
were based on typical examples found in e-commerce websites and sourced from
Brignull’s “Hall of Shame.”333https://www.deceptive.design/hall-of-shame Refer
to the visual guide on OSF444https://osf.io/65wzr or the Supplementary
Materials for screen shots (Appendix A) and a full list (Appendix C) of each
DP case.

####  2.3.3. Tasks

We developed a user flow (Figure 1) representing typical tasks in an online
shopping environment that would also lead participants to experience all DPs
within the website (Figure 3). Specifically, the tasks were: (i) Browse the
homepage; (ii) Browse the shop for a set of items: a computer, a bag, and
cleaning supplies; (iii) Check-out and sign up; and (iv) Cancel your
membership. Participants were asked to carry out the tasks in this order.
However, they were not forced to complete each task nor prevented from
exploring the website. In doing so, they were additionally able to experience
other DPs placed throughout the website. The DPs that could be experienced per
task are outlined in Figure 1. The tasks were rigorously developed to ensure
realism and encounters with all DPs, as well as timed in pilot tests with
Japanese lab members. Participants were given a 3-minute limit for the last
task.

###  2.4. Data Collection

####  2.4.1. Questionnaire Instruments and Measures

We used pre- and post-task questionnaires. We used valence and arousal to
assess affective state before and after the experience. This duo is among the
most common subjective measures of emotion, with arousal in particular found
to thoroughly explain affective states (Mauss and Robinson, 2009). For self-
reporting, we used the valence and arousal dimensions from the face-based
tactile version (Iturregui-Gallardo and Méndez-Ulrich, 2020) of the Self-
Assessment Manikin (SAM) (Bradley and Lang, 1994). This language-free self-
report instrument uses a 9-point semantic differential scale, where 1 is low
valence/arousal and 9 is high valence/arousal. In the post-task questionnaire,
we used the Japanese translation (Yamanouchi, 2015) of the 10-item System
Usability Scale (SUS) (Lewis, 2018) to assess acceptability (Bangor et al.,
2008) with a 5-point Likert scale.

Figure 3. A participant experiences an unintentional premium membership sign-
up after making a purchase (left) and an unnecessary location disclosure
request pop-up (right).

Two images of a participant interacting with the website. In the first, the
participant finds themselves unintentionally signed up for a premium
membership. In the second, they consider a pop-up asking for location
disclosure.

####  2.4.2. Observation Checklist and Deceptibility Metric

We created a checklist for observers to record whether each participant
_noticed_ each DP and their _reaction_ to it. To account for priming (Head et
al., 1988), the checklist distinguished between the participant first
experiencing the website, after being prompted to comment on the page (as part
of the concurrent think-aloud protocol), and the semi-structured interview,
which involved revealing the focus on DPs and the user flow (refer to 2.4.3).
The checklist also accounted for the chance that participants were not tricked
by certain DPs or found alternative paths through the website, thus missing
certain DPs. The _noticed_ options were: on their own; after prompting; during
the reveal; missed chance; and did not notice. The _reaction_ options were:
felt manipulated or without a choice, and felt negative (“deceptive”);
disturbed or interrupted and wanted to avoid it (“disruptive”); accepted it
(“no big deal”); and no reaction, i.e., tricked.

We used the _noticed_ data to create a deceptibility metric based on the lead
researcher’s expertise in DPs and as a designer and scholar with over 15 years
experience in UX. We added the ratios of each option with weights: .7 for
noticing on their own; .25 for after prompted, and .05 for after learning
about DPs. We multiplied this sum by the quotient of 10/7 and inverted the
product for a measure between 0 and 1, where 0 is undeceived and 1 is fully
deceived.

####  2.4.3. Semi-Structured Interview

We used semi-structured interviews (Blandford et al., 2016) with a funnel
approach (Wilson, 2014) to avoid priming responses while also capturing prior
familiarity with and understanding of DPs. The questions were: _Have you heard
of “dark patterns” before? Have you experienced dark patterns before? Were
there any aspects of the design that you felt were deceptive during operation?
What other feelings or impressions do you have about the design?_ Refer to
Appendix B for the Japanese versions. The participant was shown a PDF (refer
Appendix A) of the user flow and asked to point out specific aspects of the
design that aligned with their perceptions and feelings. The observer recorded
their responses in the checklist.

###  2.5. Data Analysis

We generated descriptive statistics by participant, DP class, DP case, and DP
category. We used the acceptability interpretation of the SUS from Bangor et
al. (2008): 51.6 as unacceptable, 51.7–71 as marginal, and 71.1–100 as
acceptable. We ran exploratory inferential statistics to evaluate affect
before and after the experience (paired t-test) and relationships between
deceptibility and other variables (parametric and/or non-parametric, depending
on the data types and normality results via Shapiro-Wilk tests). One
participant did not supply “before” affect scores and another did not fill out
the SUS; their data was excluded from analyses using these measures. Quotes
were sourced from the interviews.

##  3\. Results

###  3.1. Deception: Ability, Noticeability, Trickery, and Deceptibility (RQ1)

####  3.1.1. Task Completion

Two DP cases affected the path through the website: (i) a Hidden Subscription
to the “CyberSelect” premium membership through Trick Wording and
Pre“Un”Selection of a checkbox at checkout and (ii) Untranslation of the
account cancellation page (Hard to Cancel). For (i), half unintentionally
signed up for “CyberSelect.” Of the 42% that noticed, 20% accepted it, 6% felt
it was deception, and 75% felt it was disruptive. Most (n=27, 90%) were unable
to complete (ii). Of the 35% that noticed, deceptive (72%) or disruptive
(15%), but some accepted it (13%).

####  3.1.2. Noticeability and Reactions to Trickery

Descriptive statistics on the noticeability and reactions to all 25 DP cases
are in Supplementary Tables 2 and 3 (refer to Appendix D). Most (58%) DP cases
were unnoticeable (M=17.3, SD=7, MD=19.3, IQR=12). When noticed, 47% cases
were deemed “no big deal” (M=6.4, SD=6, MD=5, IQR=3.5), 29% were deceptive
(M=3.7, SD=4.9, MD=2, IQR=2), and 24% were disruptive (M=2.6, SD=2.9, MD=2,
IQR=2). The most noticeable were the homepage call-outs (n=25, 83%), the low
stock indicators (n=24, 80%), the testimonials on the product pages (n=23,
77%), and Sneaking an Item (n=23, 77%). The least noticeable were the
CyberSelect membership fee featuring Alphabet Soup and the Social Proof on the
search dropdown (only three each).

Descriptive statistics for DP category and class are in Supplementary Table 4
(Appendix D). The most noticeable category was Obstruction (80%) and the least
was Forced Action (73%). Obstruction was most deceptive (53%), Interface
Interference was most disruptive (68%), and Urgency was most acceptable (41%).
For classes, Sneaking an Item (Sneaking) and Testimonial (Social Proof) were
most noticeable (23%), while Alphabet Soup (Linguistic Dead-End) and
Misleading Reference Pricing (Interface Interference) were least noticeable
(90%). Sneaking an Item (Sneaking) (58%), Hard to Cancel (Obstruction) (53%),
and Untranslation (Linguistic Dead-End) (48%) were most deceptive. Trick
Questions (Interface Interference) were most disruptive (53%). Low Stock, High
Demand (Urgency) (67%) and Testimonials (Social Proof) (63%) were most
acceptable.

####  3.1.3. Deceptibility Metric

We calculated deceptibility metrics for participants and DPs (refer to
Appendix D).

Participant deceptibility ran the gamut: a Shapiro-Wilk test did not find a
statistically significant departure from normality, _W_(30) = .98, _p_ = .783,
indicating a bell curve. The highest score was 0.92 and the lowest was 0.40
(M=0.7, SD=0.1, MD=0.7, IQR=0.1). There were no statistically significant
correlations to age, gender, or education. This indicates that everyone was
susceptible to DPs, erring on the side of being more deceived overall (skewed
-0.4).

The deceptibility of DP cases (Figure 4, right) varied (M=0.7, SD=0.2, MD=0.7,
IQR=0.3) but were skewed high (-0.6). A Shapiro-Wilk test indicated a
statistically significant departure from normality, _W_(25) = .92, _p_ = .047.
The most deceptive were the CyberSelect fee with Alphabet Soup (0.96), search
dropdown with Social Proofs and False Hierarchies (0.93), and membership price
comparison chart (0.92). The least were sneaking in an item (0.25), the
product page testimonials (0.26), and the low stock indicators (0.28).

Figure 4. Deceptibility metric scores for classes (left) and cases (right).

Two graphs featuring the deceptibility metric scores for all classes and
cases.

At a higher level, the deceptibility of DP categories and classes (Figure 4,
left) also varied (M=0.6, SD=0.2, MD=0.7, IQR=0.2). A Shapiro-Wilk test found
a statistically significant departure from normality, _W_(28) = .92, _p_ =
.033, skewed high (-0.7). The most deceptive categories were Forced Action
(0.76), Interface Interference (0.75) and Linguistic Dead-Ends (0.75). The
least was Obstruction (0.40). The most deceptive classes were Alphabet Soup
(0.96), Misleading Reference Pricing (0.90), Hidden Information (0.86) and
ConfirmShaming (0.86). The least were Sneaking an Item (0.25), Testimonials
(0.26), Trick Questions (0.35), and Hard to cancel (0.40).

In short, some DPs were more deceptive than others, though most were
deceptive. Later, we will discuss the nuances of these results given the
noticeability results and reactions, as well as the feelings of participants,
up next.

###  3.2. Feelings: Mood and Judgments (RQ2)

The SUS scores indicated that most participants did not accept the website
(M=35.5, SD=32.5, MD=32.5, IQR=32.5, hi=85, lo=0). 69% (n=20) deemed the
website unacceptable, with three accepting it and six with marginal scores.

A paired-t test showed a statistically significant large difference between
valence before (M=6.9, SD=1.4) and after (M=4.6, SD=2.5), _t_(28) = 5, _p_ ¡
.001. Since the data was non-normal, we confirmed this result with a Wilcoxon
Signed-Rank test, which indicated the same for before (M=7, n=29) and after
(MD=5, n=29), _Z_ = -3.7, _p_ ¡ .001, _r_ = -0.8. In short, valence decreased
significantly for most people right after experiencing the website. Results of
a paired-t test for arousal also indicated a statistically significant large
difference before (M=2.9, SD=2) and after (M=5.5, SD=2.4), _t_(28) = 5.5, _p_
¡ .001. Arousal greatly rose for most people after the experience. Taken
together, the valence and arousal results suggest that people became
significantly more agitated after the experience.

Quotes from participants help understand these results. As A2 explained, “I
was concerned about being solicited for information. The location request pop-
up worried me, now that I think about it, because I pressed it without paying
attention to anything.” The particularly problematic Untranslation DP was
summed up by A11: “I couldn’t delete my account. The page was written in
English.” A20 provided further context: “I feel that the content of the
English text may be important, or that I am being deceived.” B6 helped make
sense of why Social Proofs—fake reviews and testimonials by fake people—were
generally noticeable and also not considered deceptive: “The reviews were
mostly good … it’s word-of-mouth.” Sneaking in an Item was among the most
noticeable and least deceptive according to the deceptibility matrix, but
among those who were not tricked, most (58%) considered it deceptive. A27
explained: “That cloth comes with the cleaner. They force you to buy it. They
don’t mention it in the description.” We will discuss this variability and the
nuanced relationship between noticeability and trickery as represented in the
deceptibility metric next.

##  4\. Discussion

Dark patterns and deceptive designs are multifarious UI phenomena. In the case
of our online shopping simulation, nearly two-thirds went unnoticed (RQ1). For
the rest, noticeability and deceptibility varied (RQ1), as did people’s
reactions and feelings towards each pattern (RQ2). We now attempt to bring a
depth of understanding to these findings by considering particular examples.

Almost no one was able to delete their account. This shows the power of the
newly found Untranslation DP as a mode of Obstruction. Untranslation is tied
to knowledge of the local linguistic context and relies on assumptions of
language ability. We used English in our simulation, given its global reach
and power, as well as statistics on Japan’s low English proficiency
555https://www.nippon.com/en/japan-data/h01843/. The statistics bore out in
our case. Other nations with low English proficiency may also be at risk of
Untranslation, but this has yet to be explored.

A gestalt of DPs led half of participants to unknowingly sign up for the
premium member service. Yet, rather than deceptive, those that realized found
it disruptive. This may point to attribution biases (Heider, 2013) that may be
self-serving (Miller and Ross, 1975). Whether people blamed themselves or
attributed the problem to the system will need to be researched. At least, our
participants assumed good intentions, which raises the issue of user
vulnerability to such DPs.

Reactions to the Social Proofs raise the issue of subversive influences.
Repeated exposure to media stimuli can lead to unconscious shifts in
preference by way of familiarity (Cox and Cox, 2002). Social Proofs, among the
most and least noticeable, may be implicated, given the social media age we
live within. Most participants deemed these DPs acceptable. This may be
explained by social mobilization effects, where even small profile pictures
can have massive effects on behaviour, perhaps especially when representing
“friends” and influencers (Bond et al., 2012). The interviews also suggest a
perceived benefit of “word-of-mouth” information. Japan has been characterized
as a collectivist market culture, where social bonds are prioritized and group
acceptance of commercial offerings dictate market value (Synodinos, 2001). We
must be wary of taking advantage of people’s implicit trust in social
indicators. Similarly, Urgency DPs were also considered noticeable and
acceptable, perhaps because of the store context and repeated exposure to
these rather widespread DPs in the wild.

Perhaps the most subversive DP was Alphabet Soup, found in two membership fee
displays. This DP was noticed only by a handful of participants and was a top-
scorer on the deceptibility metric (0.96 and 0.92). This study is the first to
empirically explore this novel DP and its sibling Untranslation, so it is
difficult to contextualize these results within previous research. However,
the National Consumer Affairs Center of Japan has recently drawn attention to
this DP in the form of the ¥ symbol, which is used for both Chinese and
Japanese currencies666https://www.kokusen.go.jp/news/data/n-20230419_2.html.
Consumers and companies must be cautious of conflating these symbols, as their
emergence represents a sublime form of deception for the Asian market.

Misleading Reference Pricing, a form of Interface Interference, was also
highly unnoticeable and deceptive, according to our metric. Indeed, virtually
no one noticed it, even when shown the user flow and told about DPs. This is a
common DP in shopping websites and may escape the notice of the average
consumer, allowing it to live on even when laws change in response to advocacy
initiatives. We will need to raise attention to this DP and critically engage
designers, purveyors, and stakeholders (Gray et al., 2021b) in light of
regulatory shifts around less deceptive forms of DPs.

Despite its name, “Sneaking” DPs were the least deceptive and easily noticed.
Obstruction, too. These may be more “blunt” forms of DPs, irritants rather
than deceptive. Controlled experimental work with and without these DPs may
confirm and provide further context on these findings from a consumer
perspective (Voigt et al., 2021).

###  4.1. Limitations

We used a simulation that was developed to the level of commercial viability,
containing DPs based on real cases from live websites. Nevertheless, we
recognize that this limits the ecological validity of our results. Our
simulation also contained significantly more DPs than the average 3.9
previously found for apps in the Japanese Google Play Store (Hidaka et al.,
2023). This may have lessened the realism of the experience. At the same time,
our results confirm that most DPs went unnoticed. Still, future work should
study UIs with more realistic levels of DPs. Importantly, other kinds of UI,
such as smartphone apps and video games, as well as other kinds of online
stores, such as clothing stores and grocery markets, should be explored, as
the level and range of DPs may vary by platform.

We acknowledge limitations related to recruitment. We relied on a third-party
recruiter publicly known to participants (NHK). This allowed us to recruit a
diverse and random selection of participants from the Japanese populace.
However, participants may have self-selected in or out of the study based on
the recruiter. We also recruited a relatively small sample (N=30) and did not
have an experimental design with a control setup, limiting the
generalizability of our findings. Finally, we did not control or capture
relevant metrics, such as daily Internet usage, type of degree or career, or
knowledge of English (for the Untranslation DP).

Finally, this was a pilot study. Our protocol was not preregistered, and we
adapted our procedure and methods based on the pilot tests, notably switching
from retrospective to concurrent protocol due to timing issues and adding
options to the observer checklist. A preregistered study will be conducted
next. We also created a novel metric of deceptibility that could be applied to
individuals as well as DPs. This weighted measure was useful for determining
relative deceptibility, especially by taking into account priming and exposure
to DP stimuli. Still, it rests on our operationalizations of performance and
arbitrary weights. This metric should be vetted and discussed with experts in
design and DPs, modified, if needed, and then tested empirically. We also did
not measure affective reactions to specific DPs; targeted measurement can do
so in future.

##  5\. Conclusion

We have taken a first step towards better understanding the deceptibility and
UX of DPs commonly deployed in online stores for the Japanese context. We have
demonstrated variability in perceptions and deceptions while highlighting an
overall trend of deceptibility for most Japanese participants and DPs. The
next step is experimental work with larger sample sizes and real commercial
UI.

###### Acknowledgements.

Our sincere gratitude to NHK, especially Tatsuro Imono and Yuya Higashi, for
funding part of this research (development and participant fees) and handling
the recruitment of participants. We sincerely thank Émilie Fabre for
development assistance. We thank Tokyo Tech Innovation and Dai Senoo for
supporting this research in various ways. We thank Peter Pennefather for pre-
reviewing this manuscript. Katie Seaborn and Takao Fujii conscientiously
dissent to in-person participation at CHI this year; read their positionality
statement here: https://bit.ly/chi24statement

## References

  * (1)
  * Alshammari et al. (2015) Thamer Alshammari, Obead Alhadreti, and Pamela Mayhew. 2015.  When to ask participants to think aloud: A comparative study of concurrent and retrospective think-aloud methods.  _International Journal of Human Computer Interaction_ 6, 3 (2015), 48–64.
  * Bangor et al. (2008) Aaron Bangor, Philip T. Kortum, and James T. Miller. 2008.  An empirical evaluation of the System Usability Scale.  _International Journal of Human–Computer Interaction_ 24, 6 (2008), 574–594.  https://doi.org/10.1080/10447310802205776
  * Berens et al. (2022) Benjamin Maximilian Berens, Heike Dietmann, Chiara Krisam, Oksana Kulyk, and Melanie Volkamer. 2022.  Cookie Disclaimers: Impact of Design and Users’ Attitude. In _Proceedings of the 17th International Conference on Availability, Reliability and Security_ (Vienna, Austria) _(ARES ’22)_. ACM, New York, NY, USA, Article 12, 20 pages.  https://doi.org/10.1145/3538969.3539008
  * Bermejo Fernandez et al. (2021) Carlos Bermejo Fernandez, Dimitris Chatzopoulos, Dimitrios Papadopoulos, and Pan Hui. 2021.  This Website Uses Nudging: MTurk Workers’ Behaviour on Cookie Consent Notices.  _Proceedings of the ACM on Human-Computer Interaction_ 5, CSCW2, Article 346 (oct 2021), 22 pages.  https://doi.org/10.1145/3476087
  * Blandford et al. (2016) Ann Blandford, Dominic Furniss, and Stephann Makri. 2016.  _Qualitative HCI Research: Going Behind the Scenes_.  Morgan & Claypool Publishers, Kentfield, CA, USA.
  * Bond et al. (2012) Robert M. Bond, Christopher J. Fariss, Jason J. Jones, Adam D.I. Kramer, Cameron Marlow, Jaime E. Settle, and James H. Fowler. 2012.  A 61-million-person experiment in social influence and political mobilization.  _Nature_ 489, 7415 (2012), 295–298.  https://www.nature.com/articles/nature11421
  * Bongard-Blanchy et al. (2023) Kerstin Bongard-Blanchy, Jean-Louis Sterckx, Arianna Rossi, Anastasia Sergeeva, Vincent Koenig, Salvador Rivas, and Verena Distler. 2023.  Analysing the Influence of Loss-Gain Framing on Data Disclosure Behaviour: A Study on the Use Case of App Permission Requests. In _Proceedings of the 2023 European Symposium on Usable Security_ (Copenhagen, Denmark) _(EuroUSEC ’23)_. ACM, New York, NY, USA, 112–125.  https://doi.org/10.1145/3617072.3617108
  * Borberg et al. (2022) Ida Borberg, Rene Hougaard, Willard Rafnsson, and Oksana Kulyk. 2022.  “So I sold my soul”: Effects of dark patterns in cookie notices on end-user behavior and perceptions. In _Workshop on Usable Security and Privacy (USEC)_ (San Diego, CA, USA), Vol. 3. HATS, Bloomintgton, IN, USA, 11 pages.  https://doi.org/10.14722/usec.2022.23026
  * Bösch et al. (2016) Christoph Bösch, Benjamin Erb, Frank Kargl, Henning Kopp, and Stefan Pfattheicher. 2016.  Tales from the dark side: Privacy dark strategies and privacy dark patterns.  _Proceedings on Privacy Enhancing Technologies_ 2016, 4 (2016), 237–254.  https://doi.org/10.1515/popets-2016-0038
  * Bradley and Lang (1994) Margaret M. Bradley and Peter J. Lang. 1994.  Measuring emotion: The Self-Assessment Manikin and the semantic differential.  _Journal of Behavior Therapy and Experimental Psychiatry_ 25, 1 (1994), 49–59.  https://www.sciencedirect.com/science/article/abs/pii/0005791694900639
  * Carter et al. (2008) Scott Carter, Jennifer Mankoff, Scott R. Klemmer, and Tara Matthews. 2008.  Exiting the cleanroom: On ecological validity and ubiquitous computing.  _Human–Computer Interaction_ 23, 1 (2008), 47–99.  https://doi.org/10.1080/07370020701851086
  * Chaudhary et al. (2022) Akash Chaudhary, Jaivrat Saroha, Kyzyl Monteiro, Angus G. Forbes, and Aman Parnami. 2022.  “Are You Still Watching?”: Exploring Unintended User Behaviors and Dark Patterns on Video Streaming Platforms. In _Proceedings of the 2022 ACM Designing Interactive Systems Conference_ (Virtual Event, Australia) _(DIS ’22)_. ACM, New York, NY, USA, 776–791.  https://doi.org/10.1145/3532106.3533562
  * Conti and Sobiesk (2010) Gregory Conti and Edward Sobiesk. 2010.  Malicious Interface Design: Exploiting the User. In _Proceedings of the 19th International Conference on World Wide Web_ (Raleigh, North Carolina, USA) _(WWW ’10)_. ACM, New York, NY, USA, 271–280.  https://doi.org/10.1145/1772690.1772719
  * Cox and Cox (2002) Dena Cox and Anthony D. Cox. 2002.  Beyond first impressions: The effects of repeated exposure on consumer liking of visually complex and simple product designs.  _Journal of the Academy of Marketing Science_ 30, 2 (2002), 119–130.  https://journals.sagepub.com/doi/abs/10.1177/03079459994371
  * Cranor (2022) Lorrie Faith Cranor. 2022.  Cookie Monster.  _Commun. ACM_ 65, 7 (June 2022), 30–32.  https://doi.org/10.1145/3538639
  * Di Geronimo et al. (2020) Linda Di Geronimo, Larissa Braz, Enrico Fregnan, Fabio Palomba, and Alberto Bacchelli. 2020.  UI dark datterns and where to find them: A study on mobile applications and user perception. In _Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems_ (Honolulu, HI, USA) _(CHI ’20)_. ACM, New York, NY, USA, 1–14.  https://doi.org/10.1145/3313831.3376600
  * Fansher et al. (2018) Madison Fansher, Shruthi Sai Chivukula, and Colin M. Gray. 2018.  #darkpatterns: UX Practitioner Conversations About Ethical Design. In _Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems_ (Montreal, QC, Canada) _(CHI EA ’18)_. ACM, New York, NY, USA, 1–6.  https://doi.org/10.1145/3170427.3188553
  * Feng et al. (2023) Jiaying Feng, Fan Mo, Yuki Yada, Tsuneo Matsumoto, Nao Fukushima, Fukuyo Kido, and Hayato Yamana. 2023.  Analysis of Dark Pattern-related Tweets from 2010. In _2023 IEEE 8th International Conference on Big Data Analytics (ICBDA)_. IEEE, Piscataway, NJ, USA, 100–106.  https://doi.org/10.1109/ICBDA57405.2023.10104855
  * Graßl et al. (2021) P.A.J. Graßl, H.K. Schraffenberger, F.J. Zuiderveen Borgesius, and M.A. Buijzen. 2021.  Dark and bright patterns in cookie consent requests.  _Journal of Digital Social Research_ 3, 1 (2021), 1–38.  https://doi.org/10.33621/jdsr.v3i1.54
  * Gray et al. (2021a) Colin M. Gray, Shruthi Sai Chivukula, Kassandra Melkey, and Rhea Manocha. 2021a.  Understanding “Dark” Design Roles in Computing Education. In _Proceedings of the 17th ACM Conference on International Computing Education Research_ (Virtual Event, USA) _(ICER 2021)_. ACM, New York, NY, USA, 225–238.  https://doi.org/10.1145/3446871.3469754
  * Gray et al. (2018) Colin M. Gray, Yubo Kou, Bryan Battles, Joseph Hoggatt, and Austin L. Toombs. 2018.  The Dark (Patterns) Side of UX Design. In _Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems_ (Montreal, QC, Canada) _(CHI ’18)_. ACM, New York, NY, USA, 1–14.  https://doi.org/10.1145/3173574.3174108
  * Gray et al. (2023) Colin M. Gray, Lorena Sanchez Chamorro, Ike Obi, and Ja-Nae Duane. 2023.  Mapping the Landscape of Dark Patterns Scholarship: A Systematic Literature Review. In _Companion Publication of the 2023 ACM Designing Interactive Systems Conference_ (Pittsburgh, PA, USA) _(DIS ’23 Companion)_. ACM, New York, NY, USA, 188–193.  https://doi.org/10.1145/3563703.3596635
  * Gray et al. (2021b) Colin M. Gray, Cristiana Santos, Nataliia Bielova, Michael Toth, and Damian Clifford. 2021b.  Dark Patterns and the Legal Requirements of Consent Banners: An Interaction Criticism Perspective. In _Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems_ (Yokohama, Japan) _(CHI ’21)_. ACM, New York, NY, USA, Article 172, 18 pages.  https://doi.org/10.1145/3411764.3445779
  * Gunawan et al. (2021) Johanna Gunawan, Amogh Pradeep, David Choffnes, Woodrow Hartzog, and Christo Wilson. 2021.  A Comparative Study of Dark Patterns Across Web and Mobile Modalities.  _Proceedings of the ACM on Human-Computer Interaction_ 5, CSCW2, Article 377 (oct 2021), 29 pages.  https://doi.org/10.1145/3479521
  * Habib et al. (2022) Hana Habib, Megan Li, Ellie Young, and Lorrie Cranor. 2022.  “Okay, whatever”’: An evaluation of cookie consent interfaces. In _Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems_ (New Orleans, LA, USA) _(CHI ’22)_. ACM, New York, NY, USA, Article 621, 27 pages.  https://doi.org/10.1145/3491102.3501985
  * Harry (2023) Brignull Harry. 2023.  _Deceptive Patterns: Exposing the Tricks Tech Companies Use to Control You_.  Testimonium Ltd, London, UK.
  * Head et al. (1988) Thomas C. Head, Ricky W. Griffin, Thomas S. Bateman, Lynn Lohman, and Valerie L Yates. 1988.  The priming effect in task design research.  _Journal of Management_ 14, 1 (1988), 33–39.  https://doi.org/10.1177/014920638801400104
  * Heider (2013) Fritz Heider. 2013.  _The Psychology of Interpersonal Relations_.  Psychology Press, London, UK.
  * Hidaka et al. (2023) Shun Hidaka, Sota Kobuki, Mizuki Watanabe, and Katie Seaborn. 2023.  Linguistic Dead-Ends and Alphabet Soup: Finding Dark Patterns in Japanese Apps. In _Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems_ (Hamburg, Germany) _(CHI ’23)_. ACM, New York, NY, USA, Article 3, 13 pages.  https://doi.org/10.1145/3544548.3580942
  * Iturregui-Gallardo and Méndez-Ulrich (2020) Gonzalo Iturregui-Gallardo and Jorge Luis Méndez-Ulrich. 2020.  Towards the creation of a tactile version of the Self-Assessment Manikin (T-SAM) for the emotional assessment of visually impaired people.  _International Journal of Disability, Development and Education_ 67, 6 (2020), 657–674.  https://doi.org/10.1080/1034912X.2019.1626007
  * Kyi et al. (2023) Lin Kyi, Sushil Ammanaghatta Shivakumar, Cristiana Teixeira Santos, Franziska Roesner, Frederike Zufall, and Asia J. Biega. 2023.  Investigating Deceptive Design in GDPR’s Legitimate Interest. In _Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems_ (Hamburg, Germany) _(CHI ’23)_. ACM, New York, NY, USA, Article 583, 16 pages.  https://doi.org/10.1145/3544548.3580637
  * Lewis (2018) James R Lewis. 2018.  The System Usability Scale: Past, present, and future.  _International Journal of Human–Computer Interaction_ 34, 7 (2018), 577–590.  https://doi.org/10.1080/10447318.2018.1455307
  * Luguri and Strahilevitz (2021) Jamie Luguri and Lior Jacob Strahilevitz. 2021.  Shining a light on dark patterns.  _Journal of Legal Analysis_ 13, 1 (2021), 43–109.  https://doi.org/10.1093/jla/laaa006
  * M. Bhoot et al. (2021) Aditi M. Bhoot, Mayuri A. Shinde, and Wricha P. Mishra. 2021.  Towards the Identification of Dark Patterns: An Analysis Based on End-User Reactions. In _Proceedings of the 11th Indian Conference on Human-Computer Interaction_ (Online, India) _(IndiaHCI ’20)_. ACM, New York, NY, USA, 24–33.  https://doi.org/10.1145/3429290.3429293
  * Mathur et al. (2019) Arunesh Mathur, Gunes Acar, Michael J. Friedman, Eli Lucherini, Jonathan Mayer, Marshini Chetty, and Arvind Narayanan. 2019.  Dark Patterns at Scale: Findings from a Crawl of 11K Shopping Websites.  _Proceedings of the ACM on Human–Computer Interaction_ 3, CSCW, Article 81 (nov 2019), 32 pages.  https://doi.org/10.1145/3359183
  * Mathur et al. (2021) Arunesh Mathur, Mihir Kshirsagar, and Jonathan Mayer. 2021.  What Makes a Dark Pattern… Dark? Design Attributes, Normative Considerations, and Measurement Methods. In _Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems_ (Yokohama, Japan) _(CHI ’21)_. ACM, New York, NY, USA, Article 360, 18 pages.  https://doi.org/10.1145/3411764.3445610
  * Mauss and Robinson (2009) Iris B. Mauss and Michael D. Robinson. 2009.  Measures of emotion: A review.  _Cognition & Emotion_ 23, 2 (Feb. 2009), 209–237.  https://doi.org/10.1080/02699930802204677
  * Miller and Ross (1975) Dale T. Miller and Michael Ross. 1975.  Self-serving biases in the attribution of causality: Fact or fiction?  _Psychological Bulletin_ 82, 2 (1975), 213.
  * Monge Roffarello and De Russis (2022) Alberto Monge Roffarello and Luigi De Russis. 2022.  Towards Understanding the Dark Patterns That Steal Our Attention. In _Extended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems_ (New Orleans, LA, USA) _(CHI EA ’22)_. ACM, New York, NY, USA, Article 274, 7 pages.  https://doi.org/10.1145/3491101.3519829
  * Nazarov and Baimukhambetov (2022) Dmitry Nazarov and Yerkebulan Baimukhambetov. 2022.  Clustering of Dark Patterns in the User Interfaces of Websites and Online Trading Portals (E-Commerce).  _Mathematics_ 10, 18 (2022), 3219.  https://doi.org/10.3390/math10183219
  * OECD (2022) OECD. 2022.  _Dark commercial patterns_.  Number 336 in OECD Digital Economy Papers. OECD Publishing, Paris, France. 96 pages.  https://doi.org/10.1787/44f5e846-en
  * OECD (2023) OECD. 2023.  _Consumer Vulnerability in the Digital Age_.  Number 355 in OECD Digital Economy Papers. OECD Publishing, Paris, France. 80 pages.  https://doi.org/10.1787/4d013cc5-en
  * Schäfer et al. (2023) René Schäfer, Paul Miles Preuschoff, and Jan Borchers. 2023.  Investigating Visual Countermeasures Against Dark Patterns in User Interfaces. In _Proceedings of Mensch Und Computer 2023_ (Rapperswil, Switzerland) _(MuC ’23)_. ACM, New York, NY, USA, 161–172.  https://doi.org/10.1145/3603555.3603563
  * Soe et al. (2020) Than Htut Soe, Oda Elise Nordberg, Frode Guribye, and Marija Slavkovik. 2020.  Circumvention by design - Dark patterns in cookie consent for online news outlets. In _Proceedings of the 11th Nordic Conference on Human-Computer Interaction: Shaping Experiences, Shaping Society_ (Tallinn, Estonia) _(NordiCHI ‘20)_. ACM, New York, NY, USA, Article 19, 12 pages.  https://doi.org/10.1145/3419249.3420132
  * Stanton et al. (2017) Neville A. Stanton, Paul M. Salmon, Laura A. Rafferty, Guy H. Walker, Chris Baber, and Daniel P. Jenkins. 2017.  _Human Factors Methods: A Practical Guide for Engineering and Design_.  CRC Press, Boca Raton, Florida, USA.
  * Synodinos (2001) Nicolaos E. Synodinos. 2001.  Understanding Japanese consumers: Some important underlying factors.  _Japanese Psychological Research_ 43, 4 (Nov. 2001), 235–248.  https://doi.org/10.1111/1468-5884.00181
  * Utz et al. (2019) Christine Utz, Martin Degeling, Sascha Fahl, Florian Schaub, and Thorsten Holz. 2019.  (Un)Informed Consent: Studying GDPR Consent Notices in the Field. In _Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security_ (London, United Kingdom) _(CCS ’19)_. ACM, New York, NY, USA, 973–990.  https://doi.org/10.1145/3319535.3354212
  * van den Haak et al. (2003) Maaike van den Haak, Menno de Jong, and Peter Jan Schellens. 2003.  Retrospective vs. concurrent think-aloud protocols: Testing the usability of an online library catalogue.  _Behaviour & Information Technology_ 22, 5 (2003), 339–351.  https://doi.org/10.1080/0044929031000
  * van Nimwegen and de Wit (2022) Christof van Nimwegen and Jesse de Wit. 2022.  Shopping in the dark: Effects of platform choice on dark pattern recognition. In _International Conference on Human-Computer Interaction_. Springer, London, UK, 462–475.  https://doi.org/10.1007/978-3-031-05412-9_32
  * Voigt et al. (2021) Christian Voigt, Stephan Schlögl, and Aleksander Groth. 2021.  Dark patterns in online shopping: Of sneaky tricks, perceived annoyance and respective brand trust. In _International Conference on Human-Computer Interaction_. Springer, London, UK, 143–155.  https://link.springer.com/chapter/10.1007/978-3-030-77750-0_10
  * Wilson (2014) Chauncey Wilson. 2014.  _Interview Techniques for UX Practitioners: A User-Centered Design Method_.  Morgan Kaufmann, Waltham, MA, USA.
  * Yada et al. (2022) Yuki Yada, Jiaying Feng, Tsuneo Matsumoto, Nao Fukushima, Fuyuko Kido, and Hayato Yamana. 2022.  Dark patterns in e-commerce: A dataset and its baseline evaluations. In _2022 IEEE International Conference on Big Data (Big Data)_. IEEE, Piscataway, NJ, USA, 3015–3022.  https://ieeexplore.ieee.org/document/10020800
  * Yamanouchi (2015) Shigeru Yamanouchi. 2015.  _エンジニアのための人を対象とする研究計画入門 (Introduction to Human Subjects Research Planning for Engineers)_.  Maruzen, Tokyo, Japan, Chapter 科学的合理性と倫理的妥当性 (Scientific rationality and Ethical Relevance), 74–118.


# GarmentTracking: Category-Level Garment Pose Tracking

Han Xue1,2, Wenqiang Xu2, Jieyi Zhang2, Tutian Tang2, Yutong Li2, Wenxin Du2,
Ruolin Ye3, Cewu Lu$\dagger$1,2
1Shanghai Qi Zhi Institute 2Shanghai Jiao Tong University 3 Cornell University
2{xiaoxiaoxh, vinjohn, yi_eagle, tttang, davidliyutong, mnkmYuki,
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
###### Abstract

Garments are important to humans. A visual system that can estimate and track
the complete garment pose can be useful for many downstream tasks and real-
world applications. In this work, we present a complete package to address the
category-level garment pose tracking task: (1) A recording system VR-Garment,
with which users can manipulate virtual garment models in simulation through a
VR interface. (2) A large-scale dataset VR-Folding, with complex garment pose
configurations in manipulation like flattening and folding. (3) An end-to-end
online tracking framework GarmentTracking, which predicts complete garment
pose both in canonical space and task space given a point cloud sequence.
Extensive experiments demonstrate that the proposed GarmentTracking achieves
great performance even when the garment has large non-rigid deformation. It
outperforms the baseline approach on both speed and accuracy. We hope our
proposed solution can serve as a platform for future research. Codes and
datasets are available in https://garment-tracking.robotflow.ai.

††footnotetext: $\dagger$ Cewu Lu is the corresponding author, the member of
Qing Yuan Research Institute and MoE Key Lab of Artificial Intelligence, AI
Institute, Shanghai Jiao Tong University, China and Shanghai Qi Zhi Institute.

##  1 Introduction

Garments are one of the most important deformable objects in daily life. A
vision system for garment pose estimation and tracking can benefit downstream
tasks like MR/AR and robotic manipulation[17, 7]. The category-level garment
pose estimation task is firstly introduced in GarmentNets [11], which aims to
recover the full configuration of an unseen garment from a single static
frame. Unlike the non-rigid tracking methods [27, 19, 16, 34, 35, 10, 9] which
can only recover the geometry of the visible regions, pose estimation task can
also reconstruct the occluded parts of the object. Another line of works [15,
14, 21, 28, 29, 30, 22] (non-rigid 4D reconstruction which can reconstruct
complete object geometry) cannot be directly applied on garments, since they
assume the object has a watertight geometry. In contrast, garments have thin
structures with holes.

In this paper, we propose a new task called Category-level Garment Pose
Tracking, which extends the single-frame pose estimation setting in [11] to
pose tracking in dynamic videos. Specifically, we focus on the pose tracking
problem in garment manipulation (_e.g_. flattening, folding). In this setting,
we do not have the priors of the human body like previous works for clothed
humans [31, 41, 26, 18]. Therefore, we must address the extreme deformation
that manipulated garments could undergo.

Figure 1: The pipeline of our Virtual Realty recording system (VR-Garment).
(a) A volunteer needs to put on a VR headset and VR gloves. (b) By following
the guidance of a specially designed UI, the volunteer begins to collect data
efficiently. (c) After recording, we re-render multi-view RGB-D images with
Unity[6] and obtain masks and deformed garment meshes with NOCS labels.

To tackle the garment pose tracking problem, we need a dataset of garment
manipulation with complete pose annotations. However, such a dataset does not
exist so far to the best of our knowledge. To build such a dataset, we turn to
a VR-based solution due to the tremendous difficulty of garment pose
annotation in the real world [10]. We first create a real-time VR-based
recording system named VR-Garment. Then the volunteer can manipulate the
garment in a simulator through the VR interface. With VR-Garment, we build a
large-scale garment manipulation dataset called VR-Folding. Compared to the
single static garment configuration (_i.e_. grasped by one point) in
GarmentNets, our manipulation tasks include flattening and folding, which
contain much more complex garment configurations. In total, our VR-Folding
dataset contains 9767 manipulation videos which consist of 790K multi-view
RGB-D frames with full garment pose and hand pose annotations on four garment
categories selected from the CLOTH3D [8] dataset.

With the VR-Folding dataset, we propose an end-to-end online tracking method
called GarmentTracking to perform category-level garment pose tracking during
manipulation. For the garment pose modeling, we follow GarmentNets [11] to
adopt the normalized object coordinate space (NOCS) for each category.
Nevertheless, tracking garment pose raises new challenges compared to single-
frame pose estimation: (1) How to fuse inter-frame geometry and correspondence
information? (2) How to make the tracking prediction robust to pose estimation
errors? (3) How to achieve tracking in real-time? To address these challenges,
we conduct GarmentTracking in three stages, namely NOCS predictor, NOCS
refiner, and warp field mapper. Firstly, it predicts per-frame features and
fuses them for canonical coordinate prediction. Then it refines the predicted
canonical coordinates and the geometry with a NOCS refiner to reduce the
accumulated errors. Finally, it maps the prediction in canonical space to the
task space (_i.e_. coordinate frame of the input point cloud).

Since no previous work is designed for the tracking setting, we use
GarmentNets [11] as a single-frame prediction baseline for comparison. We also
perform extensive ablative experiments to reveal the efficacy of our design
choices. Finally, we collect real-world data on garment manipulation and show
the qualitative results of our method. In our design, we avoid the
computationally expensive Marching Cubes [25] for reconstructing the canonical
mesh frame by frame, so that we can achieve tracking at 15 FPS with an RTX
3090 GPU (5 times faster than the baseline approach).

We summarize our contributions as follows:

1). We propose a VR-based garment manipulation recording system named VR-
Garment. It can synchronize human operations into the simulator and collect
garment manipulation data.

2). We propose a large-scale garment manipulation dataset named VR-Folding for
pose tracking. During manipulation, garments exhibit diverse configurations.

3). We propose a real-time end-to-end framework named GarmentTracking for
category-level garment pose tracking. It can serve as a strong baseline for
further research. We also demonstrate its generalization ability to real-world
garment recordings with models trained by simulated data.

##  2 Related Work

Category-level Object Pose Estimation and Tracking. Object pose is the
configuration of the object posited in the observation space. For the rigid
object, we can describe its pose in 6 degrees of freedom (DOFs), _i.e_. 3 for
translation and 3 for rotation. However, for the non-rigid object, like
garments, the object pose can be of near-infinite DOFs.

On the other hand, the category-level object pose estimation task aims to
learn a model that can predict unseen object poses of the same category [38,
20, 23, 39, 24]. The concept is first introduced to estimate rigid object pose
[38]. In [38], Wang _et al_. proposed a Normalized Object Coordinate Space
(NOCS) as a category-specific canonical representation. Following the idea, Li
_et al_. [20] proposed a hierarchical NOCS representation for articulated
objects.

To handle the near-infinite DOF nature and the category-level generalization
requirement, GarmentNets [11] also defined NOCS for each garment category.
They predicted the garment pose by mapping the reconstructed mesh from
canonical space to task space. However, GarmentNets [11] treats each frame
individually, which hampers its stability for inter-frame prediction, and its
ability to infer complex poses from sequential movements. Our GarmentTracking
is proposed for these tracking issues.

Non-rigid Tracking and Reconstruction. Tracking and reconstructing non-rigid
deforming objects is an important research area in computer vision and
graphics. One line of works [27, 19, 16, 34, 35, 10, 9] perform free-form
tracking, which does not assume any geometric prior. For example,
DynamicFusion [27] used a hierarchical node graph structure and an efficient
GPU solver to reconstruct the visible surface of the object. Deepdeform [10]
and Bozic _et al_. [9] leveraged learning-based correspondences to track
deformable objects. However, unlike pose estimation methods which reconstruct
the complete configuration of objects, all these methods can not reconstruct
occluded parts.

Another line of works [15, 14, 21, 28, 29, 30, 22] can perform 4D
reconstruction from RGB-D videos, which captures the complete geometry of the
object both in space and time. Unfortunately, the shape representations of
these methods have limitations when adapting to garment pose reconstruction
under large deformations. For example, Fusion4D [15], Motion2fusion [14],
4DComplete [21], OcclusionFusion [22], NPMs [29] and OccupancyFlow [28]
heavily rely on watertight object modeling such as SDF, TSDF, or occupancy
grids to reconstruct object surfaces. Such modeling is not suitable for
objects with open and thin structures like garments.

Garment-related Dataset. Current garment-related datasets can be divided into
asset datasets [8, 41] and task datasets [10, 37, 26, 11]. Asset datasets
provide garment models for different tasks. For example, GarmentNets [11]
proposed a simulation dataset for category-level pose-estimation task based on
CLOTH3D [8]. We also build our VR-Folding dataset based on [8]. Other task
datasets do not require complete garment models [26]. For example, CAPE [26]
deals with the clothed human reconstruction task. However, the human body
limits the possible garment states. DeepDeform [10] dataset contains simple
scenes where a person lifts one garment with minor deformations, and it only
annotates sparse keypoint correspondences between frames. A real-world cloth-
folding dataset proposed by Verleysen _et al_. [37] contains videos of cloth-
folding actions, but it only annotates the contour of garments in 2D images.
Our VR-Folding dataset is the first dataset designed for category-level
garment pose tracking in manipulation, and it contains dynamic scenes which
include complex human actions and garment configurations.

Figure 2: The overview of GarmentTracking. Given the per-point NOCS coordinate
of the first frame and a rough canonical shape (mesh NOCS), our tracking
method takes two frames of the partial point cloud as input. In stage 1, the
NOCS predictor will generate an inter-frame fusion feature and predict raw
NOCS coordinates. In stage 2, the NOCS refiner will refine the NOCS
coordinates and the canonical shape simultaneously. In stage 3, the warp field
mapper will predict the warp field which maps from canonical space to task
space.

##  3 VR-Folding Dataset

To build the VR-Folding dataset, we develop a real-time data recording system
called VR-Garment (Fig. 1). In this way, we can combine human experience and
the benefits of simulation environments (_i.e_. easy access to ground-truth
poses) to efficiently collect a large amount of data with natural and complex
poses. We will describe the system design and the operation procedures in Sec.
3.1 and Sec. 3.2. Then we will describe the data statistics in Sec. 3.3.

###  3.1 VR-Garment

In this section, we will first describe the hardware and software setup of the
system and then the recording procedure for volunteers to operate. The
recording system is illustrated in Fig. 1.

Hardware. On the hardware side, our recording system needs an HTC Vive Pro [1]
VR headset and Noitom Hi5 [2] VR gloves which can track finger poses in the
real world and reproduce them with virtual hands in the simulator through a VR
interface.

Software. We developed our VR recording framework based on Unity [6] for its
good support of mainstream VR devices. The cloth physics simulation in Unity
is achieved by Obi [3]. Specifically, we implemented a simple UI and a
grasping system in VR, which allow users to grasp or release any point on the
garment surface when two virtual fingertips make contact with the garment.

Recording Procedure. Firstly, a volunteer must wear a VR headset and VR
gloves. Then he should observe a garment instance randomly dropping on a table
in Unity. Next, he performs a pre-defined manipulation task (_e.g_. folding).
In the manipulation process, we save the deformed garment mesh and hand poses
for each frame. When the task is done, the volunteer will use special gestures
(_e.g_. fist) to send commands for moving to the next garment instance. After
recording, we re-render multi-view RGB-D images in Unity with the saved
animation data and generate corresponding ground-truth annotations (_e.g_.
garment poses, masks). Note that all the rendering settings (_e.g_. lights,
cameras, textures _etc_.) can be customized even after recording.

###  3.2 Task Definition

In a typical cloth folding process, we operate this task in two stages, namely
flattening and folding:

Flattening: Firstly, a garment will drop on the virtual table in Unity. Then
our system will randomly choose one point on the garment surface. Next, the
volunteer will grasp that point with one hand and lift the garment in the air
(an initial configuration similar to that in GarmentNets [11]). Next, the
volunteer will try to grasp and fling the garment repeatedly with two hands
until the garment is smoothed and in a flattened T-pose (see Fig. 1). Please
see the supplementary files for more details about the task.

Folding: Firstly, a garment in a flattened T-pose will be placed on the
virtual table. Then the volunteer will repeatedly perform pick-and-place
actions with both hands until the garment is folded. Though people may have
different preferred steps to achieve folding, we have defined general rules
for each category and asked the volunteers to follow them. Please see the
supplementary files for more details.

###  3.3 Data Statistics

All the garment meshes used in our system are from CLOTH3D [8]. We choose 4
categories from CLOTH3D dataset, namely Shirt, Pants, Top and Skirt. For
flattening task, we recorded 5871 videos which contain 585K frames in total.
For folding task, we recorded 3896 videos which contain 204K frames in total.
As shown in Fig. 1, the data for each frame include multi-view RGB-D images,
object masks, full garment meshes, and hand poses. Please see the
supplementary files for more statistics on the dataset.

##  4 Method

This paper proposes an end-to-end online tracking method called
GarmentTracking for category-level garment pose tracking. As shown in Fig. 2,
given a first-frame garment pose (point NOCS, _i.e_. canonical coordinates of
partial point cloud) and a rough canonical shape (mesh NOCS, _i.e_. sampled
points from mesh in canonical space) of an instance, it takes point cloud
sequences as input and generates complete garment geometry with inter-frame
correspondence (_i.e_. NOCS coordinates). Specifically, GarmentTracking can be
divided into three stages. In the first stage (Sec. 4.1), the network will
predict canonical coordinates for the partial input point cloud. In the second
stage (Fig. 3), the network will refine the predicted canonical coordinates
and the input canonical shape. In the third stage (Sec. 4.3), the network will
use the refined canonical shape and canonical coordinates to predict a warp
field that maps from canonical space to task space (_i.e_. the coordinate
frame of the input point cloud).

###  4.1 Canonical Coordinate Prediction

####  4.1.1 Normalized Garment Canonical Space

Following the definition of garment representation in GarmentNets [11], we use
Normalized Object Coordinate Space (NOCS) coordinates as an intermediate
representation for object states in a category. As shown in Fig. 2, the rest
state of a garment is the T-pose defined by the garment worn by a person
(provided by CLOTH3D [8] dataset).

####  4.1.2 3D Feature Extractor

The thin structure and near-infinite DOF nature of garments may result in many
complicated poses (_e.g_. multi-layer cloth stacked together) that require
feature extraction for granular local details. Our method uses a high-
resolution sparse 3D convolution network (ResUNet3D) proposed by FCGF [12] to
extract the per-point feature from the raw point cloud. ResUNet3D is a UNet-
like network with skip connections and residual blocks. Please refer to the
supplementary files for further details of the network.

####  4.1.3 Inter-frame Feature Fusion with Transformer

After extracting the feature from the extractor, we apply the inter-frame
feature fusion with Transformer [36].

Feature Matching Inspired by PTTR [40], we perform feature matching with self-
attention and cross-attention modules based on Transformer [36]. In general,
we first use a self-attention module to individually aggregate point features
for the two input frames. Then we use a cross-attention module to perform
feature matching between two frames. Intuitively, the self-attention operation
can have a global understanding of the current frame, and the cross-attention
operation can capture cross-frame correlations and generate a relation-
enhanced fusion feature. The self-attention and cross-attention modules are
based on the relation attention module (RAM) proposed by PTTR [40]. Please see
the supplementary files for more details on the relation attention module.

NOCS Prediction After obtaining the per-point fusion feature via the cross-
attention module, we predict the per-point canonical coordinate with MLP. We
follow GarmentNets [11] and formulate this problem as a classification task
instead of a regression task. Specifically, we divide each axis into 64 bins
and the network independently predicts each axis’s classification score.
During training, we use a cross-entropy loss to supervise the classification
scores.

NOCS Coordinates for Positional Embedding We have per-point NOCS coordinate
prediction from the previous frame, which contains clearer geometric and
structural information. We use it for positional embedding [36], which will be
added to input features before feeding into transformers. The positional
embeddings for two frames are calculated as Eq. 1:

$\mathbf{emb}_{1}=f_{1}([\mathbf{P}_{1}^{xyz},\mathbf{P}_{1}^{nocs}]),\mathbf{emb}_{2}=f_{2}([\mathbf{P}_{2}^{xyz}]),$
(1)

where $\mathbf{P}_{1}^{xyz}$ and $\mathbf{P}_{2}^{xyz}$ are the partial input
point clouds of the two frames, and $\mathbf{P}_{1}^{nocs}$ is the predicted
per-point NOCS coordinates of the partial point cloud in the previous frame.
Here $f_{1}(\cdot)$ and $f_{2}(\cdot)$ are learned MLP. By fusing NOCS
coordinates into positional embedding, the transformer network will
incorporate positional and semantic information from previous frames. Besides,
empirically speaking, utilizing intermediate representations like NOCS
coordinates instead of complete garment poses can increase the robustness
against noisy predictions during long-term tracking.

###  4.2 NOCS Refiner

Figure 3: PC-Mesh Fusion Refiner

Since the canonical shape can be generated by other methods like GarmentNets
[11], or augmented with noise, it might be inaccurate. On the other hand, the
NOCS coordinate predictions can also be noisy. Such inaccuracy could cause
errors to be accumulated during tracking. To mitigate this problem, we propose
a NOCS PC (Point Cloud)-Mesh intertwined refiner, NOCS refiner. As shown in
Fig. 3, the predicted NOCS coordinates can reveal the cues of the input point
cloud, such as scales and offsets, while the canonical shape can provide
information about the complete geometry. Thus they can complement each other.
We describe the NOCS refiner in two parts (PC refiner and Mesh refiner):

PC Refiner: Firstly, the predicted NOCS classification scores and the per-
point fusion feature from the transformer will be concatenated and fed into a
Mini-Pointnet [32]. Next, the dense feature will be fused with the global mesh
feature generated by Mesh Refiner with concatenation. Finally, we use MLP to
predict the final delta logits with the fused dense feature. We use cross-
entropy loss to supervise the refined classification logits during training.

Mesh Refiner: Firstly, we use a Mini-Pointnet to extract dense features from
raw canonical shapes. Then we concatenate the dense mesh feature generated by
PC Refiner with the global feature from the partial point cloud to obtain the
fused dense feature. Next, we use an MLP with global pooling to extract the
final global shape feature. Finally, we predict the global scale factor and
offset for the canonical shape with the global shape feature and an MLP.
Finally, we use L2 loss to supervise the refined mesh points during training.

###  4.3 Warping from Canonical To Task Space

####  4.3.1 Feature Scattering with Canonical Coordinates

After obtaining the refined canonical (NOCS) coordinate prediction (Fig. 3) of
a partial point cloud, we scatter the per-point feature generated by
Transformer (Sec. 4.1) into a $32^{3}$ feature volume. The “scatter” operation
is performed by copying the feature vector to the target location in volume
with predicted NOCS coordinates. All features mapped to the same volume index
will be aggregated with a channel-wise maximum operation. And all the volume
locations with no corresponding feature vectors are filled with zeros. Then
the feature volume will be fed into a 3D UNet [13], then we can obtain a dense
feature volume $\mathcal{V}$ for warp field prediction.

####  4.3.2 Warp Field Prediction

Finally, we map the refined canonical shape (Fig. 3) from canonical space to
task space. The output contains the full configuration of the garment,
including the occluded parts. It is achieved by warp field prediction[11],
which is an implicit neural function $w(p;\mathcal{V})\in\mathbf{R}^{3}$ that
takes a query point $p$ in the canonical space as input and infers the
corresponding location of $p$ in task space. Here $w(\cdot)$ is a learned MLP.
We use L2 loss to supervise the warp field prediction. In training, the query
points are sampled from the canonical mesh surface. In inference, the query
points are generated by our Mesh Refiner (Fig. 3).

##  5 Experiments

###  5.1 Implementation Details

We implement our method with Pytorch [4] and use Adam optimizer with a
learning rate of 0.0001. The training stage takes about 150 epochs to
converge, which lasts for 1-3 days on an RTX 3090 GPU, depending on the
training dataset sizes for different categories. We randomly sample 4000
points from the input partial point cloud and 6000 points from the input
canonical mesh surface for each frame. During training, we randomly add noise
to the partial point-cloud canonical coordinates $\mathbf{P}_{1}^{nocs}$ of
the previous frame by randomly generating a NOCS scale factor
$\mathbf{s}_{pc}\in[0.8,1.2]^{3}$ and a global NOCS offset
$\mathbf{o}_{pc}\in[0,0.1]^{3}$. We also add noise to the input canonical mesh
by randomly generating a global NOCS scale factor
$\mathbf{s}_{mesh}\in[0.8,1.2]^{3}$ during training. Please see the
supplementary files for further details on training, inference, and network
structure.

###  5.2 Metrics

NOCS Coordinate Distance ($D_{nocs}$). We calculate the point-wise L2 distance
between the predicted NOCS coordinate of the partial point cloud with the
ground-truth NOCS labels. This metric evaluates the quality of per-point NOCS
coordinate prediction for input partial point cloud.

Chamfer Distance ($D_{chamf}$). We calculate the Chamfer distance in
centimeters between the reconstructed mesh points and the ground-truth mesh
points in task space. This metric can evaluate the quality of surface
reconstruction.

Correspondence Distance ($D_{corr}$, $A_{d}$). We calculate point-wise L2
distance in centimeters between the reconstructed mesh and the ground-truth
mesh for each frame in task space. The correspondences are based on the NOCS
coordinates (_i.e_. each point on the predicted mesh will find the closest
point on the ground-truth mesh in NOCS). This metric can evaluate the quality
of garment pose estimation. In practice, we find the variance of the error
distribution in different frames is very large, which makes the mean
correspondence distance $D_{corr}$ across all frames dominated by the worst
cases. So we additionally introduce $A_{d}$ which represents the accuracy
(_i.e_. ratio of frames) with $D_{corr}<d$.

###  5.3 Experiment Results

####  5.3.1 Main Experiments

Baselines In Tab. 1, we compare GarmentNets with two settings of our method:

GarmentNets [11]: As the only-existing method for category-level garment pose
estimation, GarmentNets focused on the single-frame setting. We adapt it for
tracking by prediction frame by frame.

Ours (GT): Our tracking method given the ground-truth first-frame garment pose
and the ground-truth canonical mesh as initialization.

Ours (Pert.): Our tracking method when the first-frame garment pose and the
input canonical shape are perturbed with noise. Specifically, we use the same
noise distribution in training (Sec. 5.1) which adds global NOCS scale and
offset to the first-frame canonical coordinates of partial point-cloud and the
canonical mesh. Additionally, we add per-point Gaussian noise ($\delta$=0.05)
to the input canonical coordinates of the first frame during inference.

Type | Method | Init. | Folding | Flattening
---|---|---|---|---
$A_{3cm}\uparrow$ | $A_{5cm}\uparrow$ | $D_{corr}\downarrow$ | $D_{chamf}\downarrow$ | $D_{nocs}\downarrow$ | $A_{5cm}\uparrow$ | $A_{10cm}\uparrow$ | $D_{corr}\downarrow$ | $D_{chamf}\downarrow$ | $D_{nocs}\downarrow$
Shirt | GarmentNets [11] | N/A | 0.8% | 21.5% | 6.40 | 1.58 | 0.221 | 13.2% | 59.4% | 10.54 | 3.54 | 0.135
Ours | GT | 29.8% | 85.8% | 3.88 | 1.16 | 0.051 | 30.7% | 83.4% | 8.63 | 1.78 | 0.105
Ours | Pert. | 29.0% | 85.9% | 3.88 | 1.18 | 0.052 | 25.4% | 81.6% | 8.94 | 1.85 | 0.109
Pants | GarmentNets [11] | N/A | 16.2% | 69.5% | 4.43 | 1.30 | 0.162 | 1.5% | 42.4% | 12.54 | 4.19 | 0.185
Ours | GT | 47.3% | 94.0% | 3.26 | 1.07 | 0.039 | 31.3% | 78.2% | 8.97 | 1.64 | 0.113
Ours | Pert. | 42.8% | 93.6% | 3.35 | 1.10 | 0.039 | 30.7% | 76.9% | 9.55 | 2.71 | 0.143
Top | GarmentNets [11] | N/A | 10.3% | 53.8% | 5.19 | 1.51 | 0.148 | 21.6% | 57.6% | 9.98 | 2.13 | 0.174
Ours | GT | 37.9% | 85.9% | 3.75 | 0.99 | 0.051 | 36.5% | 69.0% | 9.41 | 1.59 | 0.113
Ours | Pert. | 36.6% | 86.1% | 3.76 | 1.00 | 0.051 | 33.5% | 68.1% | 9.61 | 1.62 | 0.116
Skirt | GarmentNets [11] | N/A | 1.1% | 30.3% | 6.95 | 1.89 | 0.239 | 0.1% | 7.9% | 18.48 | 5.99 | 0.287
Ours | GT | 23.5% | 71.3% | 4.61 | 1.33 | 0.060 | 5.4% | 39.4% | 16.09 | 2.02 | 0.199
Ours | Pert. | 22.8% | 70.6% | 4.72 | 1.36 | 0.060 | 2.3% | 35.5% | 16.55 | 2.15 | 0.207

Table 1: Quantitative results on VR-Folding dataset.

Results Tab. 1 summarizes the quantitative results on the VR-Folding dataset.
In general, our method outperforms GarmentNets in all metrics by a large
margin. On the challenging $A_{3cm}$ metric in Folding task and $A_{5cm}$ in
Flattening task, GarmentNets has very low performance (_e.g_. $0.8\%$ in Shirt
Folding), while our method achieves much higher scores (_e.g_. $29.0\%$ in
Shirt Folding), which proves that our method can generate more accurate
predictions in videos compared to GarmentNets. Our method also outperforms
GarmentNets on mean correspondence distance $D_{corr}$ and chamfer distance
$D_{chamf}$, which proves that our method can do well in both pose estimation
and surface reconstruction tasks. Even with perturbation on first-frame poses
(Ours with Pert. in Tab. 1), our method only shows minor performance loss
(_e.g_. $37.9\%\rightarrow 36.6\%$ in Top Folding) compared to using ground-
truth as first-frame pose.

We also present some qualitative results in Fig. 4 and Fig. 5. We can see from
Fig. 5 that the prediction results of GarmentNets are very unstable because it
performs mesh reconstruction for each frame individually and can not utilize
the information from previous frames. Conversely, our method can leverage
input canonical mesh and inter-frame information to predict more stable and
accurate results. Besides, GarmentNets suffers from ambiguity brought by
symmetry (_e.g_. take a front side as a back side), which hampers its ability
to predict accurate canonical coordinates (see Fig. 4). In contrast, our
method can predict much more accurate canonical coordinates (_e.g_. $D_{nocs}$
0.162 v.s. 0.039 for Pants Folding in Tab. 1).

Figure 4: The canonical coordinate prediction results on the VR-Folding
dataset. Figure 5: The qualitative results of pose estimation for unseen
instances in VR-Folding dataset. In the long sequence tracking (shown in the
lower part), our prediction still keeps high consistency with GT, while
GarmentNets outputs a series of meshes that lack stability.

####  5.3.2 Ablation Study

NOCS Positional Embedding. In our method, NOCS positional embedding (Sec.
4.1.3) is the crucial design choice to leverage inter-frame correspondence
information. As shown in Fig. 4, if we remove the NOCS positional embedding
from our network, the network will suffer from the same ambiguity problem as
GarmentNets due to symmetry.

Type | Method | Folding | Flattening
---|---|---|---
$A_{3cm}\uparrow$ | $A_{5cm}\uparrow$ | $D_{corr}\downarrow$ | $D_{chamf}\downarrow$ | $D_{nocs}\downarrow$ | $A_{5cm}\uparrow$ | $A_{10cm}\uparrow$ | $D_{corr}\downarrow$ | $D_{chamf}\downarrow$ | $D_{nocs}\downarrow$
Shirt | Ours | 29.0% |  85.9% | 3.88 | 1.18 | 0.052 | 25.4% | 81.6% | 8.94 | 1.85 | 0.109
Ours w.o. PC refiner | 4.1% | 25.3% | 7.76 | 1.70 | 0.115 | 0.6% | 27.4% | 17.42 | 2.89 | 0.229
Ours w.o. Mesh refiner | 26.8% | 83.6% | 3.92 | 1.19 | 0.048 | 23.5% | 81.2% | 9.18 | 1.88 | 0.110
Ours w. PointNet++ [33] | 2.2% | 34.7% | 6.53 | 1.54 | 0.085 | 13.3% | 53.2% | 14.21 | 1.93 | 0.174
Pants | Ours |  42.8% | 93.6% | 3.35 | 1.10 | 0.039 | 30.7% | 76.9% | 9.55 | 2.71 | 0.143
Ours w.o. PC refiner | 23.1% | 70.0% | 4.84 | 1.30 | 0.072 | 5.8% | 49.2% | 13.72 | 3.46 | 0.165
Ours w.o. Mesh refiner | 33.5% | 92.2% | 3.52 | 1.18 | 0.039 | 22.5% | 75.2% | 9.76 | 2.78 | 0.148
Ours w. PointNet++ [33] | 8.9% | 73.6% | 4.91 | 1.33 | 0.066 | 8.0% | 69.1% | 10.12 | 2.81 | 0.125

Table 2: Results of ablative experiments on VR-Folding dataset.

NOCS Refiner. Unlike rigid object tracking, garment tracking has a higher
demand for avoiding error accumulation in long videos, because the error
distribution of predicted pose in testing can be very different from that in
training due to the near-infinite DOF. As shown in Tab. 2, our proposed PC
Refiner (Fig. 3) greatly influences the performance due to its ability to
refine NOCS coordinate predictions in each frame. Besides, the Mesh Refiner
(Fig. 3) also contributes to a slight performance improvement, indicating that
our network has more tolerance for canonical mesh errors than NOCS coordinate
errors.

Feature Extractor. As shown in Tab. 2, after we replace our feature extractor
(_i.e_. ResUNet3D [12] based on sparse 3D convolution) with PointNet++ [33],
the overall performance drops a lot. Thus high-resolution 3D convolution
network should be a better choice for this task.

####  5.3.3 Robustness

Robustness against Noise We test our method under different levels of noise
perturbation by increasing the initial pose noise level described in Sec.
5.3.1 by 1 or 2 times. Specifically, we augment the point-cloud NOCS
coordinates of the first frame with a global scaling factor $s_{pc}$, a global
offset $o_{pc}$, and Gaussian noise standard deviation $\delta$. Besides, we
also augment the canonical mesh with global scaling factor $s_{mesh}$. As
shown in Fig. 6 (left), We can see that our method performs well at surface
reconstruction (_i.e_. $D_{chamf}$) under high noise level. Please see the
supplementary files for details of the noise parameters.

Robustness under Large Frame Interval Tracking at large frame intervals with
huge object deformations can be very challenging. Therefore, we uniformly drop
frames in videos and only keep part (_i.e_. 1/2, 1/4, 1/6, and 1/8) of the
frames. The results are shown in Fig. 6 (right), we can see that our method is
more robust against missing frames on Folding task compared to Flattening
task.

Figure 6: The robustness experiments.

####  5.3.4 Tracking Speed

On a single RTX 3090 GPU, GarmentNets takes 100ms to pass the backbone, 7ms
for volume query, and 170ms for Marching Cubes [25]. It results in a runtime
of 3.6 FPS. While in our design, we adopt a faster backbone that costs only
45ms and eliminates the time-consuming Marching Cubes. Our NOCS refiner and
warp field prediction take 12ms and 7ms respectively. Our method can achieve
15 FPS during inference, which is $\sim 5$ times faster than [11].

####  5.3.5 Generalization Ability

Neural Prediction as First-Frame Pose In order to evaluate the generalization
ability of our method, we directly use GarmentNets prediction (_i.e_.
canonical coordinates and mesh) as the first-frame pose during inference. As
shown in Tab. 3, our method still outperforms GarmentNets by a large margin
without any data augmentation related to GarmentNets during training.

Type | Method | $A_{3cm}\uparrow$ | $A_{5cm}\uparrow$ | $D_{corr}\downarrow$ | $D_{chamf}\downarrow$ | $D_{nocs}\downarrow$
---|---|---|---|---|---|---
Shirt | Ours* | 25.4% | 78.9% | 4.04 | 1.18 | 0.052
GarmentNets | 0.8% | 21.5% | 6.40 | 1.58 | 0.221
Pants | Ours* | 45.1% | 92.2% | 3.33 | 1.16 | 0.040
GarmentNets | 16.2% | 69.5% | 4.43 | 1.30 | 0.162
Top | Ours* | 21.1% | 61.9% | 4.82 | 1.11 | 0.065
GarmentNets | 10.3% | 53.8% | 5.19 | 1.51 | 0.148
Skirt | Ours* | 14.7% | 65.9% | 5.36 | 1.46 | 0.078
GarmentNets | 1.1% | 30.3% | 6.95 | 1.89 | 0.239

Table 3: Results of our method using GarmentNets prediction as first-frame
pose on Folding task.

Real World Experiments We collect some real-world RGB-D videos of garment
manipulation with Realsense L515[5] LiDAR cameras. Our method can directly
track garment pose for novel garments in the real-world with a model trained
only on our simulated data. Please see the supplementary files for more
qualitative results.

##  6 Conclusion and Future Works

In this work, we propose a complete framework for garment pose tracking,
including the data collection (_i.e_. VR-Garment system), dataset (_i.e_. VR-
Folding), and a strong approach (_i.e_. GarmentTracking) which is both
quantitatively and qualitatively better than the baseline approach. As a
platform, we believe VR-Garment can innovate the dataset collection for other
kinds of deformable objects. As a manipulation dataset, we are interested in
using VR-Folding for robot imitation learning. As a strong baseline, we hope
GarmentTracking can facilitate future research in this challenging direction.

## Acknowledgement

This work was supported by the National Key Research and Development Project
of China (2021ZD0110704), Shanghai Municipal Science and Technology Major
Project (2021SHZDZX0102), Shanghai Qi Zhi Institute, Shanghai Science and
Technology Commission (21511101200) and OpenBayes.

## References

  * [1] HTC Vive Pro Headset.  https://www.vive.com/us/product/vive-pro/.
  * [2] Noitom Hi5 VR Glove.  https://www.noitom.com/hi5-vr-glove.
  * [3] Obi: Unified particle physics for Unity.  http://obi.virtualmethodstudio.com/.
  * [4] Pytorch.  https://pytorch.org/.
  * [5] RealSense L515 LiDAR camera.  https://www.intelrealsense.com/lidar-camera-l515/.
  * [6] Unity Game Engine.  http://www.unity.com/.
  * [7] Yahav Avigal, Lars Berscheid, Tamim Asfour, Torsten Kröger, and Ken Goldberg.  Speedfolding: Learning efficient bimanual folding of garments.  In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1–8. IEEE, 2022.
  * [8] Hugo Bertiche, Meysam Madadi, and Sergio Escalera.  Cloth3d: clothed 3d humans.  In European Conference on Computer Vision, pages 344–359. Springer, 2020.
  * [9] Aljaz Bozic, Pablo Palafox, Michael Zollhöfer, Angela Dai, Justus Thies, and Matthias Nießner.  Neural non-rigid tracking.  Advances in Neural Information Processing Systems, 33:18727–18737, 2020.
  * [10] Aljaz Bozic, Michael Zollhofer, Christian Theobalt, and Matthias Nießner.  Deepdeform: Learning non-rigid rgb-d reconstruction with semi-supervised data.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7002–7012, 2020.
  * [11] Cheng Chi and Shuran Song.  Garmentnets: Category-level pose estimation for garments via canonical space shape completion.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3324–3333, 2021.
  * [12] Christopher Choy, Jaesik Park, and Vladlen Koltun.  Fully convolutional geometric features.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8958–8966, 2019.
  * [13] Özgün Çiçek, Ahmed Abdulkadir, Soeren S Lienkamp, Thomas Brox, and Olaf Ronneberger.  3d u-net: learning dense volumetric segmentation from sparse annotation.  In International conference on medical image computing and computer-assisted intervention, pages 424–432. Springer, 2016.
  * [14] Mingsong Dou, Philip Davidson, Sean Ryan Fanello, Sameh Khamis, Adarsh Kowdle, Christoph Rhemann, Vladimir Tankovich, and Shahram Izadi.  Motion2fusion: Real-time volumetric performance capture.  ACM Transactions on Graphics (TOG), 36(6):1–16, 2017.
  * [15] Mingsong Dou, Sameh Khamis, Yury Degtyarev, Philip Davidson, Sean Ryan Fanello, Adarsh Kowdle, Sergio Orts Escolano, Christoph Rhemann, David Kim, Jonathan Taylor, et al.  Fusion4d: Real-time performance capture of challenging scenes.  ACM Transactions on Graphics (ToG), 35(4):1–13, 2016.
  * [16] Kaiwen Guo, Feng Xu, Tao Yu, Xiaoyang Liu, Qionghai Dai, and Yebin Liu.  Real-time geometry, albedo, and motion reconstruction using a single rgb-d camera.  ACM Transactions on Graphics (ToG), 36(4):1, 2017.
  * [17] Huy Ha and Shuran Song.  Flingbot: The unreasonable effectiveness of dynamic manipulation for cloth unfolding.  In Conference on Robot Learning, pages 24–33. PMLR, 2022.
  * [18] Fangzhou Hong, Liang Pan, Zhongang Cai, and Ziwei Liu.  Garment4d: Garment reconstruction from point cloud sequences.  Advances in Neural Information Processing Systems, 34:27940–27951, 2021.
  * [19] Matthias Innmann, Michael Zollhöfer, Matthias Nießner, Christian Theobalt, and Marc Stamminger.  Volumedeform: Real-time volumetric non-rigid reconstruction.  In European conference on computer vision, pages 362–379. Springer, 2016.
  * [20] Xiaolong Li, He Wang, Li Yi, Leonidas J Guibas, A Lynn Abbott, and Shuran Song.  Category-level articulated object pose estimation.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3706–3715, 2020.
  * [21] Yang Li, Hikari Takehara, Takafumi Taketomi, Bo Zheng, and Matthias Nießner.  4dcomplete: Non-rigid motion estimation beyond the observable surface.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 12706–12716, 2021.
  * [22] Wenbin Lin, Chengwei Zheng, Jun-Hai Yong, and Feng Xu.  Occlusionfusion: Occlusion-aware motion estimation for real-time dynamic 3d reconstruction.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1736–1745, 2022.
  * [23] Liu Liu, Wenqiang Xu, Haoyuan Fu, Sucheng Qian, Qiaojun Yu, Yang Han, and Cewu Lu.  Akb-48: a real-world articulated object knowledge base.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14809–14818, 2022.
  * [24] Liu Liu, Han Xue, Wenqiang Xu, Haoyuan Fu, and Cewu Lu.  Toward real-world category-level articulation pose estimation.  IEEE Transactions on Image Processing, 31:1072–1083, 2022.
  * [25] William E Lorensen and Harvey E Cline.  Marching cubes: A high resolution 3d surface construction algorithm.  ACM siggraph computer graphics, 21(4):163–169, 1987.
  * [26] Qianli Ma, Jinlong Yang, Anurag Ranjan, Sergi Pujades, Gerard Pons-Moll, Siyu Tang, and Michael J Black.  Learning to dress 3d people in generative clothing.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6469–6478, 2020.
  * [27] Richard A Newcombe, Dieter Fox, and Steven M Seitz.  Dynamicfusion: Reconstruction and tracking of non-rigid scenes in real-time.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 343–352, 2015.
  * [28] Michael Niemeyer, Lars Mescheder, Michael Oechsle, and Andreas Geiger.  Occupancy flow: 4d reconstruction by learning particle dynamics.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 5379–5389, 2019.
  * [29] Pablo Palafox, Aljaž Božič, Justus Thies, Matthias Nießner, and Angela Dai.  Npms: Neural parametric models for 3d deformable shapes.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 12695–12705, 2021.
  * [30] Pablo Palafox, Nikolaos Sarafianos, Tony Tung, and Angela Dai.  Spams: Structured implicit parametric models.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12851–12860, 2022.
  * [31] Gerard Pons-Moll, Sergi Pujades, Sonny Hu, and Michael J Black.  Clothcap: Seamless 4d clothing capture and retargeting.  ACM Transactions on Graphics (ToG), 36(4):1–15, 2017.
  * [32] Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas.  Pointnet: Deep learning on point sets for 3d classification and segmentation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 652–660, 2017.
  * [33] Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas.  Pointnet++: Deep hierarchical feature learning on point sets in a metric space.  Advances in neural information processing systems, 30, 2017.
  * [34] Miroslava Slavcheva, Maximilian Baust, Daniel Cremers, and Slobodan Ilic.  Killingfusion: Non-rigid 3d reconstruction without correspondences.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1386–1395, 2017.
  * [35] Miroslava Slavcheva, Maximilian Baust, and Slobodan Ilic.  Sobolevfusion: 3d reconstruction of scenes undergoing free non-rigid motion.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2646–2655, 2018.
  * [36] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  Advances in neural information processing systems, 30, 2017.
  * [37] Andreas Verleysen, Matthijs Biondina, and Francis Wyffels.  Video dataset of human demonstrations of folding clothing for robotic folding.  The International Journal of Robotics Research, 39(9):1031–1036, 2020.
  * [38] He Wang, Srinath Sridhar, Jingwei Huang, Julien Valentin, Shuran Song, and Leonidas J Guibas.  Normalized object coordinate space for category-level 6d object pose and size estimation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2642–2651, 2019.
  * [39] Han Xue, Liu Liu, Wenqiang Xu, Haoyuan Fu, and Cewu Lu.  Omad: Object model with articulated deformations for pose estimation and retrieval.  arXiv preprint arXiv:2112.07334, 2021.
  * [40] Changqing Zhou, Zhipeng Luo, Yueru Luo, Tianrui Liu, Liang Pan, Zhongang Cai, Haiyu Zhao, and Shijian Lu.  Pttr: Relational 3d point cloud object tracking with transformer.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8531–8540, 2022.
  * [41] Heming Zhu, Yu Cao, Hang Jin, Weikai Chen, Dong Du, Zhangye Wang, Shuguang Cui, and Xiaoguang Han.  Deep fashion3d: A dataset and benchmark for 3d garment reconstruction from single images.  In European Conference on Computer Vision, pages 512–530. Springer, 2020.