# Do You Need a Hand? – a Bimanual Robotic Dressing Assistance Scheme

Jihong Zhu1,3, Michael Gienger2, Giovanni Franzese3, and Jens Kober3 1J. Zhu
is with University of York, the UK<EMAIL_ADDRESS>Gienger is with
Honda Research Institute Europe, Germany<EMAIL_ADDRESS>Zhu,
G. Franzese, and J. Kober are with Cognitive Robotics, 3mE, Delft University
of Technology, Netherlands<EMAIL_ADDRESS>
###### Abstract

Developing physically assistive robots capable of dressing assistance has the
potential to significantly improve the lives of the elderly and disabled
population. However, most robotics dressing strategies considered a single
robot only, which greatly limited the performance of the dressing assistance.
In fact, healthcare professionals perform the task bimanually. Inspired by
them, we propose a bimanual cooperative scheme for robotic dressing
assistance. In the scheme, an interactive robot joins hands with the human
thus supporting/guiding the human in the dressing process, while the dressing
robot performs the dressing task. We identify a key feature that affects the
dressing action and propose an optimal strategy for the interactive robot
using the feature. A dressing coordinate based on the posture of the arm is
defined to better encode the dressing policy. We validate the interactive
dressing scheme with extensive experiments and also an ablation study. The
experiment video is available on
https://sites.google.com/view/bimanualassitdressing/home

##  I Introduction

The shortage of caregivers is a pressing challenge worldwide due to a decrease
in birth rates combined with an increase in life expectancy: according to the
WHO, by 2030, 1 in 6 people in the world will be aged 60 years or over. By
2050, the world’s population of people aged 60 years and older will double
[1]. Assistive robots capable of physically aiding humans are promising to
tackle this challenge. Among various tasks that the caregiver performs on a
daily basis, the dressing was reported to be the greatest burden while the
least automated [2]. In this paper, we tackle the problem of dressing a human
arm with a bimanual robot setup.

Robot-assisted dressing is challenging as it involves direct physical
interaction with flexible clothes and humans. More specifically, it differs
from a conventional human-robot interaction task such as co-manipulation or
co-carrying [3, 4, 5] where the object allows immediate force propagation. In
the dressing assistance, the object i.e. the cloth, is soft and deformable,
thus, the interaction force is hard to obtain. Due to the absence of direct
force feedback and heavy occlusion during dressing, arm movement estimation
during dressing is also difficult.

One common assumption to simplify the task is to consider static arm posture
during dressing. Then the main challenge becomes adapting the dressing policy
to different static postures. Learning from demonstrations (LfD) is often used
in obtaining a generalized policy [6, 7, 8].

However, a static arm posture is a very strong assumption. To make the
proposed scheme work in practice, we need to consider dynamic poses during
dressing. Since the task largely depends on postures, arm movement estimation
is crucial.

Computer vision is a popular way to track arm postures. It is also used in the
early work in assistive dressing [9] for tracking upper limb movement.
However, the inevitable occlusions during dressing render most of the existing
vision-based human posture tracking algorithms fail [10, 11]. Additional
sensory modules, such as proximity sensors may be employed for posture
estimation [12].

To overcome the difficulties faced with conventional assistive dressing
strategies, we draw inspiration from human-to-human dressing assistance from
healthcare professionals111https://youtu.be/Wn1OEfQt0ow?t=93. Our proposed
framework resembles the method adapted by health professionals. It requires
two robots to perform the dressing assistance task. An interactive robot
$\mathcal{I}$ that reaches out for holding hands with the human, then
supports/guides the human to facilitate the dressing process. The dressing
robot $\mathcal{D}$ performs the main task, dressing. Hand holding not only
makes the dressing task easier to execute by supporting/guiding the human arm
movement but also results in the tracking of the arm posture with
proprioceptive sensors only.

The main contribution of the paper is the proposal of a bimanual dressing
assistance framework, which is unprecedented in previous robotics research. In
the framework, instead of considering interaction forces during dressing which
is often hard to obtain in practice [13], we offer a novel perspective by
analysing dressing geometrically. Benefiting from this new insight, we
identify a key feature in the human posture that affects dressing policy. The
feature is then used for designing an optimal stretch policy for the
interactive robot. For the dressing, in contrast to encoding the movement in
Cartesian space, we define a task coordinate system for flexible policy
encoding from demonstrations.

The paper is organized as follows: In Sect. II we survey the related work in
three subcategories, namely: robotic dressing assistance, bimanual
manipulation, and arm poster estimation. Sect. III provides an overview of the
overall framework. Sect. IV introduces the feature that affects the dressing
strategy and then designs an optimal strategy for the interactive robot
$\mathcal{I}$ based on the feature. With the information on the hand position
given by the $\mathcal{I}$, we can solve the arm posture with proprioceptive
sensors only. Later in Sect. V, we propose a dressing coordinate dependent on
arm postures, then learn the dressing policy from expert demonstrations in the
dressing coordinate. The overall framework is validated with robotic
experiments in Sect. VI. Finally in Sect. VII, we conclude.

##  II Related Work

###  II-A Robotic Dressing Assistance

Robotic dressing assistance is a recurrent topic that broadly covers dressing
tops (t-shirt, jacket, etc.) [14], trousers [15] and even shoes [16, 17].
Since in this paper, our focus is dressing the upper body, we review recent
advances, particularly in this area.

Dressing assistance is a skill mastered by humans. Therefore, LfD-based
approaches are popular in obtaining motor skills in dressing. Pignat and
Calinon learned a hidden semi-Markov model [6] from human demonstrations to
encode the dressing skills. The authors of [7] implemented an incremental
learning of Task-Parameterized Gaussian Mixture Models (TP-GMM) that allows
generalization of the dressing task. More recently, [8] proposed a method to
reduce the number of demonstrations needed for training the TP-GMM to encode a
dressing policy in different static postures.

Recent research also explores the interaction between a robot and a human in
the dressing process with reinforcement learning in a simulation environment
[18, 19]. The authors of [20] learned an outcome classifier for dressing tasks
with simulated haptic data. Instead of relying on simulations, data collected
in the real dressing scenarios suggests that the interaction force between the
cloth and the human allows for predicting dressing outcomes with high accuracy
[21]. Later, Erickson et al. [22]conducted an investigation with physics-based
simulation suggesting that the interaction force can be recovered with only
end-effector measurements.

Further research in this area is concerned with cloth state modeling [23],
safe motion planning[24, 25], grasping point selection [26] and novel gripper
design for dressing assistance [27].

In the above-mentioned research, the dressing tasks were mostly conducted with
a single manipulator. The only papers considering bimanual dressing are [19,
14]. However, they consider the bimanual in a completely different setup where
each robot is dressing one human arm respectively. Therefore, they adopt the
one-robot-to-one-arm setup just like the rest of the papers. In our proposal,
we have instead a two-robot-to-one-arm setup, where a robot is guiding humans
to facilitate the dressing task of the other robot.

###  II-B Bimanual Manipulation

In the proposed framework, we adopt implicit master-slave coordination between
the interactive and the dressing robots. The dressing policy is dependent on
the human arm posture which is controlled by the interactive robot from the
applied guidance force on the human hand. Such task setup is referred to as
asymmetrical bimanual manipulation in the bimanual taxonomy literature [28].

The asymmetrical bi-manual task that was investigated in the previous work
includes: vegetable peeling [29], cloth manipulation [30], assembly [31] and
cooking [32]. These tasks usually do not require interaction with a human as
in the dressing assistance.

More relevant bimanual assistive tasks: Xu et al. designed an autonomous
wheelchair with two robots to assist humans in daily tasks [33]. Connan et al.
employed a humanoid robot to perform tasks such as removing the lid,
unscrewing a bottle, and pouring water [34]. Edsinger and Kemp discussed three
key themes in bimanual manipulation design in assistive tasks: cooperative
manipulation, tasks relevant features, let the body do the thinking and
analysed them in different assistive scenarios [35]. These key themes remain
very relevant to us in designing the framework for this paper.

To date, robotics dressing assistance (specifically dressing clothes) has
never been considered a bimanual task that requires both robots. However, as
we argued in Sect. I, the additional robot can provide support and guidance to
the arm being dressed, which makes the dressing easier and more comfortable.
As the scheme is employed by caregiving professionals, humans might feel more
contented dressing in such a composition. By adopting the bimanual setup in
dressing, our work is the first of its kind and represents a paradigm shift in
thinking of the dressing assistance task.

###  II-C Arm Posture Estimation

The dressing policy is highly dependent on human arm posture. As the guidance
force at hand will change the human arm posture in our framework, a robust arm
posture tracking/estimation is required for dressing policy generation.

Due to heavy occlusion during dressing, the task is particularly challenging
for popular vision-based deep learning posture tracking algorithm [36, 37, 38,
39].

Human modeling is employed for tracking the human posture under occlusions. A
preliminary study in dressing assistance considers human modeling combined
with visual tracking in personalized dressing: the authors of [9] adopted a
mixture of Gaussians for modeling, and later a Gaussian process latent
variable model (GP-LVM) was employed for posture estimation in [10]. Recently,
the author of [11] tackled the occlusion problem by employing a recurrent
neural network trained on human-human interaction data to predict the elbow
position during dressing. Rather than relying on vision alone, previous works
also explored capacitive proximity sensing [12] for posture tracking.

In this paper, we will demonstrate that tracking is made easy by hand-holding.
With the hand position always known, arm posture tracking can be formulated as
an inverse kinematic problem that has analytical solutions and is solvable in
real-time with proprioceptive sensors.

Figure 1: The schematic diagram of cooperative dressing framework. The
framework contains three blocks: a posture estimation, a dressing motion
generator, and an optimal stretch controller. The posture estimation takes in
an initial posture of the human arm and the current hand position and outputs
the estimated current arm posture. The posture is then fed into the dressing
motion generator together with the current dressing robot position to derive
the next step dressing policy. The optimal stretch controller takes the
initial human arm posture and the current hand posture yields the stretching
policy for the interactive robotics arm.

##  III Framework Overview

We envision a bimanual dressing assistance system that:

  1. 1.

provides support/guidance to the human arm so as to facilitate dressing,

  2. 2.

tracks the arm posture without additional sensors (vision/force/proximity),

  3. 3.

can be programmed easily without expert knowledge of robotics,

  4. 4.

and flexible to arms of different lengths.

Fig. 1 presents the overall cooperative dressing scheme that fulfills the
above goals. The framework contains three parts: an optimal stretch
controller, a posture estimation, and a dressing motion generator.

The optimal stretch controller is designed by analysing dressing behaviours
under different arm postures to facilitate dressing. The optimal stretch
controller takes in the initial human arm posture together with the current
hand position and yields the stretching policy for the interactive robot (see
Sect. IV-B).

Benefiting from the hand holding, the posture tracking is done by the posture
estimation module which takes in the same inputs as the stretch controller and
outputs the estimated current arm posture (see Sect. IV-C). It relies solely
on proprioceptive sensors.

Using the estimated posture, we define a posture-dependent dressing coordinate
and transform the robot movement in the Cartesian coordinate to the dressing
coordinate for learning the dressing motion generator from human
demonstrations. The dressing can be programmed by non-expert users with
kinesthetic teaching. The defined dressing coordinate enables flexible
encoding of the dressing strategy to adapt to different arm lengths. Inputs to
the dressing motion generator are the estimated posture together with the
current dressing robot position. Using the inputs, it generates the next step
dressing policy (see Sect. V).

The scheme described above is realized with a bimanual robot setup illustrated
in Fig. 2. The dressing robot grips the cloth and executes the dressing
policy. The interactive robot is equipped with a SoftHand from qbrobotics that
provides human-like hand-holding. It supports and guides the human in the
dressing process.

Figure 2: The bimanual robot setup for the cooperative dressing. We employ two
Franka Emika robots. The dressing robot grips the cloth and executes the
dressing policy. The interactive robot is equipped with a SoftHand from
qbrobotics that provides human-like hand-holding. It guides the human in the
dressing process.

In addition, we employ the following assumptions:

  * •

The starting arm posture is known,

  * •

The shoulder remains static during dressing.

##  IV Interactive robot – Optimal Stretch

In this section, we motivate a key feature in the human arm posture that
affects the dressing policy. Based on this feature, we design an optimal
stretch controller for the interactive robot. Finally, we present a scheme for
the estimation of human arm posture during dressing.

###  IV-A The feature affects dressing – elbow angle

The dressing gets complicated around the elbow. If we simplify the forearm and
the upper arm as two straight lines, then the discontinuity between these two
lines is the linking point (elbow) which connects both lines. The degree of
discontinuity can be represented by the elbow angle $\psi\leq\pi$. The angle
$\psi$ is a pure geometrical feature that influences the dressing policy.

If we neglect force and only consider geometry, dressing resembles a hotwire
game. The cloth can be simplified into a ring that represents the armhole
(more formally referred to as the armscye in textile literature), and the arm
is the wire that the ring needs to pass through. The goal of dressing is to
transport the ring to a position above the shoulder.

Figure 3: Illustration of diminishing rigidity on the armscye of the cloth and
its influence on the dressing strategy. We consider the arm plane defined by
the hand, elbow and shoulder. The arm plane is divided into two parts: the
inner arm area with an angle less than $\pi$ and the outer area with an angle
more than $\pi$.

However, a major difference that complicates the dressing task is the nature
of the ring. In a hotwire game, the ring is rigid. All points on the ring move
simultaneously with the grasping/control point on the ring. In dressing, due
to the deformation nature of the armscye, we observe the property of
diminishing rigidity [40] in the armscye – i.e. that the effect of gripper
motion along the armscye diminishes as the distance from the gripper
increases. On the bottom left of Fig. 3, we display this effect with color
intensity.

In Fig. 3, we define the arm plane: the plane given by shoulder, elbow, and
hand positions. The plane is divided by the arm into two parts, inner arm area
(angle between the forearm and upper arm less than $\pi$) and outer arm area
(angle between the forearm and upper arm larger than $\pi$). To go past a
corner (in dressing case the elbow), we distinguish two different
strategies222The trivial case of the shoulder, elbow, and hand is in a line:
we cannot define a plane anymore, yet there is also no difference between an
inner and outer arm area.:

  * •

inner strategy: the projection of the path on the arm plane is within the
inner arm area. This strategy offers a shorter path to go around the elbow

  * •

outer strategy: the projection of the path on the arm plane is within the
outer arm area. This strategy offers a longer path to go around the elbow.

When $\psi$ is small, taking the inner strategy, the far end of armscye where
we have very little control needs to travel much longer to go pass the elbow
(See Fig. 3). There is a high possibility it fails and results in the armscye
getting stuck. It is more plausible in this case to take the outer strategy,
although travels longer in distance, it makes sure that the end with the least
control is required to move as little as possible to minimize the chance of
getting stuck. When $\psi$ is larger, the distance difference between an inner
and outer strategy diminishes and becomes zero when the arm is fully
stretched: $\psi=\pi$.

To conclude, elbow angle $\psi$ affects the dressing strategy. When $\psi$ is
large, the degree of discontinuity becomes small (the arm resembles a straight
line), and we are more likely to take the inner strategy to have a smaller
path length. When $\psi$ is small, the degree of discontinuity gets large (the
arm is more bend), and we will more likely to take the outer strategy to avoid
getting stuck. We will show this effect with expert demonstration data on
different static arm postures (with different elbow angles) in Sect. VI-A.

###  IV-B Optimal stretch based on the elbow angle

The outer strategy requires a longer dressing trajectory, and when $\psi$ is
small, there is always the risk of getting stuck even taking the outer
strategy. Thus, the objective of the interactive robot is to guide the human
arm so that $\psi$ increases as much as possible to avoid getting stuck and
facilitate dressing.

The interactive robot is holding the human hand. Besides the singular
configuration where the arm is fully stretched, the robot is able to guide the
hand to move in all directions in 3D which is represented by the blue sphere
in Fig. 4. In the figure, the arm posture is given by the shoulder, elbow, and
hand joint positions: $\bm{P}=\\{\bm{p}_{s},\bm{p}_{e},\bm{p}_{h}\\}$. The
movement direction $\delta\bm{d}$ that maximizes the increase of the elbow
angle, here referred to as $\delta\bm{d}^{*}$, is aligned with the direction
of $\overrightarrow{\bm{p}_{s}\bm{p}_{h}}$. Proving the optimal direction is
straightforward, since $\psi\propto|\bm{d}+\delta\bm{d}|$, finding
$\delta\bm{d}^{*}$ that maximizes $\psi$ is equivalent to:

$\delta\bm{d}^{*}=\max_{\delta\bm{d}}|\bm{p}_{s}\bm{p}_{h}+\delta\bm{d}|,~{}\text{for}~{}\delta\rightarrow
0,~{}\text{and}~{}\bm{d}\in\mathrm{O}(3)$

With the triangle inequality theorem, we have:

$|\bm{p}_{s}\bm{p}_{h}+\delta\bm{d}|\leq|\bm{p}_{s}\bm{p}_{h}|+|\delta\bm{d}|$

The maximum value for $|\bm{p}_{s}\bm{p}_{h}+\delta\bm{d}|$ is when the
equality holds. And it only holds when $\delta\bm{d}$ and
$\bm{p}_{s}\bm{p}_{h}$ are in the same direction.

Therefore, to increase the elbow angle as much as possible, the guidance force
direction from the interactive robot should always be aligned with the
direction that connects the hand and the shoulder.

Figure 4: The optimal direction that maximizes the increase of the elbow angle
$\psi$. The arm posture is given by the shoulder, elbow, and hand joint
positions: $\bm{P}=\\{\bm{p}_{s},\bm{p}_{e},\bm{p}_{h}\\}$. The optimal
direction $\delta\bm{d}^{*}$ is aligned with the direction that connects the
hand and the shoulder and marked with a red solid arrow. The triangle
inequality theorem is demonstrated with another random direction
$\delta\bm{d}$ and the resulting shoulder-to-hand distance is marked with a
black dash line.

Knowing the optimal direction to move, we further implement the optimal
stretch controller with Cartesian impedance control. The end-effector dynamics
of a conventional Cartesian impedance controller is [41]:

$\bm{\Lambda}\ddot{\bm{x}}+\bm{D}\dot{\bm{x}}+\bm{K}(\bm{x}-\bm{x}_{d})=\bm{f}^{\bm{x}}_{\text{ext}},$
(1)

where the $\bm{\Lambda}$ is the Cartesian inertia matrix, $\bm{K}$ is a non-
diagonal stiffness matrix, and $\bm{D}$ is the corresponding critical damping
matrix[42]. $\bm{x}_{d}$ is the desired end-effector position. $\bm{x}$,
$\dot{\bm{x}}$, $\ddot{\bm{x}}$ and $\bm{f}^{\bm{x}}_{\text{ext}}$ are current
end-effector position, velocity, acceleration, and external force in the base
Cartesian coordinate.

Given a rotation matrix $\bm{\Gamma}$ of a local stiffness matrix with respect
to the global coordinate, the equivalent stiffness matrix in global
coordinates is

$\bm{K}_{\text{global}}=\bm{\Gamma}^{T}\bm{K}_{\text{local}}\bm{\Gamma}$ (2)

where in our framework, the rotation matrix $\bm{\Gamma}$ transforms the
$x$-axis of the global coordinates in the direction of end effector movements
that maximizes the increase of the human elbow angle $\psi$, i.e.
$\overrightarrow{\bm{p}_{s}\bm{p}_{h}}$.

Additionally, we choose

$\bm{K}_{\text{local}}=\begin{bmatrix}k_{x}&0&0\\\ 0&0&0\\\
0&0&0\end{bmatrix},$

in order to have a non-zero stiffness only in the direction of movement and
complete compliance in the other two perpendicular directions: such a
controller will provide a guiding force in the optimal direction of hand
movement that maximizes the increase of the elbow angle $\psi$. At the same
time, the human can disagree and move easily in the plane perpendicular to the
force direction.

###  IV-C Human arm posture estimation

With the guidance force upon the hand, the human arm posture is changing
during dressing. For successfully dressing the human arm, we propose a vision-
free estimation of the human arm posture.

Consider two coordinate frames $\mathcal{O}^{\mathcal{D}}$,
$\mathcal{O}^{\mathcal{I}}$ located at the base of each robot respectively.
Any point $\bm{x}^{\mathcal{I}}$ in $\mathcal{O}^{\mathcal{I}}$ can be
transformed into $\mathcal{O}^{\mathcal{D}}$ by:

$\bm{x}^{D}=\bm{R}\bm{x}^{I}+\bm{t}.$ (3)

where $\bm{R}$ is a $3$ by $3$ rotation matrix and $\bm{t}$ is the translation
vector. We can obtain $\bm{R}$ and $\bm{t}$ between two coordinates by
calibration.

The human hand is joined by the interactive robot $\mathcal{I}$. Thus its
position can be regarded as the same as the end-effector position of the
interactive robot $\bm{x}^{I}_{\text{end-eff}}$. Using (3), we can track the
human hand position in $\mathcal{O}^{\mathcal{D}}$ given $\bm{R}$ and
$\bm{t}$:

$\bm{p}_{h}^{\mathcal{D}}=\bm{R}\bm{x}^{I}_{\text{end-eff}}+\bm{t}.$ (4)

Assuming the initial arm posture in $\mathcal{O}^{\mathcal{D}}$ is known and
the shoulder does not move during the dressing, we can move the origin of
$\mathcal{O}^{\mathcal{D}}$ to the shoulder by translation, we denote this
frame as shoulder frame. From now on, for simplicity, unless explicitly
denoted with superscript ${}^{\mathcal{I}},^{\mathcal{D}}$, all positions are
with regard to the shoulder frame with its origin at the shoulder.

Since the shoulder is assumed static during dressing, the posture estimation
problem boils down to recovering the elbow position $\bm{p}_{e}$ given the
hand position $\bm{p}_{h}$. A schematic diagram is shown in Fig. 5.

Figure 5: A schematic diagram showing the inputs and outputs of the human
posture estimation. Figure 6: Schematic representation of human arm from the
shoulder to the hand joint with four joint angles. A spherical joint located
at the shoulder with two degrees of freedom ($\alpha$, $\beta$), and another
spherical joint at the elbow with two degrees of freedom ($\phi$, $\gamma$).

The human arm has seven degrees of freedom. Only four out of seven contribute
to the displacement of the hand $\bm{p}_{h}$ [43]. The four joint angles
$\bm{q}=[\alpha~{}\beta~{}\phi~{}\gamma]^{T}$ are depicted in Fig. 6. A
spherical joint located at the shoulder with two degrees of freedom ($\alpha$,
$\beta$), and another spherical joint at the elbow with two degrees of freedom
($\phi$, $\gamma$). The shoulder, elbow, and hand representation in the
shoulder frame can be transformed into the angle representation and vice versa
(given forearm and upper arm lengths). The hand position can be expressed as:

$\bm{p}_{h}=\bm{f}(\bm{q})$ (5)

The hand movement:

$\bm{p}_{h}+\Delta\bm{p}_{h}=\bm{f}(\bm{q}+\Delta\bm{q})=\bm{f}(\bm{q})+\bm{J}(\bm{q})\Delta\bm{q}+\text{h.o.t.},$
(6)

where

$\bm{J}(\bm{q})=\frac{\partial\bm{f}}{\partial\bm{q}}(\bm{q})\in\mathbb{R}^{3\times
4}$ (7)

This then leads to:

$\Delta\bm{p}_{h}=\bm{J}(\bm{q})\Delta\bm{q}.$ (8)

For a given $\Delta\bm{p}_{h}$, (8) has infinite solutions of $\Delta\bm{q}$.
We aim to find the $\Delta\bm{q}$ from (8) with good accuracy as compared to
the human movement while enabling fast computation to be implemented in real-
time. Thus, we define a quadratic optimization with an analytical solution:

$\displaystyle\min_{\Delta\bm{q}}\bm{L}=\Delta\bm{q}^{T}\bm{Q}\Delta\bm{q}$
(9) $\displaystyle\text{s.t. }\Delta\bm{p}_{h}=\bm{J}(\bm{q})\Delta\bm{q},$

where $\bm{Q}$ is a diagonal weighting matrix with non-zero diagonal elements
for solving the optimal $\Delta\bm{q}$. Physically, $\bm{Q}$ indicates how the
hand velocity is distributed on each joint, a larger element on the diagonal
indicates the corresponding joint is less likely to move during arm movements.
We find $\bm{Q}$ using the arm movement data recorded during the stretch and
the detail is presented in Sect. VI-B.

The solution to the optimization problem (9) is (detailed derivation in the
Appendix):

$\Delta\bm{q}^{*}=\bm{J}^{+}\Delta\bm{p}_{h}-({\bm{\mu}^{T}\bm{Q}\Delta\bm{q}_{h}}/{\bm{\mu}^{T}\bm{Q}\bm{\mu}})\bm{\mu},$

where $\bm{J}^{+}$ is Moore–Penrose inverse of the $\bm{J}$ and $\bm{\mu}$ is
the null space vector of $\bm{J}$333Detail solution can be found in the
appendix. Current postures in angle representation are then:

$\hat{\bm{q}}_{T}=\hat{\bm{q}}_{T-1}+\Delta\bm{q}^{*}.$ (10)

Note that this is a recursive estimation as depicted in Fig. 5. Only the
initial value of $\bm{q}$ is known such that $\hat{\bm{q}}_{0}=\bm{q}_{0}$.
The analytical solution can be computed fast online which allows us to use the
estimation scheme in real-time during dressing.

Lastly, the angle representation $\hat{\bm{q}}_{T}$ is transformed into
current positions of the shoulder, elbow, and hand
$\\{\hat{\bm{p}}_{s},\hat{\bm{p}}_{e},\hat{\bm{p}}_{h}\\}_{T}$ for deriving
the dressing policy. We evaluate the accuracy of the estimation scheme in
Sect. VI-B.

##  V Dressing Robot – Dressing Coordinate-based Learning

The dressing is highly dependent on the human arm posture. Previous research
explored such dependency in learning the dressing policy. In [44], the authors
placed in simulation haptic sensor on each segment of the arm as observation
in deep reinforcement learning. Others define task parameters as local
coordinates at the shoulder, elbow, and hand and then encode the dressing
policy with TP-GMM [7, 6, 8]. Similarly, we explore such dependencies and
define an arm-posture-dependent coordinate system for encoding the dressing
policy with LfD. In the following subsections, we first present the dressing
coordinate then the LfD in the new coordinate system.

###  V-A Dressing coordinate

The dressing motion follows the arm from the hand to the elbow and finally
reaches the shoulder. The monotonic motion naturally leads us to introduce a
scalar quantity to model the dressing progress. We denote the progress scalar
as $s$.

The dressing motion is additionally constrained by the size of the armscye.
Thus we use a polar coordinate around the forearm and the upper-arm for
encoding the constraint: $l$, the distance to the arm, and $\theta$, the angle
around the arm. The formulation yields two cylinder coordinates on the forearm
and the upper-arm. However, the complexity arises from the elbow where we
require a smooth transition from the forearm coordinate to the upper arm
coordinate (see Fig. 7a).

(a)

(b)

Figure 7: The dressing coordinate is a cylinder coordinate defined around the
arm, with a smoothing transitional corner around the elbow. The sub-figure (a)
presents the full definition of the coordinate and (b) shows the progress
curve in three segments.

To enable the smooth transition, we use a part of a horn torus (A torus where
the distance from the center of the tube to the center of the torus equals the
radius of the tube) for connecting the two cylinder coordinate. As a result of
the torus define around the elbow, the progress curve is divided into three
pieces: a forearm segment, an elbow arc for a smooth transition to the upper
arm, and an upper-arm segment as shown in Fig. 7b.

To convert a point on the dressing path $\bm{x}\in\mathrm{R}^{3}$ in Cartesian
coordinate into the dressing coordinate $[s,l,\theta]$, we first project
$\bm{x}$ onto the arm plane given by the shoulder $\bm{p}_{s}$, elbow
$\bm{p}_{e}$ and hand positions $\bm{p}_{h}$:

$\bm{x}_{\text{arm}}=\bm{x}+\frac{\bm{v}(\bm{p}_{h}\cdot\bm{v}-\bm{x}\cdot\bm{v})}{|\bm{v}|^{2}},$
(11)

where $\bm{v}$ is the vector perpendicular to the arm plane.

We further find the progress scalar of $\bm{x}_{\text{arm}}$ on the progress
curve. The progress curve is divided into $3$ segments by $2$ transitional
points: $\bm{p}_{he}$ (transition from the forearm to elbow) and $\bm{p}_{es}$
(transition from elbow to upper arm) as shown by stars on Fig. 7b. The
distance from $\bm{p}_{h}$ to $\bm{p}_{he}$ is denoted $|d_{1}|$, and from
$\bm{p}_{s}$ to $\bm{p}_{es}$ is $|d_{2}|$.

We need to determine which segments $\bm{x}_{\text{arm}}$ belongs to. To do
that, we compute the distance $\bm{x}_{\text{arm}}$ project onto the line
defined by the forearm and upper-arm segments respectively:

$\displaystyle\begin{split}d_{\text{forearm}}&=\frac{(\bm{x}_{\text{arm}}-\bm{p}_{h})\cdot(\bm{p}_{e}-\bm{p}_{h})}{|(\bm{p}_{e}-\bm{p}_{h})|}\\\
d_{\text{upperarm}}&=\frac{(\bm{x}_{\text{arm}}-\bm{p}_{s})\cdot(\bm{p}_{e}-\bm{p}_{s})}{|(\bm{p}_{e}-\bm{p}_{s})|}\end{split}$
(12)

The value of $d_{\text{forearm}}$ and $d_{\text{upperarm}}$ is then compared
against $d_{1}$ and $d_{2}$ so that:

$\bm{x}_{\text{arm}}\text{ on the }\begin{cases}\text{forearm}&\text{if
}d_{\text{forearm}}<|d_{1}|\text{ and }d_{\text{upperarm}}>|d_{2}|\\\
\text{upperarm}&\text{if }d_{\text{upperarm}}<|d_{2}|\text{ and
}d_{\text{forearm}}>|d_{1}|\\\ \text{elbow}&\text{otherwise}\end{cases}$

Depending on which segment $\bm{x}_{\text{arm}}$ belongs to, we find the
corresponding $\bm{x}_{\text{curve}}$ (pink triangles on Fig. 8) on the
progress curve for $\bm{x}_{\text{arm}}$. For the forearm and upper arm
segment, calculating $\bm{x}_{\text{curve}}$ is straightforward. For the elbow
segment, we find the line connecting the center of the elbow arc to
$\bm{x}_{\text{arm}}$. The intersection point between the line and the
progress curve is $\bm{x}_{\text{curve}}$ (See Fig. 8).

We then define $s=0$ for $\bm{x}_{\text{curve}}=\bm{p}_{h}$, $s=1$ for
$\bm{x}_{\text{curve}}=\bm{p}_{s}$, for any $\bm{x}_{\text{curve}}$, we have:

$s=\begin{cases}\vspace{0.8mm}\frac{d_{\text{forearm}}}{|d_{\text{curve}}|}&\text{if
$\bm{x}_{\text{curve}}$ is on the forearm}\\\
\frac{|d_{\text{curve}}|-d_{\text{upperarm}}}{|d_{\text{curve}}|}&\text{if
$\bm{x}_{\text{curve}}$ is on the upperarm}\\\
\vspace{0.8mm}\frac{d_{1}+\omega r}{|d_{\text{curve}}|}&\text{if
$\bm{x}_{\text{curve}}$ is on the upperarm}\end{cases}$

where $|d_{\text{curve}}|$ is the full length of the progress curve from
$\bm{p}_{h}$ to $\bm{p}_{s}$, $\omega$ is the angle of $\bm{x}_{\text{curve}}$
on the elbow arc and $r$ is the radius of the elbow segment on the progress
curve.

Figure 8: Graphic illustration of finding the corresponding point on the
progress curve $\bm{x}_{\text{curve}}$ given a random point $\bm{x}$ on the
dressing path in Cartesian coordinate. The point $\bm{x}$ is first projected
onto the arm plane to get $\bm{x}_{\text{arm}}$. Then we determine which
segment $\bm{x}_{\text{arm}}$ belongs to on the progress curve. Finally we
find the corresponding $\bm{x}_{\text{curve}}$.

Once we found the projection of the dressing position on the progress curve
$\bm{x}_{\text{curve}}$ and the progress scalar $s$, we can calculate the
distance to the arm $l$ and the angle around the arm $\theta$ as follows.

The length $l$ of $\bm{x}$ is:

$l=|\bm{x}-\bm{x}_{\text{curve}}|$ (13)

We define $\bm{v}$ as the reference $0$° for the angle $\theta$ computation.
The $\theta$ is defined as

$\theta=\angle(\bm{x}_{\text{curve}}-\bm{x},\bm{v})$ (14)

Note that $\bm{v}$ has two directions, one pointing towards to the human body
and the other pointing outwards. We select the former to avoid flipping angles
from $0$ to $2\pi$ during the dressing motion.

By converting the Cartesian coordinate to the dressing coordinate and
obtaining the motion generation model in the dressing coordinate, we can:

  * •

limit the policy space to be around the arm by imposing the maximum value for
the length $l$,

  * •

ensure the motion convergence to the vicinity of the shoulder by progress
scalar $s$,

  * •

transfer the trained motion generation model on one arm to another as the data
is relative to the arm posture.

Note one prerequisite for the last statement is that the ratio between the
forearm and upper arm is similar among people.

###  V-B LfD in dressing coordinate

For LfD in dressing coordinate, we employ Gaussian Mixture Models (GMM) and
Gaussian Mixture Regression (GMR) for learning and generating the dressing
policy [45]. The GMM/GMR-based imitation learning offers fast movement
retrieval and versatile inputs/outputs arrangement which is desirable for our
task. Please note that the LfD algorithm is not a part of our contribution and
it is possible to use different methods/formulations for learning the policy
in the dressing coordinate.

We first brief describe GMM/GMR-based LfD then show how to apply it to
learning the dressing policy.

GMM models the joint distribution of the demonstration data. We denote the
demonstrated data $\bm{\xi}$:

$\bm{\xi}=\begin{bmatrix}\bm{\xi}^{\mathcal{I}},\bm{\xi}^{\mathcal{O}}\end{bmatrix}^{T}.$
(15)

with $\bm{\xi}^{\mathcal{I}}$ is the input and $\bm{\xi}^{\mathcal{O}}$ the
output. The probability density function $p(\bm{\xi}_{i})$ is estimated with
$K$ Gaussian distributions:

$p(\bm{\xi}_{i})=\sum_{k=1}^{K}\pi_{k}\mathcal{N}(\bm{\xi}_{i}\big{|}\bm{\mu}_{k},\Sigma_{k})$

where $\bm{\mu}_{k}$ and $\Sigma_{k}$ are mean and variance of the $k$th
Gaussian. To determine an optimal number of Gaussian $K$ for GMM, we make use
of the Bayesian Information Criterion (BIC) [46] for balancing the model
complexity and representation quality.

Once $K$ is selected, we can initialize the GMM with K-Means clustering and
then employ the Expectation-Maximization algorithm to iterative compute the
model parameters
$\big{\\{}\pi_{k},\bm{\mu}_{k},\bm{\Sigma}_{k}\big{\\}}^{K}_{k=1}$.

The resulting $K$ Gaussian parameters can be decomposed as:

$\bm{\mu}_{k}=\begin{bmatrix}\bm{\mu}_{k}^{\mathcal{I}}\\\
\bm{\mu}_{k}^{\mathcal{O}}\end{bmatrix},\quad\bm{\Sigma}_{k}=\begin{bmatrix}\bm{\Sigma}_{k}^{\mathcal{I}}&\hat{\bm{\Sigma}}_{k}^{\mathcal{I}\mathcal{O}}\\\
\bm{\Sigma}_{k}^{\mathcal{O}\mathcal{I}}&\hat{\bm{\Sigma}}_{k}^{\mathcal{O}}\end{bmatrix}.$
(16)

Given input data the output is retrieved using GMR by conditioning on the
input data [47].

For encoding dressing with GMM/GMR in the dressing coordinate, we demonstrate
the dressing task with static arm postures, the arm posture, and the dressing
path is recorded for converting the dressing path into the dressing
coordinate. We use the progress scalar $s$ and the elbow angle $\phi$ as
inputs. The outputs are the differences in the distance to the arm $\delta l$,
and the angle difference around the arm $\delta\theta$. The differences are
computed with regard to the starting position in the demonstration. For all
demonstrations, we start from above the hand of the human.

##  VI Experiments

We conduct $3$ different experiments to investigate various aspects of the
proposed framework.

In Sect. VI-A, we investigate the effect of the elbow angle on the dressing
strategy using expert demonstration data of static arm postures with different
elbow angles. Then in Sect. VI-B we evaluate the arm posture estimation scheme
to show that it is accurate enough for the dressing task.

Lastly, in Sect. VI-C we showcase our interactive dressing framework with real
robot experiments. To demonstrate the flexibility of the defined dressing
coordinate, we train the policy from demonstrations on a mannequin and apply
the policy to a human arm. We conduct an ablation study where we disable the
stretch movement and dress with a static human arm. Furthermore, we test the
framework under cases where the human is not fully compliant with the
interactive robot and show that the overall framework still works well. In
these experiments, the robot dresses the human in short and long sleeves
shirts with different materials. Note that the automatic dressing of long
sleeves shirts are unprecedented in previous research.

###  VI-A Effect of the elbow angle on dressing strategies

In Sect. IV-A we discussed the effect of the elbow angle on the dressing
strategy with diminished rigidity. In this section, we aim to show this effect
with expert demonstration data.

We collect demonstration data through kinesthetic teaching with a Franka robot
under different arm postures. The dressing paths and arm postures are recorded
in the process. To validate whether the elbow angle affects the dressing
strategies, we project the path onto the arm plane. In Fig. 9 we present $6$
postures with different elbow angles and the corresponding dressing path on
the arm plane.

Figure 9: Static arm postures with different elbow angles and the
corresponding expert demonstrations of dressing with these postures.

The projection of the arm postures (in black solid line) and the corresponding
demonstrated dressing path (colored dash line) on the arm plane are shown in
Fig. 10. The red dot is a transition point corresponding to the middle of the
elbow arc in Fig. 7b on the progress curve.

(a)

(b)

(c)

(d)

(e)

(f)

Figure 10: The projection of the arm postures and the corresponding
demonstrated dressing path on the arm plane. The red dot is a transition point
corresponding to the middle of the elbow arc on the progress curve.

In addition, we provide a quantitative result summarized in Tab. I. We
calculate the distance between the transition point and the elbow. When the
transition point is on the inner arm plane, the distance is positive which
indicates that the dressing employs an inner strategy. When the transition
point is on the outer arm plane, we set the distance negative which indicates
that the dressing employs an outer strategy (see Sect. IV-B and Fig. 3 for the
definition of inner/outer strategy).

TABLE I: Elbow angle and corresponding distance between the transition point and the elbow on the arm plane. elbow angle (°) | 80.47 | 81.65 | 87.49 | 106.07 | 143.46 | 149.30
---|---|---|---|---|---|---
distance (m) | -0.099 | -0.11 | -0.10 | -0.07 | 0.107 | 0.105

We can observe both from Fig. 10 and Tab. I that the outer dressing strategy
is employed when the elbow angle is small, which justifies our analysis in
Sect. IV-A.

###  VI-B Evaluation of the posture estimation scheme

As the dressing path is highly dependent on the arm posture, posture
estimation is crucial to the success of dressing. In this section, we provide
a quantitative evaluation of the posture estimation scheme. We consider

  1. 1.

the shoulder is static during dressing which is a common assumption in
assistive dressing literature [10, 6],

  2. 2.

the initial arm posture (in the dressing robot frame $\mathcal{O}^{D}$) is
known.

With the above assumptions and real-time hand position recovery from the
interactive robot (see. Sect. IV-C), the only estimated quantity in the full
posture description $\bm{P}=\\{\bm{p}_{s},\bm{p}_{e},\bm{p}_{h}\\}$ is the
elbow position $\bm{p}_{e}$.

Thus in the evaluation, for a series of arm postures during movements
$\\{\bm{P}_{0},\bm{P}_{1},\cdots,\bm{P}_{T}\\}$, we use the maximum error of
the elbow as the performance indicator of our posture estimation scheme:

$\text{error}_{\max}=\max_{i=1}^{T}{|\bm{p}_{e}(i)-\hat{\bm{p}}_{e}(i)|}$

where $\hat{\bm{p}}_{e}(i)$ is the estimated elbow position at the
$i^{\text{th}}$ instance.

One tuneable parameter that affects the estimation performance is the diagonal
weight matrix $D$ in (9). We compute $D$ using the movement data
$\\{\bm{P}_{0},\bm{P}_{1},\cdots,\bm{P}_{T}\\}$. The data is first transformed
to angle representation $\\{\bm{q}_{0},\bm{q}_{1},\cdots,\bm{q}_{T}\\}$ (refer
to Sect. IV-C and Fig. 5 for the definition of angle representation and $4$
angles $\bm{q}=[\alpha~{}\beta~{}\phi~{}\gamma]^{T}$ associated). We then
compute the difference between adjacent angles and yield:

$\\{\delta\bm{q}_{1},\delta\bm{q}_{2},\cdots,\delta\bm{q}_{T}\\},$ (17)

where

$\delta\bm{q}_{i}=[\delta\alpha_{i}~{}\delta\beta_{i}~{}\delta\phi_{i}~{}\delta\gamma_{i}]^{T}=\bm{q}_{i}-\bm{q}_{i-1}.$
(18)

The weighting diagonal matrix $\bm{Q}$ in (9) is calculated as:

$\bm{Q}=\text{diag}\\{\frac{1}{\sum_{i=1}^{T}\delta\alpha_{i}^{2}},\frac{1}{\sum_{i=1}^{T}\delta\beta^{2}},\frac{1}{\sum_{i=1}^{T}\delta\phi_{i}^{2}},\frac{1}{\sum_{i=1}^{T}\delta\gamma_{i}^{2}}\\}$

For the evaluation, we conduct experiments to collect arm movement data with
an XSens motion capture suit. During the data collection phase, one person is
asked to stretch the arm of the other. The posture data was collected using
the suit and regarded as ground truth. We then compared the estimated elbow
position with the ground truth. We collected $4$ movement patterns from
different starting poses. The maximum error during the stretch motion is
presented in Tab. II.

TABLE II: The maximum error in elbow estimation during the stretch. case number | 1 | 2 | 3 | 4
---|---|---|---|---
$\text{error}_{\max}$ (m) | 0.028 | 0.026 | 0.032 | 0.028

Figure 11 shows the ground truth and the estimated elbow position ($x,y,z$) in
solid and dash line respectively in all $4$ cases.

(a) case 1

(b) case 2

(c) case 3

(d) case 4

Figure 11: The ground truth and the estimated elbow position ($x,y,z$) in
solid and dash line respectively in test cases.

We observe that generally, the deviation from ground truth is larger as the
time step increases, this is probably due to the recursive nature of the
estimation scheme. The error may accumulate with time. Despite this, the
maximum error is around $3$ centimeters which is much smaller than the size of
the armscye. It is compatible with the results from [11] which proposed a
vision-based estimation of the elbow position during dressing under occlusion.

###  VI-C Interactive dressing

In this section, we show the experimental results of our interactive dressing
scheme. We present experiments with three types of clothes: a short and rigid
sleeve shirt, a long and rigid sleeve shirt, and a long and soft sleeve shirt.

To show the flexibility of our framework, our demonstrations were collected on
the mannequin and the learned dressing policy was executed on the human arm.
In demonstrations, we always start around above the hand and with the same
orientation444The orientation can be seen in the start configuration in
dressing in Fig. 14. The arm length of both the human and the mannequin is
listed in Tab. III, although having different lengths, the ratio between hand-
elbow and elbow-shoulder is similar. Since we encode the data in the dressing
coordinate which is independent of the arm lengths, thus enables direct
transfer of the dressing policy.

TABLE III: Arm lengths of the human and the mannequin | hand - elbow (cm) | elbow - shoulder (cm) | ratio
---|---|---|---
mannequin | 26 | 25 | 1.04
human | 30 | 29 | 1.03

Fig. 12 shows the demonstration and transformed data in the dressing
coordinate. For the angle around the arm $\theta$ and the distance to the arm
$l$, we present the changes during dressing $\delta\theta$ and $\delta l$
instead of the absolute values. The difference is computed with regard to the
starting position which is around above the hand. For postures with smaller
elbow angles, we can observe a change in the angles around the arm $\theta$
since it needs to move to the back of the arm to execute an outer strategy.

(a) Expert demonstrations on different postures

(b) Demonstrations transformed into the dressing coordinates

Figure 12: The expert demonstrations on different postures used for training
dressing policy and the transformed data in the dressing coordinate.

From Fig. 12b we observe that the progress scalar $s$ is a monotonic
increasing quantity. Consider a target $s_{\text{target}}$, we model $s$ with
a dynamical system with a constant increment $c>0$:

$s_{T+1}=s_{T}+\min(c,~{}(s_{\text{target}}-s_{T}))$ (19)

We respectively trained two GMM for the changes in angles around the arm
$\delta\theta$ and changes in the distance to the arm $\delta l$ with inputs
progress scalar $s$ and the elbow angle $\phi$555Please note that other
formulations may be also possible, such as to learn the joint distribution of
$[s,~{}\phi,~{}l,~{}\theta]$. We use $8$ Gaussians for both $\delta l$ and
$\delta\theta$. The resulting mixture models are presented in 3D surf plots in
Fig. 13. We can observe a smaller elbow angle displays larger changes in
$\theta$ which is a clear indication for taking the outer strategy. For a
large elbow angle, the $\theta$ change is almost zero which resembles an inner
strategy. Due to the diminishing rigidity, the end effector rotation has a
very limited effect during dressing. In the experiment, we consider only the
$z$ axis orientation of the end-effector of the dressing robot. The $z$ axis
rotation is applied to avoid collision with the arm during dressing. We can
use a constant incremental rotation with regard to $s$ or a separate GMM with
the same inputs ($[s,~{}\phi]$) for encoding the difference $dz$ in $z$ axis
rotation (The training procedure is the same as $\delta\theta$ and $\delta
l$).

(a) The 3D surface plot of GMM for encoding $\delta l$

(b) The 3D surface plot of GMM for encoding $\delta\theta$

Figure 13: The surface plot for GMM with elbow angles $\psi$ and progress
scalar $s$ as inputs and outputs changes in the distance to the arm $\delta l$
and changes in angles around the arm $\delta\theta$ respectively.

We implement the policy with our optimal stretch and also posture estimation
scheme. Three types of experiments are conducted:

  1. 1.

Human passively follows the interactive robot

  2. 2.

Human is not fully compliant with the guiding force and moves in other
directions

  3. 3.

Ablation study: human arm remains static

We tested in different starting postures and different clothes as shown in
Fig. 14. In the figure, we present only starting and end configurations, the
full dressing sequences are recorded in the video accompanying the paper. We
tested the dressing while the human passively follows the guiding force and
also moves in other directions (not fully compliant). When the human is not
fully compliant, in the figure, we also display with high transparency a
nominal end configuration when the human is compliant. The deviation from the
nominal end configuration is indicated by a blue arrow. With these results, we
prove that humans can move in other directions if desired and the framework is
robust against non-compliant behavior and still succeed in dressing.

In the ablation study, we demonstrate cases where the elbow angle is small,
and the cloth tends to get stuck around the elbow, thus reiterating the
importance of having a bimanual setup for dressing. However, the result also
suggests that if the human starts in these postures and refuses to move along
with the guiding force, the dressing could fail.

Figure 14: Starting and/or end configurations for $3$ different experiments:
human passive/compliant, human not fully compliant, and the ablation study
where the human remains static. For experiments where the human is not fully
compliant, we also display with high transparency a nominal end configuration
when the human is compliant. The deviation from the nominal end configuration
is indicated by a blue arrow.

##  VII Conclusion

Traditional assistive dressing considers a one-robot-to-one-arm setup, which

  * •

either assumes static arm posture or requires additional sensors and
algorithms for tracking the arm posture,

  * •

renders the human arm hanging in the air during the dressing process which is
often tiring,

  * •

may fail/get stuck at the elbow for some initial arm postures.

In the light of the following limitations, and inspired by strategies taken by
caregiving experts, we designed a bimanual assistive dressing framework with a
two-robot-to-one-arm setup that

  * •

allows tracking of arm postures without additional sensors,

  * •

provides support and guiding force for the arm being dressed,

  * •

stretches the human arm while compliant to obtain postures for the ease of
dressing.

We approached the design by first analysing the effect of elbow angle in
dressing. Based on this effect, we proposed an optimal stretch controller for
the interactive robot. As the interactive robot holds the human hand, the arm
posture can be estimated in real-time with inverse kinematics. For the
dressing robot, we utilize the dependency of the dressing policy on the arm
posture and introduced a dressing coordinate defined by the arm posture for
easy encoding of the dressing policy from demonstrations.

The adoption of the dressing coordinate greatly enhances the flexibility of
the LfD. The trained policy can adapt to different arm lengths as long as the
forearm to upper-arm ratio is similar. The framework allows dressing different
types of clothes, even long sleeves which is unprecedented in previous
research. The dressing remains robust even though human is not fully compliant
with the guidance from the interactive robot.

Since this is the first time such a bimanual setup was employed in dressing a
human, there are two major limitations of the proposed framework:

  * •

The assumption of the shoulder remains static is not always true, since the
interactive robot is stretching the arm, the shoulder is more likely to move
forward. A forward-moving shoulder will subsequently increase the length of
the arm (as now the hand can reach further when fully stretched). The
increased length may render the arm posture estimation scheme unsolvable thus
the overall scheme will fail.

  * •

The elbow angle is identified as the crucial parameter that affects dressing,
however, there is no explicit coordination between the dressing and
interactive robot to make sure that when the dressing robot is going through
the elbow, the interactive robot already makes the elbow angle sufficiently
large to avoid getting stuck.

It is difficult to solve the first limitation purely from the algorithm side,
one possibility is to combine inverse kinematics with vision-based tracking
for more accurate posture estimation. The second limitation is even trickier,
to make the coordination explicit, a thorough analysis of dressing on
different elbow angles needs to be conducted to determine a boundary condition
of the elbow angle, then additionally, an assumption has to be placed on the
human side (passive for instance) to make sure that the interactive robot will
always succeed in bringing the human arm into favorable postures before
dressing arm reach the elbow.

Another possible direction for future work is safety analysis. We did not
explicitly analyze the safety of the overall system. However, for assistive
robots, it is crucial to ensure human safety during robotic assistance.
Inspirations for conducting system-level safety analysis can be drawn from
[48].

Despite the above-mentioned limitations, our proposal is the first that
considers interactive dressing assistance with a bimanual setup. The setup is
inspired by caregiving experts conducting the task, which makes the scheme
receptive to humans. It represents a paradigm shift in thinking of the
dressing task from a one-robot-to-one-arm setup to a two-robot-to-one-arm
setup.

## Acknowledgments

This work was supported by Honda Research Institute Europe GmbH as part of the
project: Learning Physical Human-Robot Cooperation Tasks.

## Appendix

In the Appendix, we provide the analytical solution to (9) in Sect. IV-C:

$\displaystyle\min_{\Delta\bm{q}}\bm{L}=\Delta\bm{q}^{T}\bm{Q}\Delta\bm{q}$
$\displaystyle\text{s.t. }\Delta\bm{p}_{h}=\bm{J}\Delta\bm{q}$

Since $\bm{J}\in\mathbb{R}^{3\times 4}$, $\Delta\bm{p}_{h}=\bm{J}\Delta\bm{q}$
has infinite number of solutions:

$\Delta\bm{q}=\bm{J}^{+}\Delta\bm{p}_{h}+\lambda\bm{\mu}=\Delta\bm{q}_{h}+\lambda\bm{\mu}$
(20)

where $\bm{J}^{+}$ is the Moore–Penrose inverse of the $\bm{J}$ and $\bm{\mu}$
is the vector in the null space of $\bm{J}$ (e.g., $\bm{J}\bm{\mu}=\bm{0}$),
and $\lambda\in\mathbb{R}$ is a gain. Taking (21) into $\bm{L}$:

$\displaystyle\bm{L}$
$\displaystyle=(\Delta\bm{q}_{h}+\lambda\bm{\mu})^{T}\bm{Q}(\Delta\bm{q}_{h}+\lambda\bm{\mu})$
(21)
$\displaystyle=(\bm{\mu}^{T}\bm{Q}\bm{\mu})\lambda^{2}+2\bm{\mu}^{T}\bm{Q}\Delta\bm{q}_{h}\lambda+\Delta\bm{q}_{h}^{T}\bm{Q}\Delta\bm{q}_{h}$

which is quadratic in $\lambda$.

The original optimization problem can be transformed into:

$\min_{\lambda}\bm{L}=(\bm{\mu}^{T}\bm{Q}\bm{\mu})\lambda^{2}+2\bm{\mu}^{T}\bm{Q}\Delta\bm{q}_{h}\lambda+\Delta\bm{q}_{h}^{T}\bm{Q}\Delta\bm{q}_{h}\\\
$ (22)

Let us denote $\bm{\mu}^{T}\bm{Q}\bm{\mu}=a$,
$\bm{\mu}^{T}\bm{Q}\Delta\bm{q}_{h}=b$ and
$\Delta\bm{q}_{h}^{T}\bm{Q}\Delta\bm{q}_{h}=c$:

$\displaystyle\bm{L}$ $\displaystyle=a\lambda^{2}+2b\lambda+c$ (23)
$\displaystyle=a(\lambda^{2}+2\frac{b}{a}\lambda+c)$
$\displaystyle=a((\lambda+\frac{b}{a})^{2}+c-\frac{b^{2}}{a^{2}})$

The solution that minimize $\bm{L}$ is:

$\displaystyle\lambda$
$\displaystyle=-\frac{b}{a}=-{\bm{\mu}^{T}\bm{Q}\Delta\bm{q}_{h}}/{\bm{\mu}^{T}\bm{Q}\bm{\mu}}$
(24)

Taking (24) into (21) we can obtain the optimal solution:

$\Delta\bm{q}^{*}=\bm{J}^{+}\Delta\bm{p}_{h}-({\bm{\mu}^{T}\bm{Q}\Delta\bm{q}_{h}}/{\bm{\mu}^{T}\bm{Q}\bm{\mu}})\bm{\mu}$
(25)

The first term ensures that the constraint is respected; on the other hand,
the second term generates a null-space transition that minimizes the total
weighted joint displacement.

## References

  * [1] “Ageing and health,” Oct 2021. [Online]. Available: https://www.who.int/news-room/fact-sheets/detail/ageing-and-health
  * [2] B. J. Dudgeon, J. M. Hoffman, M. A. Ciol, A. Shumway-Cook, K. M. Yorkston, and L. Chan, “Managing activity difficulties at home: a survey of medicare beneficiaries,” _Archives of physical medicine and rehabilitation_ , vol. 89, no. 7, pp. 1256–1261, 2008.
  * [3] D. J. Agravante, A. Cherubini, A. Sherikov, P.-B. Wieber, and A. Kheddar, “Human-humanoid collaborative carrying,” _IEEE Transactions on Robotics_ , vol. 35, no. 4, pp. 833–846, 2019.
  * [4] S. Tarbouriech, B. Navarro, P. Fraisse, A. Crosnier, A. Cherubini, and D. Sallé, “Admittance control for collaborative dual-arm manipulation,” in _2019 IEEE Int. Conf. on Advanced Robotics_. IEEE, 2019.
  * [5] L. van der Spaa, M. Gienger, T. Bates, and J. Kober, “Predicting and optimizing ergonomics in physical human-robot cooperation tasks,” in _2020 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2020, pp. 1799–1805.
  * [6] E. Pignat and S. Calinon, “Learning adaptive dressing assistance from human demonstration,” _Robotics and Autonomous Systems_ , vol. 93, pp. 61–75, 2017\.
  * [7] J. Hoyos, F. Prieto, G. Alenyà, and C. Torras, “Incremental learning of skills in a task-parameterized Gaussian mixture model,” _Journal of Intelligent & Robotic Systems_, vol. 82, no. 1, pp. 81–99, 2016.
  * [8] J. Zhu, M. Gienger, and J. Kober, “Learning task-parameterized skills from few demonstrations,” _IEEE Robotics and Automation Letters_ , vol. 7, no. 2, pp. 4063–4070, 2022.
  * [9] Y. Gao, H. J. Chang, and Y. Demiris, “User modelling for personalised dressing assistance by humanoid robots,” in _2015 IEEE/RSJ Int. Conf. on Intelligent Robots and Systems_. IEEE, 2015, pp. 1840–1845.
  * [10] F. Zhang, A. Cully, and Y. Demiris, “Probabilistic real-time user posture tracking for personalized robot-assisted dressing,” _IEEE Transactions on Robotics_ , vol. 35, no. 4, pp. 873–888, 2019.
  * [11] G. Chance, A. Jevtić, P. Caleb-Solly, G. Alenya, C. Torras, and S. Dogramadzi, ““elbows out”—predictive tracking of partially occluded pose for robot-assisted dressing,” _IEEE Robotics and Automation Letters_ , vol. 3, no. 4, pp. 3598–3605, 2018.
  * [12] Z. Erickson, M. Collier, A. Kapusta, and C. C. Kemp, “Tracking human pose during robot-assisted dressing using single-axis capacitive proximity sensing,” _IEEE Robotics and Automation Letters_ , vol. 3, no. 3, pp. 2245–2252, 2018.
  * [13] Y. Wang, D. Held, and Z. Erickson, “Visual haptic reasoning: Estimating contact forces by observing deformable object interactions,” _arXiv preprint arXiv:2208.05632_ , 2022.
  * [14] T. Tamei, T. Matsubara, A. Rai, and T. Shibata, “Reinforcement learning of clothing assistance with a dual-arm robot,” in _2011 11th IEEE-RAS Int. Conf. on Humanoid Robots_. IEEE, 2011, pp. 733–738.
  * [15] K. Yamazaki, R. Oya, K. Nagahama, K. Okada, and M. Inaba, “Bottom dressing by a dual-arm robot using a clothing state estimation based on dynamic shape changes,” _International Journal of Advanced Robotic Systems_ , vol. 13, no. 1, p. 5, 2016.
  * [16] G. Canal, E. Pignat, G. Alenyà, S. Calinon, and C. Torras, “Joining high-level symbolic planning with low-level motion primitives in adaptive hri: Application to dressing assistance,” in _2018 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2018, pp. 3273–3278.
  * [17] A. Jevtić, A. F. Valle, G. Alenyà, G. Chance, P. Caleb-Solly, S. Dogramadzi, and C. Torras, “Personalized robot assistant for support in dressing,” _IEEE transactions on cognitive and developmental systems_ , vol. 11, no. 3, pp. 363–374, 2018.
  * [18] A. Kapusta, Z. Erickson, H. M. Clever, W. Yu, C. K. Liu, G. Turk, and C. C. Kemp, “Personalized collaborative plans for robot-assisted dressing via optimization and simulation,” _Autonomous Robots_ , vol. 43, no. 8, pp. 2183–2207, 2019.
  * [19] A. Clegg, Z. Erickson, P. Grady, G. Turk, C. C. Kemp, and C. K. Liu, “Learning to collaborate from simulation for robot-assisted dressing,” _IEEE Robotics and Automation Letters_ , vol. 5, no. 2, pp. 2746–2753, 2020.
  * [20] W. Yu, A. Kapusta, J. Tan, C. C. Kemp, G. Turk, and C. K. Liu, “Haptic simulation for robot-assisted dressing,” in _2017 IEEE international conference on robotics and automation (ICRA)_. IEEE, 2017, pp. 6044–6051.
  * [21] A. Kapusta, W. Yu, T. Bhattacharjee, C. K. Liu, G. Turk, and C. C. Kemp, “Data-driven haptic perception for robot-assisted dressing,” in _2016 25th IEEE international symposium on robot and human interactive communication (RO-MAN)_. IEEE, 2016, pp. 451–458.
  * [22] Z. Erickson, A. Clegg, W. Yu, G. Turk, C. K. Liu, and C. C. Kemp, “What does the person feel? learning to infer applied forces during robot-assisted dressing,” in _2017 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2017, pp. 6058–6065.
  * [23] L. Twardon and H. Ritter, “Active boundary component models for robotic dressing assistance,” in _2016 IEEE/RSJ Int. Conf. on Intelligent Robots and Systems (IROS)_. IEEE, 2016, pp. 2811–2818.
  * [24] S. Li, N. Figueroa, A. J. Shah, and J. A. Shah, “Provably safe and efficient motion planning with uncertain human dynamics.” in _Robotics: Science and Systems_ , 2021.
  * [25] S. Li, T. Stouraitis, M. Gienger, S. Vijayakumar, and J. A. Shah, “Set-based state estimation with probabilistic consistency guarantee under epistemic uncertainty,” _IEEE Robotics and Automation Letters_ , vol. 7, no. 3, pp. 5958–5965, 2022.
  * [26] F. Zhang and Y. Demiris, “Learning grasping points for garment manipulation in robot-assisted dressing,” in _2020 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2020, pp. 9114–9120.
  * [27] M. Dragusanu, S. Marullo, M. Malvezzi, G. M. Achilli, M. C. Valigi, D. Prattichizzo, and G. Salvietti, “The dressgripper: A collaborative gripper with electromagnetic fingertips for dressing assistance,” _IEEE Robotics and Automation Letters_ , vol. 7, no. 3, pp. 7479–7486, 2022.
  * [28] F. Krebs and T. Asfour, “A bimanual manipulation taxonomy,” _IEEE Robotics and Automation Letters_ , 2022.
  * [29] L. P. Ureche and A. Billard, “Constraints extraction from asymmetrical bimanual tasks and their use in coordinated behavior,” _Robotics and autonomous systems_ , vol. 103, pp. 222–235, 2018.
  * [30] A. Colomé and C. Torras, “Dimensionality reduction for dynamic movement primitives and application to bimanual manipulation of clothes,” _IEEE Transactions on Robotics_ , vol. 34, no. 3, pp. 602–615, 2018.
  * [31] F. Suárez-Ruiz, X. Zhou, and Q.-C. Pham, “Can robots assemble an ikea chair?” _Science Robotics_ , vol. 3, no. 17, p. eaat6385, 2018.
  * [32] J. Liu, Y. Chen, Z. Dong, S. Wang, S. Calinon, M. Li, and F. Chen, “Robot cooking with stir-fry: Bimanual non-prehensile manipulation of semi-fluid objects,” _IEEE Robotics and Automation Letters_ , vol. 7, no. 2, pp. 5159–5166, 2022.
  * [33] J. Xu, G. G. Grindle, B. Salatin, J. J. Vazquez, H. Wang, D. Ding, and R. A. Cooper, “Enhanced bimanual manipulation assistance with the personal mobility and manipulation appliance (permma),” in _2010 IEEE/RSJ International Conference on Intelligent Robots and Systems_. IEEE, 2010, pp. 5042–5047.
  * [34] M. Connan, M. Sierotowicz, B. Henze, O. Porges, A. Albu-Schäffer, M. A. Roa, and C. Castellini, “Learning to teleoperate an upper-limb assistive humanoid robot for bimanual daily-living tasks,” _Biomedical Physics & Engineering Express_, vol. 8, no. 1, p. 015022, 2021.
  * [35] A. Edsinger and C. C. Kemp, “Two arms are better than one: A behavior based control system for assistive bimanual manipulation,” in _Recent progress in robotics: Viable robotic service to human_. Springer, 2007, pp. 345–355.
  * [36] A. Toshev and C. Szegedy, “Deeppose: Human pose estimation via deep neural networks,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2014, pp. 1653–1660.
  * [37] Z. Cao, T. Simon, S.-E. Wei, and Y. Sheikh, “Realtime multi-person 2d pose estimation using part affinity fields,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 7291–7299.
  * [38] R. A. Güler, N. Neverova, and I. Kokkinos, “Densepose: Dense human pose estimation in the wild,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 7297–7306.
  * [39] U. Iqbal, A. Doering, H. Yasin, B. Krüger, A. Weber, and J. Gall, “A dual-source approach for 3d human pose estimation from single images,” _Computer Vision and Image Understanding_ , vol. 172, pp. 37–49, 2018.
  * [40] D. Berenson, “Manipulation of deformable objects without modeling and simulating deformation,” in _2013 IEEE/RSJ International Conference on Intelligent Robots and Systems_. IEEE, 2013, pp. 4525–4532.
  * [41] N. Hogan, “Impedance control: An approach to manipulation,” in _1984 American control conference_. IEEE, 1984, pp. 304–313.
  * [42] L. Sciavicco and B. Siciliano, _Modelling and control of robot manipulators_. Springer Science & Business Media, 2001.
  * [43] M. Desmurget and C. Prablanc, “Postural control of three-dimensional prehension movements,” _Journal of neurophysiology_ , vol. 77, no. 1, pp. 452–464, 1997.
  * [44] A. Clegg, W. Yu, J. Tan, C. K. Liu, and G. Turk, “Learning to dress: Synthesizing human dressing motion via deep reinforcement learning,” _ACM Transactions on Graphics (TOG)_ , vol. 37, no. 6, pp. 1–10, 2018.
  * [45] S. Calinon, F. Guenter, and A. Billard, “On learning, representing, and generalizing a task in a humanoid robot,” _IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics)_ , vol. 37, no. 2, pp. 286–298, 2007.
  * [46] G. Schwarz, “Estimating the dimension of a model,” _The annals of statistics_ , 1978.
  * [47] M. Muhlig, M. Gienger, S. Hellbach, J. J. Steil, and C. Goerick, “Task-level imitation learning using variance-based movement optimization,” in _2009 IEEE International Conference on Robotics and Automation_. IEEE, 2009, pp. 1177–1184.
  * [48] J. A. McDermid, Y. Jia, and I. Habli, “Towards a framework for safety assurance of autonomous systems,” in _Artificial Intelligence Safety 2019_. CEUR Workshop Proceedings, 2019, pp. 1–7.