#  AGIC: Approximate Gradient Inversion Attack on Federated Learning

Jin Xu, Chi Hong, Jiyue Huang, Lydia Y. Chen, Jérémie Decouchant  Delft
University of Technology, The Netherlands
<EMAIL_ADDRESS>{c.hong, j.huang-4, y.chen-10,
<EMAIL_ADDRESS>
###### Abstract

Federated learning is a private-by-design distributed learning paradigm where
clients train local models on their own data before a central server
aggregates their local updates to compute a global model. Depending on the
aggregation method used, the local updates are either the gradients or the
weights of local learning models, e.g., FedAvg aggregates model weights.
Unfortunately, recent reconstruction attacks apply a gradient inversion
optimization on the gradient update of a single mini-batch to reconstruct the
private data used by clients during training. As the state-of-the-art
reconstruction attacks solely focus on single update, realistic adversarial
scenarios are overlooked, such as observation across multiple updates and
updates trained from multiple mini-batches. A few studies consider a more
challenging adversarial scenario where only model updates based on multiple
mini-batches are observable, and resort to computationally expensive
simulation to untangle the underlying samples for each local step. In this
paper, we propose AGIC, a novel Approximate Gradient Inversion Attack that
efficiently and effectively reconstructs images from both model or gradient
updates, and across multiple epochs. In a nutshell, AGIC (i) approximates
gradient updates of used training samples from model updates to avoid costly
simulation procedures, (ii) leverages gradient/model updates collected from
multiple epochs, and (iii) assigns increasing weights to layers with respect
to the neural network structure for reconstruction quality. We extensively
evaluate AGIC on three datasets, namely CIFAR-10, CIFAR-100 and ImageNet. Our
results show that AGIC increases the peak signal-to-noise ratio (PSNR) by up
to 50% compared to two representative state-of-the-art gradient inversion
attacks. Furthermore, AGIC is faster than the state-of-the-art simulation-
based attack, e.g., it is 5x faster when attacking FedAvg with 8 local steps
in between model updates.

††footnotetext:  This document is a preprint of a paper accepted at the 41st
International Symposium on Reliable Distributed Systems (SRDS 2022).

###### Index Terms:

reconstruction attack, federated learning, federated averaging

##  I Introduction

Federated learning (FL) [1, 2, 3, 4, 5] is a popular collaborative learning
paradigm that aims at providing accurate predictive models while preserving
the clients’ data privacy and reducing communication costs. In a FL system,
the training data of a client never leaves its initial premises. In each
global round, the server first sends the most recent model to clients, which
then train the local model on their private data and send gradients or model
updates back to the server. At the end of a global round, the server is able
to update the global model by aggregating all the gradients or model updates
it has received.

An attacker that would compromise the server would observe model parameters
and their updates, but would not have access to training samples. Therefore,
FL is often assumed to safely protect the clients’ local data. However, recent
works have shown that the observation of model parameters and gradient updates
might allow attributes of local samples to be leaked [6], class
representatives to be inferred [7, 8], and even real training samples to be
reconstructed [9, 10].

Gradient inversion attacks that directly reconstruct training samples based on
a model and gradient updates on it result in the most serious data leakages.
Most gradient inversion attacks are optimization-based. In an optimization-
based gradient inversion attack, the adversary randomly initializes dummy
samples, and executes forward and backward propagation on them to obtain dummy
gradients. The dummy samples are then optimized to minimize the sum of the
distance between the observed real gradients and the dummy gradients, and
regularization terms. To improve the reconstruction performance, some works
define new distance functions and regularization terms [10], while others
exploit prior knowledge, e.g., by using batch normalization data [11] or pre-
trained generative models [12].

Gradient inversion attacks are designed to reconstruct samples from gradient
updates. However, in practice, federated learning systems often use federated
averaging (FedAvg) [4] where clients send model updates after conducting
multiple local steps, each executed over a mini-batch, in order to further
reduce the communication overhead. Few previous works have discussed how to
attack model updates generated by FedAvg. These works either use simulation
[10], which is slow and incompatible with label inference, a sub-task that
significantly improves the reconstruction performance [13], or make an
averaging approximation [14] to estimate the gradient of a full batch for
FedAvg, which does not extend to the most general case where mini-batches are
used.

In this paper, we describe AGIC, a novel Approximate Gradient Inversion Attack
that specifically targets federated learning systems based on FedAvg’s model
updates, and is also compatible with gradient updates. Overall, AGIC makes the
following contributions.

First, to avoid the computational overhead of simulation attacks, AGIC
leverages a one-batch approximation, which assimilates multiple local steps
executed over multiple mini-batches as a single local step executed over a
single aggregated mini-batch. The corresponding gradients of that larger mini-
batch can also be approximated based on the received model update. Thanks to
this one-batch approximation, AGIC is faster than previous simulation-based
attacks on FedAvg and is compatible with label inference.

Second, based on the observation that the attacker is able to observe updates
across multiple epochs, AGIC better exploits the information contained in
multiple updates that have been computed over common data samples. AGIC
matches the collected updates with specific training samples and jointly
optimizes with the updates to improve the reconstruction quality of the
specific training samples.

Finally, AGIC assigns different weights to the gradients of different layers
in the distance function to optimize, inspired by the results of recent works
on model compression that have shown that model layers have unequal effects on
model accuracy and other metrics [15, 16]. Interestingly, AGIC uses a layer
weight modifier for convolutional neural networks (CNN) that use ReLU as
activation function, to balance the contribution of each convolution layer,
because ReLU imports zeros into gradients.

Our experiments with three datasets show that AGIC outperforms two
representative state-of-the-art baselines in both reconstruction quality and
efficiency. When attacking gradient updates, AGIC’s peak signal-to-noise ratio
is up to 50% higher than with both baselines. When attacking FedAvg model
updates configured with 8 local steps, AGIC reconstructs samples five times
faster than the only applicable baseline, a state-of-the-art simulation-based
attack.

The remainder of this paper is organized as follows. Section II provides some
background on federated learning and gradient inversion attacks. Section III
introduces our system and threat models, and an overview of AGIC. Section IV
describes AGIC’s three key features. Section V evaluates AGIC’s performance.
Finally, Section VII concludes this paper.

##  II Background

###  II-A Federated learning

Federated learning (FL) [5, 2] is a collaborative learning paradigm that
allows distributed data owners (called clients) to jointly learn machine
learning models without centrally pooling their data. Clients train the model
using their own local data and rely on a central server to aggregate their
learning results. The FL system progresses over multiple global rounds. At the
beginning of a global round, the server sends the latest global model to
clients. Clients then train the latest model on their own data, and send their
training results, i.e., gradients or model updates, back to the server.

The local training process typically follows the stochastic gradient descent
(SGD) algorithm, which iterates through the data using batches. A batch might
contain the full local dataset, but in the more general case a batch is a
subset of the local dataset and is also called a mini-batch. During an epoch,
clients iterate over their complete training data once. A client may use its
full dataset in a single batch during one round, which is then regarded as an
epoch. We focus on the more general mini-batch case, where the full dataset is
split into multiple mini-batches, and the client iterates over these mini-
batches in multiple global rounds during an epoch.

Figure 1 depicts two typical local training procedures of FL learning systems
that respectively send gradient updates after one local step (top), or send
model updates after multiple local steps (bottom), which is the case with
federated averaging (FedAvg) [4]. With the first method, the clients execute a
single local training step on a batch of local samples and their labels
$(X,y)$ with the latest received global model $W$, and send back the
corresponding gradient update $\nabla W$. With FedAvg, the clients run $T$
local steps on $T$ batches, which means that they will update their local
model parameters $T$ times, and then send the trained model weights $W_{T}$
back to the server. In practical scenarios, FedAvg is more commonly used
because it reduces the amount of data transmitted over the network, in
particular because a client can process several batches during a global round,
i.e., before sending its results to the server. In the rest of this paper, we
will note $\Delta W{=}W_{T}{-}W$, the difference between the updated model
sent by the client at the end of a round and the global model it received at
its beginning.

Figure 1: A global federated learning round with gradient update and with
model update.

(a) Attack based on gradient updates.

(b) Simulation-based attack based on model updates with 2 local steps using
simulation.

Figure 2: Gradient inversion attacks based on gradient updates and model
updates.

###  II-B Gradient inversion attacks

Even though the clients and the server only exchange intermediate learning
results such as models or gradient updates, it has been shown that sensitive
information from local training datasets can be inferred in FL systems [6, 8].
Furthermore, it has been shown that an attacker that obtains gradient updates
can launch a gradient inversion attack to recover the training samples that
were used by clients to generate the gradients [9]. Gradient inversion attacks
generally assume the attacker is an honest-but-curious server [10].

Most existing gradient inversion attacks solve an optimization problem, as
illustrated in Figure 2(a). After retrieving gradient updates $\nabla W$, the
attacker generates dummy samples $(\hat{X},\hat{y})$ and minimizes the
distance between the received gradients $\nabla W$ and its dummy gradients
$\nabla W^{\prime}$, which are retrieved by feeding dummy samples through the
obtained model in one forward-backward pass. During the optimization process,
the values of the dummy samples are optimized, so that at the end of the
attack the dummy samples approximate the training samples.

An optimization-based gradient inversion attack typically relies on an
optimization objective such as:

$\min_{\hat{X}}\textnormal{Dist}(\frac{\partial\mathcal{L}(F(W,\hat{X}),\hat{y})}{\partial
W},\nabla W)+\textnormal{Reg}(\hat{X},\hat{y})$ (1)

Formula 1 uses the following notations. $W$ is the current values of the
trainable parameters of the attacked neural network. $\nabla
W=\frac{\partial\mathcal{L}(F(W,X),y)}{\partial W}$ is the gradient update
received from a client. $F$ is the forward propagation function of the model.
$\mathcal{L}$ is the loss function of the model. $\hat{X}$ is the generated
reconstruction samples and $\hat{y}$ is the labels of the samples. Dist is a
distance function such as the L2 distance [9] and the cosine distance [10],
which are the two most widely used distance functions in gradient inversion
attacks. Reg refers to regularization terms. For attacks on image
classification tasks, additional regularization terms can be leveraged to
generate more natural images. For example, total variation [10] is used to
reduce image noise, and clipping terms [14] are used to prevent abnormal
values that are out of range for a pixel. The sum of the distance and
regularization terms for generated samples forms the objective to be
minimized.

In the previous paragraph, we mention that both dummy samples $\hat{X}$ and
dummy labels $\hat{y}$ are simultaneously optimized to recover the local
training data $(X,y)$. In reality, to avoid jointly optimizing on the labels
and reducing the complexity of the optimization problem, recent analytical
approaches infer the labels before conducting the optimization by analyzing
the distribution of the gradient tensor of the last fully connected layer [13,
11]. Label inference is a crucial step to improve reconstruction quality of
the most recent gradient inversion attacks.

Designing accurate gradient inversion attacks on model updates with mini-
batches is more difficult than on gradient updates. Indeed, assuming a batch
size $B$ for each local step, an adversary attacking gradient updates needs to
reconstruct $B$ samples. With model updates, from $T$ local-step model updates
respectively computed over $T$ different mini-batches of size $B$, the
adversary needs to reconstruct $TB$ samples for an identical update size,
i.e., number of model parameters.

While most gradient inversion attacks can only be applied to gradient updates,
a recent attack on model updates [10] simulates the execution of $T$ local
steps in each optimization iteration, retrieves the dummy model update
$W_{T}^{\prime}$ after the last local step, and optimizes the dummy samples in
all mini-batches $\hat{X}_{1},...,\hat{X}_{T}$ based on the difference between
the dummy model and the observe model $\Delta W_{T}^{\prime}=W_{T}^{\prime}-W$
and the real observed model difference $\Delta W_{T}$. Figure 2(b) illustrates
a simulation-based gradient inversion attack based on FedAvg model updates.

##  III Overview of AGIC

This section defines our system and threat models, and provides an overview of
AGIC, our gradient inversion attack.

###  III-A System and adversarial models

We consider a FL system with one server that aggregates updates and multiple
clients. We assume all clients to be correct and that the server is honest-
but-curious. Therefore, our adversary observes the communications between the
server and the clients and can retrieve both models and model updates. The
attack can occur at any phase of the training process to target untrained, in-
training or well-trained networks. The attacker may observe updates from
different epochs during training. During an epoch a client iterates over its
whole dataset, possibly using different mini-batches.

We focus on attacking convolutional neural networks (CNN), which are used for
image classification, one of the most popular uses of FL systems. For example,
it can be used by medical institutes to collaboratively train a diagnosis
model [17] without sharing patient-related information.

###  III-B Attack overview

To successfully run gradient inversion attacks on FedAvg, AGIC relies on three
key features: the one-batch approximation for federated averaging, the
leveraging of model updates from multiple epochs, and the assignment of layer
weights in the distance function.

Figure 3 illustrates AGIC and its key features. Compared to the simulation
attack on FedAvg, which was presented in Figure 2(b), this Figure shows AGIC
key features. First, AGIC uses a one-batch approximation so that mini-batches
$\hat{X}_{1},\hat{X}_{2}$ within a two-step FedAvg update compose a larger
batch $\hat{X}_{1}{+}\hat{X}_{2}$ (cf. §IV-A). Figure 3 also illustrates two
FedAvg model updates from two different epochs $e_{i},e_{j}$ that have used
overlapping samples in their first local step, i.e., first mini-batch,
respectively $X_{1}$ and $X_{3}$. Tensors $\hat{X}_{1}$ and $\hat{X}_{3}$ thus
have overlapping components and can be jointly optimized. (cf. §IV-B).
$\alpha_{layer}$ represent the different layer weights that AGIC uses in its
distance function (cf. §IV-C).

Figure 3: Overview of AGIC.

Base loss function used during optimization. The base loss function that AGIC
uses to reconstruct images from an update is the negative cosine distance
between the dummy gradients $\nabla
W^{\prime}=\frac{\partial\mathcal{L}(F(\hat{X},W),y)}{\partial W}$ and the
received real gradients $\nabla W$, to which is added a total variation (TV)
regularization term that reduces image noises with weight $\zeta_{TV}$ [18]:

$\min_{\hat{X}}1-\cos(\nabla W^{\prime},\nabla
W)+\zeta_{TV}\textnormal{TV}(\hat{X})$ (2)

$\cos(\nabla W^{\prime},\nabla W)=\frac{\nabla W^{\prime}\cdot\nabla
W}{\parallel\nabla W^{\prime}\parallel_{2}\ \parallel\nabla W\parallel_{2}}$
(3)

The dummy samples are optimized based on the gradient values from back
propagation of the loss. AGIC modifies this base loss function to adapt to
different attack scenarios.

AGIC is a novel approximate gradient inversion attack that is applicable to
both model updates and gradient updates. We focus on the more challenging
model update scenario, but our methods (namely, leveraging multiple epochs and
choosing layer weights) also apply to gradient updates.

##  IV Key implementation features of AGIC

This section details AGIC’s three key features that enable its use with FedAvg
and increase its accuracy: one-batch approximation, use of multiple updates,
and layer weights.

###  IV-A One-batch approximation for FedAvg

As previously explained, accurately reconstructing samples from FedAvg’s model
updates is more difficult than with gradient updates because a model update is
computed using more samples. When attacking FedAvg with possibly multiple
local steps, AGIC is faster and more accurate than simulation-based gradient
inversion attacks thanks to its one-batch approximation. This approximation
allows AGIC not to use computationally expensive simulation and to be
compatible with label inference, as there is no need to assign inferred labels
to different local steps.

The one-batch approximation is equivalent to assuming that the model remains
mostly identical after each local step. This assumption has been discussed in
large mini-batch training [19]. A similar first-order approximation has been
used in meta learning [20]. It should be noted that if model parameters change
drastically in each local step, e.g. when the learning rate of local SGD is
$1\times 10^{-2}$, the one-batch approximation should not be used. In our
experiments, with a learning rate of $1\times 10^{-4}$, the approximation
brings good performance (cf. V-B).

Figure 4 illustrates AGIC’s one-batch approximation on FedAvg’s updates.
Compared to the multi-step simulation of [10], which is presented in Figure
2(b), AGIC’s approximation only runs one local step with the aggregated batch,
and its loss function is based on approximated gradients instead of models
difference $\Delta W$. More precisely, given all the mini-batches used in one
FedAvg update $X_{1},...,X_{n}$ and their inferred labels $y_{1},...,y_{n}$,
and local learning rate $\mu$, AGIC makes the approximation that all the mini-
batches compose a larger batch $X_{A}=[X_{1},...,X_{n}]$ to produce the
received model update:

$\Delta
W\approx\sum_{j}^{N}-\mu\frac{\partial\mathcal{L}(F(W,X_{j}),y_{j})}{\partial
W}=-\mu\frac{\partial\mathcal{L}(F(W,X_{A}),y_{A})}{\partial W}$ (4)

Figure 4: Gradient inversion attack with AGIC’s one-batch approximation on
FedAvg model updates. Here, clients use 2 local steps before sharing model
updates with the server.

Based on Formula 4, AGIC computes the approximated gradients $\nabla
W=\frac{\Delta W}{-\mu}$ for aggregated batch $X_{A}$ from the model updates
given then local learning rate. Then AGIC initializes dummy samples
$\hat{X}_{A}$ as the size of $X_{A}$, and optimizes on the distance between
dummy gradients from $\hat{X}_{A}$ and the computed gradients from received
model updates. Because the objective function is the distance between
gradients, any improvement method designed for gradient updates can be
directly applied on FedAvg model updates with AGIC’s approximation.

(a) Pre-reconstruct from updates from multiple epochs separately.

(b) Measure similarity between image pairs of same inferred label.

(c) Match most similar image pairs with greedy algorithm.

(d) Match updates and conduct joint reconstruction.

Figure 5: Reconstruction from updates from multiple epochs. We use updates
with batch size 1 for illustration purposes.

###  IV-B Leveraging updates from multiple epochs

A training sample is used in many global rounds, each round producing an
update, and thus is involved in multiple updates that are communicated to the
server. The attacker can therefore observe several updates that are based on
some specific samples. AGIC demonstrates that the adversary can better
reconstruct the samples by making use of the updates it observes.

Prior work by Geng et al. has shown that reconstruction quality of a specific
batch is improved if jointly optimizing with more pairs of model parameters
$W_{k}$ and their corresponding gradient update from that batch
$W^{\prime}_{k}$ [14]. However, this method only works on full batch gradient
descent and does not provide a solution to the general case where the client
uses multiple mini-batches to go over its data as in an epoch. In contrast,
AGIC is able to reconstruct training samples using multiple model updates or
gradient updates from different mini-batches across epochs during training.

AGIC processes updates from multiple epochs in two steps: (i) update matching,
which matches the collected updates whose corresponding samples are
overlapping; and (ii) joint reconstruction, which reconstructs the samples
from the matching updates. We discuss those two steps in the following, and
Figure 5 illustrates them with a batch size equal to 1.

Step 1: Update matching. AGIC first matches updates that have overlapping
samples so that they can be jointly reconstructed afterwards. If the training
process does not shuffle the samples used between epochs, or if the entire
local dataset is used in a full batch, then the matching process is trivial.
In the general case, a more complex algorithm is required to match gradient
updates of single mini-batches or FedAvg model updates based on multiple mini-
batches. AGIC’s matching mechanism first conducts a reconstruction process for
each update to obtain a first reconstruction of images, and then identifies
the most similar pairs of reconstructed images.

As the adversary observes updates from two complete epochs, the pre-
reconstructed samples from the first epoch’s updates can be mapped one-to-one
with the samples from the updates of the second epoch. Equivalently said, for
any sample $X_{c}$, two updates from the two epochs have been trained on it.

To match pairs of reconstructed images, we evaluate the similarity of pairs of
images by processing the images with an average pooling layer that takes
average value of each small part of an image in order to mitigate noises, and
computing their mean squared error. We refer the reader to §V-D, which
demonstrates the performance improvement that using a pooling layer brings.

We use a simple greedy algorithm to find a one-to-one mapping between
reconstructed images from two epochs. First, we measure the similarity of each
image pair and sort them. From the highest similarity to the lowest one, we
check each pair of still unmatched images and associate them to each other.
Using this method, all pre-reconstructed images from one epoch can be matched
to those from another epoch. For any matched image pairs, which are then
considered to reconstruct the same sample, AGIC executes a joint
reconstruction using the all updates they participated in (Step 2).

It is easy to extend this method to leverage more than two epochs, by
successively computing matching update pairs from consecutive epoch pairs. A
specific sample is then associated to the image pairs reconstructed from
updates from consecutive epochs, and to one reconstructed image per epoch. A
sample is therefore associated to exactly one update per epoch.

Since label inference can be applied before optimizing for each update, AGIC
uses a label-based filter. Based on their inferred label, AGIC computes the
similarity between image pairs that have the same inferred label to reduce the
search space and limit the possibility of incorrect matching, thereby
increasing accuracy. For a multi-sample batch, generated either from gradient
update directly or from the one-batch approximation of model update, label
inference returns a list of labels for all samples. Therefore, whether two
samples can be matched depends on whether their inferred label lists share at
least a common label.

Step 2: Joint reconstruction with multiple updates. AGIC jointly optimizes
updates that have been identified as having been trained over a common sample,
in order to improve reconstruction quality. We use gradients during the
optimization process, obtained either from the one-batch approximation of
model update or directly from received gradient update. Our distance function
uses the summation of distance of all pairs of input gradients and dummy
gradients, with weights $\gamma_{i}$ assigned. A decreasing weight is assigned
to the gradients generated later during the training process, because the
reconstruction quality of gradient inversion attacks is higher in the earliest
training phases. These gradient weights are set as experimental
hyperparameters (§V details the values we use).

Let us assume that after the update matching step, AGIC has identified that
gradients $\nabla W_{i}$ with corresponding model parameters $W_{i}$ from
different epochs have been trained over a given sample $X_{c}$. We note
$\nabla W^{\prime}_{i}$ the dummy gradients from the dummy batches. AGIC’s
distance function with multiple gradient updates is the following.

$\min_{\hat{X}_{k}}\sum_{k}(1-\frac{\nabla W^{\prime}_{k}\cdot\nabla
W_{k}}{\parallel\nabla W^{\prime}_{k}\parallel_{2}\ \parallel\nabla
W_{k}\parallel_{2}})\cdot\gamma_{k}$ (5)

The reconstruction results of AGIC’s joint reconstruction are of higher
quality than those that would be obtained by separately optimizing based on
single updates. To illustrate why, let us consider two gradient updates from
two epochs that are respectively trained from batches $X_{1},X_{2}$ with
overlapping sample $X_{c}$. In this situation, dummy sample $\hat{X}_{c}$ is a
common component of dummy batches $\hat{X}_{1},\hat{X}_{2}$ and always has a
single value in each dummy batch during optimization. Therefore, both updates
provide information that AGIC uses to optimize the sample.

If updates are observed in two or more complete epochs then all training
samples can be better recovered thanks to the update matching and joint
reconstruction steps. Note that an update is used $B$ times to jointly
reconstruct different samples if the batch size is equal to $B$.

TABLE I: PSNR results for reconstructing size-4 batches from untrained ResNet20-4, with different layer assignment functions. The decreasing functions pass through $(1,1)$ and $(N_{conv},0.5)$, and the increasing functions pass through $(1,1)$ and $(N_{conv},2)$, where the number of convolutional layers $N_{conv}=21$ in ResNet20-4. The two points and function expressions determine the used functions. Monotonicity | Convexity | Function | CIFAR-10 | CIFAR-100
---|---|---|---|---
None | None | $y=1$ | 14.836 | 15.190
Decrease | Convex | $-a\log x+b$ | 14.182 | 14.332
Decrease | Concave | $-ae^{x}+b$ | 14.727 | 15.175
Decrease | None | $-ax+b$ | 14.354 | 14.471
Increase | Convex | $ae^{x}+b$ | 15.255 | 16.001
Increase | Concave | $a\log x+b$ | 15.505 | 16.311
Increase | None | $ax+b$ | 15.601 | 16.702

###  IV-C Assigning layer weights

It has been observed that different layers of a neural network have variable
impact on model accuracy [16], energy consumption [15], and can be selectively
pruned for model compression [21]. Similarly, we observe that gradients from
different layers have different effect on the performance of a gradient
inversion attack.

Based on these previous works, AGIC assigns different weights $\alpha_{i}$ to
layers in the distance function used in the gradient inversion attack. For a
CNN that has $N_{conv}$ convolutional layers before the fully connected
layers, suppose that the input gradients can be decomposed by layers from
first to last as $\nabla W=(\nabla L_{1},...,\nabla L_{N_{conv}},\nabla
L_{fc})$ and so do the reconstructed gradients $\nabla W^{\prime}$. AGIC’s
negative cosine similarity objective with layer weights $\alpha_{i}$ is
indicated in Formula 6.

$\min_{\hat{X}}1-\frac{\sum_{i}\alpha_{i}(\nabla L^{\prime}_{i}\cdot\nabla
L_{i})}{\sqrt{\sum_{i}\alpha_{i}\parallel\nabla
L^{\prime}_{i}\parallel_{2}^{2}}\sqrt{\sum_{i}\alpha_{i}\parallel\nabla
L_{i}\parallel_{2}^{2}}}$ (6)

Linear assignment. It is impractical to separately select a weight for each
layer using tedious methods like grid search. Therefore, we use a function to
assign the weights to convolutional layers, and set weights of fully connected
layers to the average of convolutional layer weights.

Both the shape and the range of the weight function affect performance. For
the range, we first determine that the function passes through points $(1,1)$
and $(N_{conv},\beta)$. Parameter $\beta$ controls the range of the function
and $\beta=1$ implies homogeneous weights. We also look into various function
shapes, such as linear functions $y{=}ax{+}b$, exponential-like functions
$y{=}ae^{x}{+}b$ and logarithmic-like functions $y{=}a\log x{+}b$ to cover
various function shapes (e.g., concave or convex). The two points $(1,1)$ and
$(N_{conv},\beta)$ determine values of coefficients $a,b$ in the various
functions.

Table I presents some results obtained with different function shapes. We
experimentally observe that an increasing linear weight function maximizes the
reconstruction performance, and therefore determines a layer’s weight using
Formula 7.

$l_{i}=1+\frac{\beta-1}{N_{conv}-1}(i-1)$ (7)

In practice, the properties of the attacked network change during the training
process. For example, the magnitude of gradient values becomes smaller and the
reconstruction quality deteriorates. In consequence, the best value for
parameter $\beta$ changes according to the attacked training phase (as
illustrated in Figure 9).

ReLU modifier. We can directly set $\alpha_{i}{=}l_{i}$ in the general case,
but we additionally propose a layer weight modifier for CNNs that apply ReLU
after convolution layers, as $\textnormal{ReLU}(x)=\max{(0,x)}$ is the most
commonly used activation function in current CNN designs such as ResNet [22].
ReLU is more popular than sigmoid functions, because it simplifies gradient
computation in backward propagation and avoids gradient vanishing and gradient
inversion.

When the input of ReLU is negative, its output is set to zero, and as a
result, after back propagation, the gradients of input are also equal to zero.
Therefore, a proportion of zeros exists in each convolution layer’s gradients.
These zeros limit the performance of gradient inversion attacks, because it is
then easier for the optimization process to get its dummy gradients to achieve
these zeros, which appear as soon as the corresponding output of a
convolutional layer is negative. Thus, layers that have more zero gradients
contribute less in the reconstruction.

To remove the effect of zeros imported by ReLU and balance the contribution of
each convolutional layer, AGIC relies on a layer weight modifier based on the
proportion of zeros of each layer. Assuming that the proportion of zeros in a
convolutional layer’s gradients is $p_{i}$, its modifier zero-based
coefficient is computed as $z_{i}{=}\frac{1}{1-p_{i}}$.

In conclusion, the layer weight for the $i$-th convolutional layer is equal to
a linearly assigned weight $l_{i}$, which is multiplied by a zero-based weight
$z_{i}$ if ReLU is used, and the layer weight of a fully connected layer is
the average of the linearly assigned weights of convolutional layers:

$\displaystyle\alpha_{i}$ $\displaystyle=l_{i}\cdot\frac{1}{1-p_{i}}$
$\displaystyle i\textnormal{-th convolutional layer}$ (8)
$\displaystyle\alpha_{fc}$
$\displaystyle=\frac{\sum_{i=1}^{N_{conv}}l_{i}}{N_{conv}}$ fully connected
layers

##  V Performance evaluation

TABLE II: Reconstruction quality of gradient inversion attacks on untrained ResNet20-4 on CIFAR-10 and CIFAR-100, and trained ResNet50 on ImageNet, on FedAvg model updates. Here, FedAvg runs on 4 size-1 mini-batches and 4 local steps. Method | simulation | CIFAR-10, untrained | CIFAR-100, untrained | ImageNet, trained
---|---|---|---|---
|  | PSNR↑ | SSIM↑ | LPIPS↓ | PSNR↑ | SSIM↑ | LPIPS↓ | PSNR↑ | SSIM↑ | LPIPS↓
DLG-Adam | $\checkmark$ | 13.611 | 0.391 | 0.123 | 15.114 | 0.463 | 0.096 | 8.044 | 0.028 | 1.314
InvG | $\checkmark$ | 14.108 | 0.414 | 0.117 | 15.465 | 0.480 | 0.092 | 10.593 | 0.167 | 0.779
AGIC (this work, one-batch) | $\times$ | 14.250 | 0.414 | 0.115 | 15.437 | 0.475 | 0.093 | 10.522 | 0.161 | 0.791
AGIC (this work, one-batch + layers) | $\times$ | 16.182 | 0.529 | 0.112 | 19.133 | 0.672 | 0.050 | 10.699 | 0.177 | 0.780
TABLE III: Reconstruction quality of gradient inversion attacks on FedAvg, with local step 4 and batch size 2, on ResNet20-4 on CIFAR-100. The matching batches from two epochs are assumed to have the same samples. A ’1’ in the Epochs column means that the updates are collected after one epoch, while ’1, 2’ means that the matching updates are collected after one epoch and two epochs. Method | Simulation | Epochs | PSNR↑ | SSIM↑ | LPIPS↓
---|---|---|---|---|---
DLG-Adam | $\checkmark$ | 1 | 12.196 | 0.292 | 0.087
InvG | $\checkmark$ | 1 | 14.398 | 0.419 | 0.065
AGIC (this work) | $\times$ | 1 | 15.651 | 0.486 | 0.055
AGIC (this work) | $\times$ | 1,2 | 16.664 | 0.533 | 0.040

(a) DLG-Adam

(b) InvG

(c) AGIC (this work)

(d) Ground truth

Figure 6: Reconstructed CIFAR-10 images from untrained ResNet20-4 with batch
size 1.

(a) DLG-Adam

(b) InvG

(c) AGIC (this work)

(d) Ground truth

Figure 7: Reconstructed ImageNet images from untrained ResNet50 with batch size 1. TABLE IV: Reconstruction quality of gradient inversion attacks on single gradient updates, reconstructing CIFAR-10 images from untrained ResNet20-4 and ImageNet images from untrained ResNet50. Dataset |  Batch size | Method | PSNR↑ | SSIM↑ | LPIPS↓
---|---|---|---|---|---
CIFAR-10 |  1 | DLG-Adam | 19.908 | 0.727 | 0.034
InvG | 20.671 | 0.753 | 0.031
AGIC (this work) | 31.341 | 0.963 | 0.004
CIFAR-10 |  4 | DLG-Adam | 14.034 | 0.412 | 0.110
InvG | 14.421 | 0.433 | 0.103
AGIC (this work) | 17.183 | 0.586 | 0.092
ImageNet |  1 | DLG-Adam | 11.311 | 0.094 | 0.751
InvG | 12.832 | 0.214 | 0.691
AGIC (this work) | 15.573 | 0.366 | 0.670
ImageNet |  4 | DLG-Adam | 9.879 | 0.070 | 0.801
InvG | 11.640 | 0.216 | 0.720
AGIC (this work) | 12.836 | 0.302 | 0.721

###  V-A Setup

Hardware. We use a machine equipped with an AMD EPYC 7542 32-core CPU, 256GB
of RAM and an NVIDIA GeForce RTX 2080 Ti GPU for our experiments.

Datasets. We conduct experiments on three image datasets: CIFAR-10 (size
$32\times 32$, 10 classes) [23], CIFAR-100 (size $32\times 32$, 100 classes)
[23], and ImageNet (size $224\times 224$, 1000 classes) [24]. All three
datasets contain 3-channel colored images. The datasets are normalized to mean
$0$ and standard deviation $1$. For the experiments, we randomly select 100
mini-batches from the validation set of each dataset. We attack ResNet trained
on the datasets [22]: ResNet20-4 on CIFAR-10 and CIFAR-100, and ResNet50 on
ImageNet.

Baselines. We compare our attack with two baselines. The first is an improved
version of DLG [9], namely DLG-Adam, which uses the Adam optimizer instead of
L-BFGS in the original version, because L-BFGS often fails to converge while
attacking large networks like ResNet [10]. The other baseline is Inverting
Gradients [10], which we call InvG. The two baselines cover the two main
distance functions applied in gradient inversion attacks.

We do not compare our attack to those based on pretrained models, e.g., an
image generative model [12], because we do not assume the attacker to have
access to additional knowledge. We also do not consider a batch normalization
(BN) regularization term [11], because it requires that each synchronous BN
layer has cross-node communication during forward propagation, which is very
time-consuming and impractical in FL. Our methods are however orthogonal to
these methods that leverage extra information and could be combined with them.

Hyperparameters. The Adam optimizer with learning rate 0.1 is used based on
the gradient values on dummy samples during optimization. $\zeta_{TV}$ is
equal to $10^{-4}$ for untrained ResNet20-4, $10^{-2}$ for trained ResNet20-4,
$10^{-3}$ for untrained ResNet50, and $10^{-1}$ for trained ResNet50. For
FedAvg, the learning rate of local SGD is $\mu=1\times 10^{-4}$. The kernel
size of the averaging pooling layer used in update matching is $2\times 2$ and
the stride of it is 2. Regarding epoch weights $\gamma_{i}$ in multi-epoch
reconstruction, $\gamma=1$ for a batch from the first epoch, and $\gamma=0.1$
for the other batches. For layer weights, $\beta=50$ for untrained networks,
and $\beta$=2 for trained networks. The optimization has 10,000 iterations.

Because label inference is an independent subtask that has already been well
explored and for which very accurate methods already exist, we assume that
labels are correctly inferred in experiments of gradient updates, if not
specified in other ways. For our experiments with FedAvg, we apply the label
inference method from [11]. Labels in a mini-batch are assumed not to be
duplicated, following the experiment setup of previous works [10, 11].

Metrics. We use three metrics to measure the similarity between the
reconstructed images and the real training samples: (i) peak signal-to-noise
ratio (PSNR), (ii) structural similarity index measure (SSIM) [25], and (iii)
LPIPS, a perceptual image similarity score [26]. Higher PSNR, higher SSIM, and
lower LPIPS all indicate better reconstruction quality.

###  V-B Overall performance of AGIC

We first report the overall performance of AGIC on FedAvg model updates and
Gradient updates. Table II shows the attack results on single FedAvg model
updates. The one-batch approximation and layer weights are used with this FL
setting. AGIC with the one-batch approximation outperforms the other two
baselines that are based on simulation of FedAvg. Table III further shows the
attack results on FedAvg with updates from multiple epochs. One can also
observe that AGIC further increases its reconstruction quality when
reconstructing from matching updates from multiple epochs.

AGIC can also attack gradient updates, the most common setup in gradient
inversion attacks, as shown in Table IV. These results are interesting because
only layer weights are used in this experiment, and AGIC significantly
outperforms baseline methods, with different datasets and batch sizes. AGIC
improves the PSNR over CIFAR-10’s by up to almost 50%. The reconstructed
CIFAR-10 samples in Figure 6 are visually almost identical to the ground
truth. Figure 7 shows two examples of reconstructed ImageNet samples, where
AGIC also clearly outperforms the baselines.

842$14$$14.2$$14.4$$14.6$# local stepsPSNRInvGAGIC (this work) (a)
Reconstruction quality.

842$0$$10$$20$$30$# local stepstime cost (min)InvGAGIC (this work) (b)
Computation time for attacks with 10k iterations.

Figure 8: Reconstruction qualities and computation times for the simulation-
based attack InvG and for AGIC on CIFAR-10 images from untrained ResNet20-4.
Each model update is generated with 8 images, e.g., for the 8-step scenario
each update is generated from 8 size-1 mini-batches.

###  V-C Computation time reduction with the one-batch approximation

We further compare the one-batch approximation of AGIC with the simulation
approach of InvG, for attacking FL systems using FedAvg. We look into the case
where a FedAvg update is generated from 8 images, but they are divided into
mini-batches with different sizes in different experiments, which implies that
the number of local steps also changes. Figure 8 show that AGIC outperforms
InvG in both quality and speed with all settings. It can also be noticed that
both methods have worse performance when the number of local steps increases,
if the total number of samples are fixed.

When the number of local steps increases, the time cost of attacks based on
simulation also increases, while the one-batch approximation implies that AGIC
has a constant time cost. When FedAvg has 8 local steps, AGIC is 5 times
faster than InvG: AGIC requires 6.7 minutes on an NVIDIA GeForce RTX 2080 Ti
GPU instead of 33.2 minutes with InvG.

TABLE V: Matching success rate of 1000 CIFAR-100 images from epochs 1 and 2, with different update matching methods and batch sizes. Labels | Pooling | Batch size 1 | Batch size 4
---|---|---|---
$\times$ | $\times$ | 0.819 | 0.509
$\times$ | $\checkmark$ | 0.873 | 0.547
$\checkmark$ | $\times$ | 0.988 | 0.672
$\checkmark$ | $\checkmark$ | 0.981 | 0.705
TABLE VI: Reconstruction quality with 200 CIFAR-100 images using gradients from epochs 1 and 2, using label inference and update matching, and equal layer weights. The model is ResNet20-4 and the batch size equals 1. Epochs | PSNR↑ | SSIM↑ | LPIPS↓
---|---|---|---
1 | 20.379 | 0.665 | 0.026
2 | 19.085 | 0.626 | 0.022
1,2 | 21.464 | 0.697 | 0.016

###  V-D Reconstruction quality improvement with update matching and joint
reconstruction

We now focus on reconstructing samples from multiple epochs. We first conduct
experiments on our pre-reconstruction matching method with 2,000 iterations.
We consider matching with and without using an average pooling layer, and with
and without restriction on inferred labels. Table V presents the matching rate
with pre-reconstructed CIFAR-100 images. It shows that with pre-
reconstruction, the matching rate is very high (0.988) when the batch size is
1. When the batch size is 4, the matching rate decreases (0.672), because
reconstruction quality decreases when the batch size increases, but it is
still high. Using inferred labels provides strong prior knowledge for gradient
matching, even when the batch size is larger than 1, which implies that labels
may be incorrectly inferred. A pooling layer successfully mitigates noise,
except in the case where the batch size equals 1 and labels are used, which
already has an almost perfect accuracy.

We further evaluate on a full setup that includes label inference the matching
of gradient updates and joint reconstruction. Pre-reconstruction runs 2,000
iterations. Gradient updates are matched based on pre-reconstructed single
images. Table VI shows that the reconstruction results based on two updates
from two epochs are better than with single updates from either epoch (e.g.,
PSNR of 21.464 instead of 20.379 or 19.085).

$10^{0}$$10^{1}$$10^{2}$$10$$12$$14$$16$$18$$\beta$: last layer weight / first
layer weightPSNRuntrained, with $z_{i}$ from ReLUuntrained5 epochs10
epochstrained Figure 9: Effect of different layer weights on PSNR with
different experimental settings, including untrained and trained networks.
Figure 10: Reconstructions of a CIFAR-10 image of a horse from untrained
ResNet20-4 with a weight ratio $\beta$ that ranges from 1 (left) to 20
(right).

###  V-E Reconstruction quality improvement with layer weights

We investigate the effect of layer weights on reconstruction quality by
varying the ratio $\beta$ between the weights of the last convolutional layer
and the first one, using ResNet20-4 on CIFAR-10. Figure 9 presents our
results. The effect of layer weights is very significant on the untrained
network. As training continues, the effect of weights weakens, and the best
$\beta$ value is also lower. However, assigning larger weights to later
convolutional layers ($\beta>1$) still improves performance, compared with the
equally assigned weights at $\beta=1$, even in trained networks. Figure 10
shows the reconstructed images of one CIFAR-10 sample on untrained ResNet20-4,
with increasing $\beta$ values. One can notice that the reconstruction quality
improves when $\beta$ increases.

Figure 9 also compares attacking untrained ResNet20-4 with and without ReLU
weight modifier $z_{i}$. The zero-proportion-based weight modifier
consistently improves the attack performance, bringing up to 2.0% improvement
on PSNR.

TABLE VII: Comparison of AGIC and other gradient inversion attacks on (horizontal) federated learning systems.  Method |  Main loss term |  Other loss terms |  Optimizer |  Label inference capability |  Extra knowledge |  Trained network |  Multiple epochs’ updates |  FedAvg
---|---|---|---|---|---|---|---|---
DLG [9] |  L2 |  None |  L-BFGS |  No use |  No |  $\times$ |  $\times$ |  $\times$
iDLG [13] |  L2 |  None |  L-BFGS |  One |  No |  $\times$ |  $\times$ |  $\times$
Inverting Gradients (InvG in this paper) [10] |  Cosine |  Total variation |  Adam |  One |  No |  $\checkmark$ |  $\times$ |  $\checkmark$
CPL [27] |  L2 |  Label-based |  L-BFGS |  One |  No |  $\times$ |  $\times$ |  $\times$
R-GAP [28] |  Recursively |  N/A |  N/A |  One |  No |  $\times$ |  $\times$ |  $\times$
Grad-Inversion [11] |  L2 |  Total variation, L2 of input, BN, group consistency |  Adam |  Multiple |  BN statistics, pretrained image alignment model |  $\checkmark$ |  $\times$ |  $\times$
Geng et al. [14] |  L2 |  Clipping, scaling |  L-BFGS |  Multiple |  No |  $\times$ |  $\times$ |  Only full batch
GIAS [12] |  Cosine |  Total variation |  Adam |  One |  Pretrained generative model |  $\times$ |  $\times$ |  $\times$
AGIC (this work) |  Cosine |  Total variation |  Adam |  Multiple |  No |  $\checkmark$ |  $\checkmark$ |  $\checkmark$

##  VI Related Work

###  VI-A Gradient inversion attack on gradient updates

Gradient inversion attacks on federated learning systems reconstruct training
samples from gradient updates. Deep Leakage from Gradients (DLG) [9] is the
first gradient inversion attack. DLG jointly optimizes on the samples and on
the labels, which complicates the optimization objective and impairs the
reconstruction quality if the labels are inaccurately optimized. Improved DLG
(iDLG) [13] proposes to conduct label inference before DLG’s optimization and
shows that labels can be inferred with perfect accuracy using some analytical
method when the batch size is equal to 1. Following iDLG, several label
inference methods explained how to infer labels in larger batches with higher
accuracy [11, 14, 29]. Label inference is now an essential part of current
gradient inversion attacks since it significantly improves the reconstruction
quality.

Most existing gradient inversion attacks are optimization-based: they
formulate an optimization problem and minimize the distance between the real
observed gradient update and a dummy gradient update in order to optimize the
reconstructed samples. Therefore, one way to improve the performance of a
gradient inversion attack is to design a better objective function. Geiping et
al. uses the cosine distance instead of L2 distance in DLG [10]. CPL adds a
label-based regularization term to improve the optimization stability [27].
Another way to improve the reconstruction performance is to leverage prior
knowledge in specific scenarios. GradInversion adds a batch normalization
regularizer, because the attacker is assumed to know batch normalization
statistics, and it designs a group consistency regularization term leveraging
multiple reconstruction processes with different initialization seeds to find
an enhanced reconstruction result [11]. GIAS assumes a known data distribution
and improves the reconstruction quality using a pre-trained generative model
[12].

R-GAP is a gradient inversion attack that is not based on optimization.
Instead, R-GAP recursively reconstructs each layer’s input from the last layer
to first by solving linear equations, which is limited to a batch size equal
to 1 [28]. The works we have discussed so far consider horizontal FL systems
[30, 3] where client datasets share a given feature space. Differently, CAFE
is a gradient inversion attack [31] for vertical FL systems where client
datasets have different feature spaces [32, 33].

Table VII compares our attack, AGIC, with existing gradient inversion attacks,
and shows that it is applicable in more general federated learning scenarios.

###  VI-B Gradient inversion attack on FedAvg model updates

Although using FedAvg model updates is more practical than using gradient
updates, only a handful of gradient inversion attacks on FedAvg’s model
updates have been described.

To be compatible with the multiple local steps of FedAvg, Geiping et al.
simulate the local training process of multiple local steps to obtain a dummy
model update, and then minimize the distance between the dummy model update
and the real model update [10]. However, this method is incompatible with
label inference because it is unable to associate correct labels to each mini-
batch, as label inference returns a set of labels for all mini-batches. In
addition, the computation time of simulation-based attacks also increases with
the number of local steps FedAvg uses, because the computation graph for a
model update then also gets larger [10]. Geng et al. present an attack that
targets FedAvg systems that use full batch gradient descent [14]. Their attack
divides the received model update by the number of local steps to approximate
the model update of each local step, which is then used to approximate the
full batch’s gradient update. However, this attack cannot cover the common
case where FedAvg uses multiple mini-batches.

AGIC addresses the limitations of previous attacks on FedAvg. First, its one-
batch approximation is compatible with label inference. Second, AGIC can be
used in common FedAvg scenarios that use several mini-batches per round, and
it is faster than simulation-based attacks.

##  VII Conclusion

In this paper, we presented AGIC, an accurate and fast gradient inversion
attack that can leverage both model updates and gradient updates across
multiple epochs. AGIC uses a one-batch approximation to convert model updates
into approximate gradient updates of a larger mini-batch. AGIC also leverages
updates from multiple epochs with update matching to jointly reconstruct
specific training samples accurately. Finally, AGIC assigns different weights
to layers in the objective function, which significantly improves the
reconstruction quality. Compared to previous works, AGIC can be used with more
general federating learning system settings. Our experiments demonstrate that
AGIC reconstructs samples with up to 50% PSNR improvement compared to state-
of-the-art baselines, and is up to 5x faster than simulation-based attacks,
which is the only baseline that can attack FedAvg.

## References

  * [1] Keith Bonawitz et al.  “Towards federated learning at scale: System design”  In _Proceedings of Machine Learning and Systems_ 1, 2019, pp. 374–388
  * [2] Keith Bonawitz et al.  “Practical secure aggregation for privacy-preserving machine learning”  In _ACM SIGSAC CCS_ , 2017
  * [3] Jakub Konečnỳ et al.  “Federated learning: Strategies for improving communication efficiency”  In _arXiv preprint arXiv:1610.05492_ , 2016
  * [4] Brendan McMahan et al.  “Communication-efficient learning of deep networks from decentralized data”  In _Artificial intelligence and statistics_ , 2017, pp. 1273–1282  PMLR
  * [5] H Brendan McMahan, Eider Moore, Daniel Ramage and Blaise Agüera Arcas  “Federated learning of deep networks using model averaging”  In _arXiv preprint arXiv:1602.05629_ Technical report, 2016
  * [6] Luca Melis, Congzheng Song, Emiliano De Cristofaro and Vitaly Shmatikov  “Exploiting unintended feature leakage in collaborative learning”  In _2019 IEEE Symposium on Security and Privacy (SP)_ , 2019, pp. 691–706  IEEE
  * [7] Briland Hitaj, Giuseppe Ateniese and Fernando Perez-Cruz  “Deep models under the GAN: information leakage from collaborative deep learning”  In _Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security_ , 2017, pp. 603–618
  * [8] Zhibo Wang et al.  “Beyond inferring class representatives: User-level privacy leakage from federated learning”  In _IEEE INFOCOM 2019-IEEE Conference on Computer Communications_ , 2019, pp. 2512–2520  IEEE
  * [9] Ligeng Zhu and Song Han  “Deep leakage from gradients”  In _Federated learning_ Springer, 2020, pp. 17–31
  * [10] Jonas Geiping, Hartmut Bauermeister, Hannah Dröge and Michael Moeller  “Inverting gradients-how easy is it to break privacy in federated learning?”  In _Advances in Neural Information Processing Systems_ 33, 2020, pp. 16937–16947
  * [11] Hongxu Yin et al.  “See through Gradients: Image Batch Recovery via GradInversion”  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 16337–16346
  * [12] Jinwoo Jeon et al.  “Gradient inversion with generative image prior”  In _Thirty-Fifth Conference on Neural Information Processing Systems_ , 2021
  * [13] Bo Zhao, Konda Reddy Mopuri and Hakan Bilen  “idlg: Improved deep leakage from gradients”  In _arXiv preprint arXiv:2001.02610_ , 2020
  * [14] Jiahui Geng et al.  “Towards General Deep Leakage in Federated Learning”  In _arXiv preprint arXiv:2110.09074_ , 2021
  * [15] Tien-Ju Yang, Yu-Hsin Chen and Vivienne Sze  “Designing energy-efficient convolutional neural networks using energy-aware pruning”  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 5687–5695
  * [16] Sara Elkerdawy et al.  “To filter prune, or to layer prune, that is the question”  In _Proceedings of the Asian Conference on Computer Vision_ , 2020
  * [17] Nicola Rieke et al.  “The future of digital health with federated learning”  In _NPJ digital medicine_ 3.1 Nature Publishing Group, 2020, pp. 1–7
  * [18] Leonid I Rudin, Stanley Osher and Emad Fatemi  “Nonlinear total variation based noise removal algorithms”  In _Physica D: nonlinear phenomena_ 60.1-4 Elsevier, 1992, pp. 259–268
  * [19] Priya Goyal et al.  “Accurate, large minibatch sgd: Training imagenet in 1 hour”  In _arXiv preprint arXiv:1706.02677_ , 2017
  * [20] Chelsea Finn, Pieter Abbeel and Sergey Levine  “Model-agnostic meta-learning for fast adaptation of deep networks”  In _International conference on machine learning_ , 2017, pp. 1126–1135  PMLR
  * [21] Shi Chen and Qi Zhao  “Shallowing deep networks: Layer-wise pruning based on feature representations”  In _IEEE transactions on pattern analysis and machine intelligence_ 41.12 IEEE, 2018, pp. 3048–3056
  * [22] Kaiming He, Xiangyu Zhang, Shaoqing Ren and Jian Sun  “Deep residual learning for image recognition”  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778
  * [23] Alex Krizhevsky and Geoffrey Hinton  “Learning Multiple Layers of Features from Tiny Images”  In _MSc thesis, University of Toronto_ , 2009
  * [24] Olga Russakovsky et al.  “Imagenet large scale visual recognition challenge”  In _International journal of computer vision_ 115.3 Springer, 2015, pp. 211–252
  * [25] Zhou Wang, Alan C Bovik, Hamid R Sheikh and Eero P Simoncelli  “Image quality assessment: from error visibility to structural similarity”  In _IEEE transactions on image processing_ 13.4 IEEE, 2004, pp. 600–612
  * [26] Richard Zhang et al.  “The unreasonable effectiveness of deep features as a perceptual metric”  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 586–595
  * [27] Wenqi Wei et al.  “A framework for evaluating client privacy leakages in federated learning”  In _European Symposium on Research in Computer Security_ , 2020  Springer
  * [28] Junyi Zhu and Matthew Blaschko  “R-gap: Recursive gradient attack on privacy”  In _arXiv preprint arXiv:2010.07733_ , 2020
  * [29] Trung Dang et al.  “Revealing and Protecting Labels in Distributed Training”  In _Advances in Neural Information Processing Systems_ 34, 2021
  * [30] Murat Kantarcioglu and Chris Clifton  “Privacy-preserving distributed mining of association rules on horizontally partitioned data”  In _IEEE transactions on knowledge and data engineering_ 16.9 IEEE, 2004, pp. 1026–1037
  * [31] Xiao Jin et al.  “Catastrophic Data Leakage in Vertical Federated Learning”  In _Advances in Neural Information Processing Systems_ 34, 2021
  * [32] Jaideep Vaidya and Chris Clifton  “Privacy preserving association rule mining in vertically partitioned data”  In _Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining_ , 2002, pp. 639–644
  * [33] Stephen Hardy et al.  “Private federated learning on vertically partitioned data via entity resolution and additively homomorphic encryption”  In _arXiv preprint arXiv:1711.10677_ , 2017


# TieBot: Learning to Knot a Tie from Visual Demonstration through a Real-to-
Sim-to-Real Approach

Weikun Peng1, Jun Lv2, Yuwei Zeng1, Haonan Chen3, Siheng Zhao3,
Jichen Sun2, Cewu Lu2, Lin Shao1†
1 School of Computing, National University of Singapore
2Department of Computer Science, Shanghai Jiao Tong University
3Department of Computer Science and Technology, Nanjing University
$\dagger$ Corresponding author<EMAIL_ADDRESS>
###### Abstract

The tie-knotting task is highly challenging due to the tie’s high deformation
and long-horizon manipulation actions. This work presents _TieBot_ , a Real-
to-Sim-to-Real learning from visual demonstration system for the robots to
learn to knot a tie. We introduce the Hierarchical Feature Matching approach
to estimate a sequence of tie’s meshes from the demonstration video. With
these estimated meshes used as subgoals, we first learn a teacher policy using
privileged information. Then, we learn a student policy with point cloud
observation by imitating teacher policy. Lastly, our pipeline learns a
residual policy when the learned policy is applied to real-world execution,
mitigating the Sim2Real gap. We demonstrate the effectiveness of _TieBot_ in
simulation and the real world. In the real-world experiment, a dual-arm robot
successfully knots a tie, achieving 50% success rate among 10 trials. Videos
can be found on our website.

##  1 Introduction

Learning cloth manipulation holds great utility across a wide range of
applications. One intriguing domain is robotic tie knotting. Service robots
must be adept at tasks like aiding the elderly or individuals with
disabilities in dressing for certain social events. Teaching robots to knot
ties, as a special case of cloth manipulation, typically pushes the limits of
robotic cloth manipulation. This offers valuable insights for tie knotting and
the broader field of robotic cloth manipulation.

Cloth manipulation presents challenges for robots due to its high-dimensional
state and complex dynamics. Extracting and modeling state information are
difficult problems. In contrast, humans have accumulated extensive knowledge
about cloth manipulation. These priors make learning from demonstration (LfD)
a promising direction. LfD empowers a robot to acquire a policy from expert
demonstrations, significantly reducing the need to design task-specific reward
functions manually. Consequently, LfD stands as a potent and efficient
framework for instructing robots in the execution of complex skills.

However, existing LfD methods struggle with tie-knotting tasks. Kinesthetic
demonstration or teleoperation suffers from the complexity of tie-knotting
tasks. Tie-knotting tasks require bi-manual operations, placing high demand on
human operators’ skills and equipment. For instance, Zhang et. al use VR
headsets for teleoperation [1]. Thus, simple behavior cloning may be
significantly labor-intensive. Learning from visual demonstration is usually
an easier approach in terms of collecting demonstration data. But this
approach also leads to embodiment gaps. Therefore, researchers attempt to find
some object-centric representations that robots can utilize to generate
correct actions, overcoming embodiment gaps. Several methods attempt to learn
a general visual representation of some simple pick-place skills via large-
scale pre-training on actionless videos [2, 3, 4]. These works present strong
generalizations on the learned visual representations, but none of them shows
the ability to learn dexterous manipulation skills that can knot a tie. Other
methods such as [5, 6, 7, 8] try to leverage object trajectories or keypoints
as representations to guide the policy learning. Such representations are
indeed sufficient to describe simple object motions but fail to capture the
tie’s complex topology and subtle dynamics.

Figure 1: Our proposed _TieBot_ performs a tie-knotting task. We leverage
cloth simulation to recover the cloth’s state from human demonstration and
learn a goal-condition policy to accomplish the tie-knotting task.

Compared to the existing LfD work mentioned in the previous paragraph, our
insight is that mesh is the most suitable representation for tie-knotting
tasks and other complex cloth manipulation tasks. It captures accurate
geometric structures and physics properties of the tie, which is crucial for
tie-knotting tasks. It also disentangles irrelevant information in the visual
demonstrations, such as environment background, object colors, and so forth,
enabling the learned policy to apply to different test settings. Therefore, we
propose a Real-to-Sim-to-Real LfD framework. First, we propose a Hierarchical
Feature Matching method to iteratively estimate the tie’s meshes with cloth
simulation from the demonstrated video. We use a cloth simulator called
_DiffClothAI_ [9] that supports intersection-free contact for cloth to
maintain the tie’s topological structure during the estimation process. These
estimated meshes from the demonstrated video are then used as subgoals. To
learn where to grasp the tie and where to pull the tie from point clouds
observations in simulation, we adopt a teacher-student training paradigm
similar to [10]. Lastly, our pipeline learns a residual policy when applying
the policy to real-world settings, mitigating the sim2real gap.

In summary, we make the following contributions: 1) We introduced a systematic
LfD framework for a dual-arm robot to learn to knot to tie. 2) We proposed a
Hierarchical Feature Matching approach to estimate the tie’s mesh with high
deformation from the demonstrated RGB-D video using cloth simulation. 3) With
estimated meshes as subgoals, we presented a teacher-student training paradigm
to learn grasping points and placing points from point cloud observations in
simulation. 4) We conduct experiments in simulation and the real-world to
demonstrate the effectiveness and advances of our pipeline. To the best of our
knowledge, this work is the first effort to develop a robot system,
integrating perception, modeling, and robot learning. It successfully complete
the tie-knotting task, which is one of the most complex and challenging cloth
manipulation tasks.

##  2 Related Work

###  2.1 Cloth Manipulation

Previous work mainly addresses short-horizon cloth manipulation tasks that
only involve simple pick-place actions. There are several approaches to
learning cloth manipulation skills. One approach is using model-free RL or
learned dynamics model to learn cloth unfolding, rope rearranging, and
dressing assistance tasks on raw sensor input [11, 12, 13, 14, 15]. Other
approaches will collect and annotate data from images [16, 17] or generate
demonstration trajectories in simulation [18] to learn policy. Because of the
short-horizon and simple actions features of tasks, it’s also possible to
infer correct actions from some visual representations, such as flow between
current observation and target images [19].

In contrast, tie-knotting tasks require flipping or rotating a part of the
tie, which makes it difficult to annotate robot actions or design action
primitives. Therefore, collecting and annotating robot actions on observations
is infeasible. It’s also difficult to generate demonstrations or directly
apply RL in simulation since the trajectories of tie-knotting tasks are much
longer and the possible state space is much larger. Thus, in this work, we
choose to learn skills from human demonstration.

###  2.2 Learning from Visual Demonstration

One line of research explores pre-training neural representations from
actionless videos [20, 21, 2, 22, 3, 4, 23]. This approach aims at learning
general representations for different actions, whereas none of them shows the
ability to learn dexterous manipulation skills that can knot a tie. Another
line of research attempts to learn from visual priors extracted from visual
demonstrations, such as object trajectories [5, 24], hand poses [25, 26],
keypoints positions [27, 6], graph relations [7], or affordances [28]. The
third approach is to learn a video or trajectory prediction model to guide
policy learning [29, 30, 31, 32, 8]. These approaches require in-domain
demonstrations, placing restrictions on visual demonstrations. Moreover, the
prediction model may suffer from the long-horizon feature of tie-knotting
tasks. ORION is the most closely related work, which builds a graph
representation from object motions that can generalize across diverse test
environments [33]. However, simple graph representations cannot capture the
tie’s complex topology and subtle dynamics during the tie-knotting process.

Consequently, we propose explicitly modeling the demonstration as a sequence
of meshes. Mesh can accurately describe the tie’s structure and dynamics,
which is crucial to learning correct robot actions and generalizing them to
different test scenarios.

###  2.3 Cloth State Estimation

One cloth state estimation method directly predicts cloth states using deep
learning [34, 35, 36]. Non-rigid point cloud registration methods such as
coherent point drifting are also applied for linear deformable object tracking
[37, 38, 39]. However, purely vision-based methods do not guarantee correct
cloth topology due to the lack of physics prior. Huang et al. propose a method
to reconstruct and track cloth state with a dynamics model [40]. However, this
method requires known actions, which cannot be accessed easily from human
demonstration sometimes.

Therefore, we propose a Hierarchical Feature Matching method to iteratively
estimate the tie mesh in the demonstration video with cloth simulation. Cloth
simulation provides important physics prior for state estimation, such as non-
penetration, which is crucial for maintaining correct topology.

##  3 Technical Approach

Figure 2: _TieBot_ utilizes simulation to estimate the tie’s meshes from the
demonstrated video. Then, using mesh sequences as subgoals, we introduce how
to generate the robot’s actions to manipulate the tie. The pipeline finally
learns a residual policy to reduce the sim2real gap.

This work presents a Real-to-Sim-to-Real LfD framework called _TieBot_ to
guide a dual-arm robot shown in Fig. 1 to knot the tie from an RGB-D
demonstration video. An overview of our proposed method is in Fig. 2. We first
describe the procedure to estimate the tie’s mesh sequences from the
demonstrated video (Sec. 3.1). Using the tie’s mesh sequences as subgoals, we
introduce a pipeline to generate robot actions to manipulate the tie, using
teacher-student training paradigm (Sec. 3.2). Lastly, we discuss learning
residual policy to mitigate Sim2Real gaps(Sec. 3.3).

###  3.1 Real2Sim

To better estimate the tie’s mesh, we propose to integrate cloth simulation
into our pipeline, which provides important physical prior such as non-
penetration in the estimation process. We segment the tie in the demonstrated
RGB-D video using _Track-Anything_ [41] and transform the associated segmented
depth images into point clouds. Meanwhile, a tie’s mesh is loaded into the
_DiffClothAI_. At time step $t$, we use the tie mesh’s vertices denoted as
$\mathcal{X}^{S}_{t}$ to describe the tie’s shape. From the RGB images and
segmented point clouds denoted as $\\{\mathcal{I}^{D}_{t}\\}$ and
$\\{\mathcal{X}^{D}_{t}\\}$, Real2Sim pipeline estimates tie’s mesh sequences
$\\{\mathcal{X}^{S}_{t}\\}$ with simulation. The pipeline manually aligns the
mesh with the initial frame. We assume the initial mesh fully overlaps with
the initial point cloud.

####  3.1.1 Local Feature Matching

Figure 3: Local Feature matching between two images. A hand caused a gap along
the length of the tie during the demonstration.

If $\mathcal{X}^{S}_{t-1}$ and $\mathcal{X}^{D}_{t-1}$ are aligned and there
are correspondences between $\mathcal{X}^{D}_{t-1}$ and $\mathcal{X}^{D}_{t}$,
we can build up the correspondences from the tie’s mesh to the next
demonstrated point cloud $\mathcal{X}^{D}_{t}$ and move the tie’s vertices to
align $\mathcal{X}^{S}_{t}$ towards $\mathcal{X}^{D}_{t}$.

Here we adopt an off-the-shelf feature matching model called  _LoFTR_ [42] to
build up correspondences between two RGB images $\mathcal{I}_{t-1}$ and
$\mathcal{I}_{t}$ as shown in Fig 3. Typically, _LoFTR_ can provide more than
a hundred reliable correspondences between two images, which cover almost
every visible part of the tie. From the correspondences between
$\mathcal{I}_{t-1}$ and $\mathcal{I}_{t}$, we can find the feature points on
$\mathcal{X}^{D}_{t-1}$ and their corresponding feature points on
$\mathcal{X}^{D}_{t}$. Then, to control the mesh in _DiffClothAI_ to align it
with $\mathcal{X}^{D}_{t}$, we need to define several vertices on the mesh as
control vertices $\mathcal{V}$. Since $\mathcal{X}^{S}_{t-1}$ aligns well with
$\mathcal{X}^{D}_{t-1}$, we map the feature points on $\mathcal{X}^{D}_{t-1}$
to nearest vertices on $\mathcal{X}^{S}_{t-1}$. These vertices are assigned as
control vertices $\mathcal{V}$. Finally, we control $\mathcal{V}$ to move to
the positions of feature points on $\mathcal{X}_{t}^{D}$ to align
$\mathcal{X}_{t}^{S}$ towards $\mathcal{X}^{D}_{t}$ in _DiffClothAI_.

However, vanilla local feature matching cannot create correspondences in
occluded regions, which is common in tie-knotting tasks. We lose the motion
information due to occlusion, and the estimation will deviate. Therefore, we
propose to add global keypoints information to amend this pipeline.

####  3.1.2 Keypoints Detection

Figure 4: The oriented keypoints to represent the state of the tie. The x,y,z
axis are represented by the red, green, blue arrow, respectively.

Keypoints detection can directly build correspondences between mesh vertices
and the point cloud. Thus, it will not be affected by occlusion. We define
five keypoints along the tie’s surface and the corresponding five key vertices
on the mesh, shown in Fig. 4. For each keypoint as the origin, we define the
local frame as follows. The z direction is the surface normal from the tie’s
positive side to the negative side. The x direction is the direction of the
tie’s middle skeleton. The y direction is derived using the right-hand rule.
These five keypoints, in a predefined order, play the role of the skeleton
definition.

Then, we train Pointnet++ [43] to predict the keypoints and associated local
frames on the demonstrated point clouds. However, the high-dimensional state
makes it challenging to generate sufficient training data to cover all the
states encountered in the knotting procedure. A successful tie-knotting
trajectory occupies only a small portion of the whole state space of the tie.
Thus, uniformly applying random actions on the initial tie’s mesh in the
simulation to produce training data fails to cover these states. In contrast,
we generate training data based on the current estimated mesh. When we detect
the chamfer distance between $\mathcal{X}^{S}_{t}$ and $\mathcal{X}^{D}_{t}$
is larger than a threshold, we backtrack to the previous time step, gather the
tie’s shape $\mathcal{X}_{t-1}$ and apply random actions to the tie’s mesh at
$t-1$ in the simulation to generate annotated training data and train the
keypoints prediction network.

####  3.1.3 Hierarchical Feature Matching(HFM)

Finally, we combine them as Hierarchical Feature Matching(HFM) for state
estimation. Control vertices assigned in local feature matching and key
vertices will be used together to pull the mesh to target positions specified
by local feature matching and keypoints detection. Local feature matching
provides detailed motion of vertices, while global keypoints indicate a global
tie’s structure. This global structural information ensures the estimation
won’t deviate too much, alleviating the shortcomings of the local feature
matching method. We use this method to estimate the tie’s meshes from
demonstration and output a sequence of meshes
$\\{\mathcal{X}_{t}^{\mathcal{G}}\\}$. The next part will use these meshes as
subgoals to guide robot action generation.

###  3.2 Learn@Sim

Our pipeline begins to sequentially generate feasible robotic actions in the
simulation to guide the tie $\\{\mathcal{X}^{S}_{t}\\}$ towards these
subgoals. Since the tie-knotting task is a long-horizon task with multiple
grasp and pull actions, we aim to learn where to grasp the tie and where to
pull the tie.

For where to pull, we apply a simple strategy. Once we identify the grasping
vertices, we pull these vertices to the positions of those vertices on the
subgoal. For where to grasp, we adopt a similar teacher-student training
paradigm in [10] to ease policy learning. Directly learning from high
dimensional observations such as point cloud is data-inefficient because the
policy needs to simultaneously learn which features to extract from visual
observations and what the high-rewarding actions are. On the contrary,
learning a policy via RL from sufficient state information would be much
easier, as suggested by [10]. Therefore, we first use privileged information
to learn a teacher policy, and then train a student policy imitating teacher
policy with point clouds as observations.

##### Teacher Policy

We first learn a teacher policy to select proper grasping points using
privileged information. The state $s$ contains the previous tie’s vertices
positions and the point-wise displacement for each tie vertices to the
subgoal. The action $a$ is one or two grasping vertices of all the tie mesh’s
vertices. The reward function $\mathcal{R}$ is defined in equation 1.

Note that we specify the action space as the discrete space (vertex index of
the tie). Although there are multiple 6D poses of the robotic grippers to
grasp one vertex position of the mesh, the learned policy still reflects the
overall grasping quality of these 6D poses associated with one vertex. In the
engineering practice, we record each grasping pose offline so that once we
figure out the grasping vertices on the tie’s mesh at each timestep, we can
automatically produce the feasible grasping poses concerning specific hardware
platforms using inverse kinematics.

$\mathcal{R}(s,a)=\begin{cases}C_{1},&\text{if knotting-tie succeeds}\\\
-C_{2},&\text{if fails to reach any subgoal along the trajectory}\\\
C_{3}-\|\mathcal{X}^{S}_{t}-\mathcal{X}^{\mathcal{G}}_{t}\|,&\text{Otherwise}\\\
\end{cases}$ (1)

For the reward function, here $C_{1},C_{2},C_{3}$ are constant positive
values. $\mathcal{X}_{t}^{S}$ is the result tie mesh. The failure to reach the
subgoal is due to the distance
$\|\mathcal{X}^{S}_{t}-\mathcal{X}^{\mathcal{G}}_{t}\|$ is larger than a given
threshold, the tie could not be pulled close to the subgoal by grasping on the
wrong selected vertex; otherwise, it will return
$C_{3}-\|\mathcal{X}^{S}_{t}-\mathcal{X}^{\mathcal{G}}_{t}\|$ for intermediate
steps or $C_{1}$ for the final step.

##### Student Policy

To learn actions from point clouds, we train a student policy to imitate
teacher policy. We add some perturbations to the size and positions of the
mesh and update the associated trajectories accordingly to generate training
data in the simulation. We render point clouds from meshes in PyBullet [44] as
the input of our policy network $\pi^{sim}$ and output the grasping points and
placing points positions. We use Pointnet++ [43] as the policy network and
train it in a supervised learning manner.

###  3.3 Sim2Real

When the robot knots the tie in the real world, the robot receives a segmented
point cloud of the tie denoted as $\mathcal{X}^{real}_{t}$ at each time step.
Instead of directly applying the action $\pi^{sim}(\mathcal{X}^{real}_{t})$ in
the real world, we train a residual policy that takes in the real point cloud
$\mathcal{X}^{real}_{t}$ and the output of
$\pi^{sim}(\mathcal{X}^{real}_{t})$, outputs small offsets to the positions of
predicted grasping points and placing points. Combing the small offsets and
predicted grasping and placing points positions, we finally generate the
action in real setting. We follow the training process of residual policy
described in [45].

##  4 Experiments

In this section, we introduce our experimental setup and conduct quantitative
and qualitative evaluations to demonstrate the effectiveness of our approach.
Our experiments focus on answering the following questions.

  * •

How do our pipeline and baseline methods perform on tie-knotting task?

  * •

How does HFM compare to other cloth state estimation methods?

  * •

Can our HFM apply to other cloth manipulation tasks?

Considering the complexity of the entire system, we provide additional
experiment results, along with detailed explanations of submodules, in the
supplementary materials and website.

###  4.1 Comparing _TieBot_ and baseline

We first evaluate the whole pipeline of _TieBot_ and a baseline method in a
tie-knotting task. We estimate a sequence of meshes from one human
demonstration video. Then, we divide the whole trajectory into 6 parts with 6
subgoals. Our teacher policy learns to select proper grasping points using PPO
[46], and student policy imitates the teacher policy to infer grasping points
and placing points from the point cloud. We evaluate _TieBot_ and the baseline
method 10 times for each of the two different ties in _DiffClothAI_ and
evaluate _TieBot_ on two real ties with a dual-arm robot.

##### Metrics

We compare the success rate between our pipeline and ATM [8]. In simulation
experiments, if the distance of the final tie’s mesh to the target tie’s mesh
is smaller than a threshold, we consider it a success. In real-world
experiments, if the little end of the tie is inserted into the hole, as shown
in the final stage in Fig. 5, we consider it a success. Since the tie-knotting
task is long-horizon, we also compute the averaged number of achieved subgoals
for further evaluation.

##### ATM

ATM proposes to model tasks as points trajectories [8]. It first learns a
trajectory prediction model, and then learns policy with the learned
prediction model using imitation learning. Following similar experiment
settings in ATM, we collect 100 demonstration videos in simulation to train
the trajectory prediction module. Then, we use the 45 demonstration videos
with ground truth action annotations to train the policy network and test the
policy in simulation. The action is the 3D offset of the grasping vertices.

Success Rate / Average Achieved Subgoals | Ours | ATM
---|---|---
normal tie | 60% / 5.1 | 0% / 0.0
larger tie(unseen) |  30% / 4.3 | 0% / 0.0
real tie1(softer) | 50% / 5.0 | NA / NA
real tie2(harder, unseen) | 30% / 4.15 | NA / NA
Table 1: Success rate and average achieved subgoals of policy rollouts

##### Experiment Result

We test _TieBot_ on two real ties that differ in materials: one is softer, and
the other one is harder. We tested each of them 10 times. We also test
_TieBot_ and ATM on two ties with different sizes in simulation 10 times for
each. The quantitative results are shown in Tab. 1, and qualitative results of
_TieBot_ are shown in Fig. 5. This comparison suggests that object
trajectories are insufficient to represent subtle dynamics and topology of the
tie in tie-knotting tasks. ATM quickly deviates from the correct trajectory
since it cannot capture the subtle dynamics of the tie. Therefore, it fails to
achieve even one subgoal. This comparison suggests that explicitly modeling
the tie in meshes is necessary. For qualitative evaluation, in Fig. 5, we can
see that although the tie in the demonstration video, the mesh in the
simulation, and the tie used for real robot manipulation are different, our
policy can overcome these gaps and learn feasible robot policy.

Figure 5: The results of _TieBot_ at different stages. We show different sides
of the tie in red and blue and manipulation action in yellow to better
visualize.

###  4.2 Evaluating Hierarchical Feature Matching(HFM)

Real2Sim is the most important part of our pipeline. Without accurate state
estimation, particularly estimating the correct topology for the tie, it’s
impossible to learn feasible policy to finish the task. To illustrate the
importance of different components of HFM and its performance against other
cloth state estimation methods, we design three experiments in simulation to
test baseline methods and the ablation versions of HFM.

##### Coherent Point Drift

Coherent Point Drift(CPD) is a non-rigid point cloud registration algorithm.
We employ the CPD to predict the target positions of the mesh vertices in the
target point cloud and directly align the mesh to the target positions.

##### Ablated Version

Ours w/o KP stands for only using local feature matching; Ours w/o LF stands
for using local feature matching and the predicted keypoints positions; Ours
w/o FM stands for only using predicted keypoints positions and local frames.

##### Experiment Result

The qualitative results are shown in Fig. 6. We can find that either CPD or
ablated versions of HFM cannot estimate the target mesh correctly among these
three experiments. We also compute the L2 Distance between the vertices of the
target mesh and estimated mesh as a quantitative evaluation shown in Tab. 2.
It also suggests that the performance will degrade if we cancel some parts of
HFM, while CPD deviates a lot from the correct states.

L2 Distance | Ours | Ours w/o FM | Ours w/o KP | Ours w/o LF | CPD
---|---|---|---|---|---
exp1 | 0.0248 | 0.0732 | 0.2424 | 0.0512 | 0.1384
exp2 | 0.0053 | 0.0107 | 0.0123 | 0.0088 | 0.0661
exp3 | 0.0032 | 0.0093 | 0.0053 | 0.0049 | 0.1049
Table 2: Quantitative results of ablation study and comparison to CPD in
simulation Figure 6: The visualization of the ablation study of HFM in
simulation. We put a cross sign in the image’s bottom-right corner to indicate
failures of estimating the correct target state, according to human
evaluation. Red and blue colors represent different sides of the mesh.

###  4.3 Apply HFM on Other Cloth Manipulation Tasks

We demonstrate that HFM can be applied to other cloth manipulation tasks. One
is a different way to knot a tie. The other one is to fold a towel. We
visualize the estimation results in Fig. 7. The results show that HFM can be
applied to different cloth manipulation tasks.

Figure 7: The visualization of a differentway to knot a tie and the towel
folding. The fist row is the human demonstration of tie-knotting, the second
row is the estimated states in simulation. While the third row is showing the
towel folding. To better visualize, we show the manipulation action in yellow
dots and arrows.

##  5 Conclusion

This work introduces a Real-to-Sim-to-Real LfD framework called _TieBot_ for
the robots to learn to knot a tie from visual demonstration. _TieBot_
introduces the Hierarchical Feature Matching approach to iteratively estimate
a sequence of tie’s meshes from the demonstrated video. _TieBot_ adopts a
teacher-student training paradigm to learn grasping points and placing points
from point clouds. Lastly, our pipeline learns a residual policy when the
imitated policy is applied to real-world execution, mitigating the Sim2Real
gap. We demonstrate that a dual-arm robot successfully knots the tie with our
framework, achieving 50% success rate over 10 trials.

Nonetheless, our pipeline has some limitations. First, our pipeline still
requires manually setting the initial state of the tie at the beginning of the
Real2Sim stage. Second, our Real2Sim module requires training keypoints
detection models iteratively, which takes a lot of time. Third, due to the
hardware limits, the last step in the real-world experiments shown in Fig. 5
is hardcode action. Better cloth mesh reconstruction methods, video tracking
methods, and more dexterous robot arms may alleviate these issues.

## References

  * Zhang et al. [2018] T. Zhang, Z. McCarthy, O. Jow, D. Lee, X. Chen, K. Goldberg, and P. Abbeel.  Deep imitation learning for complex manipulation tasks from virtual reality teleoperation.  In _2018 IEEE International Conference on Robotics and Automation (ICRA)_ , pages 5628–5635, 2018.  doi:10.1109/ICRA.2018.8461249.
  * Nair et al. [2022] S. Nair, A. Rajeswaran, V. Kumar, C. Finn, and A. Gupta.  R3m: A universal visual representation for robot manipulation.  _arXiv preprint arXiv:2203.12601_ , 2022.
  * Radosavovic et al. [2023] I. Radosavovic, T. Xiao, S. James, P. Abbeel, J. Malik, and T. Darrell.  Real-world robot learning with masked visual pre-training.  In _Conference on Robot Learning_ , pages 416–426. PMLR, 2023.
  * Shaw et al. [2023] K. Shaw, S. Bahl, and D. Pathak.  Videodex: Learning dexterity from internet videos.  In _Conference on Robot Learning_ , pages 654–665. PMLR, 2023.
  * Seita et al. [2022] D. Seita, Y. Wang, S. Shetty, E. Li, Z. Erickson, and D. Held.  ToolFlowNet: Robotic Manipulation with Tools via Predicting Tool Flow from Point Clouds.  In _Conference on Robot Learning (CoRL)_ , 2022.
  * Das et al. [2021] N. Das, S. Bechtle, T. Davchev, D. Jayaraman, A. Rai, and F. Meier.  Model-based inverse reinforcement learning from visual demonstrations.  In _Conference on Robot Learning_ , pages 1930–1942. PMLR, 2021.
  * Kumar et al. [2023] S. Kumar, J. Zamora, N. Hansen, R. Jangir, and X. Wang.  Graph inverse reinforcement learning from diverse videos.  In _Conference on Robot Learning_ , pages 55–66. PMLR, 2023.
  * Wen et al. [2023] C. Wen, X. Lin, J. So, K. Chen, Q. Dou, Y. Gao, and P. Abbeel.  Any-point trajectory modeling for policy learning, 2023.
  * Yu et al. [2023] X. Yu, S. Zhao, S. Luo, G. Yang, and L. Shao.  Diffclothai: Differentiable cloth simulation with intersection-free frictional contact and differentiable two-way coupling with articulated rigid bodies.  In _2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2023.
  * Chen et al. [2023] T. Chen, M. Tippur, S. Wu, V. Kumar, E. Adelson, and P. Agrawal.  Visual dexterity: In-hand reorientation of novel and complex object shapes.  _Science Robotics_ , 8(84):eadc9244, 2023.  doi:10.1126/scirobotics.adc9244.  URL https://www.science.org/doi/abs/10.1126/scirobotics.adc9244.
  * Ha and Song [2022] H. Ha and S. Song.  Flingbot: The unreasonable effectiveness of dynamic manipulation for cloth unfolding.  In _Conference on Robot Learning_ , pages 24–33. PMLR, 2022.
  * Wu et al. [2019] Y. Wu, W. Yan, T. Kurutach, L. Pinto, and P. Abbeel.  Learning to manipulate deformable objects without demonstrations.  _arXiv preprint arXiv:1910.13439_ , 2019.
  * Deng et al. [2022] Y. Deng, C. Xia, X. Wang, and L. Chen.  Deep reinforcement learning based on local gnn for goal-conditioned deformable object rearranging.  In _2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , pages 1131–1138. IEEE, 2022.
  * Wang et al. [2023] Y. Wang, Z. Sun, Z. Erickson, and D. Held.  One policy to dress them all: Learning to dress people with diverse poses and garments.  In _Robotics: Science and Systems (RSS)_ , 2023.
  * Yan et al. [2020] M. Yan, Y. Zhu, N. Jin, and J. Bohg.  Self-supervised learning of state estimation for manipulating deformable linear objects.  _IEEE robotics and automation letters_ , 5(2):2372–2379, 2020.
  * Avigal et al. [2022] Y. Avigal, L. Berscheid, T. Asfour, T. Kröger, and K. Goldberg.  Speedfolding: Learning efficient bimanual folding of garments.  In _2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , pages 1–8. IEEE, 2022.
  * Seita et al. [2019] D. Seita, N. Jamali, M. Laskey, A. K. Tanwani, R. Berenstein, P. Baskaran, S. Iba, J. Canny, and K. Goldberg.  Deep transfer learning of pick points on fabric for robot bed-making.  In _The International Symposium of Robotics Research_ , pages 275–290. Springer, 2019.
  * Deng et al. [2023] Y. Deng, K. Mo, C. Xia, and X. Wang.  Learning language-conditioned deformable object manipulation with graph dynamics.  _arXiv preprint arXiv:2303.01310_ , 2023.
  * Weng et al. [2022] T. Weng, S. M. Bajracharya, Y. Wang, K. Agrawal, and D. Held.  Fabricflownet: Bimanual cloth manipulation with a flow-based policy.  In _Conference on Robot Learning_ , pages 192–202. PMLR, 2022.
  * Sermanet et al. [2018] P. Sermanet, C. Lynch, Y. Chebotar, J. Hsu, E. Jang, S. Schaal, S. Levine, and G. Brain.  Time-contrastive networks: Self-supervised learning from video.  In _2018 IEEE International Conference on Robotics and Automation (ICRA)_ , pages 1134–1141. IEEE, 2018.
  * Ma et al. [2022] Y. J. Ma, S. Sodhani, D. Jayaraman, O. Bastani, V. Kumar, and A. Zhang.  Vip: Towards universal visual reward and representation via value-implicit pre-training.  _arXiv preprint arXiv:2210.00030_ , 2022.
  * Xiao et al. [2022] T. Xiao, I. Radosavovic, T. Darrell, and J. Malik.  Masked visual pre-training for motor control.  _arXiv preprint arXiv:2203.06173_ , 2022.
  * Mendonca et al. [2023] R. Mendonca, S. Bahl, and D. Pathak.  Structured world models from human videos.  2023\.
  * Vecerik et al. [2023] M. Vecerik, C. Doersch, Y. Yang, T. Davchev, Y. Aytar, G. Zhou, R. Hadsell, L. Agapito, and J. Scholz.  Robotap: Tracking arbitrary points for few-shot visual imitation.  _arXiv preprint arXiv:2308.15975_ , 2023.
  * Bahl et al. [2022] S. Bahl, A. Gupta, and D. Pathak.  Human-to-robot imitation in the wild.  2022\.
  * Bharadhwaj et al. [2023] H. Bharadhwaj, A. Gupta, S. Tulsiani, and V. Kumar.  Zero-shot robot manipulation from passive human videos.  _arXiv preprint arXiv:2302.02011_ , 2023.
  * Xiong et al. [2021] H. Xiong, Q. Li, Y.-C. Chen, H. Bharadhwaj, S. Sinha, and A. Garg.  Learning by watching: Physical imitation of manipulation skills from human videos.  In _2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , pages 7827–7834. IEEE, 2021.
  * Bahl et al. [2023] S. Bahl, R. Mendonca, L. Chen, U. Jain, and D. Pathak.  Affordances from human videos as a versatile representation for robotics.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 13778–13790, 2023.
  * Schmeckpeper et al. [2020] K. Schmeckpeper, A. Xie, O. Rybkin, S. Tian, K. Daniilidis, S. Levine, and C. Finn.  Learning predictive models from observation and interaction.  In _European Conference on Computer Vision_ , pages 708–725. Springer, 2020.
  * Escontrela et al. [2024] A. Escontrela, A. Adeniji, W. Yan, A. Jain, X. B. Peng, K. Goldberg, Y. Lee, D. Hafner, and P. Abbeel.  Video prediction models as rewards for reinforcement learning.  _Advances in Neural Information Processing Systems_ , 36, 2024.
  * Du et al. [2024] Y. Du, S. Yang, B. Dai, H. Dai, O. Nachum, J. Tenenbaum, D. Schuurmans, and P. Abbeel.  Learning universal policies via text-guided video generation.  _Advances in Neural Information Processing Systems_ , 36, 2024.
  * Ko et al. [2023] P.-C. Ko, J. Mao, Y. Du, S.-H. Sun, and J. B. Tenenbaum.  Learning to act from actionless videos through dense correspondences.  _arXiv preprint arXiv:2310.08576_ , 2023.
  * Zhu et al. [2024] Y. Zhu, A. Lim, P. Stone, and Y. Zhu.  Vision-based manipulation from single human video with open-world object graphs.  _arXiv preprint arXiv:2405.20321_ , 2024.
  * Chi and Song [2021] C. Chi and S. Song.  Garmentnets: Category-level pose estimation for garments via canonical space shape completion.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 3324–3333, 2021.
  * Xue et al. [2023] H. Xue, W. Xu, J. Zhang, T. Tang, Y. Li, W. Du, R. Ye, and C. Lu.  Garmenttracking: Category-level garment pose tracking.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 21233–21242, 2023.
  * Huang et al. [2022] Z. Huang, X. Lin, and D. Held.  Mesh-based dynamics with occlusion reasoning for cloth manipulation.  _arXiv preprint arXiv:2206.02881_ , 2022.
  * Tang et al. [2017] T. Tang, Y. Fan, H.-C. Lin, and M. Tomizuka.  State estimation for deformable objects by point registration and dynamic simulation.  In _2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , pages 2427–2433. IEEE, 2017.
  * Tang et al. [2018] T. Tang, C. Wang, and M. Tomizuka.  A framework for manipulating deformable linear objects by coherent point drift.  _IEEE Robotics and Automation Letters_ , 3(4):3426–3433, 2018.
  * Chi and Berenson [2019] C. Chi and D. Berenson.  Occlusion-robust deformable object tracking without physics simulation.  In _2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , pages 6443–6450. IEEE, 2019.
  * Huang et al. [2023] Z. Huang, X. Lin, and D. Held.  Self-supervised cloth reconstruction via action-conditioned cloth tracking.  In _2023 IEEE International Conference on Robotics and Automation (ICRA)_ , pages 7111–7118. IEEE, 2023.
  * Yang et al. [2023] J. Yang, M. Gao, Z. Li, S. Gao, F. Wang, and F. Zheng.  Track anything: Segment anything meets videos, 2023.
  * Sun et al. [2021] J. Sun, Z. Shen, Y. Wang, H. Bao, and X. Zhou.  LoFTR: Detector-free local feature matching with transformers.  _CVPR_ , 2021.
  * Qi et al. [2017] C. R. Qi, L. Yi, H. Su, and L. J. Guibas.  Pointnet++: Deep hierarchical feature learning on point sets in a metric space.  _arXiv preprint arXiv:1706.02413_ , 2017.
  * Coumans and Bai [2016–2021] E. Coumans and Y. Bai.  Pybullet, a python module for physics simulation for games, robotics and machine learning.  http://pybullet.org, 2016–2021.
  * Silver et al. [2018] T. Silver, K. Allen, J. Tenenbaum, and L. Kaelbling.  Residual policy learning.  _arXiv preprint arXiv:1812.06298_ , 2018.
  * Schulman et al. [2017] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov.  Proximal policy optimization algorithms.  _arXiv preprint arXiv:1707.06347_ , 2017.