# TaCL: Improving BERT Pre-training with
Token-aware Contrastive Learning

Yixuan Su♠ Fangyu Liu♠ Zaiqiao Meng♠ Tian Lan♢
Lei Shu♡ Ehsan Shareghi♣♠ Nigel Collier♠
♠Language Technology Lab, University of Cambridge ♢Beijing Institute of
Technology
♡Amazon AWS AI ♣ Department of Data Science and AI, Monash University
<EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>
Work was done prior to joining Amazon.

###### Abstract

Masked language models (MLMs) such as BERT have revolutionized the field of
Natural Language Understanding in the past few years. However, existing pre-
trained MLMs often output an anisotropic distribution of token representations
that occupies a narrow subset of the entire representation space. Such token
representations are not ideal, especially for tasks that demand discriminative
semantic meanings of distinct tokens. In this work, we propose TaCL (Token-
aware Contrastive Learning), a novel continual pre-training approach that
encourages BERT to learn an isotropic and discriminative distribution of token
representations. TaCL is fully unsupervised and requires no additional data.
We extensively test our approach on a wide range of English and Chinese
benchmarks. The results show that TaCL brings consistent and notable
improvements over the original BERT model. Furthermore, we conduct detailed
analysis to reveal the merits and inner-workings of our approach.111Our code
and pre-trained models are publicly available at
https://github.com/yxuansu/TaCL

##  1 Introduction

Since the rising of BERT (Devlin et al., 2019), masked language models (MLMs)
have become the de facto backbone for almost all natural language
understanding (NLU) tasks. Despite their clear success, many existing language
models pre-trained with MLM objective suffer from the anisotropic problem
Ethayarajh (2019). That is, their token representations reside in a narrow
subset of the representation space, therefore being less discriminative and
less powerful in capturing the semantic differences of distinct tokens.

Figure 1: An overview of TaCL. The student learns to make the representation
of a masked token closer to its “reference” representation produced by the
teacher (solid arrow) and away from the representations of other tokens in the
same sequence (dashed arrows).

Recently, great advancement has been made in continually training MLMs with
unsupervised sentence-level contrastive learning, aiming at creating more
discriminative sentence-level representations (Giorgi et al., 2021; Carlsson
et al., 2021; Yan et al., 2021; Kim et al., 2021; Liu et al., 2021b; Gao et
al., 2021). However, such representations are only evaluated as sentence
embeddings and there is no evidence that they will benefit other well-
established NLU tasks. We show that these approaches hardly bring any benefit
to challenging tasks like SQuAD Rajpurkar et al. (2016, 2018).

In this paper, we argue that the key of obtaining more discriminative and
transferrable representations lies in learning contrastive and isotropic
token-level representations. To this end, we propose TaCL (Token-aware
Contrastive Learning), a new continual pre-training approach that encourages
BERT to learn discriminative token representations. Specifically, our approach
involves two models (a student and a teacher) that are both initialized from
the same pre-trained BERT. During the learning stage, we freeze the parameters
of the teacher and continually optimize the student model with (1) the
original BERT pre-training objectives (masked language modelling and next
sentence prediction) and (2) a newly proposed TaCL objective. The TaCL loss is
obtained by contrasting the student representations of masked tokens against
the “reference” representations produced by the teacher without masking the
input tokens. In Figure 1, we provide an overview of our approach.

We extensively test our approach on a wide range of English and Chinese
benchmarks and illustrate that TaCL brings notable performance improvements on
most evaluated datasets (section 3.1.1). These results validate that more
discriminative and isotropic token representations lead to better model
performances. Additionally, we highlight the benefits of using our token-level
method compared to current state-of-the-art sentence-level contrastive
learning techniques on NLU tasks (section 3.2.1). We further analyze the inner
workings of TaCL and its impact on the token representation space (section
3.2.2).

Our work, to the best of our knowledge, is the first effort on applying
contrastive learning to improve token representations of Transformer models.
We hope the findings of this work could facilitate further development of
methods on the intersection of contrastive learning and representation
learning at a more fine-grained granularities.

##  2 Token-aware Contrastive Learning

Our approach contains two models, i.e., a student $S$ and a teacher $T$, both
of which are initialized from the same pre-trained BERT. During learning, we
freeze $T$ and only optimize the parameters of $S$. Given an input sequence
$x=[x_{1},...,x_{n}]$, we randomly mask $x$ with the same procedure as in
Devlin et al. (2019) and feed the masked sequence $\tilde{x}$ into the student
model to produce the contextual representation
$\tilde{h}=[\tilde{h}_{1},...,\tilde{h}_{n}]$. Meanwhile, the teacher model
takes the original sequence $x$ as input and produces the representation
$h=[h_{1},...,h_{n}]$ (see Figure 1). The proposed token-aware contrastive
learning objective $\mathcal{L}_{\textup{TaCL}}$ is then defined as

$-\sum_{i=1}^{n}\mathds{1}(\tilde{x}_{i})\log\frac{\exp(\textup{sim}(\tilde{h}_{i},h_{i})/\tau)}{\sum_{j=1}^{n}\exp(\textup{sim}(\tilde{h}_{i},h_{j})/\tau)},$
(1)

where $\mathds{1}(\tilde{x}_{i})=1$ if $\tilde{x}_{i}$ is a masked token,
otherwise $\mathds{1}(\tilde{x}_{i})=0$. $\tau$ is a temperature hyper-
parameter and $\textup{sim}(\cdot,\cdot)$ computes the cosine similarity.
Intuitively, the student learns to make the representation of a masked token
closer to its “reference” representation produced by the teacher and away from
other tokens in the same sequence. As a result, the token representations
learnt by the student are more discriminative with respect to distinct tokens,
therefore better following an isotropic distribution. Similar to Devlin et al.
(2019), we also adopt the masked language modelling
$\mathcal{L}_{\textup{MLM}}$ and next sentence prediction
$\mathcal{L}_{\textup{NSP}}$ objectives. The overall learning objective
$\mathcal{L}$ of the student model during the continual pre-training stage is
defined as

$\mathcal{L}=\mathcal{L}_{\textup{TaCL}}+\mathcal{L}_{\textup{MLM}}+\mathcal{L}_{\textup{NSP}}.$
(2)

Note that the learning of the student is fully unsupervised and can be
realized using the original pre-training corpus. After the learning is
completed, we fine-tune the student model on downstream tasks.

##  3 Experiment

English Benchmark | Model | GLUE | SQuAD 1.1 | SQuAD 2.0
---|---|---|---|---
CoLA | SST-2 | MPRC | STS-B | QQP | MNLI | QNLI | RTE | Ave. | EM | F1 | EM | F1
| Base size models |
$\textup{BERT}_{\textup{base}}\parallel$ | 52.1 | 93.5 | 88.9 | 85.8 | 71.2 | 84.6/83.4 | 90.5 | 66.4 | 79.6 | 80.8 | 88.5 | - | -
$\textup{BERT}_{\textup{base}}$ $\ddagger$ | 52.2 | 92.4 | 89.0 | 86.4 | 73.2 | 84.6/84.5 | 90.3 | 63.2 | 79.8 | 80.9 | 88.4 | 73.4 | 76.8
+MT$\ddagger$ | 51.9 | 92.5 | 89.3 | 87.1 | 75.8 | 84.2/84.0 | 90.6 | 64.1 | 80.0 | 81.0 | 88.5 | 73.2 | 76.3
$\textup{TaCL}_{\textup{base}}$ | 52.4 | 92.3 | 90.8 | 89.0 | 80.7 | 84.4/84.3 | 91.1 | 62.8 | 81.2 | 81.6 | 89.0 | 74.4 | 77.5
| Large size models |
$\textup{BERT}_{\textup{large}}\parallel$ | 60.5 | 94.9 | 89.3 | 86.5 | 72.1 | 86.7/85.9 | 92.7 | 70.1 | 82.1 | 84.1 | 90.9 | 78.7 | 81.9
$\textup{BERT}_{\textup{large}}$ $\ddagger$ | 61.6 | 93.6 | 90.2 | 89.0 | 81.8 | 86.4/86.1 | 92.6 | 67.2 | 83.6 | 84.0 | 90.8 | 77.9 | 81.0
+MT $\ddagger$ | 62.0 | 93.8 | 90.5 | 89.1 | 82.5 | 86.3/86.3 | 92.2 | 66.5 | 83.7 | 83.9 | 90.9 | 77.8 | 80.7
$\textup{TaCL}_{\textup{large}}$ | 61.1 | 94.1 | 92.0 | 89.7 | 82.5 |  86.5/85.9 | 92.4 | 70.5 | 84.7 | 84.2 | 91.1 | 78.7 | 81.9
|  |  |  |  |  |  |  |  |  |  |  |  |  |

Chinese Benchmark | Model | Ontonotes | MSRA | Resume | Weibo | PKU | CityU | AS
---|---|---|---|---|---|---|---|---
Dev | Test | Dev | Test | Dev | Test | Dev | Test | Test | Test | Test
|  ${\spadesuit}$ and ${\diamondsuit}$ published in Li et al. (2020) and Meng et al. (2019) |
$\textup{BERT}_{\textup{base}}$ | - | 80.14♠ | - | 94.95♠ | - | 95.53♠ | - | 68.20♠ | 96.50♢ | 97.60♢ | 96.50♢
$\textup{BERT}_{\textup{base}}\ddagger$ | 78.29 | 80.23 | 94.13 | 94.97 | 95.37 | 95.70 | 70.63 | 67.98 | 96.51 | 97.83 | 96.58
+MT $\ddagger$ | 78.42 | 80.36 | 94.20 | 95.01 | 95.29 | 95.62 | 70.81 | 68.02 | 96.53 | 97.79 | 96.54
$\textup{TaCL}_{\textup{base}}$ | 79.73 | 82.42 | 94.58 | 95.44 | 96.23 | 96.45 | 72.32 | 69.54 | 96.75 | 98.18 | 96.75
|  |  |  |  |  |  |  |  |  |  |  |

Table 1: Benchmark Results. $\parallel$: published in Devlin et al. (2019);
and $\ddagger$: models from our implementations.

We test our approach on a wide range of benchmarks in two languages. For
English benchmarks, we evaluate the $\textup{BERT}_{\textup{base}}$ and
$\textup{BERT}_{\textup{large}}$ models. For Chinese benchmarks, we test the
$\textup{BERT}_{\textup{base}}$ model.222All models are officially released by
Devlin et al. (2019). After initializing the student and teacher, we
continually pre-train the student on the same Wikipedia corpus as in Devlin et
al. (2019) for 150k steps. The training samples are truncated with a maximum
length of 256 and the batch size is set as 256. The temperature $\tau$ in Eq.
(1) is set as 0.01. Same as Devlin et al. (2019), we optimize the model with
Adam optimizer Kingma and Ba (2015) with weighted decay, and an initial
learning rate of 1e-4 (with warm-up ratio of 10%).

###  3.1 Evaluation Benchmarks

For English benchmarks, we use the GLUE dataset Wang et al. (2019) which
contains a variety of sentence-level classification tasks covering textual
entailment (RTE and MNLI), question-answer entailment (QNLI), paraphrase
(MRPC), question paraphrase (QQP), textual similarity (STS-B), sentiment
(SST-2), and linguistic acceptability (CoLA). Our evaluation metrics are
Spearman correlation for STS-B, Matthews correlation for CoLA, and accuracy
for the other tasks; the macro average score is also reported. Additionally,
we conduct experiments on SQuAD 1.1 Rajpurkar et al. (2016) and 2.0 Rajpurkar
et al. (2018) datasets that evaluate the model’s performance on the token-
level answer-extraction task. The dev set results of Exact-Match (EM) and F1
scores are reported.

For Chinese benchmarks, we evaluate our model on two token-level labelling
tasks, including name entity recognition (NER) and Chinese word segmentation
(CWS). For NER, we use the Ontonotes Weischedel et al. (2011), MSRA Levow
(2006), Resume Zhang and Yang (2018), and Weibo He and Sun (2017) datasets.
For CWS, we use the PKU, CityU, and AS datasets from SIGHAN 2005 Emerson
(2005) for evaluation. The standard F1 score is used for evaluation.

Baselines: We compare against two baselines: (1) the original BERT used to
initialize the student and teacher; (2) BERT+MT (BERT with more training)
which is acquired by continually pre-training the original BERT on Wikipedia
for 150k steps333The number of steps is set the same as our TaCL training.
using the original BERT pre-training objectives.

####  3.1.1 Benchmark Results

Table 1 reports the results on English and Chinese benchmarks.444For all
tasks, the average results over five runs are reported. We observe that, on
most sequence-level classification tasks in GLUE, TaCL outperforms BERT and
BERT+MT. Additionally, on all token-level benchmarks (SQuAD, NER, and CWS),
TaCL consistently and notably surpasses other baselines. These results
indicate that the learning of an isotropic token representation space is
beneficial for the model’s performance, especially on the token-centric tasks.

Model |  $\mathcal{L_{\textup{MLM}}}$ \+ $\mathcal{L_{\textup{NSP}}}$ | CL | SQuAD 1.1 | SQuAD 2.0
---|---|---|---|---
BERT | ✓ | $\times$ | 80.8/88.5 | 73.4/76.8
Sentence-Level Contrastive Methods
DeCLUTR | $\times$ | Sen. | 79.9/87.6 | 72.1/75.4
SimCSE | $\times$ | Sen. | 80.2/88.0 | 72.5/75.7
MirrorBERT | $\times$ | Sen. | 80.3/88.1 | 72.7/75.9
Ablated Models
model-1 | ✓ | Sen. | 80.5/88.3 | 73.1/76.5
model-2 | $\times$ | Tok. | 81.3/88.7 | 73.8/77.1
TaCL | ✓ | Tok. | 81.6/89.0 | 74.4/77.5
|  |  |  |

Table 2: Comparison of various sentence- and token-level contrastive learning
methods. “Sen.” or “Tok.” denotes training with sentence- or token-level
contrastive objectives. Scores of (EM/F1) are reported.

###  3.2 Analysis

In this section, we present further comparisons and in-depth analysis of the
proposed approach.

####  3.2.1 Sentence-Level vs. Token-Level CL

We compare TaCL against existing sentence-level contrastive learning methods,
including DeCLUTR Giorgi et al. (2021), SimCSE Gao et al. (2021), and
MirrorBERT Liu et al. (2021b). We also include two ablated models to study the
effect of different combinations of pre-training objectives. Specifically, the
ablated model-1 is initialized with BERT and trained with the original BERT
objectives ($\mathcal{L}_{\textup{MLM}}$ and $\mathcal{L}_{\textup{NSP}}$)
plus the sentence-level contrastive objective as proposed in Liu et al.
(2021b). The ablated model-2 is initialized with BERT and trained only with
the proposed token-aware contrastive objective of Eq. (1). Note that all
compared models have the same size as the $\textup{BERT}_{\textup{base}}$
model.

Table 2 shows the performance of different models on SQuAD. We observe
decreased performance of existing sentence-level contrastive methods compared
with the original BERT. This could be attributed to the fact that such methods
only focus on learning sentence-level representations while ignoring the
learning of individual tokens. This behaviour is undesired for tasks like
SQuAD that demands informative token representations. Nonetheless, the ablated
model-1 shows that the original BERT pre-training objective
($\mathcal{L}_{\textup{MLM}}$ and $\mathcal{L}_{\textup{NSP}}$) remedies, to
some extent, the performance degeneration caused by the sentence-level
contrastive methods. On the other hand, the ablated model-2 demonstrates that
our token-aware contrastive objective helps the model to achieve improved
results by learning better token representations.

Figure 2: Layer-wise representation self-similarity.

####  3.2.2 Token Representation Self-similarity

To analyze the token representations learnt by TaCL and BERT, we follow
Ethayarajh (2019) and define the averaged self-similarity of the token
representations within one sequence $x=[x_{1},...,x_{n}]$ as,

$s(x)=\frac{1}{n(n-1)}\sum_{i=1}^{n}\sum_{j=1,j\neq
i}^{n}\textup{cosine}(h_{i},h_{j}),$ (3)

where $h_{i}$ and $h_{j}$ are the token representations of $x_{i}$ and $x_{j}$
produced by the model. Intuitively, a lower $s(x)$ indicates that the
representations of tokens within the sequence $x$ are less similar to each
other, therefore being more discriminative.

We sample 50k sentences from both Chinese and English Wikipedia and compute
the self-similarity of representations over different layers. Figure 2 plots
the results of $\textup{TaCL}_{\textup{base}}$ and
$\textup{BERT}_{\textup{base}}$ averaged over all sentences. We see that, in
the intermediate layers, the self-similarity of TaCL is higher than BERT’s. In
contrast, at the top layer (layer 12), TaCL’s self-similarity becomes notably
lower than BERT’s, demonstrating that the final output token representations
of TaCL are more discriminative.

Figure 3: Self-similarity Matrix Visualization: (a) BERT and (b) TaCL. (best
viewed in color)

##### Qualitative Analysis.

We sample one sentence from Wikipedia and visualize the self-similarity matrix
$M$ (where $M_{i,j}=\textup{cosine}(h_{i},h_{j})$) produced by
$\textup{BERT}_{\textup{base}}$ and $\textup{TaCL}_{\textup{base}}$. The
results are shown in Figure 3, where a darker color denotes a higher self-
similarity score.555The entries $M_{i,i}$ in the diagonal have a $1.0$ self-
similarity by definition, as $\textup{cosine}(h_{i},h_{i})=1.0$. We see that,
as compared with BERT (Fig. 3(a)), the self-similarities of TaCL (Fig. 3(b))
are much lower in the off-diagonal entries. This further highlights that the
individual token representations of TaCL are more discriminative, which in
return leads to improved model performances as demonstrated (section 3.1.1,
section 3.2.1).

##  4 Conclusion

In this work, we proposed TaCL, a novel approach that applies token-aware
contrastive learning for the continual pre-training of BERT. Extensive
experiments are conducted on a wide range of English and Chinese benchmarks.
The results show that our approach leads to notable performance improvement
across all evaluated benchmarks. We then delve into the inner-working of TaCL
and demonstrate that our performance gain comes from a more discriminative
distribution of token representations.

## Acknowledgments

The first author would like to thank Jialu Xu and Piji Li for the insightful
discussions and support. Many thanks to our anonymous reviewers and area
chairs for their suggestions and comments.

## Ethical Statement

We honor and support the ACL code of Ethics. Language model pre-training aims
to improve the system’s performance on downstream NLU tasks. All pre-training
corpora and pre-trained models used in this work are publically available. In
addition, all evaluated datasets are from previously published works, and in
our view, do not have any attached privacy or ethical issues.

## References

  * Carlsson et al. (2021) Fredrik Carlsson, Amaru Cuba Gyllensten, Evangelia Gogoulou, Erik Ylipää Hellqvist, and Magnus Sahlgren. 2021.  Semantic re-tuning with contrastive tension.  In _International Conference on Learning Representations_.
  * Chen et al. (2020) Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020.  A simple framework for contrastive learning of visual representations.  In _International conference on machine learning_ , pages 1597–1607. PMLR.
  * Chopra et al. (2005) Sumit Chopra, Raia Hadsell, and Yann LeCun. 2005.  Learning a similarity metric discriminatively, with application to face verification.  In _2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2005), 20-26 June 2005, San Diego, CA, USA_ , pages 539–546. IEEE Computer Society.
  * Clark et al. (2020) Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020.  ELECTRA: pre-training text encoders as discriminators rather than generators.  In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net.
  * Das et al. (2021) Sarkar Snigdha Sarathi Das, Arzoo Katiyar, Rebecca J Passonneau, and Rui Zhang. 2021\.  Container: Few-shot named entity recognition via contrastive learning.  _arXiv preprint arXiv:2109.07589_.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers)_ , pages 4171–4186. Association for Computational Linguistics.
  * Emerson (2005) Thomas Emerson. 2005.  The second international chinese word segmentation bakeoff.  In _Proceedings of the Fourth SIGHAN Workshop on Chinese Language Processing, SIGHAN@IJCNLP 2005, Jeju Island, Korea, 14-15, 2005_. ACL.
  * Ethayarajh (2019) Kawin Ethayarajh. 2019.  How contextual are contextualized word representations? comparing the geometry of bert, elmo, and GPT-2 embeddings.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019_ , pages 55–65. Association for Computational Linguistics.
  * Gao et al. (2021) Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021.  SimCSE: Simple contrastive learning of sentence embeddings.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 6894–6910, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Giorgi et al. (2021) John Giorgi, Osvald Nitski, Bo Wang, and Gary Bader. 2021.  DeCLUTR: Deep contrastive learning for unsupervised textual representations.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 879–895, Online. Association for Computational Linguistics.
  * Gururangan et al. (2020) Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A Smith. 2020.  Don’t stop pretraining: Adapt language models to domains and tasks.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 8342–8360.
  * He and Sun (2017) Hangfeng He and Xu Sun. 2017.  F-score driven max margin neural network for named entity recognition in chinese social media.  In _Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2017, Valencia, Spain, April 3-7, 2017, Volume 2: Short Papers_ , pages 713–718. Association for Computational Linguistics.
  * Jia et al. (2021) Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. 2021.  Scaling up visual and vision-language representation learning with noisy text supervision.  In _Proceedings of the 38th International Conference on Machine Learning_ , volume 139 of _Proceedings of Machine Learning Research_ , pages 4904–4916. PMLR.
  * Kim et al. (2021) Taeuk Kim, Kang Min Yoo, and Sang-goo Lee. 2021.  Self-guided contrastive learning for BERT sentence representations.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 2528–2540, Online. Association for Computational Linguistics.
  * Kingma and Ba (2015) Diederik P. Kingma and Jimmy Ba. 2015.  Adam: A method for stochastic optimization.  In _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_.
  * Lan et al. (2021) Tian Lan, Deng Cai, Yan Wang, Yixuan Su, Xian-Ling Mao, and Heyan Huang. 2021\.  Exploring dense retrieval for dialogue response selection.  _CoRR_ , abs/2110.06612.
  * Levow (2006) Gina-Anne Levow. 2006.  The third international chinese language processing bakeoff: Word segmentation and named entity recognition.  In _Proceedings of the Fifth Workshop on Chinese Language Processing, SIGHAN@COLING/ACL 2006, Sydney, Australia, July 22-23, 2006_ , pages 108–117. Association for Computational Linguistics.
  * Lewis et al. (2020) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020.  BART: denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020_ , pages 7871–7880. Association for Computational Linguistics.
  * Li et al. (2020) Xiaonan Li, Hang Yan, Xipeng Qiu, and Xuanjing Huang. 2020.  FLAT: chinese NER using flat-lattice transformer.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020_ , pages 6836–6842. Association for Computational Linguistics.
  * Liu et al. (2021a) Fangyu Liu, Ehsan Shareghi, Zaiqiao Meng, Marco Basaldella, and Nigel Collier. 2021a.  Self-alignment pretraining for biomedical entity representations.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 4228–4238, Online. Association for Computational Linguistics.
  * Liu et al. (2021b) Fangyu Liu, Ivan Vulić, Anna Korhonen, and Nigel Collier. 2021b.  Fast, effective, and self-supervised: Transforming masked language models into universal lexical and sentence encoders.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 1442–1459, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Liu et al. (2021c) Qianchu Liu, Fangyu Liu, Nigel Collier, Anna Korhonen, and Ivan Vulić. 2021c.  MirrorWiC: On eliciting word-in-context representations from pretrained language models.  In _Proceedings of the 25th Conference on Computational Natural Language Learning_ , pages 562–574, Online. Association for Computational Linguistics.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized BERT pretraining approach.  _CoRR_ , abs/1907.11692.
  * Liu and Liu (2021) Yixin Liu and Pengfei Liu. 2021.  SimCLS: A simple framework for contrastive learning of abstractive summarization.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_ , pages 1065–1072, Online. Association for Computational Linguistics.
  * Meng et al. (2021a) Yu Meng, Chenyan Xiong, Payal Bajaj, Saurabh Tiwary, Paul Bennett, Jiawei Han, and Xia Song. 2021a.  Coco-lm: Correcting and contrasting text sequences for language model pretraining.  _arXiv preprint arXiv:2102.08473_.
  * Meng et al. (2019) Yuxian Meng, Wei Wu, Fei Wang, Xiaoya Li, Ping Nie, Fan Yin, Muyu Li, Qinghong Han, Xiaofei Sun, and Jiwei Li. 2019.  Glyce: Glyph-vectors for chinese character representations.  In _Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada_ , pages 2742–2753.
  * Meng et al. (2021b) Zaiqiao Meng, Fangyu Liu, Ehsan Shareghi, Yixuan Su, Charlotte Collins, and Nigel Collier. 2021b.  Rewire-then-probe: A contrastive recipe for probing biomedical knowledge of pre-trained language models.  _CoRR_ , abs/2110.08173.
  * Oord et al. (2018) Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018.  Representation learning with contrastive predictive coding.  _arXiv preprint arXiv:1807.03748_.
  * Pan et al. (2021) Lin Pan, Chung-Wei Hang, Haode Qi, Abhishek Shah, Saloni Potdar, and Mo Yu. 2021\.  Multilingual bert post-pretraining alignment.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 210–219.
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021.  Learning transferable visual models from natural language supervision.  _arXiv preprint arXiv:2103.00020_.
  * Radford et al. (2019) Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019.  Language models are unsupervised multitask learners.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _J. Mach. Learn. Res._ , 21:140:1–140:67.
  * Rajpurkar et al. (2018) Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018.  Know what you don’t know: Unanswerable questions for squad.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 2: Short Papers_ , pages 784–789. Association for Computational Linguistics.
  * Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016.  Squad: 100, 000+ questions for machine comprehension of text.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA, November 1-4, 2016_ , pages 2383–2392. The Association for Computational Linguistics.
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019.  Sentence-BERT: Sentence embeddings using Siamese BERT-networks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3982–3992, Hong Kong, China. Association for Computational Linguistics.
  * Sermanet et al. (2018) Pierre Sermanet, Corey Lynch, Yevgen Chebotar, Jasmine Hsu, Eric Jang, Stefan Schaal, and Sergey Levine. 2018.  Time-contrastive networks: Self-supervised learning from video.  In _2018 IEEE International Conference on Robotics and Automation, ICRA 2018, Brisbane, Australia, May 21-25, 2018_ , pages 1134–1141. IEEE.
  * Su et al. (2021a) Yixuan Su, Deng Cai, Yan Wang, David Vandyke, Simon Baker, Piji Li, and Nigel Collier. 2021a.  Non-autoregressive text generation with pre-trained language models.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, EACL 2021, Online, April 19 - 23, 2021_ , pages 234–243. Association for Computational Linguistics.
  * Su et al. (2021b) Yixuan Su, Deng Cai, Qingyu Zhou, Zibo Lin, Simon Baker, Yunbo Cao, Shuming Shi, Nigel Collier, and Yan Wang. 2021b.  Dialogue response selection with hierarchical curriculum learning.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021_ , pages 1740–1751. Association for Computational Linguistics.
  * Su et al. (2022) Yixuan Su, Tian Lan, Yan Wang, Dani Yogatama, Lingpeng Kong, and Nigel Collier. 2022\.  A contrastive framework for neural text generation.  _CoRR_ , abs/2202.06417.
  * Su et al. (2021c) Yixuan Su, Zaiqiao Meng, Simon Baker, and Nigel Collier. 2021c.  Few-shot table-to-text generation with prototype memory.  In _Findings of the Association for Computational Linguistics: EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 16-20 November, 2021_ , pages 910–917. Association for Computational Linguistics.
  * Su et al. (2021d) Yixuan Su, Lei Shu, Elman Mansimov, Arshit Gupta, Deng Cai, Yi-An Lai, and Yi Zhang. 2021d.  Multi-task pre-training for plug-and-play task-oriented dialogue system.  _CoRR_ , abs/2109.14739.
  * Su et al. (2021e) Yixuan Su, David Vandyke, Simon Baker, Yan Wang, and Nigel Collier. 2021e.  Keep the primary, rewrite the secondary: A two-stage approach for paraphrase generation.  In _Findings of the Association for Computational Linguistics: ACL/IJCNLP 2021, Online Event, August 1-6, 2021_ , volume ACL/IJCNLP 2021 of _Findings of ACL_ , pages 560–569. Association for Computational Linguistics.
  * Su et al. (2021f) Yixuan Su, David Vandyke, Sihui Wang, Yimai Fang, and Nigel Collier. 2021f.  Plan-then-generate: Controlled data-to-text generation via planning.  In _Findings of the Association for Computational Linguistics: EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 16-20 November, 2021_ , pages 895–909. Association for Computational Linguistics.
  * Su et al. (2021g) Yixuan Su, Yan Wang, Deng Cai, Simon Baker, Anna Korhonen, and Nigel Collier. 2021g.  PROTOTYPE-TO-STYLE: dialogue generation with style-aware editing on retrieval memory.  _IEEE ACM Trans. Audio Speech Lang. Process._ , 29:2152–2161.
  * Vulić et al. (2021) Ivan Vulić, Edoardo Maria Ponti, Anna Korhonen, and Goran Glavaš. 2021\.  LexFit: Lexical fine-tuning of pretrained language models.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 5269–5283, Online. Association for Computational Linguistics.
  * Wang et al. (2019) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019.  GLUE: A multi-task benchmark and analysis platform for natural language understanding.  In _7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019_. OpenReview.net.
  * Wang et al. (2021) Shufan Wang, Laure Thompson, and Mohit Iyyer. 2021.  Phrase-bert: Improved phrase embeddings from bert with an application to corpus exploration.  _ArXiv_ , abs/2109.06304.
  * Wang and Gupta (2015) Xiaolong Wang and Abhinav Gupta. 2015.  Unsupervised learning of visual representations using videos.  In _2015 IEEE International Conference on Computer Vision, ICCV 2015, Santiago, Chile, December 7-13, 2015_ , pages 2794–2802. IEEE Computer Society.
  * Weischedel et al. (2011) Ralph Weischedel, Sameer Pradhan, Lance Ramshaw, Martha Palmer, Nianwen Xue, Mitchell Marcus, Ann Taylor, Craig Greenberg, Eduard Hovy, and Robert Belvin. 2011\.  Ontonotes release 4.0. ldc2011t03.  In _Philadelphia, Penn.: Linguistic Data Consortium ._
  * Wu et al. (2020) Zhuofeng Wu, Sinong Wang, Jiatao Gu, Madian Khabsa, Fei Sun, and Hao Ma. 2020.  Clear: Contrastive learning for sentence representation.  _arXiv preprint arXiv:2012.15466_.
  * Xu et al. (2021) Hu Xu, Gargi Ghosh, Po-Yao Huang, Dmytro Okhonko, Armen Aghajanyan, Florian Metze, Luke Zettlemoyer, and Christoph Feichtenhofer. 2021.  Videoclip: Contrastive pre-training for zero-shot video-text understanding.  _arXiv preprint arXiv:2109.14084_.
  * Xu et al. (2019) Hu Xu, Bing Liu, Lei Shu, and Philip Yu. 2019.  BERT post-training for review reading comprehension and aspect-based sentiment analysis.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 2324–2335, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Yan et al. (2021) Yuanmeng Yan, Rumei Li, Sirui Wang, Fuzheng Zhang, Wei Wu, and Weiran Xu. 2021.  ConSERT: A contrastive framework for self-supervised sentence representation transfer.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 5065–5075, Online. Association for Computational Linguistics.
  * Yang et al. (2021) Jianwei Yang, Yonatan Bisk, and Jianfeng Gao. 2021.  Taco: Token-aware cascade contrastive learning for video-text alignment.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 11562–11572.
  * Yang et al. (2019) Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2019.  Xlnet: Generalized autoregressive pretraining for language understanding.  In _Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada_ , pages 5754–5764.
  * Zhang and Yang (2018) Yue Zhang and Jie Yang. 2018.  Chinese NER using lattice LSTM.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long Papers_ , pages 1554–1564. Association for Computational Linguistics.
  * Zhong et al. (2021) Ming Zhong, Yang Liu, Yichong Xu, Chenguang Zhu, and Michael Zeng. 2021.  Dialoglm: Pre-trained model for long dialogue understanding and summarization.  _CoRR_ , abs/2109.02492.

##  Appendix A Statistics of Evaluated Benchmarks

###  A.1 English Benchmarks

Dataset | Train | Test | Evaluation Metric
---|---|---|---
CoLA | 8.5k | 1k | Matthews correlation
SST-2 | 67k | 1.8k | accuracy
MRPC | 3.7k | 1.7K | accuracy
STS-B | 7k | 1.4k | Spearman correlation
QQP | 364k | 391k | accuracy
MNLI | 393k | 20k | matched/mismatched accuracy
QNLI | 105k | 5.4k | accuracy
RTE | 2.5k | 3k | accuracy
|  |  |

Table 3: GLUE Statistics

Dataset | Train | Dev | Evaluation Metric
---|---|---|---
1.1 | 87.6k | 10.6k | Exact-Match/F1
2.0 | 130.3k | 11.9k | Exact-Match/F1
|  |  |

Table 4: SQuAD Statistics

###  A.2 Chinese Benchmarks

Dataset | Train | Dev | Test | Evaluation Metric
---|---|---|---|---
Ontonotes | 15.7k | 4.3k | 4.3k | F1
MSRA | 37.0k | 9.3k | 4.4k | F1
Resume | 3.8k | 0.5k | 0.5k | F1
Weibo | 1.4k | 0.3k | 0.3k | F1
|  |  |  |

Table 5: NER Dataset Statistics

Dataset | Train | Test | Evaluation Metric
---|---|---|---
PKU | 19.1k | 1.9k | F1
CityU | 53.0k | 1.5k | F1
AS | 708.9k | 14.4k | F1
|  |  |

Table 6: CWS Dataset Statistics

##  Appendix B Related Work

##### Pre-trained Language Models.

Since the introduction of BERT Devlin et al. (2019), the research community
has witnessed remarkable progress in the field of language model pre-training
on a large amount of free text. Such advancements have led to significant
progresses in a wide range of natural language understanding (NLU) tasks Liu
et al. (2019); Yang et al. (2019); Clark et al. (2020); Lan et al. (2021) and
text generation tasks Radford et al. (2019); Lewis et al. (2020); Raffel et
al. (2020); Su et al. (2021a, e, g, d, f, c); Zhong et al. (2021)

##### Contrastive Learning.

Generally, contrastive learning methods distinguish observed data points from
fictitious negative samples. They have been widely applied to various computer
vision areas, including image Chopra et al. (2005); Oord et al. (2018) and
video Wang and Gupta (2015); Sermanet et al. (2018). Recently, Chen et al.
(2020) proposed a simple framework for contrastive learning of visual
representations (SimCLR) based on multi-class N-pair loss. Radford et al.
(2021); Jia et al. (2021) applied the contrastive learning approach for
language-image pretraining. Xu et al. (2021); Yang et al. (2021) proposed a
contrastive pre-training approach for video-text alignment.

In the field of NLP, numerous approaches have been proposed to learn better
sentence-level Reimers and Gurevych (2019); Wu et al. (2020); Meng et al.
(2021a); Liu et al. (2021b); Gao et al. (2021); Su et al. (2021b) and lexical-
level (Liu et al., 2021a; Vulić et al., 2021; Liu et al., 2021c; Wang et al.,
2021) representations using contrastive learning. Different from our work,
none of these studies specifically investigates how to utilize contrastive
learning for improving general-purpose token-level representations. Beyond
representation learning, contrastive learning has also been applied to other
NLP applications such as NER (Das et al., 2021) and summarisation (Liu and
Liu, 2021), knowledge probing for pre-trained language models (Meng et al.,
2021b), and open-ended text generation (Su et al., 2022).

##### Continual Pre-training.

Many researchers Xu et al. (2019); Gururangan et al. (2020); Pan et al. (2021)
have investigated how to continually pre-train the model to alleviate the
task- and domain-discrepancy between the pre-trained models and the specific
target task. In contrast, our proposed approach studies how to apply continual
pre-training to directly improve the quality of model representations which is
transferable and beneficial to a wide range of benchmark tasks.

Figure 4: Example 2: self-similarity matrix visualization of (a) BERT and (b)
TaCL. (best viewed in color) Figure 5: Example 3: self-similarity matrix
visualization of (a) BERT and (b) TaCL. (best viewed in color) Figure 6:
Example 4: self-similarity matrix visualization of (a) BERT and (b) TaCL.
(best viewed in color)

##  Appendix C More Self-similarity Visualizations

In Figure 4, 5, and 6, we provide three more comparisons between the self-
similarity matrix produced by TaCL and BERT (the example sentences are
randomly sampled from Wikipedia).666All results are generated by models with
base size. From the figures, we can draw the same conclusion as in section
section 3.2.2, that the token representations of BERT follow an anisotropic
distribution and are less discriminative. On the other hand, the token
representations of TaCL better follow an isotropic distribution, therefore
different tokens become more distinguishable with respect to each other.


1in1in

# Learning Keypoints from Synthetic Data for Robotic Cloth Folding ††thanks:
This research is supported by the Research Foundation Flanders (FWO) under
Grant numbers 1S56022N (TL) and 1SD4421N (VDG)

Thomas Lips, Victor-Louis De Gusseme and Francis wyffels
AI and Robotics Lab (AIRO), IDLab,Ghent University - imec
<EMAIL_ADDRESS>
###### Abstract

Robotic cloth manipulation is challenging due to its deformability, which
makes determining its full state infeasible. However, for cloth folding, it
suffices to know the position of a few semantic keypoints. Convolutional
neural networks (CNN) can be used to detect these keypoints, but require large
amounts of annotated data, which is expensive to collect. To overcome this, we
propose to learn these keypoint detectors purely from synthetic data, enabling
low-cost data collection. In this paper, we procedurally generate images of
towels and use them to train a CNN. We evaluate the performance of this
detector for folding towels on a unimanual robot setup and find that the grasp
and fold success rates are 77% and 53%, respectively. We conclude that
learning keypoint detectors from synthetic data for cloth folding and related
tasks is a promising research direction, discuss some failures and relate them
to future work. A video of the system, as well as the codebase, more details
on the CNN architecture and the training setup can be found at
https://github.com/tlpss/workshop-icra-2022-cloth-keypoints.git.

###### Index Terms:

Synthetic Data, Procedural Data Generation, Keypoint Detection, Robotic Cloth
Folding

Figure 1: Left - Examples of the synthetic data used to train the keypoint
detector. Center/Right - Successful folds on towels from the evaluation set.
Each image pair shows the initial state with the detected keypoints (yellow
dots) and the final state after attempting to grasp and fold.

##  I Introduction

Robotic manipulation of deformable objects is challenging, both in terms of
perception, control and modelling [1]. This is a.o. caused by their high-
dimensional state [2]. Nonetheless, any general-purpose robotic manipulation
system would encounter such objects. Cloth, for example, is omnipresent in
household settings. Although significant progress has been made, cloth
manipulation remains a challenging task in terms of perception and control [3,
4, 1, 2].

Supervised deep learning (DL) has the potential to tackle these perception
tasks and to generalise to a wide range of settings. However, DL is far from
data-efficient and requires large numbers of labelled data to learn and
generalise. Procedural data generation is an appealing alternative to manually
collecting these datasets: Synthetic data is cheap to generate, has perfect
annotations and can be used to generate all desired variations. The drawback
is that the resulting network often has a reduced performance when transferred
to the real world. This is due to the remaining differences between the
simulation and the real world, referred to as the reality gap.

In this paper, we tackle towel folding, a standard task for cloth manipulation
[5, 6]. We approach this task by using a convolutional neural network (CNN) as
keypoint detector to estimate the 2D positions of the towel corners from a
single RGB image and executing a scripted, open-loop grasp and quasi-static
fold trajectory based on these semantic keypoints. We train the keypoint
detector entirely on synthetic data and evaluate the zero-shot sim-to-real
performance on a unimanual robotic setup under various conditions. Using the
terminology of [7, 3], we assume the towels are unfolded but not necessarily
perfectly flattened. Recent progress in cloth unfolding has made this a
realistic assumption [8].

Our contributions are as follows:

  * •

We show that synthetic data can be used to train convolutional neural networks
to detect keypoints for cloth folding.

  * •

We extensively evaluate the performance of our keypoint detector and unimanual
folding system.

  * •

We provide qualitative insights into the failure cases.

##  II Related Work

###  II-A Robotic Cloth Folding

Cloth manipulation has been extensively studied [2, 3]. For cloth folding in
particular, [9, 4, 6] have devised complete pipelines. These works rely on
segmentation and a combination of polygonal approximations and template
matching or corner detectors to localise keypoints (landmarks) of unfolded
cloth items and subsequently fold them. Although these systems perform very
well in their test settings, the segmentation algorithms assume a known
background color distribution, which strongly limits the potential to
generalise to diverse environments. Furthermore the perception pipeline can
take up to 5 seconds [4] to detect the keypoints.

###  II-B Learning Keypoint Representations for Robotic Control

Recently, CNNs were trained to detect [10, 11, 12, 13] and even discover [14,
15] semantic keypoints end-to-end for various robotic tasks. As the number of
detected keypoints can often vary, keypoint detectors usually output spatial
heatmaps instead of cartesian coordinates [16, 17].

###  II-C Synthetic Data for Computer Vision

Procedural data generation for training perception models has been used to
learn state representations for various robotic tasks [11, 12, 18]. The main
limitation is the induced reality gap, which limits the performance on the
real-world target domain. Tobin et al. [19] introduced domain randomization as
a way to overcome the reality gap by enlarging the distribution from which the
data is generated to ensure it entails the real-world data. However, more
recent work has stressed the importance of more realistic image synthesis [20,
18] and showed that over-randomizing can result in loss of context and
performance [21]. Most authors find that even with domain randomization, the
mismatch between both distributions is still too large and resort to
finetuning with real-world data to increase performance [22]. The factors that
cause this reality gap are not well-known and one usually attempts to close it
with task-specific tuning [22, 18].

##  III Method

###  III-A Synthetic Dataset Generation

We use Blender [23] and BlenderProc [24] to generate data samples. For each
image, we build a new 3D scene and randomize object geometries, materials,
lighting, and camera pose. Examples can be found in Fig. 1. The towel geometry
is modelled as a rectangular mesh. We created a procedural material that
combines a random HSV colour and a Perlin noise texture to omit the need for
realistic fabric textures, which are difficult to obtain. Additionally, we
sample up to 5 distractor objects from a subset of the Thingi10k dataset [25]
and add them randomly to the scene, as this was observed to reduce false
positives on the manipulator and other objects present in the scene. The
ground plane material combines a random HSV base colour with a texture from
PolyHaven111https://polyhaven.com/ to introduce spatial patterns. To generate
complex and realistic lighting of the scene, we use 360-degree images as
environment textures, which are also obtained from PolyHaven. The position of
the camera is sampled inside a spherical cap and its orientation is set to
point towards the centre of the scene. Finally, the scene is rendered into a
256x256 image with Cycles, Blender’s physically-based path tracer.

###  III-B Keypoint Detector

To detect the desired keypoints on an RGB image, we use a fully convolutional
neural network to predict a single spatial heatmap that contains all corners
of the cloth. Inspired by Vecerik et Al. [10], a U-net architecture [26] is
used to combine spatial resolution conserving paths with downsampling paths to
obtain a large receptive field. In the spatial bottleneck, ResNet-inspired
skip connections [27] are used. Bilinear upsampling is used for the upsampling
layers. All hidden layers use the ReLU activation function. The final layer
uses a sigmoid, of which the outputs are interpreted as the probability
function of having a keypoint centred on that location [16]. Heatmaps for the
synthetic data samples are generated using a pixel-wise maximum over 2D
Gaussian blobs that are centred on each keypoint [17]. Pixel-wise Binary Cross
Entropy is used as a training objective and keypoints are extracted using a
max-filter with a configurable receptive field, as implemented in Scikit-Image
[28]. We use Pytorch Lightning [29] to train the CNN and Weights and Biases
[30] for experiment tracking.

###  III-C Robotic Folding

To initiate a fold, a top-down RGB image is passed through the keypoint
detector network. If less than four keypoints are detected, the folding is
aborted; otherwise, the four keypoints with the highest probability are
extracted and reprojected onto the table plane. Based on these keypoints, a
local frame is defined and the scripted pregrasp pose, grasp pose and fold
trajectory in this local frame are transformed to the robot frame. The
sequence is then executed by the robot. The robot grasps the towel in the
middle of the side that needs to be folded. The folding trajectory is an arc
from that grasp point to the corresponding point on the other side of the
towel.

##  IV Experiments

###  IV-A Evaluation

To evaluate the keypoint detector, two sets of towels are used. All towels
were collected by asking a number of lab members to bring random towels to
reduce bias. The first set contains 11 towels with various colours and
material properties but with uniform textures, as modelled in the synthetic
data. These are referred to as the in-distribution towels. The second set
consists of 9 towels that are not in the distribution of the synthetic data as
they have highly non-uniform textures or very different material properties.
We refer to this additional set as the out-of-distribution towels and include
them to evaluate to what degree the neural network is able to generalise from
the synthetic data.

We use three different environment settings, in which we vary lighting and the
presence of 6 random chosen distractor objects. For each setting, the robot
attempts to fold each towel twice. The pose of the towel is manually
randomised within the workspace of the robot before each attempt. For some
larger towels, we already partially fold the towel to reduce their dimensions.
The partially-folded towels range from 0.2 m to 0.5 m in size.

As the most informative metric of a robotic system is the task performance, we
evaluate the keypoint detector by measuring the grasp success rate. Grasp
success is defined as the ability of the system to enclose the cloth at the
grasp pose. Note that this is different from [5], where grasps are only
considered successful if they are held during the entire manipulation. Fold
success is defined, as suggested in [5], as the approximate coincidence of the
opposing corners after manipulation.

###  IV-B Keypoint Detector

Using the procedure described in Section III-A, a dataset of 30,000 images is
generated, of which samples can be found in Fig. 1. The dataset is generated
on a Dell XPS 9570 laptop with a low-range Nvidia GTX 1050Ti mobile GPU.
Generating a single sample takes 1.8 s on average, of which 0.6 s is spent on
building the scene and 1.2 s on rendering.

We then train our keypoint detector to predict all visible corner keypoints
using sensible hyperparameters based on previous experience. We train for 15
epochs, which takes 55 minutes on an Nvidia V100 GPU and results in an average
precision (with 2 pixel threshold) of 80% on a synthetic validation set. We
refer to the accompanying Github page for details about the network
architecture and other training parameters.

###  IV-C Robotic Setup

We use a UR3e robot and Robotiq 2F-85 gripper. We 3D-printed fingertips with a
width of 0.08 m using flexfill TPU 98A filament to ensure compliance of the
fingers while sliding underneath the cloth for grasping. The camera is a ZED2i
that is mounted 1 m above the table and of which the extrinsics have been
determined upfront. The images are cropped to the relevant area and then
rescaled to the CNN’s input size of 256x256 pixels.

(a)

(b)

Figure 2: a) Examples of trials for which the keypoint detection failed,
resulting in a failed grasp and fold. b) Examples of trials for which the fold
failed after a successful grasp. Each image pair shows the initial state with
the detected keypoints (yellow dots) and the final state after attempting to
grasp and fold.

###  IV-D Results

The performance of the system is reported in Table I. The total grasp success
rate for the towels that are similar to those modelled in the synthetic data
is about 77%. From the success rates, it is also clear that the presence of
distractor items or changes in lighting conditions, both of which are
important real-world domain shifts, do not influence the performance of the
keypoint detector. This shows that the randomizations in the synthetic data
generation were effective in this regard.

The fold success rate is about 53%. Examples of successful folds (and hence
grasps) for various settings can be seen in Fig. 1. The fold success rate is
lower than the grasp success rate due to the limitations of scripted
trajectories and single-arm manipulation: About 80% of all fold failures are
caused by corner misalignment as the corners bend during the execution of the
fold motion ( see bottom row of Fig. 2(b)). This could be resolved with
bimanual manipulation or by enlarging the fingertips to better perform a line
fold [3]. For towels that are already partially folded, the thickness and
material properties can change, which makes the open-loop trajectory no longer
suited to fold them (see top row example in Fig. 2(b)), explaining the
remaining fold failures.

Most grasp failures are due to inaccurate or incomplete keypoint detection by
the transferred neural network, examples of which can be seen in Fig. 2(a).
These failures indicate there is still a reality gap that was not covered by
the variations introduced in our synthetic dataset, requiring additional
tuning to better match the target distribution. For the towels that contained
non-uniform textures and were hence not explicitly modelled, the grasp success
rate deteriorates to 48% (see Table I). The remaining grasp failures are due
to some towels being very light, causing the fingers to push the cloth away
instead of reaching underneath and grasping it due to a lack of friction.

Finally, as recommended in [5], we report the execution time of our robot
system: A forward pass of the neural network is, even on CPU, negligibly fast
($\ll$ 1 s) compared to the time it takes to execute the grasp and fold, which
is about 20 s.

Setting | In-distribution Towels | Out-of-distribution Towels
---|---|---
| Grasp | Fold | Grasp | Fold
Natural Light | 17/22 | 14/22 | 9/18 | 8/18
LED Light | 16/22 | 10/22 | 8/18 | 8/18
Distractors | 18/22 | 11/22 | 9/18 | 4/18
Total | 51/66 (77%) | 35/66 (53%) | 26/54 (48%) | 20/54 (37%)

TABLE I: Performance of the system for real-world robotic folding of towels
using zero-shot transfer for the keypoint detector.

##  V Conclusion and Future Work

In this paper, we used synthetic data to train a neural network to detect
keypoints as a low-dimensional state representation for cloth folding. We
transferred the detector without any finetuning (zero-shot) and measured its
performance by using the keypoints for folding towels on a robot setup. The
results indicate that using procedural data generation is a viable approach to
training keypoint detectors for cloth manipulation. However, more extensive
tuning is required to completely overcome the reality gap. In future work, we
plan to further explore what factors matter to close this gap. This will
enable increased performance and provide more principled guidelines for
procedural data generation in general.

Additionally, the gap between the grasp and fold success rates indicates that
not only the perception but also the manipulation of cloth remains a
challenging task. In future work, we therefore aim to close the control loop
to take the cloth properties into account and make the control more robust.

## Acknowledgment

The authors wish to thank the members of the Keypoints Gang, in particular
Rembert Daems and Peter De Roovere, for sharing many insights into Computer
Vision and academic research in general. This research is supported by the
Research Foundation Flanders (FWO) under Grant numbers 1S56022N (TL) and
1SD4421N (VDG).

## References

  * [1] H. Yin, A. Varava, and D. Kragic, “Modeling, learning, perception, and control methods for deformable object manipulation,” _Science Robotics_ , vol. 6, no. 54, p. eabd8803, 2021.
  * [2] J. Sanchez, J.-A. Corrales, B.-C. Bouzgarrou, and Y. Mezouar, “Robotic manipulation and sensing of deformable objects in domestic and industrial applications: a survey,” _The International Journal of Robotics Research_ , vol. 37, no. 7, pp. 688–716, 2018.
  * [3] J. Borràs, G. Alenyà, and C. Torras, “A grasping-centered analysis for cloth manipulation,” _IEEE Transactions on Robotics_ , vol. 36, no. 3, pp. 924–936, 2020.
  * [4] A. Doumanoglou, J. Stria, G. Peleka, I. Mariolis, V. Petrík, A. Kargakos, L. Wagner, V. Hlaváč, T.-K. Kim, and S. Malassiotis, “Folding clothes autonomously: A complete pipeline,” _IEEE Transactions on Robotics_ , vol. 32, no. 6, pp. 1461–1478, 2016.
  * [5] I. Garcia-Camacho, M. Lippi, M. C. Welle, H. Yin, R. Antonova, A. Varava, J. Borras, C. Torras, A. Marino, G. Alenyà, and D. Kragic, “Benchmarking bimanual cloth manipulation,” _IEEE Robotics and Automation Letters_ , vol. 5, no. 2, pp. 1111–1118, 2020.
  * [6] J. Maitin-Shepard, M. Cusumano-Towner, J. Lei, and P. Abbeel, “Cloth grasp point detection based on multiple-view geometric cues with application to robotic towel folding,” in _2010 IEEE International Conference on Robotics and Automation_ , 2010, pp. 2308–2315.
  * [7] A. Verleysen, M. Biondina, and F. Wyffels, “Video dataset of human demonstrations of folding clothing for robotic folding,” _The International Journal of Robotics Research_ , vol. 39, no. 9, pp. 1031–1036, 2020\.
  * [8] H. Ha and S. Song, “Flingbot: The unreasonable effectiveness of dynamic manipulation for cloth unfolding,” in _Conference on Robot Learning_. PMLR, 2022, pp. 24–33.
  * [9] S. Miller, J. van den Berg, M. Fritz, T. Darrell, K. Goldberg, and P. Abbeel, “A geometric approach to robotic laundry folding,” _The International Journal of Robotics Research_ , vol. 31, no. 2, pp. 249–267, 2012.
  * [10] M. Vecerik, J.-B. Regli, O. Sushkov, D. Barker, R. Pevceviciute, T. Rothörl, R. Hadsell, L. Agapito, and J. Scholz, “S3k: Self-supervised semantic keypoints for robotic manipulation via multi-view consistency,” in _Conference on Robot Learning_. PMLR, 2021, pp. 449–460.
  * [11] Z. Qin, K. Fang, Y. Zhu, L. Fei-Fei, and S. Savarese, “Keto: Learning keypoint representations for tool manipulation,” in _2020 IEEE International Conference on Robotics and Automation (ICRA)_ , 2020, pp. 7278–7285.
  * [12] J. Wang, S. Lin, C. Hu, Y. Zhu, and L. Zhu, “Learning semantic keypoint representations for door opening manipulation,” _IEEE Robotics and Automation Letters_ , vol. 5, no. 4, pp. 6980–6987, 2020.
  * [13] A. Kendall, M. Grimes, and R. Cipolla, “Posenet: A convolutional network for real-time 6-dof camera relocalization,” in _Proceedings of the IEEE international conference on computer vision_ , 2015, pp. 2938–2946.
  * [14] T. D. Kulkarni, A. Gupta, C. Ionescu, S. Borgeaud, M. Reynolds, A. Zisserman, and V. Mnih, “Unsupervised learning of object keypoints for perception and control,” _Advances in neural information processing systems_ , vol. 32, 2019\.
  * [15] B. Chen, P. Abbeel, and D. Pathak, “Unsupervised learning of visual 3d keypoints for control,” in _International Conference on Machine Learning_. PMLR, 2021, pp. 1539–1549.
  * [16] T. Jakab, A. Gupta, H. Bilen, and A. Vedaldi, “Unsupervised learning of object landmarks through conditional image generation,” in _Advances in Neural Information Processing Systems_ , vol. 31, 2018.
  * [17] X. Zhou, D. Wang, and P. Krähenbühl, “Objects as points,” _arXiv preprint arXiv:1904.07850_ , 2019.
  * [18] J. Tremblay, T. To, B. Sundaralingam, Y. Xiang, D. Fox, and S. Birchfield, “Deep object pose estimation for semantic robotic grasping of household objects,” in _Conference on Robot Learning_. PMLR, 2018, pp. 306–316.
  * [19] J. Tobin, R. Fong, A. Ray, J. Schneider, W. Zaremba, and P. Abbeel, “Domain randomization for transferring deep neural networks from simulation to the real world,” in _2017 IEEE/RSJ international conference on intelligent robots and systems (IROS)_. IEEE, 2017, pp. 23–30.
  * [20] A. Tsirikoglou, J. Kronander, M. Wrenninge, and J. Unger, “Procedural modeling and physically based rendering for synthetic data generation in automotive applications,” _arXiv preprint arXiv:1710.06270_ , 2017.
  * [21] A. Prakash, S. Boochoon, M. Brophy, D. Acuna, E. Cameracci, G. State, O. Shapira, and S. Birchfield, “Structured domain randomization: Bridging the reality gap by context-aware synthetic data,” in _2019 International Conference on Robotics and Automation (ICRA)_. IEEE, 2019, pp. 7249–7255.
  * [22] J. Tremblay, A. Prakash, D. Acuna, M. Brophy, V. Jampani, C. Anil, T. To, E. Cameracci, S. Boochoon, and S. Birchfield, “Training deep networks with synthetic data: Bridging the reality gap by domain randomization,” in _Proceedings of the IEEE conference on computer vision and pattern recognition workshops_ , 2018, pp. 969–977.
  * [23] B. O. Community, _Blender - a 3D modelling and rendering package_ , Blender Foundation, Stichting Blender Foundation, Amsterdam, 2018. [Online]. Available: http://www.blender.org
  * [24] M. Denninger, M. Sundermeyer, D. Winkelbauer, Y. Zidan, D. Olefir, M. Elbadrawy, A. Lodhi, and H. Katam, “Blenderproc,” _arXiv preprint arXiv:1911.01911_ , 2019.
  * [25] Q. Zhou and A. Jacobson, “Thingi10k: A dataset of 10,000 3d-printing models,” _arXiv preprint arXiv:1605.04797_ , 2016.
  * [26] O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in _International Conference on Medical image computing and computer-assisted intervention_. Springer, 2015, pp. 234–241.
  * [27] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778.
  * [28] S. Van der Walt, J. L. Schönberger, J. Nunez-Iglesias, F. Boulogne, J. D. Warner, N. Yager, E. Gouillart, and T. Yu, “scikit-image: image processing in python,” _PeerJ_ , vol. 2, p. e453, 2014.
  * [29] W. Falcon and The PyTorch Lightning team, “PyTorch Lightning,” 3 2019. [Online]. Available: https://github.com/PyTorchLightning/pytorch-lightning
  * [30] L. Biewald, “Experiment tracking with weights and biases,” 2020, software available from wandb.com. [Online]. Available: https://www.wandb.com/