#  SLUA: A Super Lightweight Unsupervised Word Alignment Model
via Cross-Lingual Contrastive Learning

Di Wu† Liang Ding‡ Shuo Yang‡ Dacheng Tao‡
†Peking University ‡The University of Sydney
<EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS>
###### Abstract

Word alignment is essential for the down-streaming cross-lingual language
understanding and generation tasks. Recently, the performance of the neural
word alignment models Zenkel et al. (2020); Garg et al. (2019); Ding et al.
(2019) has exceeded that of statistical models. However, they heavily rely on
sophisticated translation models. In this study, we propose a Super
Lightweight Unsupervised word Alignment (SLUA) model, in which a bidirectional
symmetric attention trained with a contrastive learning objective is
introduced, and an agreement loss is employed to bind the attention maps, such
that the alignments follow mirror-like symmetry hypothesis. Experimental
results on several public benchmarks demonstrate that our model achieves
competitive, if not better, performance compared to the state of the art in
word alignment while significantly reducing the training and decoding time on
average. Further ablation analysis and case studies show the superiority of
our proposed SLUA. Notably, we recognize our model as a pioneer attempt to
unify bilingual word embedding and word alignments. Encouragingly, our
approach achieves 16.4$\times$ speedup against GIZA++, and 50$\times$
parameter compression compared with the Transformer-based alignment methods.
We will release our code to facilitate the community.

##  1 Introduction

Word alignment, aiming to find the word-level correspondence between a pair of
parallel sentences, is a core component of the statistical machine translation
(SMT, Brown et al. (1993)). It also has benefited several downstream tasks,
_e.g._ , computer-aided translation Dagan et al. (1993), semantic role
labeling Kozhevnikov and Titov (2013), cross-lingual dataset creation Yarowsky
et al. (2001) and cross-lingual modeling Ding et al. (2020a).

Recently, in the era of neural machine translation (NMT, Bahdanau et al.
(2015); Vaswani et al. (2017)), the attention mechanism plays the role of the
alignment model in translation system. Unfortunately, Koehn and Knowles (2017)
show that attention mechanism may in fact dramatically diverge with word
alignment. The works of Li et al. (2019); Ghader and Monz (2017) also confirm
this finding.

Figure 1: Two examples of word alignment. The upper and bottom cases are the
Chinese and Japanese references, respectively.

Although there are some studies attempt to mitigate this problem, most of them
are rely on a sophisticated translation architecture Zenkel et al. (2020);
Garg et al. (2019). These methods are trained with a translation objective,
which computes the probability of each target token conditioned on source
tokens and previous target tokens. This will bring tremendous parameters and
noisy alignments. Most recent work avoids the noisy alignment of translation
models but employed too much expensive human-annotated alignments Stengel-
Eskin et al. (2019). Given these disadvantages, simple statistical alignment
tools, _e.g.,_ FastAlign Dyer et al. (2013) and GIZA++ Och and Ney
(2003)111GIZA++ employs the IBM Model 4 as default setting., are still the
most representative solutions due to its efficiency and unsupervised fashion.
We argue that the word alignment task, intuitively, is much simpler than
translation, and thus should be performed before translation rather than
inducing alignment matrix with heavy neural machine translation models. For
example, the IBM word alignment model, e.g., FastAlign, is the prerequisite of
SMT. However, related research about super lightweight neural word alignment
without NMT is currently very scarce.

Inspired by cross-lingual word embeddings (CLWEs) Luong et al. (2015b), we
propose to implement a super lightweight unsupervised word alignment model in
§3, named SLUA, which encourages the embeddings between aligned words to be
closer. We also provide the theoretical justification from mutual information
perspective for our proposed contrastive learning objective in §3.4,
demonstrating its reasonableness. Figure 1 shows an English sentence, and its
corresponding Chinese and Japanese sentences, and their word alignments. The
links indicate the correspondence between English$\Leftrightarrow$Chinese and
English$\Leftrightarrow$Japanese words. If the Chinese word “举行” can be
aligned to English word “held”, the reverse mapping should also hold.
Specifically, a bidirectional attention mechanism with contrastive estimation
is proposed to capture the alignment between parallel sentences. In addition,
we employ an agreement loss to constrain the attention maps such that the
alignments follow symmetry hypothesis Liang et al. (2006).

Our contributions can be summarized as follows:

  * •

We propose a super lightweight unsupervised alignment model (SLUA), even
merely updating the embedding matrices, achieves better alignment quality on
several public benchmark datasets compare to baseline models while preserving
comparable training efficiency with FastAlign.

  * •

To boost the performance of SLUA, we design a theoretically and empirically
proved bidirectional symmetric attention with contrastive learning objective
for word alignment task, in which we introduce extra objective to follow the
mirror-like symmetry hypothesis.

  * •

Further analysis show that the by-product of our model in training phase has
the ability to learn bilingual word representations, which endows the
possibility to unify these two tasks in the future.

Figure 2: Illustration of the SLUA, where a pair of sentences are given as
example. Each $x_{i}$ and $y_{j}$ are the representation of words in source
and target part respectively. Given $y_{j}$, we can calculate context vector
in source part. The NCE training objective is encouraging the dot product of
this context vector and $y_{j}$ to be large. The process in the other
direction is consistent. By stacking all of the soft weights, two attention
maps $A_{s\rightarrow t}$ and $A_{t\rightarrow s}$ can be produced, which will
be bound by an agreement loss to encourage symmetry.

##  2 Related Work

Word alignment studies can be divided into two classes:

#### Statistical Models

Statistical alignment models directly build on the lexical translation models
of Brown et al. (1993), also known as IBM models. The most popular
implementation of this statistical alignment model is FastAlign Dyer et al.
(2013) and GIZA++ Och and Ney (2000, 2003). For optimal performance, the
training pipeline of GIZA++ relies on multiple iterations of IBM Model 1,
Model 3, Model 4 and the HMM alignment model Vogel et al. (1996). Initialized
with parameters from previous models, each subsequent model adds more
assumptions about word alignments. Model 2 introduces non-uniform distortion,
and Model 3 introduces fertility. Model 4 and the HMM alignment model
introduce relative distortion, where the likelihood of the position of each
alignment link is conditioned on the position of the previous alignment link.
FastAlign Dyer et al. (2013), which is based on a reparametrization of IBM
Model 2, is almost the existing fastest word aligner, while keeping the
quality of alignment.

In contrast to GIZA++, our SLUA model achieves nearly 15$\times$ speedup
during training, while achieving the comparable performance. Encouragingly,
our model is at least 1.5$\times$ faster to train than FastAlign and
consistently outperforms it.

#### Neural Models

Most neural alignment approaches in the literature, such as Alkhouli et al.
(2018), rely on alignments generated by statistical systems that are used as
supervision for training the neural systems. These approaches tend to learn to
copy the alignment errors from the supervising statistical models. Zenkel et
al. (2019) use attention to extract alignments from a dedicated alignment
layer of a neural model without using any output from a statistical aligner,
but fail to match the quality of GIZA++. Garg et al. (2019) represents the
current state of the art in word alignment, outperforming GIZA++ by training a
single model that is able to both translate and align. This model is
supervised with a guided alignment loss, and existing word alignments must be
provided to the model during training. Garg et al. (2019) can produce
alignments using an end-to-end neural training pipeline guided by attention
activations, but this approach underperforms GIZA++. The performance of GIZA++
is only surpassed by training the guided alignment loss using GIZA++ output.
Stengel-Eskin et al. (2019) introduce a discriminative neural alignment model
that uses a dot-product-based distance measure between learned source and
target representation to predict if a given source-target pair should be
aligned. Alignment decisions condition on the neighboring decisions using
convolution. The model is trained using gold alignments. Zenkel et al. (2020)
uses guided alignment training, but with large number of modules and
parameters, they can surpass the alignment quality of GIZA++.

They either use translation models for alignment task, which introduces a
extremely huge number of parameters (compare to ours), making the training and
deployment of the model cumbersome. Or they train the model with the alignment
supervision, however, these alignment data is scarce in practice especially
for low resource languages. These settings make above approaches less
versatile.

Instead, our approach is fully unsupervised at word level, that is, it does
not require gold alignments generated by human annotators during training.
Moreover, our model achieves comparable performance and is at least 50 times
smaller than them, i.e., #Parameters: 4M (ours) vs. 200M (above).

##  3 Our Approach

Our model trains in an unsupervised fashion, where the word level alignments
are not provided. Therefore, we need to leverage sentence-level supervision of
the parallel corpus. To achieve this, we introduce negative sampling strategy
with contrastive learning to fully exploit the corpus. Besides, inspired by
the concept of cross-lingual word embedding, we design the model under the
following assumption: If a target token can be aligned to a source token, then
the dot product of their embedding vectors should be large. Figure-2 shows the
schema of our approach SLUA.

###  3.1 Sentence Representation

For a given source-target sentence pair $(\mathbf{s},\mathbf{t})$,
$s_{i},t_{j}\in\mathbb{R}^{d}$ represent the $i$-th and $j$-th word embeddings
for the source and target sentences, respectively. Luong et al. (2015a); Ding
et al. (2020b) illustrate that modelling the neighbour words within the local
window helps to understand the current words. Inspired by this, we perform a
extremely simple but effective mean pooling operation with the representations
of its surrounding words to capture the contextualized information. Padding
operation is used to ensure the sequence length. As a result, the final
representation of each word can be calculated by element-wisely adding the
mean pooling embedding and its original embedding:

$\begin{split}x_{i}=&\textsc{MeanPool}([s_{i}]^{win})+s_{i},\\\ \end{split}$
(1)

where $win$ is the pooling window size. We can therefore derive the sentence
level representations $(x_{1},x_{2},...,x_{|s|}),(y_{1},y_{2},...,y_{|t|})$
for $\mathbf{s}$ and $\mathbf{t}$. In addition to modeling words, modeling
structured information (such as syntactic information) may be helpful to
enhance the sentence representation Li et al. (2017); Marcheggiani and Titov
(2017); Ding and Tao (2019), thus improveing the word alignment. We leave this
exploration for future work.

###  3.2 Bidirectional Symmetric Attention

Bidirectional symmetric attention is the basic component of our proposed
model. The aim of this module is to generate the source-to-target (aka. s2t)
and target-to-source (aka. t2s) soft attention maps. The details of the
attention mechanism: given a source side word representation $x_{i}$ as query
$q_{i}\in\mathbb{R}^{d}$ and pack all the target tokens together into a matrix
$V_{t}\in\mathbb{R}^{\left|t\right|\times d}$. The attention context can be
calculate as:

$\textsc{Attention}\left(q_{i},V_{t},V_{t}\right)=(a_{t}^{i}\cdot
V_{t})^{\intercal},$ (2)

where the vector $a_{t}^{i}\in\mathbb{R}^{1\times|t|}$ represents the
attention probabilities for $q_{i}$ in source sentence over all the target
tokens, in which each element signifies the relevance to the query, and can be
derived from:

$a_{t}^{i}=\textsc{Softmax}\left(V_{t}\cdot q_{i}\right)^{\intercal}.$ (3)

For simplicity, we denote the attention context of $q_{i}$ in the target side
as $att_{t}(q_{i})$. s2t attention map $A_{s,t}\in\mathbb{R}^{|s|\times|t|}$
is constructed by stacking the probability vectors $a_{t}^{i}$ corresponding
to all the source tokens.

Reversely, we can obtain t2s attention map $A_{t,s}$ in a symmetric way. Then,
these two attention matrices $A_{s,t}$ and $A_{t,s}$ will be used to decode
alignment links. Take s2t for example, given a target token, the source token
with the highest attention weight is viewed as the aligned word.

###  3.3 Agreement Mechanism

Intuitively, the two attention matrices $A_{s,t}$ and $A_{t,s}^{T}$ should be
very close. However, the attention mechanism suffers from symmetry error in
different direction Koehn and Knowles (2017).

To bridge this discrepancy, we introduce agreement mechanism Liang et al.
(2006), acting like a mirror that precisely reflects the matching degree
between $A_{s,t}$ and $A_{t,s}$, which is also empirically confirmed in
machine translation Levinboim et al. (2015). In particular, we use an
agreement loss to bind above two matrices:

$\mathcal{L}\textit{oss}_{disagree}=\sum_{i}\sum_{j}(A^{s,t}_{i,j}-A^{t,s}_{j,i})^{2}.$
(4)

In §4.6, we empirically show this agreement can be complementary to the
bidirectional symmetric constraint, demonstrating the effectiveness of this
component.

Method | EN-FR | FR-EN | sym | RO-EN | EN-RO | sym | DE-EN | EN-DE | sym
---|---|---|---|---|---|---|---|---|---
NNSA | 22.2 | 24.2 | 15.7 | 47.0 | 45.5 | 40.3 | 36.9 | 36.3 | 29.5
FastAlign | 16.4 | 15.9 | 10.5 | 33.8 | 35.5 | 32.1 | 28.4 | 32.0 | 27.0
SLUA | 15.3 | 15.6 | 9.2 | 34.3 | 35.2 | 31.6 | 31.1 | 28.0 | 24.8

Table 1:  AER of each method in different direction. “sym” means grow-diag
symmetrization.

###  3.4 Training Objective and Theoretical Justification

Suppose that $(q_{i},att_{t}(q_{i}))$ is a pair of s2t word representation and
corresponding attention context sampled from the joint distribution
$p_{t}(q,att_{t}(q))$ (hereinafter we call it a positive pair), the primary
objective of the s2t training is to maximize the alignment degree between the
elements within a positive pair. Thus, we first define an alignment function
by using the $\mathrm{sigmoid}$ inner product as:

$\textsc{Align}(q,att_{t}(q))=\sigma(\langle q,att_{t}(q)\rangle),$ (5)

where $\sigma(\cdot)$ denotes the $\mathrm{sigmoid}$ function and
$\langle\cdot,\cdot\rangle$ is the inner product operation. However, merely
optimizing the alignment of positive pairs ignores important positive-negative
relation knowledge Mikolov et al. (2013).

To make the training process more informative, we reform the overall objective
in the contrastive learning manner Saunshi et al. (2019); Oord et al. (2018)
with Noise Contrastive Estimation (NCE) loss Mikolov et al. (2013).
Specifically, we first sample $k$ negative word representations $q_{j}$222In
the contrastive learning setting, $q_{j}$ and $att_{t}(q_{i})$ can be sampled
from different sentences. If $q_{j}$ and $att_{t}(q_{i})$ are from the same
sentence, $i\neq j$; otherwise, $j$ can be a random index within the sentence
length. For simplicity, in this paper, we use $q_{j}$ where $i\neq j$ to
denote the negative samples, although with a little bit ambiguity. from the
margin $p_{t}(q)$. Then, we can formulate the overall NCE objective as
following:

$\displaystyle\mathcal{L}\textit{oss}_{s\rightarrow t}^{i}$
$\displaystyle=-\mathop{\mathbb{E}}_{\\{att_{t}(q_{i}),q_{i},q_{j}\\}}~{}[\log$
(6)
$\displaystyle\frac{\textsc{Align}(q_{i},att_{t}(q_{i}))}{\textsc{Align}(q_{i},att_{t}(q_{i}))+\sum_{j=1}^{k}\textsc{Align}(q_{j},att_{t}(q_{i}))}],$

It is evident that the objective in Eq. (6) explicitly encourages the
alignment of positive pair $(q_{i},att_{t}(q_{i}))$ while simultaneously
separates the negative pairs $(q_{j},att_{t}(q_{i}))$.

Moreover, a direct consequence of minimizing Eq. (6) is that the optimal
estimation of the alignment between the representation and attention context
is proportional to the ratio of joint distribution and the product of margins
$\frac{p_{t}(q,att_{t}(q))}{p_{t}(q)\cdot p_{t}(att_{t}(q))}$ which is the
point-wise mutual information, and we can further have the following
proposition with repect to the mutual information:

######  Proposition 1.

The mutual information between the word representation $q$ and its
corresponding attention context $att_{t}(q)$ is lower-bounded by the negative
$\mathcal{L}\textit{oss}_{s\rightarrow t}^{i}$ in Eq. (6) as:

$I(q,att_{t}(q))\geq\log(k)-\mathcal{L}\textit{oss}_{s\rightarrow t}^{i},$ (7)

where $k$ is the number of the negative samples.

The detailed proof can be found in Oord et al. (2018). Proposition 1 indicates
that the lower bound of the mutual information $I(q,att_{t}(q))$ can be
maximized by achieving the optimal NCE loss, which provides theoretical
guarantee for our proposed method.

Our training schema over parallel sentences is mainly inspired by the
bilingual skip-gram model Luong et al. (2015b) and invertibility modeling
Levinboim et al. (2015). Therefore, the ultimate training objective should
consider both forward ($s\rightarrow t$) and backward ($t\rightarrow s$)
direction, combined with the mirror agreement loss. Technically, the final
training objective is:

$\begin{split}\mathcal{L}\textit{oss}&=\sum_{i}^{\left|t\right|}\mathcal{L}\textit{oss}_{s\rightarrow
t}^{i}+\sum_{j}^{\left|s\right|}\mathcal{L}\textit{oss}_{t\rightarrow
s}^{j}+\alpha\cdot\mathcal{L}\textit{oss}_{disagree},\end{split}$ (8)

where $\mathcal{L}\textit{oss}_{s\rightarrow t}$ and
$\mathcal{L}\textit{oss}_{t\rightarrow s}$ are symmetrical and $\alpha$ is a
loss weight to balance the likelihood and disagreement loss.

Model | EN-FR | RO-EN | DE-EN
---|---|---|---
Naive Attention | 31.4 | 39.8 | 50.9
NNSA | 15.7 | 40.3 | -
FastAlign | 10.5 | 32.1 | 27.0
SLUA | 9.2 | 31.6 | 24.8
Zenkel et al. (2020) | 8.4 | 24.1 | 17.9
Garg et al. (2019) | 7.7 | 26.0 | 20.2
GIZA++ | 5.5 | 26.5 | 18.7

Table 2: Alignment performance (with grow-diagonal heuristic) of each model.
Figure 3: An visualized alignment example. (a-c) illustrate the effects when
gradually adding the symmetric component, (d) shows the result of FastAlign,
and (e) is the ground truth. The more emphasis is placed on the symmetry of
the model, the better the alignment results model achieved. Meanwhile, as
depicted, the results of the attention map become more and more diagonally
concentrated.

##  4 Experiments

###  4.1 Datasets and Evaluation Metrics

We perform our method on three widely used datasets: English-French (EN-FR),
Romanian-English (RO-EN) and German-English (DE-EN). Training and test data
for EN-FR and RO-EN are from NAACL 2003 share tasks Mihalcea and Pedersen
(2003). For RO-EN, we merge Europarl v8 corpus, increasing the amount of
training data from 49K to 0.4M. For DE-EN, we use the Europarl v7 corpus as
training data and test on the gold alignments. All above data are lowercased
and tokenized by Moses. The evaluation metrics are Precision, Recall, F-score
(F1) and Alignment Error Rate (AER).

###  4.2 Baseline Methods

Besides two strong statistical alignment models, i.e. FastAlign and GIZA++, we
also compare our approach with neural alignment models where they induce
alignments either from the attention weights or through feature importance
measures.

#### FastAlign

One of the most popular statistical method which log-linearly reparameterize
the IBM model 2 proposed by Dyer et al. (2013).

#### GIZA++

A statistical generative model Och and Ney (2003), in which parameters are
estimated using the Expectation-Maximization (EM) algorithm, allowing it to
automatically extract bilingual lexicon from parallel corpus without any
annotated data.

#### NNSA

A unsupervised neural alignment model proposed by Legrand et al. (2016), which
applies an aggregation operation borrowed from the computer vision to design
sentence-level matching loss. In addition to the raw word indices, following
three extra features are introduced: distance to the diagonal, part-of-speech
and unigram character position. To make a fair comparison, we report the
result of raw feature in NNSA.

#### Naive Attention

Averaging all attention matrices in the Transformer architecture, and
selecting the source unit with the maximal attention value for each target
unit as alignments. We borrow the results reported in Zenkel et al. (2019) to
highlight the weakness of such naive version, where significant improvement
are achieved after introducing an extra alignment layer.

#### Others

Garg et al. (2019) and Zenkel et al. (2020) represent the current developments
in word alignment, which both outperform GIZA++. However, They both implement
the alignment model based on a sophisticated translation model. Further more,
the former uses the output of GIZA++ as supervision, and the latter introduces
a pre-trained state-of-the-art neural translation model. It is unfair to
compare our results directly with them. We report them in Table 2 as
references.

###  4.3 Setup

For our method (SLUA), all the source and target embeddings are initialized by
xavier method Glorot and Bengio (2010). The embedding size $d$ and pooling
window size are set to 256 and 3, respectively. The hyper-parameters $\alpha$
is tested by grid search from 0.0 to 1.0 at 0.1 intervals. For FastAlign, we
train it from scratch by the open-source
pipeline333https://github.com/lilt/alignment-scripts. Also, we report the
results of NNSA and machine translation based model(Sec.§4.2). All experiments
of SLUA are run on 1 `Nvidia K80` GPU. The CPU model is Intel(R) Xeon(R) CPU
E5-2620 v3 @ 2.40GHz. Both FastAlign and SLUA take nearly half a hour to train
one million samples.

###  4.4 Main Results

Table 2 summarizes the AER of our method over several language pairs. Our
model outperforms all other baseline models. Comparing to FastAlign, we
achieve 1.3, 0.5 and 2.2 AER improvements on EN-FR, RO-EN, DE-EN respectively.

Notably, our model exceeds the naive attention model in a big margin in terms
of AER (ranging from 8.2 to 26.1) over all language pairs. We attribute the
poor performance of the straightforward attention model (translation model) to
its contextualized word representation. For instance, when translating a verb,
contextual information will be paid attention to determine the form (_e.g.,_
tense) of the word, that may interfere the word alignment.

Experiment results in different alignment directions can be found in Table 1.
The grow-diag symmetrization benifits all the models.

###  4.5 Speed Comparison

Take the experiment on EN-FR dataset as an example, SLUA converges to the best
performance after running 3 epochs and taking 14 minutes totally, where
FastAlign and GIZA++ cost 21 and 230 minutes, respectively, to achieve the
best results. Notably, the time consumption will rise dozens of times in
neural translation fashion. All experiments of SLUA are run on a single Nvidia
P40 GPU.

###  4.6 Ablation Study

To further explore the effects of several components (_i.e.,_ bidirectional
symmetric attention, agreement loss) in our SLUA, we conduct an ablation
study. Table 3 shows the results on EN-FR dataset. When the model is trained
using only $\mathcal{L}\textit{oss}_{s\rightarrow t}$ or
$\mathcal{L}\textit{oss}_{t\rightarrow s}$ as loss functions, the AER of them
are quite high (20.9 and 23.3). As expected, combined loss function improves
the alignment quality significantly (14.1 AER). It is noteworthy that with the
rectification of agreement mechanism, the final combination achieves the best
result (9.2 AER), indicating that the agreement mechanism is the most
important component in SLUA.

To better present the improvements brought by adding each component, we
visualize the alignment case in Figure-3. As we can see, each component is
complementary to others, that is, the attention map becomes more diagonally
concentrated after adding the bidirectional symmetric attention and the
agreement constraint.

Setup | P | R | F1 | AER
---|---|---|---|---
$\mathcal{L}\textit{oss}_{s\rightarrow t}$ | 74.9 | 86.0 | 80.4 | 20.9
$\mathcal{L}\textit{oss}_{t\rightarrow s}$ | 71.9 | 85.3 | 77.3 | 23.3
$\mathcal{L}\textit{oss}_{s\leftrightarrow t}$ | 81.5 | 90.1 | 86.1 | 14.1
SLUA | 91.8 | 89.1 | 90.8 | 9.2

Table 3:  Ablation results on EN-FR dataset.

china | distinctive
---|---
EN | DE | EN | DE
china | chinas | distinctive | unverwechselbaren
chinese | china | distinct | besonderheiten
china’s | chinesische | peculiar | markante
republic | chinesischer | differences | charakteristische
china’ | chinesischem | diverse | einzelnen
cat | love
EN | DE | EN | DE
cat | hundefelle | love | liebe
dog | katzenfell | affection | liebt
toys | hundefellen | loved | liebe
cats | kuchen | loves | lieben
dogs | schlafen | passion | lieb

Table 4:  Top 5 nearest English (EN) and German (DE) words for each of the
following words: china, distinctive, easily, cat, love and January.

##  5 Analysis

#### Alignment Case Study

We analyze an alignment example in Figure- 4. Compared to FastAlign, our model
correctly aligns “ _do not believe_ ” in English to “ _glauben nicht_ ” in
German. Our model, based on word representation, makes better use of semantics
to accomplish alignment such that inverted phrase like “ _glauben nicht_ ” can
be well handled. Instead, FastAlign, relied on the positional assumption444A
feature $h$ of position is introduced in FastAlign to encourage alignments to
occur around the diagonal. $h(i,j,m,n)=-\left|\frac{i}{m}-\frac{j}{n}\right|$,
$i$ and $j$ are source and target indices and $m$ and $n$ are the length of
sentences pair., fails here.

#### Word Embedding Clustering

To further investigate the effectiveness of our model, we also analyze the
word embeddings learned by our model. In particular, following Collobert et
al. (2011), we show some words together with its nearest neighbors using the
Euclidean distance between their embeddings. Table- 4 shows some examples to
demonstrates that our learned representations possess a clearly clustering
structure bilingually and monolingually.

We attribute the better alignment results to the ability of our model that
could learn bilingual word representation.

Figure 4: Example of the DE-EN alignment. (a) is the result of FastAlign, and
(b) shows result of our model, which is closer to the gold alignment. The
horizontal axis shows German sentence “wir glauben nicht , da wir nur rosinen
herauspicken sollten .”, and the vertical axis shows English sentence “we do
not believe that we should cherry-pick .”.

##  6 Conclusion and Future Work

In this paper, we presented a super lightweight neural alignment model, named
SLUA, that has achieved better alignment performance compared to FastAlign and
other existing neural alignment models while preserving training efficiency.
We empirically and theoretical show its effectiveness and reasonableness over
several language pairs.

In future works, we would further explore the relationship between CLWEs and
word alignments. A promising attempt is using our model as a bridge to unify
cross-lingual embeddings and word alignment tasks. Also, it will be
interesting to design alignment model in an non-autoregressive fashion Gu et
al. (2018); Wu et al. (2020) to achieve efficient inference.

## References

  * Alkhouli et al. [2018] Tamer Alkhouli, Gabriel Bretschner, and Hermann Ney.  On the alignment problem in multi-head attention-based neural machine translation.  In WMT, 2018.
  * Bahdanau et al. [2015] Dzmitry Bahdanau, KyungHyun Cho, and Yoshua Bengio.  Neural machine translation by jointly learning to align and translate.  In ICLR, 2015.
  * Brown et al. [1993] Peter F Brown, Vincent J Della Pietra, Stephen A Della Pietra, and Robert L Mercer.  The mathematics of statistical machine translation: Parameter estimation.  Computational linguistics, 1993.
  * Collobert et al. [2011] Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa.  Natural language processing (almost) from scratch.  Journal of machine learning research, 2011.
  * Dagan et al. [1993] Ido Dagan, Kenneth Church, and Willian Gale.  Robust bilingual word alignment for machine aided translation.  In Very Large Corpora: Academic and Industrial Perspectives, 1993\.
  * Ding and Tao [2019] Liang Ding and Dacheng Tao.  Recurrent graph syntax encoder for neural machine translation.  arXiv, 2019.
  * Ding et al. [2019] Shuoyang Ding, Hainan Xu, and Philipp Koehn.  Saliency-driven word alignment interpretation for neural machine translation.  WMT, 2019.
  * Ding et al. [2020a] Liang Ding, Longyue Wang, and Dacheng Tao.  Self-attention with cross-lingual position representation.  In ACL, 2020.
  * Ding et al. [2020b] Liang Ding, Longyue Wang, Di Wu, Dacheng Tao, and Zhaopeng Tu.  Context-aware cross-attention for non-autoregressive translation.  In COLING, 2020.
  * Dyer et al. [2013] Chris Dyer, Victor Chahuneau, and Noah A Smith.  A simple, fast, and effective reparameterization of ibm model 2.  In NAACL, 2013.
  * Garg et al. [2019] Sarthak Garg, Stephan Peitz, Udhyakumar Nallasamy, and Matthias Paulik.  Jointly learning to align and translate with transformer models.  In EMNLP, 2019.
  * Ghader and Monz [2017] Hamidreza Ghader and Christof Monz.  What does attention in neural machine translation pay attention to?  In IJCNLP, 2017.
  * Glorot and Bengio [2010] Xavier Glorot and Yoshua Bengio.  Understanding the difficulty of training deep feedforward neural networks.  In ICML, 2010.
  * Gu et al. [2018] Jiatao Gu, James Bradbury, Caiming Xiong, Victor OK Li, and Richard Socher.  Non-autoregressive neural machine translation.  In ICLR, 2018.
  * Koehn and Knowles [2017] Philipp Koehn and Rebecca Knowles.  Six challenges for neural machine translation.  In WNMT, 2017.
  * Kozhevnikov and Titov [2013] Mikhail Kozhevnikov and Ivan Titov.  Cross-lingual transfer of semantic role labeling models.  In ACL, 2013.
  * Legrand et al. [2016] Joël Legrand, Michael Auli, and Ronan Collobert.  Neural network-based word alignment through score aggregation.  In WMT, 2016.
  * Levinboim et al. [2015] Tomer Levinboim, Ashish Vaswani, and David Chiang.  Model invertibility regularization: Sequence alignment with or without parallel data.  In NAACL, 2015.
  * Li et al. [2017] Junhui Li, Deyi Xiong, Zhaopeng Tu, Muhua Zhu, Min Zhang, and Guodong Zhou.  Modeling source syntax for neural machine translation.  In ACL, 2017.
  * Li et al. [2019] Xintong Li, Guanlin Li, Lemao Liu, Max Meng, and Shuming Shi.  On the word alignment from neural machine translation.  In ACL, 2019.
  * Liang et al. [2006] Percy Liang, Ben Taskar, and Dan Klein.  Alignment by agreement.  In NAACL, 2006.
  * Luong et al. [2015a] Minh-Thang Luong, Hieu Pham, and Christopher D Manning.  Effective approaches to attention-based neural machine translation.  In EMNLP, 2015.
  * Luong et al. [2015b] Thang Luong, Hieu Pham, and Christopher D Manning.  Bilingual word representations with monolingual quality in mind.  In NAACL Workshop, 2015.
  * Marcheggiani and Titov [2017] Diego Marcheggiani and Ivan Titov.  Encoding sentences with graph convolutional networks for semantic role labeling.  In EMNLP, 2017.
  * Mihalcea and Pedersen [2003] Rada Mihalcea and Ted Pedersen.  An evaluation exercise for word alignment.  In NAACL, 2003.
  * Mikolov et al. [2013] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean.  Distributed representations of words and phrases and their compositionality.  In NIPS, 2013.
  * Och and Ney [2000] Franz Josef Och and Hermann Ney.  Improved statistical alignment models.  In ACL, 2000.
  * Och and Ney [2003] Franz Josef Och and Hermann Ney.  A systematic comparison of various statistical alignment models.  Computational linguistics, 2003.
  * Oord et al. [2018] Aaron van den Oord, Yazhe Li, and Oriol Vinyals.  Representation learning with contrastive predictive coding.  arXiv, 2018.
  * Saunshi et al. [2019] Nikunj Saunshi, Orestis Plevrakis, Sanjeev Arora, Mikhail Khodak, and Hrishikesh Khandeparkar.  A theoretical analysis of contrastive unsupervised representation learning.  In ICML, 2019.
  * Stengel-Eskin et al. [2019] Elias Stengel-Eskin, Tzu-Ray Su, Matt Post, and Benjamin Van Durme.  A discriminative neural model for cross-lingual word alignment.  In EMNLP, 2019.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In NIPS, 2017.
  * Vogel et al. [1996] Stephan Vogel, Hermann Ney, and Christoph Tillmann.  HMM-based word alignment in statistical translation.  In COLING, 1996.
  * Wu et al. [2020] Di Wu, Liang Ding, Fan Lu, and Jian Xie.  Slotrefine: A fast non-autoregressive model for joint intent detection and slot filling.  In EMNLP, 2020.
  * Yarowsky et al. [2001] David Yarowsky, Grace Ngai, and Richard Wicentowski.  Inducing multilingual text analysis tools via robust projection across aligned corpora.  In HLT, 2001.
  * Zenkel et al. [2019] Thomas Zenkel, Joern Wuebker, and John DeNero.  Adding interpretable attention to neural translation models improves word alignment.  In arXiv, 2019.
  * Zenkel et al. [2020] Thomas Zenkel, Joern Wuebker, and John DeNero.  End-to-end neural word alignment outperforms GIZA++.  In ACL, 2020.


# The effect of dust cooling on low-metallicity star-forming clouds

Gustavo Dopcke, Simon C. O. Glover, Paul C. Clark and Ralf S. Klessen  Zentrum
für Astronomie der Universität Heidelberg, Institut für Theoretische
Astrophysik, Albert-Ueberle-Str. 2, 69120 Heidelberg, Germany

###### Abstract

The theory for the formation of the first population of stars (Pop III)
predicts a IMF composed predominantly of high-mass stars, in contrast to the
present-day IMF, which tends to yield stars with masses less than 1 M⊙. The
leading theory for the transition in the characteristic stellar mass predicts
that the cause is the extra cooling provided by increasing metallicity and in
particular the cooling provided at high densities by dust. The aim of this
work is to test whether dust cooling can lead to fragmentation and be
responsible for this transition. To investigate this, we make use of high-
resolution hydrodynamic simulations. We follow the thermodynamic evolution of
the gas by solving the full thermal energy equation, and also track the
evolution of the dust temperature and the chemical evolution of the gas. We
model clouds with different metallicities, and determine the properties of the
cloud at the point at which it undergoes gravitational fragmentation. We
follow the further collapse to scales of an AU when we replace very dense,
gravitationally bound, and collapsing regions by a simple and nongaseous
object, a sink particle. Our results suggest that for metallicities as small
as 10${}^{-5}\rm Z_{\odot}$, dust cooling produces low-mass fragments and
hence can potentially enable the formation of low mass stars. We conclude that
dust cooling affects the fragmentation of low-metallicity gas clouds and plays
an important role in shaping the stellar IMF even at these very low
metallicities.

###### Subject headings:

early universe — hydrodynamics — methods: numerical — stars: formation —
stars: luminosity function, mass function

##  1\. Introduction

The first burst of star formation in the Universe is thought to give rise to
massive stars, the so-called Population III, with current theory predicting
masses in the range 20-150 M⊙ (Abel et al., 2002; Bromm et al., 2002; O’Shea &
Norman, 2007; Yoshida et al., 2008). This contrasts with present-day star
formation, which tends to yield stars with masses less than 1 M⊙ (Kroupa,
2002; Chabrier, 2003), and so at some point in the evolution of the Universe
there must have been a transition from primordial (Pop. III) star formation to
the mode of star formation we see today (Pop. II/I).
When gas collapses to form stars, gravitational energy is transformed to
thermal energy and unless this can be dissipated in some fashion, it will
eventually halt the collapse. Thermal energy can be dissipated by processes
such as atomic fine structure line emission, molecular rotational or
vibrational line emission, or the heating of dust grains. In some cases, these
processes are able to cool the gas significantly during the collapse. This
temperature drop can promote gravitational fragmentation (Mac Low & Klessen,
2004; Bonnell et al., 2007) by diminishing the Jeans mass, which means that
instead of forming very massive clumps, with fragment masses corresponding to
the initial Jeans mass in the cloud, it can instead form even more fragments
with lower masses.
If the gas is cooled only by molecular hydrogen emission, numerical
simulations show that the stars should be very massive (Abel et al., 2002;
Bromm et al., 2002; O’Shea & Norman, 2007; Yoshida et al., 2008). This happens
because the H2 cooling becomes inefficient for temperatures bellow 200K and
densities above $10^{4}\rm cm^{-3}$. At this temperature and density, the mean
Jeans mass at cloud fragmentation is 1,000 times larger than in present-day
molecular clouds,

$M_{\rm frag}\approx 700M_{\odot}\left(\frac{T_{\rm frag}}{200\rm
K}\right)^{3/2}\left(\frac{n_{\rm frag}}{10^{4}\rm cm^{-3}}\right)^{-1/2},$
(1)

for an atomic gas with temperature $T_{\rm frag}$ and number density $n_{\rm
frag}$.
Metal line cooling and dust cooling are effective at lower temperatures and
larger densities, and so the most widely accepted cause for the transition
from Pop. III to Pop. II is metal enrichment of the interstellar medium by the
previous generations of stars. This suggests that there might be a critical
metallicity Zcrit at which the mode of star formation changes.
The main coolants that have been studied in the literature are CII and OI fine
structure emission (Bromm et al., 2001; Bromm & Loeb, 2003; Santoro & Shull,
2006; Frebel et al., 2007), and dust emission. C and O are identified as the
key species because in the temperature and density conditions that
characterise the early phases of Pop. III star formation, the OI and CII fine-
structure lines dominate over all other metal transitions (Hollenbach & McKee,
1989). By equating the CII or OI fine structure cooling rate to the
compressional heating rate due to free-fall collapse, one can define critical
abundances $[\rm C/H]=-3.5$ and $[\rm O/H]=-3.0$111$[\rm X/\rm
Y]=log_{10}(N_{\rm X}/N_{\rm Y})_{\star}-log_{10}(N_{\rm X}/N_{\rm
Y})_{\odot}$, for elements X and Y, where $\star$ denotes the gas in question,
and where $\rm N_{X}$ and $\rm N_{Y}$ are the mass fractions of the elements X
and Y. for efficient metal line cooling (Bromm & Loeb, 2003). However,
previous works (Jappsen et al., 2009a, b) show that this metallicity threshold
does not represent a critical metallicity: the fact that metal-line cooling
has a larger value than the compressional heating does not necessarily lead to
fragmentation.
Dust-cooling models predict a much lower critical metallicity $(\rm Z_{\rm
crit}\approx 10^{-5}\rm Z_{\odot})$. The conditions for fragmentation in the
low-metallicity dust cooling model are predicted to occur in high density gas,
where the distances between the fragments can be very small (Omukai, 2000;
Omukai et al., 2005; Schneider et al., 2002, 2006; Schneider & Omukai, 2010).
In this regime, interactions between fragments will be common, and analytic
models of fragmentation are unable to predict the mass distribution of the
fragments. A full 3D treatment, following the fragments, is needed.
Initial attempts were made by Tsuribe & Omukai (2006, 2008) and Clark et al.
(2008). However, these treatments used a tabulated equation of state, based on
results from previous one-zone chemical models (Omukai et al., 2005), to
determine the thermal energy. This approximation assumes that the gas
temperature adjusts instantaneously to a new equilibrium temperature whenever
the density changes and hence ignores thermal inertia effects. This may yield
too much fragmentation.
In this work, we improve upon these previous treatments by solving the full
thermal energy equation, and calculating the dust temperature through the
energy equilibrium equation. We assume currently that the only significant
external heat source is the CMB, and include its effects in the calculation of
the dust temperature.

##  2\. Simulations

###  2.1. Numerical method

We model the collapse of a low-metallicity gas cloud using a modified version
of the Gadget 2 (Springel, 2005) smoothed particle hydrodynamics (SPH) code.
To enable us to continue our simulation beyond the formation of the first very
high density protostellar core, we use a sink particle approach (Bate et al.,
1995), based on the implementation of Jappsen et al. (2005). Sink particles
are created once the SPH particles are bound, collapsing, and within an
accretion radius, $h_{acc}$, which is taken to be 1.0 AU. The threshold number
density for sink particle creation is $5.0\times 10^{13}\rm cm^{-3}$. At the
threshold density, the Jeans length at the minimum temperature reached by the
gas is approximately one AU, while at higher densities the gas becomes
optically thick and begins to heat up. Further fragmentation on scales smaller
than the sink particle scale is therefore unlikely to occur. For further
discussion see Clark et al. (2011).
To treat the chemistry and thermal balance of the gas, we use the same
approach as in Clark et al. (2011), with two additions: the inclusion of the
effects of dust cooling, as described below, and formation of H2 on the
surface of dust grains (see Hollenbach & McKee, 1979). The Clark et al. (2011)
chemical network and cooling function were designed for treating primordial
gas and do not include the chemistry of metals such as carbon or oxygen, or
the effects of cooling from these atoms, or molecules containing them such as
CO or H2O. We justify this approximation by noting that previous studies of
very low-metallicity gas (e.g. Omukai et al., 2005, 2010) find that gas-phase
metals have little influence on the thermal state of the gas. Omukai et al.
(2010) showed that H2O and OH are efficient coolants at $10^{8}<n<10^{10}\rm
cm^{-3}$ for their one-zone model. In their hydrodynamical calculations,
however, the collapse is faster, and the effect of H2O and OH is not
perceptible. Therefore we do not expect oxygen-bearing molecules to have a big
effect on the thermal evolution of the gas. For the metallicities and dust-to-
gas ratios considered in this study, the dominant sources of cooling are the
standard primordial coolants (H2 bound-bound emission and collision-induced
emission) and energy transfer from the gas to the dust.

Figure 1.— Results of our low-resolution simulations, showing the dependence
of gas and dust temperatures on gas density for metallicities $10^{-4}$ and
$10^{-5}$ times the solar value. In red, we show the gas temperature, and in
blue the dust temperature for the turbulent and rotating cloud. The simple
core collapse is overploted in dark red and green. The points with thinner
features are from the simulations without rotation or turbulence, while those
showing more scatter come from the simulations with rotation and turbulence.
The dashed lines show constant Jeans mass values.

####  2.1.1 Dust cooling

Collisions between gas particles and dust grains can transfer energy from the
gas to the dust (if the gas temperature $T$ is greater than the dust
temperature $T_{\rm gr}$), or from the dust to the gas (if $T_{\rm gr}>T$).
The rate at which energy is transferred from gas to dust is given by
(Hollenbach & McKee, 1979)

$\Lambda_{\rm gr}=n_{\rm gr}n\bar{\sigma}_{\rm gr}v_{\rm p}f(2kT-2kT_{\rm
gr})\>\>{\rm erg}\>{\rm s^{-1}}\>{\rm cm^{-3}},$ (2)

where $n_{\rm gr}$ is the number density of dust grains, $n$ is the number
density of hydrogen nuclei, $\bar{\sigma}_{\rm gr}$ is the mean dust grain
cross-section, $v_{\rm p}$ is the thermal speed of the proton, and $f$ is a
factor accounting for the ontribution of species other than protons, as well
as for charge and accommodation effects. We assume that $\bar{\sigma}_{\rm
gr}$ is the same as for Milky Way dust, and that the number density of dust
grains is a factor ${\rm Z}/{\rm Z_{\odot}}$ smaller than the Milky Way value.
To compute the rate at which the dust grains radiate away energy, we use the
approximation (Stamatellos et al., 2007)

$\Lambda_{\rm rad}=4\sigma_{\rm sb}n_{\rm gr}\frac{(T_{\rm gr}^{4}-T_{\rm
cmb}^{4})}{\Sigma^{2}\kappa_{R}+\kappa_{P}^{-1}},$ (3)

where $T_{\rm cmb}$ is the CMB temperature, $\sigma_{\rm sb}$ is the Stefan-
Boltzmann constant, $\kappa_{P}$ and $\kappa_{R}$ are the Planck and Rosseland
mean opacities and $\Sigma$ is the column density of gas measured along a
radial ray from the particle to the edge of the cloud.
As explained by Stamatellos et al. (2007), this expression has the correct
behaviour in the optically thin and optically thick limits, and interpolates
between these two limits in a smooth fashion. In practice, we approximate
further by assuming that the Planck and Rosseland mean opacities are equaland
by using the fact that $\Sigma\sim\rho L_{\rm J}$ for a gravitationally
collapsing gas, where $\rho$ is the mass density of the gas, and $L_{\rm J}$
is the Jeans length, given by $L_{\rm J}=(\pi c_{s}^{2}/G\rho)^{1/2}$, where
$c_{s}$ is the speed of sound in the gas. By approximating $\Sigma$ in this
fashion, we avoid the computational difficulties involved with measuring
column densities directly in the simulation, while still following the
behaviour of the gas reasonably accurately in the optically thick regime. In
any case, most of the interesting behaviour that we find in our simulations
occurs while dust cooling remains in the optically thin regime. To compute the
temperature of the dust grains, we assume that the dust is in thermal
equilibrium, and hence solve the equilibrium equation

$\Lambda_{\rm gr}-\Lambda_{\rm rad}=0.$ (4)

This equation is transcendental, so we solve it numerically.

####  2.1.2 Dust opacity

We follow the dust opacity model of Goldsmith (2001), and we calculate the
opacity as a function of the dust temperature in the same fashion as in
Banerjee et al. (2006). To convert from the frequency-dependent opacity given
in Goldsmith (2001) to our desired temperature-dependent mean opacity, we
assume that for dust with temperature $T_{\rm gr}$, the dominant contribution
to the mean opacity comes from frequencies close to a frequency $\bar{\nu}$
that is given by $h\bar{\nu}=\alpha kT_{\rm gr}$, where $\alpha=2.70$. At a
reference temperature $T_{0}=6.75$ K, this procedure yields an opacity

$\begin{array}[]{rl}\kappa(T_{0})=&3.3\times 10^{-26}\alpha(\rm n/2\rho_{\rm
gas})\\\ =&2.664\times 10^{-2}/(1+4[\rm He])\end{array}$ (5)

where [He] is the helium abundance, and $n$ is the number density of hydrogen
nuclei. At other temperatures, $\kappa\propto T_{\rm gr}^{2}$, so long as
$T_{\rm gr}<200$ K. For grain temperatures larger than 200 K, it is necessary
to account for the effects of ice-mantle evaporation, while at much higher
grain temperatures, the opacity falls off extremely rapidly due to the melting
of the grains. We account for these effects (see Semenov et al., 2003) and so
our opacity varies with dust temperature following the relationship

$\kappa=\kappa(T_{0})\times\left\\{\begin{array}[]{ll}T^{2}&\hskip
36.135pt\mbox{T $<$ 200K}\\\ T^{0}&\hskip 36.135pt\mbox{200K $<$ T $<$
1500K}\\\ T^{-12}&\hskip 36.135pt\mbox{T $>$ 1500K}\end{array}\right.$ (6)

###  2.2. Setup and Initial conditions

Resolution | Number of | Particle | Turbulence | Angular
---|---|---|---|---
Level | Particles | Mass |  | Momentum
|  |  ($10^{-5}\rm M_{\odot}$) |  ($E_{\rm turb}/|E_{\rm grav}|$) |  ($E_{\rm rot}/|E_{\rm grav}|$)
High | $40\times 10^{6}$ | $2.5$ | 0.1 | 0.02
Low | $4\times 10^{6}$ | $25.0$ | 0.1 | 0.02
|  |  | 0.0 | 0.00
Table 1Simulation properties.

We performed three sets of simulations, two at low resolution and one at high
resolution. The details are shown in Table 1. Our low resolution simulations
were performed to explore the thermal evolution of the gas during the
collapse, and had 4 million SPH particles which was insufficient to fully
resolve fragmentation. We used these simulations to model the collapse of an
initially uniform gas cloud with an initial number density of $10^{5}\>{\rm
cm^{-3}}$ and an initial temperature of $300\>{\rm K}$. We modelled two
different metallicities (10${}^{-4}{\rm Z}_{\odot}$ and 10${}^{-5}{\rm
Z}_{\odot}$). The initial cloud mass was $1000\>{\rm M_{\odot}}$, and the mass
resolution was $25\times 10^{-3}\>{\rm M_{\odot}}$. In one set of low-
resolution simulations the gas was initially at rest, while in the other, we
included small amounts of turbulent and rotational energy, with $E_{\rm
turb}/|E_{\rm grav}|=0.1$ and $\beta=E_{\rm rot}/|E_{\rm grav}|=0.02$, where
$E_{\rm grav}$ is the gravitational potential energy, $E_{\rm turb}$ is the
turbulent kinetic energy and $E_{\rm rot}$ is the rotational energy. For our
high resolution simulations, which were designed to investigate whether the
gas would fragment, we employed 40 million SPH particles. We adopted initial
conditions similar to those in the low-resolution run with turbulence and
rotation. Again, we simulated two metallicities, 10${}^{-4}{\rm Z}_{\odot}$
and 10${}^{-5}{\rm Z}_{\odot}$. The mass resolution (taken to be 100 times the
SPH particle mass) was $2.5\times 10^{-3}{\rm M}_{\odot}$.

##  3\. Analysis

###  3.1. Thermodynamical evolution of gas and dust

Figure 2.— Number density maps for a slice through the high density region.
The image shows a sequence of zooms in the density structure in the gas
immediately before the formation of the first protostar.

In Figure 1, we compare the evolution of the dust and gas temperatures in the
low-resolution simulations. The dust temperature, shown in the lower part of
the panels, varies from the CMB temperature in the low density region to the
gas temperature at much higher densities. At densities higher than
$10^{11}$–$10^{12}\>\rm cm^{-3}$, dust cooling starts to be effective and
begins to cool the gas. The gas temperature decreases to roughly 600 K in the
$10^{-5}\>\rm Z_{\odot}$ simulations, and 300 K in the Z $=10^{-4}\rm
Z_{\odot}$ case. This temperature decrease significantly increases the number
of Jeans masses present in the collapsing region, making the gas unstable to
fragmentation. The dust and the gas temperatures couple for densities higher
then $10^{13}\rm cm^{-3}$, when the compressional heating starts to dominate
again over the dust cooling. The subsequent evolution of the gas is close to
adiabatic. If we compare the results of the runs with and without rotation and
turbulence, then the most obvious difference is the much greater scatter in
the $n-T$ diagram in the former case. Variations in the infall velocity lead
to different fluid elements undergoing different amounts of compressional
heating. The overall effect is to reduce both the infall velocity and the
average compressional heating rate. This allows dust cooling to dominate at a
density that is up to five times smaller than in the case without rotation or
turbulence. The gas also reaches a lower temperature, cooling down to
$\approx$ 200K (instead of 300K) for the Z $=10^{-4}\rm Z_{\odot}$ case, and
to $\approx$ 400K (instead of 600K) for the Z $=10^{-5}\rm Z_{\odot}$ case.
This behavior shows that it is essential to use 3D simulations to follow the
evolution of the collapsing gas. A similar effect can be seen in Clark et al.
(2011).

###  3.2. Fragmentation

We follow the thermodynamical evolution of the gas up to very high densities
of order $10^{17}\rm cm^{-3}$, where the Jeans mass is $\approx 10^{-2}\rm
M_{\odot}$, and so we need a high resolution simulation to study the
fragmentation behaviour. The transport of angular momentum to smaller scales
during the collapse leads to the formation of a dense disk-like structure,
supported by rotation which then fragments into several objects. Figure 2
shows the density structure in the gas immediately before the formation of the
first protostar. The top-left panel shows a density slice on a scale
comparable to the size of the initial gas distribution. The structure is very
filamentary and there are two main overdense clumps in the center. If we zoom
in on one of the clumps, we see that its internal structure is also
filamentary. We can follow the collapse down to scales of the order of an AU,
but at this point we reach the limit of our computational approach: as the gas
collapses further, the Courant timestep becomes very small, making it
difficult to follow the further evolution of the cloud. In order to avoid this
difficulty, we replace very dense, gravitationally bound, and collapsing
regions by sink particles. Once the conditions for sink particle creation are
met, they start to form in the highest density regions (Figure 3). Due to
interactions with other sink particles that result in an increase in velocity,
some sink particles can be ejected from the high-density region, but most of
the particles still remain within the dense gas. Within 137 years of the
formation of the first sink particle, 45 sink particles have formed. At this
time, approximately $4.6\rm M_{\odot}$ of gas has been accreted by the sink
particles.

Figure 3.— Number density map showing a slice in the densest clump, and the
sink formation time evolution, for the 40 million particles simulation, and Z
= 10-4Z⊙. The box is 100AU x 100AU and the time is measured from the formation
of the first sink particle.

###  3.3. Properties of the fragments

Figure 4 shows the mass distribution of sink particles when we stop the
calculation. We typically find masses below $1\rm M_{\odot}$, with somewhat
smaller values in the $10^{-4}\rm Z_{\odot}$ case compared to the $10^{-5}\rm
Z_{\odot}$ case. Both histograms have the lowest sink particle mass well above
the resolution limit of $0.0025M_{\odot}$. Note that in both cases, we are
still looking at the very early stages of star cluster evolution. As a
consequence, the sink particle masses in Figure 4 are not the same as the
final protostellar masses – there are many mechanisms that will affect the
mass function, such as continuing accretion, mergers between the newly formed
protostars, feedback from winds, jets and luminosity accretion, etc.
Nevertheless, we can speculate that the typical stellar mass is similar to
what is observed for Pop II stars in the Milky Way. This suggests that the
transition from high-mass primordial stars to Population II stars with mass
function similar to that at the present day occurs early in the metal
evolution history of the universe, at metallicities $\rm
Z_{crit}<10^{-5}Z_{\odot}$. The number of protostars formed by the end of the
simulation, for both $10^{-4}Z_{\odot}$ (45) and $10^{-5}Z_{\odot}$ (19)
cases, is much larger than the initial number of Jeans masses (3) in the
cloud.

Figure 4.— Sink particle mass function at the end of the simulations. High and
low resolution results and corresponding resolution limits are shown. To
resolve the fragmentation, the mass resolution should be smaller than the
Jeans mass at the point in the temperature-density diagram where dust and gas
couple and the compressional heating starts to dominate over the dust cooling.
At the time shown, around 5 M⊙ of gas had been accreted by the sink particles
in each simulation.

##  4\. Conclusions

In this paper we have addressed the question of whether dust cooling can lead
to the fragmentation of low-metallicity star-forming clouds. For this purpose
we performed numerical simulations to follow the thermodynamical and chemical
evolution of collapsing clouds. The chemical model included a primordial
chemical network together with a description of dust evolution, where the dust
temperature was calculated by solving self-consistently the thermal energy
equilibrium equation.
We performed three sets of simulations, two at low resolution and one at high
resolution (Table 1). All simulations had an initial cloud mass of 1000 M⊙,
number density of 105 cm-3, and temperature of 300K. We tested two different
metallicities (10${}^{-4}{\rm Z}_{\odot}$ and 10${}^{-5}{\rm Z}_{\odot}$), and
also the inclusion of small amounts of turbulent and rotational energies.
We found in all simulations that dust can effectively cool the gas, for number
densities higher than $10^{11}\rm cm^{-3}$. An increase in metallicity implies
a higher dust-to-gas ratio, and consequently stronger cooling by dust. This is
reflected in a lower temperature of the dense gas in the higher metallicity
simulation.
For the low resolution case, we tested the effect of adding turbulence and
rotation. These diminish the infall velocity, leading to different fluid
elements undergoing different amounts of compressional heating. This lack of
heating allows the gas to reach a lower temperature.
We found that the transport of angular momentum to smaller scales lead to the
formation of a disk-like structure, which then fragmented into a number of low
mass objects.
We conclude that the dust is already an efficient coolant even at
metallicities as low as 10-5 or 10${}^{-4}Z_{\odot}$, in agreement with
previous works (Clark et al., 2008; Omukai et al., 2010; Schneider et al.,
2002, 2006; Tsuribe & Omukai, 2006, 2008). Our results support the idea that
dust cooling can play an important role in the fragmentation of molecular
clouds and the evolution of the stellar IMF.

We thank Tom Abel, Volker Brom, Kazuyuki Omukai, Raffaella Schneider, Rowan
Smith, and Naoki Yoshida for useful comments. The present work is supported by
the _Landesstiftung Baden Württemberg_ via their program International
Collaboration II (grant P-LS-SPII/18), the German _Bundesministerium für
Bildung und Forschung_ via the ASTRONET project STAR FORMAT (grant 05A09VHA),
a Frontier grant of Heidelberg University sponsored by the German Excellence
Initiative, the International Max Planck Research School for Astronomy and
Cosmic Physics at the University of Heidelberg (IMPRS-HD). All computations
described here were performed at the _Leibniz-Rechenzentrum_ , National
Supercomputer HLRB-II (_Bayerische Akademie der Wissenschaften_), and on the
HPC-GPU Cluster Kolob (University of Heidelberg) .

## References

  * Abel et al. (2002) Abel, T., Bryan, G. L., & Norman, M. L. 2002, Science, 295, 93
  * Banerjee et al. (2006) Banerjee, R., Pudritz, R. E., & Anderson, D. W. 2006, MNRAS, 373, 1091
  * Bate et al. (1995) Bate, M. R., Bonnell, I. A., & Price, N. M. 1995, MNRAS, 277, 362
  * Bonnell et al. (2007) Bonnell, I. A., Larson, R. B., & Zinnecker, H. 2007, Protostars and Planets V, 149
  * Bromm et al. (2002) Bromm, V., Coppi, P. S., & Larson, R. B. 2002, ApJ, 564, 23
  * Bromm et al. (2001) Bromm, V., Ferrara, A., Coppi, P. S., & Larson, R. B. 2001, MNRAS, 328, 969
  * Bromm & Loeb (2003) Bromm, V., & Loeb, A. 2003, Nature, 425, 812
  * Chabrier (2003) Chabrier, G. 2003, PASP, 115, 763
  * Clark et al. (2008) Clark, P. C., Glover, S. C. O., & Klessen, R. S. 2008, ApJ, 672, 757
  * Clark et al. (2011) Clark, P. C., Glover, S. C. O., Klessen, R. S., & Bromm, V. 2011, ApJ, 727, 110
  * Frebel et al. (2007) Frebel, A., Johnson, J. L., & Bromm, V. 2007, MNRAS, 380, L40
  * Goldsmith (2001) Goldsmith, P. F. 2001, ApJ, 557, 736
  * Hollenbach & McKee (1979) Hollenbach, D., & McKee, C. F. 1979, ApJS, 41, 555
  * Hollenbach & McKee (1989) —. 1989, ApJ, 342, 306
  * Jappsen et al. (2009a) Jappsen, A., Klessen, R. S., Glover, S. C. O., & Mac Low, M. 2009a, ApJ, 696, 1065
  * Jappsen et al. (2005) Jappsen, A., Klessen, R. S., Larson, R. B., Li, Y., & Mac Low, M. 2005, A&A, 435, 611
  * Jappsen et al. (2009b) Jappsen, A., Mac Low, M., Glover, S. C. O., Klessen, R. S., & Kitsionas, S. 2009b, ApJ, 694, 1161
  * Kroupa (2002) Kroupa, P. 2002, Science, 295, 82
  * Mac Low & Klessen (2004) Mac Low, M., & Klessen, R. S. 2004, Reviews of Modern Physics, 76, 125
  * Omukai (2000) Omukai, K. 2000, ApJ, 534, 809
  * Omukai et al. (2010) Omukai, K., Hosokawa, T., & Yoshida, N. 2010, ApJ, 722, 1793
  * Omukai et al. (2005) Omukai, K., Tsuribe, T., Schneider, R., & Ferrara, A. 2005, ApJ, 626, 627
  * O’Shea & Norman (2007) O’Shea, B. W., & Norman, M. L. 2007, ApJ, 654, 66
  * Santoro & Shull (2006) Santoro, F., & Shull, J. M. 2006, ApJ, 643, 26
  * Schneider et al. (2002) Schneider, R., Ferrara, A., Natarajan, P., & Omukai, K. 2002, ApJ, 571, 30
  * Schneider & Omukai (2010) Schneider, R., & Omukai, K. 2010, MNRAS, 402, 429
  * Schneider et al. (2006) Schneider, R., Omukai, K., Inoue, A. K., & Ferrara, A. 2006, MNRAS, 369, 1437
  * Semenov et al. (2003) Semenov, D., Henning, T., Helling, C., Ilgner, M., & Sedlmayr, E. 2003, A&A, 410, 611
  * Springel (2005) Springel, V. 2005, MNRAS, 364, 1105
  * Stamatellos et al. (2007) Stamatellos, D., Whitworth, A. P., Bisbas, T., & Goodwin, S. 2007, A&A, 475, 37
  * Tsuribe & Omukai (2006) Tsuribe, T., & Omukai, K. 2006, ApJ, 642, L61
  * Tsuribe & Omukai (2008) —. 2008, ApJ, 676, L45
  * Yoshida et al. (2008) Yoshida, N., Omukai, K., & Hernquist, L. 2008, Science, 321, 669


# TacFR-Gripper: A Reconfigurable Fin Ray-Based Compliant Robotic Gripper with
Tactile Skin for In-Hand Manipulation

Qingzheng Cong, Wen Fan, Dandan Zhang Q.Cong, W. Fan are with the Department
of Engineering Mathematics, University of Bristol, affiliated with Bristol
Robotics Lab. D. Zhang is with the Department of Bioengineering, Imperial
College London, and the I-X initiative.

###### Abstract

This paper introduces the TacFR-Gripper, a reconfigurable Fin Ray-based soft
and compliant robotic gripper equipped with tactile skin, which can be used
for dexterous in-hand manipulation tasks. This gripper can adaptively grasp
objects of diverse shapes and stiffness levels. An array of Force Sensitive
Resistor (FSR) sensors is embedded within the robotic finger to serve as the
tactile skin, enabling the robot to perceive contact information during
manipulation. We provide theoretical analysis for gripper design, including
kinematic analysis, workspace analysis, and finite element analysis to
identify the relationship between the gripper’s load and its deformation.
Moreover, we implemented a Graph Neural Network (GNN)-based tactile perception
approach to enable reliable grasping without accidental slip or excessive
force.

Three physical experiments were conducted to quantify the performance of the
TacFR-Gripper. These experiments aimed to i) assess the grasp success rate
across various everyday objects through different configurations, ii) verify
the effectiveness of tactile skin with the GNN algorithm in grasping, iii)
evaluate the gripper’s in-hand manipulation capabilities for object pose
control. The experimental results indicate that the TacFR-Gripper can grasp a
wide range of complex-shaped objects with a high success rate and deliver
dexterous in-hand manipulation. Additionally, the integration of tactile skin
with the GNN algorithm enhances grasp stability by incorporating tactile
feedback during manipulations. For more details of this project, please view
our website: https://sites.google.com/view/tacfr-gripper/homepage.

##  I Introduction

In recent years, soft and compliant robotic grippers have transformed the
landscape of robotics by offering unparalleled flexibility, adaptability, and
safety compared to traditional rigid grippers [1]. A typical example of this
innovation is the Fin Ray-inspired robotic gripper, which draws inspiration
from the fin movements of fish to deliver a versatile and effective gripping
mechanism [2]. It has high flexibility and can easily adapt to an object’s
contour, thus having a greater contact area for grasping [3, 4, 5]. However,
most of the existing Fin Ray-inspired soft robotic grippers do not have in-
hand manipulation functions, which lack the ability to reposition objects
within a robotic hand through the coordination of relative motions between the
object and the hand [6, 7, 8]. Therefore, our research aims to design a
reconfigurable Fin Ray-based robotic gripper, specifically for in-hand
manipulation.

Another important factor in in-hand manipulation is tactile perception [9], as
it provides essential information about the object’s pose, shape, stiffness
and other characteristics, thereby facilitating reliable manipulation.
However, there still remains a research gap in integrating tactile feedback
mechanisms with soft robotic grippers to enhance their reliability and
dexterity for in-hand manipulation of complex-shaped objects. It is important
to augment the soft and compliant gripper with integrated tactile sensing
capabilities to facilitate intricate object manipulations [10]. To this end,
we develop a soft robotic gripper with in-hand manipulation as well as tactile
sensing capabilities.

Figure 1: Overview design of the soft gripper with tactile feedback. (1-Franka
Emika Robot. 2-Reconfigurable Gripper Base driven by servo motors. 3-Fin Ray-
inspired Finger. 4-Tactile skin and visualization of the tactile image
obtained by the embedded sensor in the tactile skin.

Our research seeks to bridge this gap with the introduction of the TacFR-
Gripper, a reconfigurable Fin Ray-inspired soft robotic gripper equipped with
tactile skins. This paper details the design, development, and testing
processes of the TacFR-Gripper. We introduce the integration of the Fin Ray-
inspired soft finger and the tactile skin, which mimics the properties of
human skin by being able to measure external forces and subsequently adjust
the pose of the fingers to accommodate the object in hand. The tactile skin on
each fingertip comprises an array of sensing elements, which allow the robot
to perceive contact information during manipulation and adjust its grip
accordingly. Furthermore, our approach employs machine learning-based tactile
perception to facilitate reliable grasping, aiming to significantly enhance
tactile perception capabilities. More specifically, we use Graph Neural
Networks (GNN) for interpretable and data-efficient tactile sensing. This is
due to the fact that tactile data often possess highly irregular structures,
making them well-suited for representation as graphs [11].

In summary, the main contributions of this paper are listed as follows:

  * •

Development of a low-cost, bio-inspired, soft, and compliant robotic gripper
with in-hand manipulation capabilities.

  * •

Integration of the aforementioned robotic gripper design with tactile skin to
enhance the perception of contact force.

  * •

Utilization of a GNN-based method for reliable tactile perception during
robotic manipulation.

##  II Related Work

###  II-A Soft Robotic Gripper with Tactile Perception

Tactile sensing in soft robotic grippers can help improve object manipulation
reliability by providing direct contact information, which is crucial for
precise in-hand manipulation and feedback control. Sakuma et al. developed a
granular-jamming-based universal soft gripper [12] that was filled with
transparent objects. This gripper included an internal camera to trace markers
on its surface, enabling a 3D reconstruction of the contact surface. A soft
gripper named TaTa [13] incorporates RGB tricolor LEDs and employs an inbuilt
camera to monitor color changes on the contact surface, significantly
enhancing the resolution of the tactile image. However, integrating rigid
elements such as camera modules, circuit boards, and lights into soft fingers
without interfering with the soft gripper’s active deformation poses a
challenge.

Zhao et al. introduced GelSight Svelte [14], a soft, finger-sized device that
employs a camera to capture the reflection from a curved mirror, providing
proprioceptive and tactile information. However, the GelSight Svelte design is
limited due to its lack of compliant properties. Sandra et al. equipped a Fin
Ray-inspired finger with the GelSight tactile sensor [15], enabling the
detection of an object’s pose and facilitating tactile reconstruction. This
setup proved effective for reorienting a glass jar [10]. However, the delicate
nature of the GelSight sensor, tightly integrated with the finger’s structure,
poses durability concerns. Wear and tear, such as tearing at the skin’s
connection to the housing or elastomer shearing from intensive manipulation or
collisions, can damage the sensor. These issues potentially limit the
gripper’s service life and reduce its cost-effectiveness

To acquire comprehensive contact information while ensuring the affordability
of the gripper, we aim to integrate a flexible tactile sensor array onto the
surface of the robotic finger as tactile skin. Although tactile sensor arrays
may exhibit drawbacks like relatively low resolution compared to vision-based
tactile sensors [16, 17], they have inherent advantages in terms of
flexibility and adaptability. Their design facilitates seamless integration
into robots, and their rapid response time makes them exceptionally suited for
in-hand manipulation tasks.

###  II-B Reconfigurable Soft Gripper for In-Hand Manipulation

In recent years, reconfigurable soft grippers have gained increasing attention
in robotics, since they can adjust their shapes and functions based on task
needs. This adaptability allows for in-hand manipulation, which can
significantly enhance the versatility of the robotic gripper [18].

A versatile soft robotic gripper system [19] has been developed by Jin et al.,
which allows the fingers to be reconfigured into various poses, including
scoop, pinch, and claw, utilizing the base of the hand. Since each finger
operates independently in this design, it has advantages not only in grasping
diverse objects but also in performing intricate in-hand manipulations.
Batsuren et al. focused on a gripper with three pneumatic fingers [20], each
containing three air chambers: two for twisting in different directions and
one for grasping. This design enables the gripper to handle objects of varying
shapes and sizes. This design was later improved by inserting a stiff rod that
can be maneuvered inside the central hole of each finger to control the
bending point [21]. This new configuration significantly enhances the
dexterous grasping capabilities of the soft gripper, with a specific emphasis
on in-hand manipulations. Jain et al. developed a gripper equipped with
retractable fingernails and a versatile palm [22], thereby enhancing its
capability for various manipulation tasks. This innovative design enables the
gripper to exert up to 1.8N in normal grasping forces and to handle objects as
thin as 200$\mu$m on flat surfaces.

However, the grippers previously mentioned require molding processes that are
time-consuming and prone to manufacturing errors. Furthermore, the
functionality of these soft grippers heavily depends on the deformation of
their silicone skin, which can lead to operational inaccuracies. Therefore, we
explore the use of multi-material 3D printing technology to fabricate entire
soft grippers efficiently and cost-effectively. With this approach, we aim to
mitigate the manufacturing and operational limitations of traditional soft
grippers and facilitate precise object handling and manipulation.

##  III Methodology

###  III-A Hardware Design

####  III-A1 Sensor Array

The tactile skin on the robotic gripper comprises two distinct layers, as
shown in Fig. 1 (4). The external layer is fabricated using a multi-material
3D printing technique, utilizing a rubbery polymer known as Agilus30. The
internal layer, crucial for the sensor’s functionality, consists of a Force
Sensitive Resistor (FSR) array. This FSR array is designed to emulate human
tactile perception by incorporating a dense array of mechanoreceptors. The
sensor array employs nanomaterials that can alter their electrical properties
under slight forces, transforming physical pressure into measurable electrical
signals. Moreover, silver paste is integrated within the sensor for its
superior electrical conductivity [23], which facilitates the development of
flexible and robust circuits that are crucial for accurately detecting force
within the sensor array. Each sensor unit functions as a variable resistor,
which can dynamically alter its resistance based on the applied force, thereby
enabling precise distributed force map measurement.

####  III-A2 Robot Gripper Design

Figure 2: Hardware overview. (a) Exploded view of the robotic gripper,
3D-printed components (2, 3, 4), a four-bar mechanism, and a linear actuator
(5). (b) Kinematic diagram of the finger’s range of motion, with R1 and R2
denoting pivot radii, and ’y’ representing the linear actuator’s endpoint
displacement. $\theta$ indicates the finger’s bend angle. (c) Illustrative
configurations of the gripper that demonstrate parallel, trigonal, and
T-shaped grasps. (d) A graph of the actuator’s y-displacement against finger
bend angle $\theta$ based on mathematical modeling.

Fig. 2 (a) illustrates the mechanical structure of the robotic gripper, while
Fig. 2 (b) displays the kinematic diagram of the finger and its motion range.
The gripper consists of four main components: a Fin Ray-inspired robotic
finger, a four-bar mechanism, a linear actuator for controlling finger bending
motion, and a palm equipped with two servo motors to reposition the side
fingers relative to the palm. There is an approximately linear relationship
between the coordinates of the linear actuators and the angular displacement
of the finger in the four-bar linkage mechanism. This correlation is further
elucidated in Fig. 2 (d), where the relationship between the finger’s bending
angle $\theta$, and the displacement of the linear actuator $y$ is analyzed
through mathematical modeling.

The reconfiguration of the gripper is enabled by two servo motors assembled in
the hollow area of the palm, which grant rotational movement to the side
fingers to enable reconfigurability of the gripper. Thus, the robotic gripper
can adjust its grips for different types of objects. Fig. 2 (c) illustrates
three general configurations of the gripper. Configuration 1 engages solely in
using the side fingers to grasp targeted objects. Configuration 2 involves the
collaborative effort of all three fingers, making it suitable for grasping
objects with spherical shapes. Configuration 3 is used for grasping elongated
objects in a horizontal position. These configurations exemplify the robotic
gripper’s adaptive design and its ability to cater to a wide range of
manipulative functions in robotic applications.

Figure 3: Visualization of the maximum load analysis using ADAMS simulation
when grasping objects with different shapes.  Figure 4: Contact force for
grasping objects with different shapes, including cube, cylinder, and round
objects.  Figure 5: (a) Visualization of the reachable workspace analysis for
gripper based on Monte Carlo simulation. (b) Histograms showing the
distribution of circumcircle radius for Configuration 1 (left) and
Configuration 2, 3 (right), indicating the overall trend of graspable object
sizes. (c) 3D view of the dexterity workspace and its cross-sectional views
along the X and Y axes for objects with radius between 60 and 80 mm.

###  III-B Theoretical Analysis

####  III-B1 Force Analysis

In the maximum load analysis, simulation is carried out in ADAMS based on the
principles of theoretical mechanics and multi-body dynamics. In the
simulation, an object is positioned at the center of the robotic fingers to
evaluate grasping capabilities. Cubes, cylinders, and round objects are used
for simulation, as shown in Fig. 3.

During the simulation with the cylinder, the maximum deformation observed in
the gripper is 13.62 mm. In a similar test featuring the cube, the deformation
is marginally less, measuring 12.89 mm. When the tip of a Fin-Ray structure is
excessively pressed, it exhibits a pronounced outward deformation in the
opposite direction. This effect is particularly noticeable with shapes vastly
different from a round object such as a sphere. Given its continuous surface
in all directions of the sphere, the Fin-Ray structure can grip more tightly
without creating gaps or losing contact. This leads to a larger contact area
between the object and the gripper [24], resulting in a significantly smaller
deformation, measured at 7.6 mm.

Fig. 4 displays the gripper’s maximum force over time. For the cube and
cylinder, their maximum forces are around 25 N, while for the sphere, it is 12
N. This result closely aligns with the deformation observed. Additionally,
since the contact area for a spherical object is generally larger than that of
the cube and cylindrical ones, the maximum pressure under the same force for
the sphere should also be much smaller.

####  III-B2 Workspace Analysis

Workspace analysis is crucial in robotics for understanding the capabilities
of robotic manipulators or grippers for manipulation [25, 26, 27]. Here, we
use the Monte Carlo method to simulate a large number of potential locations
for the fingers’ tip [28, 29], taking into account the physical constraints
imposed by the joints. Fig. 5 (a) displays the potential contact points
between the gripper’s three fingers and the targeted object for grasping. We
calculate the circumradius of triangles formed by these contact points to
construct a histogram. This histogram contrasts the gripper’s graspability for
specific dimensions (radius) among its total grasping range, showcasing the
gripper’s effectiveness with various object sizes. In Fig. 5 (b), the left
side illustrates Configuration 1, tailored for grasping smaller objects using
a pinching method. The right side of the same figure represents Configurations
2 and 3, which demonstrates the gripper’s suitability for handling medium-
sized objects, specifically those with a radius between 20 to 80mm.

For visualization, Kernel Density Estimation (KDE) with a Gaussian Kernel is
used to create smooth, continuous density estimations[30]. We apply KDE to the
workspace visualization of the proposed gripper, which demonstrates where the
gripper is most capable of handling objects of various sizes. In Fig. 5 (c),
the workspace analysis and visualization of the gripper for grasping objects
with varying radii (from 60 to 80mm) is shown. Darker shades like blue
indicate lower accessibility, while brighter regions (reds and yellows)
indicate higher dexterity.

However, not all dexterity workspace is continuous like Fig. 5. When
manipulating objects with a radius of 80-100 mm, the gripper’s workspace
features two highly dexterous regions on both sides. However, these regions
are separated by an area with lower dexterity, indicating a gap in the
dexterity workspace for objects transitioning between them. For larger
objects, specifically those with a radius between 100-120 mm, the workspace
lacks such a gap but is rather confined and concentrated within a limited
area. This limitation results in a significant reduction in the gripper’s
maneuverability when handling objects that vary in size. The kinematic
structure of the reconfigurable gripper can be further enhanced through
optimization.

###  III-C Tactile Perception For Slip Detection

Figure 6: Tactile sensor readings and contact force recorded from a grip test:
Illustration of the stable period, oscillation period during grasping,
identification of the grasping starting point, and the minimum force
threshold.

####  III-C1 Threshold-Based Method

To determine the minimum force to securely grip objects without slipping, a
calibration method of gradual reduction in gripping force is employed. Fig. 6
shows the curve of the contact force during an object grasping process. We can
easily identify the initial sensor noises, the oscillation period of unstable
grasping, and the stable period of grasping. We can calculate the noise level
when the gripper hasn’t started to grasp objects. After filtering sensor
noise, we can obtain a reasonable threshold value by calculating the mean of
the remaining non-zero readings. We can infer that the lowest reading during
this oscillation represents the minimum force threshold. However, threshold-
based methods require manual tuning of thresholds to adapt to different
conditions. Therefore, we explore machine learning-based approaches, which can
adapt to new operating conditions without re-tuning.

Figure 7: GCN Architectures with 3 layers for sensor data integration. Blue
dots represent sensor nodes, connected within sensor arrays by red lines for
local feature extraction, and across arrays by green lines for data
integration.

####  III-C2 GNN-based Method

GNNs excel at learning complex and nonlinear relationships in non-Euclidean
[31]. This ability makes GNN models more accurate at detecting slip events
compared to simplistic thresholding methods that rely on fixed rules.
Additionally, they can infer the slip state smoothly even when encountering
different objects without any manual tuning of threshold parameters.

Here, we model data from tactile sensors as a graph, where each sensor unit’s
value is represented as a node. These nodes are connected to their neighboring
nodes within the same array. A 4x4 sensor array is used as an example, while
84 edges are formed within each sensor array. We use a Graph Convolutional
Network (GCN) in this paper, which is a type of GNN that applies convolutional
operations directly to graphs. This approach learns features by inspecting
neighboring nodes and aggregating node vectors through convolutional layer
[32]. The inner-connections within each layer are represented by dotted lines,
as shown in Fig. 7. Each node is additionally connected to its corresponding
nodes in the other two sensor arrays, effectively integrating data across
three sensors and adding 32 edges per sensor. These inter-array connections
are highlighted with blue lines. In total, the network comprises 348 edges,
enabling the GCN to aggregate features from multiple layers. This process
facilitates the analysis and extraction of intricate spatial relationships
among the sensor arrays. By aggregating information from each node’s immediate
neighborhood in the first layer, and incrementally broadening this scope in
subsequent layers, the GCN significantly enhances its capability to identify
complex patterns across the three distinct sensor data sets.

The effectiveness of feature learning is highly dependent on the
comprehensiveness of the training data. Thus, we collected a dataset from
experiments using a threshold-based method, categorizing samples under the
minimum force threshold as 0 (non-slip) and those above as 1 (slip), as
depicted in Fig. 6. The dataset encompasses 2100 training and 700 test
samples. In this study, a three-layer GCN model trained on this dataset could
achieve a high test accuracy of 96.2%.

##  IV Experiments

###  IV-A Gripping Capability Evaluation

The first experiment aims to assess the gripping capability of the TacFR-
Gripper. In this experiment, the gripper engages with objects of varying sizes
and repeatedly grasps them ten times to evaluate the success rate across
different configurations. The properties of the objects used in the gripping
experiments are summarized in Table I. We evaluate the success rate by
grasping different objects using three different configurations.

Table I: Summary of the objects used for grasping and slip detection tests,
and their properties including dimension and weight. The dimensions include
the objects’ radius (for a rectangular object, this refers to half of its
width).

Bottle 1 |  Bottle 2 |  Box 1 |  Mango |  Box 2
---|---|---|---|---
|  |  |  |
42 mm |  20 mm |  30 mm |  35 mm |  26 mm
1015 g |  400 g |  380 g |  300 g |  260 g

Box 3 |  Apple |  Orange |  Peper |  Glasses
---|---|---|---|---
|  |  |  |
28 mm |  35 mm |  33 mm |  35 mm |  10 mm
130 g |  94 g |  82 g |  74 g |  63 g

Bottle 3 |  potato |  tomato |  block |  corn
---|---|---|---|---
|  |  |  |
5 mm |  7 mm |  10 mm |  7 mm |  4 mm
16 g |  16 g |  14 g |  14 g |  6 g

The results of the experiment are shown in Fig. 8, where the different colors
of bars represent different configurations. The overall success rate is
approximately 80%, thanks to the gripper’s re-configurable capability,
allowing various configurations to leverage their unique strengths. Smaller
objects like the pizza model and small bottles posed challenges due to their
small dimensions. As indicated by the prior analysis of the workspace, the
gripper encounters difficulties with objects that are either too small (less
than 10mm in radius) or too large (exceeding 125mm in radius).

Configuration 1 excels in the manipulation of smaller objects but lags behind
in comparison to the others. Configuration 2 demonstrates the best overall
performance and has a high success rate in terms of grasping large and medium-
sized objects. In contrast, Configuration 3 particularly excels in grasping
larger, elongated objects. In conclusion, the gripper’s morphological
differences enabled by reconfigurable design significantly enhance its
grasping capabilities. This adaptability is crucial in real-world scenarios
where a diversity of objects demands varied manipulation strategies.

Figure 8: Success rate of grasping various objects using different gripper
configurations.

###  IV-B Performance Evaluation for Different Algorithms

In this set of experiments, our main goal is to assess the effectiveness of
two slip detection methods used for ensuring a secure grasp on objects. The
evaluation is based on three key metrics:

  * •

Success Rate: This metric evaluates how effectively the slip detection methods
prevent object slippage during the grasp. A higher success rate indicates
better performance in avoiding unintended slippage.

  * •

Time Cost: This metric measures the speed at which the gripper identifies slip
events and reacts appropriately. Quick detection and response are crucial for
efficient and timely robotic operations.

  * •

Contact Force: This metric examines the amount of force applied during
grasping. It’s crucial to maintain a secure grip without applying excessive
force, especially when manipulating delicate and soft objects.

Table II: Comparative analysis between the threshold-based method and the GNN-based method for slip detection. | Threshold Approach | GNN Approach
---|---|---
Iteration | 97.5 | 30
Contact Force (N) | 3.2 | 2.35
Success rate (%) | 80 | 80

Based on quantitative data, the GNN-based method demonstrates remarkable
efficiency, surpassing the threshold-based method (baseline) by a factor of
three. In terms of force control, the GNN-based method exhibits a substantial
36% improvement over the baseline, which indicates its superior capability in
adjusting forces during manipulation tasks. However, the GNN-based method
occasionally misinterprets insufficient gripping forces as adequate. We will
investigate the development of high-resolution tactile skin in the future.

###  IV-C Evaluation Of In-Hand Manipulation

In this final experiment, we compare the simulation results and experimental
results for in-hand manipulation. We chose three objects as examples, as shown
in Fig. 9. Based on our experiments and previous workspace analysis, we
discovered that actual displacements in the physical environment align with
our simulation. However, real-world displacements were slightly smaller than
our simulated results from workspace analysis. This discrepancy arises since
our simulations are based on ideal conditions, assuming perfect contact
between the robotic fingers and the object, with guaranteed successful
grasping. In reality, factors such as the object’s shape, weight, and surface
material can influence simulation results. We will enhance the precision of
simulation in the future.

Table III: Comparison between Maximum movement for manipulating different objects in real-world and simulation. Object Radius | Real-World Movement | Simulation Movement
---|---|---
35 mm | 74 mm | 90 mm
65 mm | 95 mm | 100 mm
12 mm | 27 mm | 35 mm
Figure 9: Comparisons between the range of in-hand manipulation between the
simulation from the workspace analysis and the physical experimental results
for manipulating objects with various sizes and shapes. $\Delta{X}$ indicates
the maximum in-hand object translation distance.  Figure 10: Illustration of
the procedures (P1-P4) for the bottle cap removal task.

To demonstrate the effective in-hand manipulation capability of the TacRF-
Gripper, we assembled the gripper on a Franka robotic arm and conducted a
bottle cap removal task. The whole task can be segmented into four procedures
(see Fig. 10, P1-P4):

  * •

Adjust Gripper Pose and Grasp (P1): This step involves determining the optimal
angle for the robotic gripper’s interaction with the bottle cap.

  * •

Bottle Cap Rotation (P2): The robotic gripper rotates the bottle cap using a
servomotor located at the base of its fingers while maintaining consistent
contact.

  * •

Reorientation and Iteration (P3): This procedure involves repositioning the
pose of the robotic gripper and then repeating Procedures P1 and P2
iteratively until the cap becomes loose.

  * •

Bottle Cap Lift-off (P4): In this final step, the robotic gripper grasps the
cap with all three fingers and performs a lift-off action using the robot arm.

To assess the performance of bottle cap removal, experiments are conducted
five times, with data collected specifically during ‘Cap Rotation’. Whenever a
slip is detected, the servomotor stops to allow for re-grasping. Therefore,
the iteration metric serves as an indicator of both the algorithm’s efficiency
and robustness in handling the task.

We compare the GNN-based and threshold-based approach for this application.
The results are summarized in Table II. According to the experimental results,
the GNN-based method outperformed the threshold-based method (baseline) in
maintaining grip force during bottle cap-twisting tasks. The GNN-based
approach leads to reduced task durations, reduced contact force, and increased
success rate.

Table IV: Comparisons between the threshold-based and GNN-based approaches for the bottle cap removal task. | Threshold Approach | GNN Approach
---|---|---
Iteration | 1060 | 780
Contact Force (N) | 5.5 | 4.3
Success rate (%) | 72 | 88

##  V Discussion and Conclusion

In this paper, we introduce the TacFR-Gripper, a reconfigurable Fin Ray-based
soft tactile gripper capable of grasping and manipulating objects of various
sizes. We incorporated a tactile sensor array onto the soft Fin-Ray finger,
which can be known as the tactile skin. We utilize a GNN-based algorithm for
tactile data processing, which significantly enhances slip detection
capabilities. Through rigorous testing and comprehensive analysis, we
demonstrate the TacFR-Gripper’s proficiency in handling a wide range of
delicate objects, and showing its potential applicability in a range of
fields, including industrial automation and medical assistance. Moreover,
experimental comparisons indicate that the proposed GNN-based tactile
perception approach outperforms threshold-based control methods in terms of
accuracy and efficiency.

In our future work, we plan to integrate tactile skin across the entire
gripper surface. More complex tasks will be carried out to further evaluate
the capabilities of the TacFR-Gripper for contact-rich dexterous manipulation.

## References

  * [1] S. Alves, M. Babcinschi, A. Silva, D. Neto, D. Fonseca, and P. Neto, “Integrated design fabrication and control of a bioinspired multimaterial soft robotic hand,” _Cyborg and Bionic Systems_ , vol. 4, p. 0051, 2023.
  * [2] E. Almanzor, T. G. Thuruthel, and F. Iida, “Automated fruit quality testing using an electrical impedance tomography-enabled soft robotic gripper,” in _2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , 2022, pp. 8500–8506.
  * [3] H. DONG, M. BOZKURTTAS, R. MITTAL, P. MADDEN, and G. V. LAUDER, “Computational modelling and analysis of the hydrodynamics of a highly deformable fish pectoral fin,” _Journal of Fluid Mechanics_ , vol. 645, p. 345–373, 2010.
  * [4] M. H. Ali, A. Zhanabayev, S. Khamzhin, and K. Mussin, “Biologically inspired gripper based on the fin ray effect,” in _2019 5th International Conference on Control, Automation and Robotics (ICCAR)_ , April 2019, pp. 865–869.
  * [5] J. Shintake, V. Cacucciolo, D. Floreano, and H. Shea, “Soft robotic grippers,” _Advanced Materials_ , vol. 30, no. 29, p. 1707035, 2018. [Online]. Available: https://onlinelibrary.wiley.com/doi/abs/10.1002/adma.201707035
  * [6] I. M. Bullock and A. M. Dollar, “Classifying human manipulation behavior,” in _2011 IEEE International Conference on Rehabilitation Robotics_. Zurich: IEEE, Jun. 2011, pp. 1–6. [Online]. Available: http://ieeexplore.ieee.org/document/5975408/
  * [7] W. Crooks, G. Vukasin, M. O’Sullivan, W. Messner, and C. Rogers, “Fin ray® effect inspired soft robotic gripper: From the robosoft grand challenge toward optimization,” _Frontiers in Robotics and AI_ , vol. 3, p. 70, 2016.
  * [8] Q. Lu, N. Baron, A. B. Clark, and N. Rojas, “Systematic object-invariant in-hand manipulation via reconfigurable underactuation: Introducing the RUTH gripper,” _The International Journal of Robotics Research_ , vol. 40, no. 12-14, pp. 1402–1418, Dec. 2021. [Online]. Available: http://journals.sagepub.com/doi/10.1177/02783649211048929
  * [9] R. Johansson and J. Flanagan, “Coding and use of tactile signals from the fingertips in object manipulation tasks,” _Nature reviews. Neuroscience_ , vol. 10, pp. 345–59, 05 2009.
  * [10] S. Q. Liu and E. H. Adelson, “Gelsight fin ray: Incorporating tactile sensing into a soft compliant robotic gripper,” in _2022 IEEE 5th International Conference on Soft Robotics (RoboSoft)_ , 2022, pp. 925–931.
  * [11] W. Fan, M. Yang, Y. Xing, N. F. Lepora, and D. Zhang, “Tac-vgnn: A voronoi graph neural network for pose-based tactile servoing,” in _2023 IEEE International Conference on Robotics and Automation (ICRA)_ , 2023, pp. 10 373–10 379.
  * [12] T. Sakuma, F. Von Drigalski, M. Ding, J. Takamatsu, and T. Ogasawara, “A universal gripper using optical sensing to acquire tactile information and membrane deformation,” in _2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2018, pp. 1–9.
  * [13] S. Li, X. Yin, C. Xia, L. Ye, X. Wang, and B. Liang, “Tata: A universal jamming gripper with high-quality tactile perception and its application to underwater manipulation,” in _2022 International Conference on Robotics and Automation (ICRA)_ , 2022, pp. 6151–6157.
  * [14] J. Zhao and E. H. Adelson, “Gelsight svelte: A human finger-shaped single-camera tactile robot finger with large sensing coverage and proprioceptive sensing,” _arXiv preprint arXiv:2309.10885_ , 2023. [Online]. Available: https://arxiv.org/pdf/2309.10885.pdf
  * [15] W. Yuan, S. Dong, and E. H. Adelson, “Gelsight: High-resolution robot tactile sensors for estimating geometry and force,” _Sensors_ , vol. 17, no. 12, p. 2762, 2017.
  * [16] Z. He, X. Zhang, S. Jones, S. Hauert, D. Zhang, and N. F. Lepora, “Tacmms: Tactile mobile manipulators for warehouse automation,” _IEEE Robotics and Automation Letters_ , 2023.
  * [17] R. Li and B. Peng, “Implementing monocular visual-tactile sensors for robust manipulation,” _Cyborg and Bionic Systems_ , 2022.
  * [18] Z. Zhakypov, F. Heremans, A. Billard, and J. Paik, “An origami-inspired reconfigurable suction gripper for picking objects with variable shape and size,” _IEEE Robotics and Automation Letters_ , vol. 3, no. 3, July 2018. [Online]. Available: https://ieeexplore.ieee.org/document/8385192
  * [19] J. H. Low, P. M. Khin, Q. Q. Han, H. Yao, Y. S. Teoh, Y. Zeng, S. Li, J. Liu, Z. Liu, P. Valdivia y Alvarado, I.-M. Chen, B. C. K. Tee, and R. C. H. Yeow, “Sensorized reconfigurable soft robotic gripper system for automated food handling,” _IEEE/ASME Transactions on Mechatronics_ , vol. 27, no. 5, pp. 3232–3243, 2022.
  * [20] K. Batsuren and D. Yun, “Soft robotic gripper with chambered fingers for performing in-hand manipulation,” _Applied Sciences_ , vol. 9, no. 15, p. 2967, Jul. 2019. [Online]. Available: http://dx.doi.org/10.3390/app9152967
  * [21] A. Pagoli, F. Chapelle, J. A. Corrales, Y. Mezouar, and Y. Lapusta, “A soft robotic gripper with an active palm and reconfigurable fingers for fully dexterous in-hand manipulation,” _IEEE Robotics and Automation Letters_ , vol. 6, no. 4, pp. 7706–7713, 2021.
  * [22] S. Jain, T. Stalin, V. Subramaniam, J. Agarwal, and P. V. y Alvarado, “A soft gripper with retractable nails for advanced grasping and manipulation,” in _2020 IEEE International Conference on Robotics and Automation (ICRA)_ , 2020, pp. 6928–6934.
  * [23] Y. Duan, S. He, J. Wu, B. Su, and Y. Wang, “Recent progress in flexible pressure sensor arrays,” _Nanomaterials_ , vol. 12, no. 14, p. 2495, 2022.
  * [24] K. Chen, T. Li, T. Yan, F. Xie, Q. Feng, Q. Zhu, and C. Zhao, “A soft gripper design for apple harvesting with force feedback and fruit slip detection,” _Agriculture_ , vol. 12, no. 11, 2022. [Online]. Available: https://www.mdpi.com/2077-0472/12/11/1802
  * [25] D. Zhang, J. Liu, A. Gao, and G.-Z. Yang, “An ergonomic shared workspace analysis framework for the optimal placement of a compact master control console,” _IEEE Robotics and Automation Letters_ , vol. 5, no. 2, pp. 2995–3002, 2020.
  * [26] J. S. Dai, D. Wang, and L. Cui, “Orientation and workspace analysis of the multifingered metamorphic hand—metahand,” _IEEE Transactions on Robotics_ , vol. 25, no. 4, pp. 942–947, 2009.
  * [27] D. Zhang, J. Liu, L. Zhang, and G.-Z. Yang, “Design and verification of a portable master manipulator based on an effective workspace analysis framework,” in _2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2019, pp. 417–424.
  * [28] J. Li, F. Zhao, X. Li, and J. Li, “Analysis of robotic workspace based on monte carlo method and the posture matrix,” in _2016 IEEE International Conference on Control and Robotics Engineering (ICCRE)_ , 2016, pp. 1–5.
  * [29] D. Zhang, F. Cursi, and G.-Z. Yang, “Wsrender: A workspace analysis and visualization toolbox for robotic manipulator design and verification,” _IEEE Robotics and Automation Letters_ , vol. 4, no. 4, pp. 3836–3843, 2019.
  * [30] SciPy Development Team, “scipy.stats.gaussian_kde — SciPy v1.11.3 Manual,” https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.gaussian_kde.html, 2023, accessed: [16/11].
  * [31] W. Fan, H. Bo, Y. Lin, Y. Xing, W. Liu, N. Lepora, and D. Zhang, “Graph neural networks for interpretable tactile sensing,” in _2022 27th International Conference on Automation and Computing (ICAC)_ , 2022, pp. 1–6.
  * [32] H. Zhu and P. Koniusz, “Simple spectral graph convolution,” in _International conference on learning representations_ , 2020.