# A Simple and Effective Method To Eliminate the Self Language Bias in
Multilingual Representations

Ziyi Yang1 , Yinfei Yang2, Daniel Cer2, Eric Darve1
1Stanford University
<EMAIL_ADDRESS>
2Google Research
<EMAIL_ADDRESS>Work done during internship at Google Research.

###### Abstract

Language agnostic and semantic-language information isolation is an emerging
research direction for multilingual representations models. We explore this
problem from a novel angle of geometric algebra and semantic space. A simple
but highly effective method “Language Information Removal (LIR)” factors out
language identity information from semantic related components in multilingual
representations pre-trained on multi-monolingual data. A post-training and
model-agnostic method, LIR only uses simple linear operations, e.g. matrix
factorization and orthogonal projection. LIR reveals that for weak-alignment
multilingual systems, the principal components of semantic spaces primarily
encodes language identity information. We first evaluate the LIR on a cross-
lingual question answer retrieval task (LAReQA), which requires the strong
alignment for the multilingual embedding space. Experiment shows that LIR is
highly effectively on this task, yielding almost 100% relative improvement in
MAP for weak-alignment models. We then evaluate the LIR on Amazon Reviews and
XEVAL dataset, with the observation that removing language information is able
to improve the cross-lingual transfer performance.

##  1 Introduction

Recently, large-scale language modeling has expanded from English to the
multilingual setting (i.a., Devlin et al. (2019); Conneau and Lample (2019);
Conneau et al. (2020)). Although these models are trained with language
modeling objectives on monolingual data, i.e. without cross-lingual
information, these multilingual systems exhibit impressive zero-shot cross-
lingual ability (Hu et al., 2020b). These observations raise many questions
and provide insight for multilingual representations learning. First, how is
the language identity information and the semantic information expressed in
the representation? Understanding their relations and underlying geometric
structure is crucial for insights into designing more effective multilingual
embedding systems. Second, how can we factor out the language identity
information from the semantic components in representations? In many
application, e.g. cross-lingual semantic retrieval, we wish to only keep the
semantic information. Third, what is the geometric relation between different
languages? Efforts have been made to answer these questions, e.g. Artetxe et
al. (2020); Chung et al. (2020); Lauscher et al. (2020). Such prior work has
addressed the problem at training time. In this work, we systematically
explore a post-training method that can be readily applied to existing
multilingual models.

Figure 1: Language Information Retrieval (LIR) removes language identification
information using principle components of the original representation space.
This mechanism is validated by LIR’s effects demonstrated in Fig. 2.

One of the first attempts in this research area, Roy et al. (2020), proposed
two concepts for language agnostic models: weak alignment v.s. strong
alignment. For multilingual systems with weak alignment, for any item in
language $L_{1}$, the nearest neighbor in language $L_{2}$ is the most
semantically “relevant” item. In the case of strong alignment, for any
representation, all semantically relevant items are closer than all irrelevant
items, regardless of their language. Roy et al. (2020) show sentence
representations from the same language tend to cluster in weak-alignment
system. Similar phenomena can be observed on other pre-trained multilingual
models like mBERT, XLM-R (Conneau et al., 2020) and CMLM (Yang et al., 2020).
Roy et al. (2020) provides carefully-designed training strategies for
retrieval-like model to mitigate this issue in order to obtain language
agnostic multilingual systems.

We systematically explore a simple post-training method we refer to as
Language Information Removal (LIR), to effectively facilitate the language
agnosticism in multilingual embedding systems. First introduced in Yang et al.
(2020) to reduce same language bias for retrieval tasks, the method uses only
linear algebra factorization and post-training operation. LIR can be
conveniently applied to any multilingual model. We show LIR yields
surprisingly large improvements in several downstream tasks, including LAReQA,
a cross-lingual QA retrieval dataset (Roy et al., 2020); Amazon Reviews, a
zero-shot cross lingual evaluation dataset; XEVAL, a collection of
multilingual sentence embedding tasks. Our results suggest that the principal
components of a multilingual system with self-language bias primarily encodes
language identification information. Implementation for LIR is available at
https://github.com/ziyi-yang/LIR.

##  2 Language Information Removal for Self Language Bias Elimination

In this section we describe Language Information Removal (LIR) to address the
self language bias in multilingual embeddings Yang et al. (2020). The first
step is to extract the language identity information for each language space.
Given a multilingual embedding system $E$, e.g. multilingual BERT, and a
collection of multilingual texts $\\{t_{L}^{i}\\}$, where $t_{L}^{i}$ denotes
the $i$th phrase in the collection for the language $L$. We construct a
language matrix ${\bm{M}}_{L}\in\mathbb{R}^{n\times d}$ for language $L$,
where $n$ denotes the number of sentences in language $L$ and $d$ denotes the
dimension of the representation. The row $i$ of ${\bm{M}}_{L}$ is the
representation of $t_{L}^{i}$ computed by $E$.

Second, we extract language identification components for each language. One
observation in multilingual systems is that representations from the same
language tend to cluster together (w.r.t representations in other languages),
even though these representations have different semantic meanings. This
phenomenon is also known as “weak alignment” (Roy et al., 2020). The
mathematical explanation for this clustering phenomenon is that
representations in the same language have shared vector space components. We
propose that these shared components essentially represent the language
identification information. Removing these language components should leave
semantic-related information in the representations.

To remove the shared components, or the language identification from the
representations, we leverage singular value decomposition (SVD) which
identifies the principal directions of a space. We use SVD instead of PCA
since SVD is more stable numerically (e.g. for Läuchli matrix). Specifically,
the SVD of a language matrix is
${\bm{M}}_{L}={\bm{U}}_{L}\bm{\Sigma}_{L}{\bm{V}}_{L}^{T}$, where the columns
of ${\bm{V}}_{L}\in\mathbb{R}^{d\times d}$ are the right singular vectors of
${\bm{M}}_{L}$. We take first $r$ columns of ${\bm{V}}_{L}$ as the language
identification components, denoted as ${\bm{c}}_{L}\in\mathbb{R}^{d\times r}$.
Different values of $r$ are explored in the next experiments section. Language
identification components are removed as follows. Given a multilingual
representation ${\bm{e}}_{L}$ in language $L$, we subtract the projection of
${\bm{e}}_{L}$ onto ${\bm{c}}_{L}$ from ${\bm{e}}_{L}$, i.e.

${\bm{e}}_{L}:={\bm{e}}_{L}-{\bm{c}}_{L}\frac{{\bm{c}}_{L}^{T}{\bm{e}}_{L}}{\|{\bm{e}}_{L}\|_{2}}$
(1)

##  3 Experiments

In the following experiments, sentences used for extracting principle
components are sampled from Wiki-40B (Guo et al., 2020). We use 10,000
sentences per language. We notice performance initially increases as more
sentences are used but then is almost unchanged after $n>10,000$. We tried
different samplings of $\\{t_{L}^{i}\\}$ and text resources other than
Wiki-40B, e.g., Tatoeba Artetxe and Schwenk (2019). The minimal differences in
performance suggest language components are stable over different domains.

###  3.1 Cross-lingual Answer Retrieval

We first examine LIR on LAReQA, a cross-lingual answer retrieval dataset
containing 11 languages (Roy et al., 2020). LAReQA consists of two retrieval
sub-datasets: XQuAD-R and MLQA-R. XQuAD-R is built by translating 240
paragraphs in the SQuAD v1.1 dev set into 10 languages and converting them to
retrieval tasks following the procedure from ReQA (Ahmad et al., 2019).
Similarly, MLQA-R is constructed by converting MLQA (Lewis et al., 2020) to QA
retrieval. In other words, each question in LAReQA has 11 relevant answers,
one in each language. Two retrieval models with self language bias are
presented in the LAReQA original paper, i.e. “En-En” and “X-X”. Specifically,
the multilingual model “En-En” finetunes mBERT for QA retrieval on the 80,000
English QA pairs from the SQuAD v1.1 train set using a ranking loss. The model
“X-X” trains on the translation (into 11 languages) of the SQuAD train set. In
one training example, the question and answer are in the same language. Since
given a question query, all positive examples are within-language, “En-En” and
“X-X” exhibit strong self-language bias and weak-alignment property.

For evaluation, we first compute the language identification components with
“En-En” and “X-X” models released by LAReQA. For testing, language
identification components are removed from question and answer embeddings
following Eq. 1. Results are shown in Table 1 and the evaluation metric is
mean average precision (MAP) of retrieval. Detailed results for each language
are provided in the appendix (Table 5). Simply applying LIR results in
significant improvements, almost $100\%$ relatively for “X-X” model on
XQuAD-R. This huge boost reveals the algebraic structure for multilingual
representation space: in weak-alignment multilingual system, the principal
components primarily encode language information. In LAReQA, each language has
one of the relevant answers. The performance improvement itself already
indicates less language bias.

| XQuAD-R | MLQA-R
---|---|---
| En-En | X-X | En-En | X-X
w/o LIR | 27.8 | 23.3 | 35.7 | 26.0
$r=1$ | 36.7 | 45.2 | 37.0 | 42.4
$r=2$ | 36.7 | 45.6 | 36.2 | 41.6
$r=3$ | 36.5 | 45.9 | 36.3 | 41.6
$r=4$ | 36.4 | 45.7 | 36.1 | 41.4
Table 1: Mean average precision (MAP) of model “En-En” and “X-X” with and
without LIR.

To further illustrate the effect of LIR, we plot the 2D PCA projection of
questions and candidates in Chinese and English for the XQuAD-R dataset.
Without LIR, as plotted on the left of Fig. 2, Chinese and English embeddings
are separated while questions and candidates in the same language are
clustering together111Note these two subfigures on the left are reproduced by
authors to imitate Figure 5 in Roy et al. (2020) in order to better
demonstrate the effects of LIR. This weak-alignment property is especially
prominent for model “X-X”. After applying LIR, the separation between the two
languages vanishes. Questions and candidates embeddings, no matter which
language they are in, group together. Both model “En-En” and “X-X” now exhibit
strong cross-lingual alignment.

Figure 2: PCA projections of English and Chinese embeddings on the XQuAD-R
dataset, with and without LIR. The two subfigures on the left are reproduced
by authors to follow Figure 5 in Roy et al. (2020).

###  3.2 Amazon Reviews

We further evaluate LIR on zero-shot transfer learning with Amazon Reviews
Dataset (Prettenhofer and Stein, 2010). In this subsection, we use
multilingual BERT (Devlin et al., 2019) as the embedding model. Following
Chidambaram et al. (2019), the original dataset is converted to a
classification benchmark by treating reviews of more than 3 stars as positive
and negative otherwise. We split 6000 English reviews in the original training
set into 90% for training and 10% for development. A logistic classifier is
trained on the English training set and then evaluated on English, French,
German and Japanese test sets (each has 6000 examples) using the same trained
model, i.e. the evaluation is zero-shot. The weights for mBERT are fixed. The
representation of a sentence/phrase is computed as the average pooling of the
transformer encoder outputs. LIR is applied in both training and evaluation
stage using the corresponding language components.

Results presented in Table 2 show that removing the language components from
multilingual representation is beneficial for cross-lingual zero-shot transfer
learning of mBERT. LIR is expected to leave only semantic-related information
in the representation so that the logistic classifier trained on English
should be conveniently transferred to other languages. Another interesting
observation is that unlike semantic retrieval, the peak performance usually
occurs at $r>1$.

| en | de | fr | jp | Avg.
---|---|---|---|---|---
w/o LIR | 80.0 | 70.4 | 73.1 | 71.7 | 73.8
$r=1$ | 80.2 | 70.9 | 74.6 | 72.6 | 74.6
$r=2$ | 80.5 | 70.9 | 75.6 | 73.1 | 75.0
$r=3$ | 80.2 | 70.8 | 75.4 | 71.8 | 74.5
$r=5$ | 80.2 | 70.8 | 76.1 | 72.0 | 74.8
$r=8$ | 80.2 | 71.1 | 75.2 | 70.8 | 74.3
$r=10$ | 80.3 | 71.0 | 76.0 | 71.2 | 74.6
$r=12$ | 80.0 | 70.9 | 76.0 | 71.4 | 74.6
Table 2: Classification accuracy on Amazon Reviews Dataset.

###  3.3 XEVAL

We have tested LIR on cross-lingual benchmarks in previous sections. In this
section, we apply LIR in XEVAL, a collection of multilingual sentence
representation benchmark (Yang et al., 2020). The training set and test set of
XEVAL are in the same language (i.e. the evaluation is not cross-lingual).
Benchmarks on XEVAL include Movie Reviews (Pang and Lee, 2005), binary SST
(sentiment analysis, Socher et al. (2013)), MPQA (opinion-polarity, Wiebe et
al. (2005)), TREC (question-type, Voorhees and Tice (2000)), CR (product
reviews, Hu and Liu (2004)), SUBJ (subjectivity/objectivity, Pang and Lee
(2004)) and SICK (both entailment and relatedness (Marelli et al., 2014)). For
this evaluation, we use mBERT as the base multilingual encoder. Still the
weights of mBERT are fixed during training and only downstream neural
structures are trained. The training, cross-validation and evaluation uses
SentEval toolkit (Conneau and Kiela, 2018).

Results are presented in Table 3. The metric is the averaging performance
across 9 datasets mentioned above. Introducing LIR is beneficial on German,
Spanish, French and Chinese. We also notice that for English dataset, removing
principal components actually hurts the performance. This observation also
echoes with findings in previous English sentence embedding works, e.g. Yang
et al. (2019b). We speculate this is because English data are dominant in
mBERT training data. Therefore mBERT representations exhibit similar behaviors
with monolingual English sentence embeddings.

| en | de | es | fr | zh | Avg.
---|---|---|---|---|---|---
w/o LIR | 80.8 | 78.1 | 78.8 | 79.1 | 79.3 | 79.2
$r=1$ | 80.4 | 78.2 | 79.0 | 79.1 | 79.3 | 79.2
$r=2$ | 80.7 | 78.5 | 79.4 | 79.3 | 79.4 | 79.5
$r=5$ | 80.6 | 78.0 | 79.4 | 78.9 | 79.3 | 79.2
$r=10$ | 80.2 | 78.4 | 79.0 | 79.0 | 78.9 | 79.1

Table 3: Results of applying LIR to XEVAL dataset. The metric is the average
of 9 downstream tasks.

###  3.4 Application to Models without Self-Language Bias

In previous sections, we have shown the great effectiveness of LIR on weak-
alignment systems. As an additional analysis, we examine LIR on multilingual
models without self language bias, i.e. models “X-X-mono” and “X-Y” introduced
in the original LAReQA paper. Model “X-X-mono” is modified from “X-X” by
ensuring that each training batch is monolingual so that in-batch negative and
positive examples are in the same language. In model “X-Y”, questions and
answers are allowed to be translated to different languages, which directly
encourage the model to regard answers in different languages from the question
as correct. With such designs in training, “X-X-mono” and “X-Y” are shown to
be without self-language bias, i.e. semantically relevant representations are
closer than all irrelevant items, regardless of their languages.

The evaluation process is similar as in Section 3.1. Results are presented in
Table 4. Applying LIR leads to a slight performance decrease for X-X-mono.
While the drop in X-Y is notable and we suspect this is because the training
process for X-Y avoids, by design, self-language bias. Rather, the principal
components of X-Y contain essential semantic-related information for the
retrieval task. This result is not negative and actually support our argument,
since for “strong alignment” multilingual systems, principal components should
both contain semantic and language-related information. Then removing
principal components will hinder the semantic retrieval. For weak-alignment
models, removing just the first component should be adequate for cross-lingual
retrieval (table 1). For tasks like classification and sentiment analysis
(tables 2 and 3), the optimal number of components to remove seems to vary on
different datasets.

| XQuAD-R | MLQA-R
---|---|---
| X-X-mono | X-Y | X-X-mono | X-Y
w/o LIR | 50.8 | 62.6 | 48.6 | 48.5
$r=1$ | 50.6 | 59.5 | 48.8 | 46.2
$r=2$ | 49.8 | 58.1 | 48.0 | 45.5
$r=3$ | 49.3 | 57.1 | 47.8 | 44.8
$r=4$ | 48.9 | 56.5 | 47.4 | 44.2
Table 4: Mean average precision (MAP) of “X-X-mono” and “X-Y” models without
language bias.

##  4 Related Work & Our Novelty

Different training methods have been proposed to obtain language agnostic
representations. LASER (Artetxe and Schwenk, 2019) leverages translation pairs
and BiLSTM encoder for multilingual sentence representation learning.
Multilingual USE (Yang et al., 2019a) uses training data such as translated
SNLI, mined multilingual QA and translation pairs to learn multilingual
sentence encoder. AMBER (Hu et al., 2020a) aligns contextualized
representations of multilingual encoders at different granularities. LaBSE
(Feng et al., 2020) finetunes a pretrained language model with the bitext
retrieval task and mined cross-lingual parallel data to obtain language
agnostic sentence representations. In contrast, LIR does not require any
parallel data for semantic alignment.

Faruqui and Dyer (2014) propose a canonical correlation analysis (CCA) based
method to add multilingual context to monolingual embeddings. The method is
post-processing and requires bilingual word translation pairs to determine the
projection vectors. In contrast, LIR is post-training and does not require
labeled data. Mrkšić et al. (2017) build semantically specialized cross-
lingual vector spaces. Like CCA, their methods requires the additional
training to adjust the original embeddings using supervised data: cross-
lingual synonyms and antonyms. Libovickỳ et al. (2019) propose that the
language-specific information of mBERT is the centroid of each language space
(the mean of embeddings). Zhao et al. (2021) propose several training
techniques to obtain language-agnostic representations, including segmenting
orthographic tokens in training data and aligning monolingual spaces by
training. In contrast, LIR is post-training and model-agnostic. Critically,
this means LIR can be conveniently and easily applied to any trained
multilingual systems without further training.

Previous explorations on principal components of the semantic space for
sentence embeddings include Arora et al. (2017) and Yang et al. (2019b),
whereby principal component removal is investigated for monolingual models and
the evaluation is only conducted on semantic similarity benchmarks. In
contrast, our work investigates the multilingual case and the evaluation is
more diverse, e.g. cross-lingual transfer learning. Mu and Viswanath (2018)
explore removing top components from English representations. However, it was
unclear prior to our work what purpose is served by removing principal
components within multilingual and cross-lingual settings. We demonstrate
these principal components represent language information for weak-alignment
multilingual models.

Compared with Yang et al. (2020), the novelty of this work is two-fold. First,
it is unclear in Yang et al. (2020) whether the assumption (i.e. principal
components contain language information) holds true for both weak and strong-
alignment multilingual models. In this work we clearly show that it is is
valid for weak-alignment models (Section 3.1). However, for strong-alignment
systems, the assumption is not quite true (Table 4). Second, in Yang et al.
(2020), the evaluation is only conducted on Tatoeba, a semantic retrieval
dataset. While in this work, evaluations are more comprehensive. Besides the
cross-lingual retrieval dataset LAReQA, our experiments include cross-lingual
zero-shot learning (Section 3.2) and monolingual transfer learning (Section
3.3). These extra results establish the effectiveness of LIR beyond the domain
of semantic retrieval.

##  5 Conclusion

In this paper, we investigate the self-language bias in multilingual systems.
We explore a simple method “Language Identity Removal (LIR)”. This method
identifies and removes the language information in multilingual semantic space
by singular value decomposition and orthogonal projection. Although as a
simple and linear-algebra-only method, LIR is highly effective in several
downstream tasks, including zero-shot transfer learning, sentiment analysis,
etc. Especially for cross-lingual retrieval, introducing LIR increases the
performance of weak-alignment multilingual systems by almost 100% relatively
in MAP.

## Acknowledgments

We would like to thank anonymous reviewers for their comments, as well as our
teammates from Descartes, Google Brain and other Google teams for their
valuable feedback.

## References

  * Ahmad et al. (2019) Amin Ahmad, Noah Constant, Yinfei Yang, and Daniel Cer. 2019.  Reqa: An evaluation for end-to-end answer retrieval models.  In _Proceedings of the 2nd Workshop on Machine Reading for Question Answering_ , pages 137–146.
  * Arora et al. (2017) Sanjeev Arora, Yingyu Liang, and Tengyu Ma. 2017.  A simple but tough-to-beat baseline for sentence embeddings.  In _5th International Conference on Learning Representations, ICLR 2017_.
  * Artetxe et al. (2020) Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2020.  On the cross-lingual transferability of monolingual representations.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4623–4637.
  * Artetxe and Schwenk (2019) Mikel Artetxe and Holger Schwenk. 2019.  Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond.  _Transactions of the Association for Computational Linguistics_ , 7:597–610.
  * Chidambaram et al. (2019) Muthu Chidambaram, Yinfei Yang, Daniel Cer, Steve Yuan, Yunhsuan Sung, Brian Strope, and Ray Kurzweil. 2019.  Learning cross-lingual sentence representations via a multi-task dual-encoder model.  In _Proceedings of the 4th Workshop on Representation Learning for NLP (RepL4NLP-2019)_ , pages 250–259, Florence, Italy. Association for Computational Linguistics.
  * Chung et al. (2020) Hyung Won Chung, Thibault Févry, Henry Tsai, Melvin Johnson, and Sebastian Ruder. 2020.  Rethinking embedding coupling in pre-trained language models.  _arXiv preprint arXiv:2010.12821_.
  * Conneau et al. (2020) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Édouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020.  Unsupervised cross-lingual representation learning at scale.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 8440–8451.
  * Conneau and Kiela (2018) Alexis Conneau and Douwe Kiela. 2018.  Senteval: An evaluation toolkit for universal sentence representations.  In _Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)_.
  * Conneau and Lample (2019) Alexis Conneau and Guillaume Lample. 2019.  Cross-lingual language model pretraining.  In _Advances in Neural Information Processing Systems_ , pages 7059–7069.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Faruqui and Dyer (2014) Manaal Faruqui and Chris Dyer. 2014.  Improving vector space word representations using multilingual correlation.  In _Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics_ , pages 462–471.
  * Feng et al. (2020) Fangxiaoyu Feng, Yinfei Yang, Daniel Cer, Naveen Arivazhagan, and Wei Wang. 2020\.  Language-agnostic bert sentence embedding.  _arXiv preprint arXiv:2007.01852_.
  * Guo et al. (2020) Mandy Guo, Zihang Dai, Denny Vrandecic, and Rami Al-Rfou. 2020.  Wiki-40b: Multilingual language model dataset.  In _LREC 2020_.
  * Hu et al. (2020a) Junjie Hu, Melvin Johnson, Orhan Firat, Aditya Siddhant, and Graham Neubig. 2020a.  Explicit alignment objectives for multilingual bidirectional encoders.  _arXiv preprint arXiv:2010.07972_.
  * Hu et al. (2020b) Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. 2020b.  Xtreme: A massively multilingual multi-task benchmark for evaluating cross-lingual generalization.  _arXiv preprint arXiv:2003.11080_.
  * Hu and Liu (2004) Minqing Hu and Bing Liu. 2004.  Mining and summarizing customer reviews.  In _Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining_ , pages 168–177.
  * Lauscher et al. (2020) Anne Lauscher, Vinit Ravishankar, Ivan Vulić, and Goran Glavaš. 2020.  From zero to hero: On the limitations of zero-shot language transfer with multilingual transformers.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 4483–4499.
  * Lewis et al. (2020) Patrick Lewis, Barlas Oguz, Ruty Rinott, Sebastian Riedel, and Holger Schwenk. 2020\.  Mlqa: Evaluating cross-lingual extractive question answering.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7315–7330.
  * Libovickỳ et al. (2019) Jindřich Libovickỳ, Rudolf Rosa, and Alexander Fraser. 2019.  How language-neutral is multilingual bert?  _arXiv preprint arXiv:1911.03310_.
  * Marelli et al. (2014) Marco Marelli, Stefano Menini, Marco Baroni, Luisa Bentivogli, Raffaella Bernardi, Roberto Zamparelli, et al. 2014.  A sick cure for the evaluation of compositional distributional semantic models.  In _LREC_ , pages 216–223.
  * Mrkšić et al. (2017) Nikola Mrkšić, Ivan Vulić, Diarmuid Ó Séaghdha, Ira Leviant, Roi Reichart, Milica Gašić, Anna Korhonen, and Steve Young. 2017.  Semantic specialization of distributional word vector spaces using monolingual and cross-lingual constraints.  _Transactions of the association for Computational Linguistics_ , 5:309–324.
  * Mu and Viswanath (2018) Jiaqi Mu and Pramod Viswanath. 2018.  All-but-the-top: Simple and effective postprocessing for word representations.  In _International Conference on Learning Representations_.
  * Pang and Lee (2004) Bo Pang and Lillian Lee. 2004.  A sentimental education: Sentiment analysis using subjectivity summarization based on minimum cuts.  In _Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics (ACL-04)_ , pages 271–278.
  * Pang and Lee (2005) Bo Pang and Lillian Lee. 2005.  Seeing stars: exploiting class relationships for sentiment categorization with respect to rating scales.  In _Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics_ , pages 115–124.
  * Prettenhofer and Stein (2010) Peter Prettenhofer and Benno Stein. 2010.  Cross-language text classification using structural correspondence learning.  In _Proceedings of the 48th annual meeting of the association for computational linguistics_ , pages 1118–1127.
  * Roy et al. (2020) Uma Roy, Noah Constant, Rami Al-Rfou, Aditya Barua, Aaron Phillips, and Yinfei Yang. 2020.  LAReQA: Language-agnostic answer retrieval from a multilingual pool.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 5919–5930, Online. Association for Computational Linguistics.
  * Socher et al. (2013) Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. 2013.  Recursive deep models for semantic compositionality over a sentiment treebank.  In _Proceedings of the 2013 conference on empirical methods in natural language processing_ , pages 1631–1642.
  * Voorhees and Tice (2000) Ellen M Voorhees and Dawn M Tice. 2000.  Building a question answering test collection.  In _Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval_ , pages 200–207.
  * Wiebe et al. (2005) Janyce Wiebe, Theresa Wilson, and Claire Cardie. 2005.  Annotating expressions of opinions and emotions in language.  _Language resources and evaluation_ , 39(2-3):165–210.
  * Yang et al. (2019a) Yinfei Yang, Daniel Cer, Amin Ahmad, Mandy Guo, Jax Law, Noah Constant, Gustavo Hernandez Abrego, Steve Yuan, Chris Tar, Yun-Hsuan Sung, et al. 2019a.  Multilingual universal sentence encoder for semantic retrieval.  _arXiv preprint arXiv:1907.04307_.
  * Yang et al. (2020) Ziyi Yang, Yinfei Yang, Daniel Cer, Jax Law, and Eric Darve. 2020.  Universal sentence representation learning with conditional masked language model.  _arXiv preprint arXiv:2012.14388_.
  * Yang et al. (2019b) Ziyi Yang, Chenguang Zhu, and Weizhu Chen. 2019b.  Parameter-free sentence embedding via orthogonal basis.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 638–648.
  * Zhao et al. (2021) Wei Zhao, Steffen Eger, Johannes Bjerva, and Isabelle Augenstein. 2021.  Inducing language-agnostic multilingual representations.  In _Proceedings of *SEM 2021: The Tenth Joint Conference on Lexical and Computational Semantics_ , pages 229–240, Online. Association for Computational Linguistics.

##  Appendix A Experimental results for each language of model “X-X” on LAReQA

Here we provide the detailed experiment results of each language on the
XQuAD-R dataset. The multilingual encoder is model “X-X”.

| w/o LIR | $r=1$ | $r=2$ | $r=3$ | $r=4$
---|---|---|---|---|---
ar | 20.5 | 40.5 | 40.4 | 40.4 | 40.0
de | 27.5 | 48.3 | 49.8 | 49.7 | 49.6
el | 20.9 | 43.5 | 43.9 | 44.1 | 44.2
en | 27.3 | 55.1 | 55.0 | 55.3 | 55.3
es | 27.6 | 52.6 | 52.8 | 52.7 | 52.6
hi | 18.6 | 36.5 | 36.8 | 37.5 | 37.3
ru | 24.9 | 48.2 | 49.6 | 49.6 | 49.4
th | 16.8 | 34.7 | 35.1 | 34.9 | 34.6
tr | 23.8 | 45.3 | 45.4 | 46.3 | 46.2
vi | 24.8 | 49.2 | 48.9 | 48.9 | 48.6
zh | 24.7 | 43.8 | 43.8 | 45.3 | 45.2
Table 5: Experimental results for each language of model “X-X” on the XQuAD-R
dataset.


# PINN-Ray: A Physics-Informed Neural Network to Model Soft Robotic Fin Ray
Fingers

Xing Wang1, Joel Janek Dabrowski2, Josh Pinskier1, Lois Liow1, Vinoth
Viswanathan1,
Richard Scalzo3, David Howard1

###### Abstract

Modelling complex deformation for soft robotics provides a guideline to
understand their behaviour, leading to safe interaction with the environment.
However, building a surrogate model with high accuracy and fast inference
speed can be challenging for soft robotics due to the nonlinearity from
complex geometry, large deformation, material nonlinearity etc. The reality
gap from surrogate models also prevents their further deployment in the soft
robotics domain.

In this study, we proposed a physics-informed Neural Networks (PINNs) named
PINN-Ray to model complex deformation for a Fin Ray soft robotic gripper,
which embeds the minimum potential energy principle from elastic mechanics and
additional high-fidelity experimental data into the loss function of neural
network for training. This method is significant in terms of its
generalisation to complex geometry and robust to data scarcity as compared to
other data-driven neural networks. Furthermore, it has been extensively
evaluated to model the deformation of the Fin Ray finger under external
actuation. PINN-Ray demonstrates improved accuracy as compared with Finite
element modelling (FEM) after applying the data assimilation scheme to treat
the sim-to-real gap. Additionally, we introduced our automated framework to
design, fabricate soft robotic fingers, and characterise their deformation by
visual tracking, which provides a guideline for the fast prototype of soft
robotics.

11footnotetext:  CSIRO Robotics, Data61, CSIRO, Australia22footnotetext:
Statistical Machine Learning Group, Data61, CSIRO, Australia33footnotetext:
Computational Modelling Group, Data61, CSIRO, Australia

##  I Introduction

Soft robotics concerns the design of robotic parts made from inherently
compliant and low-stiffness materials. Owing to their flexibility, soft
robotics offer safety advantages over rigid robotics when used to grasp or
interact with fragile objects [1]. However, the flexibility in soft robotics
results in indefinite degrees of freedom (DoF) which makes it difficult to
construct accurate and robust models that are able to achieve advanced tasks
such as real-time control or computational design optimisation [2].

Extensive research has led to several approaches to improve accuracy and speed
in soft robotics. These approaches can be classified into 3 main categories:
physics-driven modelling [3, 4], data-driven machine learning modelling [5],
and hybrid physics-guided machine learning modelling [6, 7].

Physics-driven modelling models the soft robots with specific models, such as
analytical geometry modelling [8], finite element modelling (FEM) [9], and
physical simulators [10]. Physics-rich analytical modelling using continuum
mechanics or geometry modelling, can however be extremely challenging for soft
robotics with complex geometry due to the nonlinear structure and material
properties, and potentially nonlinear actuation mechanism (e.g. pneumatic
air). This results in the sim-to-real gap where the model estimations can
differ from reality and it is typically difficult to assimilate real-world
data into the models to overcome this sim-to-real gap.

Data-driven modelling does not include any physics models but rather aims to
fit machine learning models to large multi-fidelity datasets. Artificial
neural networks are most commonly used and are applied to model soft robots’
kinematics, static, and dynamics [11]. Data-driven modelling is a mesh-free
method but requires a large amount of training data to obtain a competent
surrogate model. The experimental data can be expensive to obtain.
Experimental data can be replaced or supplemented with simulation data,
however this can be at the cost of accuracy due to the necessary approximation
in the simulation parameters. Furthermore, although these models can close the
sim-to-real gap, this is typically only applicable in proximity to where
training data was provided. Without the physics, these models can tend to be
inaccurate in scenarios where training data is scarce.

Recent developments combine physics-driven and data-driven modelling methods,
leading to a physics-guided machine learning (PGML) [12]. A highly promising
approach is the physics-informed neural network (PINN), which embeds physical
laws in the form of partial differential equations (PDEs) as constraints into
the optimisation process of data-driven artificial neural networks. As the
artificial neural network is trained to comply with the physical laws, it
provides a robust surrogate model [13]. Furthermore, the PINN does not require
large amounts of labelled data (if any) for training and this provides a
significant advantage over purely data-driven modelling approaches.

Figure 1: The framework of PINN-Ray to model the deformation of soft robotic
finger with sim-to-real treatment.

In this work, we propose a machine learning-based modelling method to predict
the deformation of soft robotics under external actuation, named PINN-Ray. The
model is trained by incorporating the principle of minimum potential energy
into the loss function, utilizing elastic mechanics theory. Additionally, we
introduce data assimilation into the model where measurements from a real Fin
Ray finger are incorporated into the training process. Our approach addresses
the limitations of current learning-based methods by demonstrating robustness
in scenarios with data scarcity. Two variations of PINN-Ray (one with and one
without data assimilation) are employed to estimate the nonlinear displacement
field of a Fin Ray finger and contrasted against the FEM. We show that our
PINN-ray with data assimilation provides superior accuracy. Additionally, we
present a real-time image processing technique that is employed to monitor the
key points indicative of Fin Ray finger deformation for more efficient
performance evaluation. In summary, the contributions of this work are as
follows:

  * •

Present a PINN-Ray that combines physics from elastic mechanics theory and
machine learning to predict the soft robotic deformation

  * •

Provide a sim-to-real treatment by a modified loss function from experimental
data assimilation

  * •

Demonstrate an image processing scheme for intrinsic key points tracking to
efficiently characterise the soft robotic deformation

  * •

Provide in-depth evaluation and comparison between PINN-Ray, FEM, and
experimental results in modelling the soft robotic deformation

The remainder of the paper is organized as follows: Section II discusses the
relevant modelling methods, then section III introduces the detailed
methodology. The experiment results and discussion are presented in Section
IV, followed by the conclusion in Section VI.

##  II Related work

Extensive research efforts have been devoted to modeling the deformation,
kinematics, statics, and dynamics of soft robotics. Geometrical modeling
stands out as a favored approach in physics-driven modeling for soft continuum
robots, which offers a robust model to capture their shape variations,
leveraging techniques such as constant curvature or specific geometric models.
For example, Alici et al. modelled the deformation of a PneuNet soft robotic
finger with the constant curvature assumption [14]. Mbakop et al. utilised a
Pythagorean Hodograph (PH) curves with prescribed lengths to reconstruct the
robot shapes [15]. Nevertheless, the scope of geometric modeling is
constrained by the physical shapes of soft robots, making it impractical for
those with intricate geometries Continuum mechanics offers another avenue for
modeling, driven by physics. It encompasses classic elastic and hyperelastic
theories, which find extensive application in the field of soft robotics. For
example, Trivedi et al. presented a model that accurately represents the large
deformations and loading of a soft robotics manipulator, using the Neo-Hookean
model for nonlinear elasticity and the Cosserat rod theory for the dynamics
[16]. The analytical approach using continuum mechanics requires precisely
account for the specific geometry and shape of soft robots, making it
challenging to generalize for other designs. This demands significant effort,
even for seasoned researchers.

Data-driven modelling has enabled the creation of surrogate models in soft
robotics, reducing the reliance on traditional physical laws. Giorell et al.
trained a data-driven neural network to model the cable-driven actuation force
that can manipulate the soft robots to a certain end position. The training
data were collected using cable tension, while an infrared camera was used to
capture the tip positions. Extensive experimental data was acquired to for
training and validation and residual between prediction and experimental data
was minimised in the loss function [17]. However, data-driven modelling method
is known for suffering from the challenges from data deficiencies.

The latest advancements in physics-informed machine learning have illuminated
the modeling of soft robotics, representing a hybrid approach that integrates
principles from both physics and data-driven machine learning. Different
strategies can be implemented to embed physical laws: train the data-driven
machine learning with raw data generated by governing equations rather than
collecting them from expensive and time-consuming experiments [18]; design a
meticulous neural network architecture to formulate physical model [19], and
lastly embed physical laws like continuum mechanics into loss function during
neural network training [20]. There have been scant endeavors to apply PINN in
the realm of soft robotics. Nava et al. proposed a novel and fully
differentiable hybrid approach to fluid-structure interaction, which includes
a U-net shaped physics-constrained neural network surrogate to simulate the
flow field and then forward swimming of a soft-body fish [21]. PINN enables an
accurate and efficient computational of swimming motion. Recently, Sun et al.
added the physics laws including the elastic, viscous and Coulomb friction
force to correct the pressure reading into recurrent neural network which was
used to indirectly sensing the bending angle/force of the soft pneumatic
actuators [7]. Their hybrid model significantly improve the prediction
accuracy for soft robotics displacement. Finally, PINNs have been used to
model wild fires with data assimilation and uncertainty quantification [22].
The data assimilation improves the accuracy of the PINN by assimilating
observations of fire-front locations into the model.

##  III Methodology

###  III-A Elastic theory

In this work, we make the following assumptions to the material property used
for soft robotics: continuity, uniformity and isotropy. The constitutional
material for soft robots are normally inherent compliant and flexible. This
study applied a linear elasticity material model to characterise the
deformations.

Considering a Fin Ray finger design that occupies a two-dimensional spatial
domain $\Omega$. Under external actuations, the Fin Ray finger will deform and
its displacement field is given by $u$ in the horizontal or $x$-direction and
$v$ in the vertical or $y$-direction [23]. The total potential energy in the
design domain is given by

$\Pi(u,v)=U+W,$ (1)

where $U$ is the total internal potential energy and $W$ is the potential
energy from external actuation such as a force or moment.

The internal energy is a result of stresses and strains in the body material
and is given by

$U=\frac{1}{2}\int_{\Omega}\sigma_{ij}\epsilon_{ij}d\Omega,$ (2)

where $\sigma_{ij}$ is the stress tensor, $\varepsilon_{ij}$ is the strain
tensor, and $i,j$ represent coordinate directions in index
notation111Introductions to index notation can be found in standard elasticity
theory books such as [24]..

The stress, strain, and deformation are related from the physical equations
[25]. Under static equilibrium conditions [24]

$\frac{\partial\sigma_{ij}}{\partial x_{j}}+f_{i}=0,$ (3)

where $f_{i}$ is the body force. The strain tensor is given by

$\varepsilon_{ij}=\frac{1}{2}\left(\frac{\partial u_{i}}{\partial
z_{j}}+\frac{\partial u_{j}}{\partial z_{i}}\right),$ (4)

where $\varepsilon_{ij}$ represents the strain tensor, $u_{i}$ denotes the
displacement component, and $z_{j}$ represents a dimension (e.g., $z_{1}\equiv
x$ and $z_{2}\equiv y$ in the two-dimensional case). The stress tensor is
given by

$\sigma_{ij}=C_{ijkl}\varepsilon_{kl}$ (5)

where $C_{ijkl}$ is the stiffness tensor that depends on the body material and
$i,j,k$, and $l$ are indices used for the index notation.

In two dimensions, the constitution equations for plain strain are derived
using Hooks law and can be simplified to the following [26]

$\displaystyle\sigma_{xx}$
$\displaystyle=\frac{E\mu}{(1+\mu)(1-\mu)}(\varepsilon_{xx}+\varepsilon_{yy})+\frac{E}{1+\mu}\varepsilon_{xx},$
(6) $\displaystyle\sigma_{yy}$
$\displaystyle=\frac{E\mu}{(1+\mu)(1-\mu)}(\varepsilon_{xx}+\varepsilon_{yy})+\frac{E}{1+\mu}\varepsilon_{yy},$
(7) $\displaystyle\sigma_{xy}$
$\displaystyle=\frac{E}{1+\mu}\varepsilon_{xy},$ (8)

where $E$ is the Young’s modulus and $\mu$ is Poisson’s ratio.

Given the form of the stress equation in (4) and strain equations in (6), (7)
and (8), the total energy in equation (1) is in the form of a partial
differential equation (PDE) of the displacement field $(u,v)$ with respect to
$x$ and $y$. The aim is to solve these equations to obtain the displacement
field resulting from external actuation (under static equilibrium conditions).

###  III-B physics-informed Neural Network

Figure 2: PINN-Ray architecture: two artificial neural networks are used to
predict u and v respectively. The minimum potential energy method is
implemented in the loss function

The PINN approach to solving the PDE is formulate the problem as an
optimisation problem. Under static equilibrium conditions, the total energy in
the system given by (1) should be zero. PINN-Ray uses a neural network
$\psi(x,y;\theta)$ parameterised by $\theta$ to estimate the displacement
field $(u,v)$ and optimise a neural network to minimise the total energy
$\Phi(u,v)$ of the system. Let $\mathcal{L}_{\Pi}(\psi(x,y;\theta)$ be a loss
function that represents the total energy in the system according to (1) (see
section III-C). The optimal parameters of the neural network are then given by

$\displaystyle\theta^{*}=\arg\min_{\theta}\mathcal{L}_{\Pi}(\psi(x,y;\theta))$
(9)

The neural network $\psi(x,y;\theta)$ takes the coordinates $(x,y)$ as inputs
and predicts the displacement field $(u,v)$. Given that a neural network is
differentiable, the partial derivatives of $u$ and $v$ with respect to $x$ and
$y$ are computed using automatic differentiation (AutoDiff) [27]. Given these
partial derivatives, the stress tensor in (4) and the strain tensors in (6),
(7) and (8) can be calculated and used to compute the total energy in (1).
Given the total energy, the neural network parameters can be optimised
according to (9). This process is illustrated in Figure 2.

In this study, the fully connected neural network architecture proposed in
[28] is used. Given an input tensor $X$ (which is $(x,y)$ in this study), the
output of an $L$-layered neural network can be computed as follows [28]

$\displaystyle A=\phi\left(XW^{1}+b^{1}\right),\quad
B=\phi\left(XW^{2}+b^{2}\right),$ $\displaystyle
H^{(1)}=\phi\left(XW^{z,1}+b^{z,1}\right),$ $\displaystyle
Z^{(k)}=\phi\left(H^{(k)}W^{z,k}+b^{z,k}\right),\quad k=1,\dots,L,$
$\displaystyle H^{(k+1)}=(1-Z^{(k)})\odot A+Z^{(k)}\odot B,\quad k=1,\dots,L,$
$\displaystyle\psi(X;\theta)=H^{(L+1)}W+b,$

where $\phi$ is an activation function, $W$ are weight matrices, and $b$ are
bias vectors. The weight matrices and bias vectors are learnable parameters
and are represented by $\theta$.

In this study, two neural networks $\psi_{u}(x,y;\theta_{u})$ and
$\psi_{v}(x,y;\theta_{v})$ are used to separately output $u$ and $v$
respectively. One neural network could have been used to output a tensor
comprising both $u$ and $v$, but we found that two separate neural networks
performed better. We configure both neural networks to use the hyperbolic
tangent function as an activation function. Furthermore, both neural networks
comprise $L=4$ layers, where each layer contains 64 neurons.

###  III-C Loss function

To perform optimisation in the neural network the energy is required to be
represented in the form of a loss function. The PINN-Ray loss function will
optimise over the total energy PDE, the boundary conditions, and data
assimilation. This loss function is given by

$\displaystyle\mathcal{L}_{\Pi}=\mathcal{L}_{\text{PDE}}+\lambda_{\text{BC}}\mathcal{L}_{\text{BC}}+\lambda_{\text{ASM}}\mathcal{L}_{\text{ASM}},$
(10)

where $\mathcal{L}_{\text{PDE}}$ is the total energy PDE loss,
$\mathcal{L}_{\text{BC}}$ is the boundary condition loss,
$\mathcal{L}_{\text{ASM}}$ is the data assimilation loss, and
$\lambda_{\text{BC}}$ and $\lambda_{\text{ASM}}$ are hyper-parameters that
specify the weighting of their respective loss component.

Working backwards through the loss terms, for the data assimilation loss, we
assume that a set of locations $(x_{\text{ASM}}^{(i)},y_{\text{ASM}}^{(i)})$
with corresponding displacements
$(u_{\text{ASM}}^{(i)},v_{\text{ASM}}^{(i)})$, $i=1,\dots,N_{\text{ASM}}$ are
available. These pairs are typically measured on a physical body. We can then
construct the following Mean Squared Error (MSE) loss function

$\displaystyle\mathcal{L}_{\text{ASM}}=\frac{1}{N_{\text{ASM}}}\sum_{i=1}^{N_{\text{ASM}}}$
$\displaystyle\bigg{[}\left(u_{\text{ASM}}^{(i)}-\psi_{u}(x_{\text{ASM}}^{(i)},y_{\text{ASM}}^{(i)})\right)^{2}$
$\displaystyle+\left(v_{\text{ASM}}^{(i)}-\psi_{v}(x_{\text{ASM}}^{(i)},y_{\text{ASM}}^{(i)})\right)^{2}\bigg{]}$
(11)

In the energy approach the free boundaries have natural boundary conditions,
which are naturally satisfied [24]. This leaves the essential boundary
conditions such as fixed boundary conditions. Fixed boundaries are associated
with no displacement and are thus Dirichlet boundary conditions where
$u|_{\text{fixed}}=0$ and $v|_{\text{fixed}}=0$. In this study, we include an
additional boundary condition associated with forcing, where a force along a
boundary causes the boundary to displace by a specific known amount. Thus,
given a set of locations
$(x_{\text{BC}}^{(i)},y_{\text{BC}}^{(i)}),~{}i=1,\dots,N_{\text{BC}}$ along
the fixed boundary and a set of locations along the forced boundary
$(x_{\text{F}}^{(j)},y_{\text{F}}^{(j)})$ with corresponding displacements
$(u_{\text{F}}^{(j)},v_{\text{F}}^{(j)})$, $j=1,\dots,N_{\text{F}}$. We thus
construct the following MSE loss function

$\displaystyle\mathcal{L}_{\text{BC}}=$
$\displaystyle\frac{1}{N_{\text{BC}}}\sum_{i=1}^{N_{\text{BC}}}\bigg{[}\left(\psi_{u}(x_{\text{BC}}^{(i)},y_{\text{BC}}^{(i)})\right)^{2}+\left(\psi_{v}(x_{\text{BC}}^{(i)},y_{\text{BC}}^{(i)})\right)^{2}\bigg{]}$
$\displaystyle+\frac{1}{N_{\text{F}}}\sum_{j=1}^{N_{\text{F}}}\bigg{[}\left(u_{\text{F}}^{(j)}-\psi_{u}(x_{\text{F}}^{(j)},y_{\text{F}}^{(j)})\right)^{2}$
$\displaystyle+\left(v_{\text{F}}^{(j)}-\psi_{v}(x_{\text{F}}^{(j)},y_{\text{F}}^{(j)})\right)^{2}\bigg{]}$
(12)

Finally, the total energy PDE loss is given by the MSE of (1), which is
computed using the neural network outputs and AutoDiff. In this particular
study, we use a forced boundary condition in $\mathcal{L}_{\text{BC}}$ which
accounts for the external energy applied to the system. We thus set $W=0$ in
(1) and the total energy is equivalent to the total internal energy $U$ in
(2). We approximate the integral in (2) with a Monte Carlo estimate. For this,
we sample a set of collocation points
$(x_{\text{C}}^{(i)},y_{\text{C}}^{(i)})$, $i=1,\dots,N_{\text{C}}$ over the
domain of $\Omega$, pass these through the neural network(s) to obtain
$(\hat{u}^{(i)},\hat{v}^{(i)})$ (which are the neural network estimates of
$(u,v)$ at the given location). The PDE loss function is then given by

$\displaystyle\mathcal{L}_{\text{PDE}}=\frac{1}{N_{C}}\sum_{i=1}^{N_{\text{C}}}\Pi(\hat{u}^{(i)},\hat{v}^{(i)})$
(13)

##  IV Experiment

###  IV-A Design space & manufacturing

The robotic finger underwent design via geometrical construction using the
Cadquery Python library, streamlined to be governed by 8 key parameters for 2D
modeling [29, 30]. Refer to the dimensions outlined in the accompanying Figure
3 (a) for specifics. Then it was discretized into triangle elements using Gmsh
library [31], where all the nodes were utilised as sampling points for
training.

Figure 3: (a) Design space of the proposed soft robotics fingers, which is
determined by 9 parameters, (b) Uniform sampling point discretization of the
design space, while the design uses parameter values of [90 mm, 30 mm, 20 mm,
20 mm, 200, 4, 2 mm, 2 mm, 2 mm]

We numerically tested the Fin Ray finger design with design parameters
$[H,L,W,L_{angle},L_{tip},N,t_{rib},t_{flex},t_{back}]$, which have values of
[90 mm, 30 mm, 20 mm, 20 ∘, 20 mm, 4, 2 mm, 2 mm, 2mm], as illustrated in
Figure 3 (b). The bottom edge is fixed as the boundary condition. We have
assigned additional markers for further experimental deformation tracking, as
shown in Figure 4.

The robotic finger underwent one-shot 3D printing, where its CAD model was
first sliced using GrabCAD. Next, the design was fabricated by the Stratasys
J850 Polyjet printer, which seamlessly integrated markers into the process in
a single step. Polyjet printer ejected liquid photopolymer into the building
tray with multiple heads, which was then solidified under ultraviolet light.
No support structure was enabled during the printing for a fast prototype. A
digital material with shore hardness 85A was selected by digitally mixing two
raw material: VeroUltra black and Agilus30 clear, whose material properties
was determined using uniaxial tensile test [32]. The properties of the
material are such that Young’s modulus is $E=11.4$ MPa and Poisson’s ratio is
$\mu=0.45$.

###  IV-B Experiment setup

Figure 4: Left: Fin Ray finger under testing in the Jimstron test rig. Upper
right: Fin Ray finger with no load. Lower right: Fin Ray finger under load.
The markers are printed with robotic finger using a multi-material 3D printing
step

To experimentally evaluate the deformation of the soft robotic finger, we used
a customised automated testing rig named Jimstron, as shown in Figure 4. The
travel platform on Jimstron primarily undergoes vertical extension or
contraction through the servo motor’s action. At the end of this platform, we
attached a 3D printed object for interacting with the soft robotic finger
affixed to the base platform in the horizontal direction. The rigid object
moved toward the finger, causing a displacement upon contact, initiating
deformation. The fixed positions of the robotic finger were adjusted to enable
contact points at various segments of the robotic finger.

An RGB camera was positioned to capture the perspective view of the soft
finger, which captured the deformation images of the soft finger after the
travel platform reached to the preset distance. The deformation was examined
by monitoring the alterations in marker coordinates. To pinpoint their actual
positions in the physical world, we set up a visual tracking system integrated
with an image processing algorithm. We opted for an Intel web camera and pre-
calibrated it using the Matlab Calibration Toolkit to acquire the intrinsic
matrix.

Figure 5:  Marker positions in pixel coordinates which are acquired by an
image processing algorithm applied to images captured by a calibrated camera.
Top: initial state; Bottom: actuated state

The movement of the finger was recorded within a 2D pixel frame and then
translated into the designated global/world frame, which is attached to the
corner of the Fin Ray finger as illustrated in Figure 5. The coordinate is in
SI units as a fixed frame of reference. An image processing algorithm was used
to analyze the coordinates of all markers, extracting their color and shape
characteristics, and locating their central positions within the pixel frame.
The video stream frames were first converted to the BGR color space. Next, a
medium blur with a size of 3 was applied to reduce noise. Following this, the
frames were transformed into the CIELAB color space for color detection.
Further geometry detection was achieved by applying a Gaussian blur operation,
followed by Hough Circle detection.

The detection positions were converted from pixel frame to world frame
afterwards using a linear transform. The marker measurements at the initial
and final static positions of the finger were were smoothed by averaging the
measurements over several video frames while the finger remained in a static
position. Given these smoothed measurements, the measured displacement is
calculated by subtracting the marker positions at the final finger position
from the marker positions at the initial finger position. The smoothed
measured locations for the initial position denoted by
$(x_{\text{M}}^{(i)},y_{\text{M}}^{(i)})$ and the measured displacements are
denoted by $(u_{\text{M}}^{(i)},v_{\text{M}}^{(i)})$, where $i=1,\dots 9$.
Note that the markers are indexed from left to right in Figure 5.

As PINN-Ray is compared with the FEM, we use the FEM mesh to generate the
collocation points $(x_{\text{C}}^{(i)},y_{\text{C}}^{(i)})$,
$i=1,\dots,N_{\text{C}}$ (randomly sampled locations over the domain $\Omega$
could also be used). A total of $N_{\text{C}}=90959$ points are used. For the
boundary along the base, a set of $N_{\text{BC}}=1000$ evenly spaced locations
are drawn from the base edge. For the forced boundary condition, the force is
applied at location (35,0) to produce a displacement of (0mm, -10mm) and thus
$N_{\text{F}}=1$ location point is drawn for this position. Finally, for the
data assimilation, the measurements of the 9 markers at the fingers initial
position are used and thus $N_{\text{ASM}}=9$.

##  V Results & Discussion

We compare two variations of PINN-Ray to COMSOL FEM. The two variations are
PINN-Ray with and without data assimilation and are referred to as PINN-
Ray${}_{\text{asm}}$ and PINN-Ray${}_{\text{std}}$) respectively. Both PINN-
Rays are trained over 60,000 epochs with a learning rate of $10^{-3}$, weight
factors of $\lambda_{\text{BC}}=\lambda_{\text{ASM}}=1000$, and using the ADAM
optimisation approach. The learning rate and weight factors were determined by
grid searches over the ranges $10^{k}$ where $k=-4,\dots,-2$ for the learning
rate and $k=-1,\dots,3$ for the weight factors. Larger weights for the
boundary and assimilation loss functions ensured that these conditions were
more strongly enforced.

To compare the methods, the Absolute Error (AE) and Mean Absolute Error (MAE)
are used. For a given displacement estimate $(\hat{u}^{(i)},\hat{v}^{(i)})$ at
the location $(x_{\text{M}}^{(i)},x_{\text{M}}^{(i)})$ and the corresponding
measured displacements $(u_{\text{M}}^{(i)},v_{\text{M}}^{(i)})$, the AE for
marker $i$ is given by

$\displaystyle\text{AE}^{(i)}=|\hat{u}^{(i)}-u_{\text{M}}^{(i)}|+|\hat{v}^{(i)}-v_{\text{M}}^{(i)}|$
(14)

The MAE is then given by

$\displaystyle\text{MAE}=\frac{1}{N_{ASM}}\sum_{i=1}^{9}\text{AE}^{(i)}$ (15)

Illustrations of the actual displacements and the estimated displacements by
the FEM, PINN-Ray${}_{\text{std}}$, and PINN-Ray${}_{\text{asm}}$ are provided
in Figure 6. The AE for these estimated displacements are illustrated in
Figure 7 and the MAE values are presented in Table I. The FEM and PINN-
Ray${}_{\text{std}}$ produce very similar results. PINN-Ray${}_{\text{std}}$
is slightly less accurate for markers further away from the base (e.g.,
markers 7, 8 and 9). PINNs have been known to struggle to propagate
information from boundary conditions further into the spatial domain and this
could possibly be improved by increasing the number of training epochs. PINN-
Ray${}_{\text{asm}}$ achieves very low AE values for all markers and provides
a significant improvement over both the FEM and PINN-Ray${}_{\text{std}}$.

(a) Finger displacement with an overlay of the FEM estimate.

(b) Finger displacement with an overlay of the PINN-Ray${}_{\text{std}}$
estimate.

(c) Finger displacement with an overlay of the PINN-Ray${}_{\text{asm}}$
estimate.

Figure 6: Photographs of the displaced finger with the estimated displacements
overlayed in blue. The black markers indicate the positions of the finger
markers.

(a) Absolute error for $u$.

(b) Absolute error for $v$.

Figure 7: Comparisons of the absolute errors for each marker produced by the FEM and PINN-Ray${}_{\text{asm}}$. TABLE I: MAE for the FEM, PINN-Ray${}_{\text{std}}$, and PINN-Ray${}_{\text{asm}}$ | FEM |  PINN-Ray${}_{\text{std}}$ |  PINN-Ray${}_{\text{asm}}$
---|---|---|---
$u$ | 1.41 | 1.37 | 0.03
$v$ | 0.39 | 0.48 | 0.00
$disp$ | 1.463 | 1.452 | 0.03

The log of the PDE loss ($\log\mathcal{L}_{\text{PDE}}$) for PINN-
Ray${}_{\text{std}}$ and PINN-Ray${}_{\text{asm}}$ are plotted over the
training epochs in Figure 8. Including data assimilation significantly lowers
the overall loss. With a lower loss, PINN-Ray${}_{\text{asm}}$ has optimised
to a lower energy level, which indicates that it has found a more accurate
solution to the PDE.

Figure 8: The log-loss ($\log\mathcal{L}_{\text{PDE}}$) over the training
epochs.

Finally, the stresses and strains in the material as estimated by PINN-
Ray${}_{\text{std}}$ and PINN-Ray${}_{\text{asm}}$ are provided in Figs. 9 and
10. As PINN-Ray${}_{\text{asm}}$ has been shown to find a more optimal PDE
solution, the estimates of the stresses and strains are likely to be more
accurate. As shown in the figures, both the stresses and strains are
underestimated by PINN-Ray${}_{\text{std}}$. Underestimating the stresses and
strains can be highly detrimental when a designer is considering yield points,
elasticity, and toughness of the material.

Figure 9: Stress estimates over the finger material. Figure 10: Strain
estimates over the finger material.

##  VI Conclusion

To address the challenges of present modelling methods for soft robotics, this
work proposes a novel physics-informed neural network that combines physic
laws and machine learning components, namely PINN-Ray. We demonstrate the
detailed components of the network architecture, including the physical laws
from elastic theory and nonlinear approximator artificial neural network. Most
importantly, we provide a sim-to-real treatment involving experimental data
assimilation to improve the modelling accuracy. The proposed method is tested
and validated on the classic Fin Ray robotic finger, whose deformation under
external actuation is captured from FEM, PINN-Ray${}_{\text{std}}$, and PINN-
Ray${}_{\text{asm}}$.

The MAE results suggest that PINN-Ray${}_{\text{std}}$ achieves competitive
accuracy compared with high-fidelity FEM. PINN-Ray${}_{\text{asm}}$ then
significantly improves accuracy with experimental data assimilation ending up
with an MAE of 0.03 as compared with 1.463 for the displacement, which
validates its effectiveness as a treatment to bridge the sim-to-real gap for
soft robotic deformation. Note that only 9 data points were assimilated to
achieve these results. Additionally, it improves the convergence speed and can
provide more accurate estimations of latent variables such as stress and
strain, which serve as the critical index to the durability evaluation of soft
robotics design.

## Acknowledgement

This work is supported by the Science and Industry Endowment Fund and CSIRO’s
Future Digital Manufacturing Fund.

## References

  * [1] D. Rus and M. T. Tolley, “Design, fabrication and control of soft robots,” _Nature_ , vol. 521, no. 7553, pp. 467–475, 2015.
  * [2] F. Chen and M. Y. Wang, “Design optimization of soft robots: A review of the state of the art,” _IEEE Robotics & Automation Magazine_, vol. 27, no. 4, pp. 27–43, 2020.
  * [3] L. Qin, H. Peng, X. Huang, M. Liu, and W. Huang, “Modeling and simulation of dynamics in soft robotics: A review of numerical approaches,” _Current Robotics Reports_ , pp. 1–13, 2023.
  * [4] C. Armanini, F. Boyer, A. T. Mathew, C. Duriez, and F. Renda, “Soft robots modeling: A structured overview,” _IEEE Transactions on Robotics_ , 2023.
  * [5] O. Yasa, Y. Toshimitsu, M. Y. Michelis, L. S. Jones, M. Filippi, T. Buchner, and R. K. Katzschmann, “An overview of soft robotics,” _Annual Review of Control, Robotics, and Autonomous Systems_ , vol. 6, pp. 1–29, 2023.
  * [6] G. Mengaldo, F. Renda, S. L. Brunton, M. Bächer, M. Calisti, C. Duriez, G. S. Chirikjian, and C. Laschi, “A concise guide to modelling the physics of embodied intelligence in soft robotics,” _Nature Reviews Physics_ , vol. 4, no. 9, pp. 595–610, 2022.
  * [7] W. Sun, N. Akashi, Y. Kuniyoshi, and K. Nakajima, “Physics-informed recurrent neural networks for soft pneumatic actuators,” _IEEE Robotics and Automation Letters_ , vol. 7, no. 3, pp. 6862–6869, 2022.
  * [8] G. Fang, C.-D. Matte, T.-H. Kwok, and C. C. Wang, “Geometry-based direct simulation for multi-material soft robots,” in _2018 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2018, pp. 4194–4199.
  * [9] X. Wang, A. Khara, and C. Chen, “A soft pneumatic bistable reinforced actuator bioinspired by venus flytrap with enhanced grasping capability,” _Bioinspiration & Biomimetics_, vol. 15, no. 5, p. 056017, 2020.
  * [10] J. Collins, S. Chand, A. Vanderkop, and D. Howard, “A review of physics simulators for robotic applications,” _IEEE Access_ , vol. 9, pp. 51 416–51 431, 2021.
  * [11] D. Kim, S.-H. Kim, T. Kim, B. B. Kang, M. Lee, W. Park, S. Ku, D. Kim, J. Kwon, H. Lee _et al._ , “Review of machine learning methods in soft robotics,” _Plos one_ , vol. 16, no. 2, p. e0246102, 2021.
  * [12] G. E. Karniadakis, I. G. Kevrekidis, L. Lu, P. Perdikaris, S. Wang, and L. Yang, “Physics-informed machine learning,” _Nature Reviews Physics_ , vol. 3, no. 6, pp. 422–440, 2021.
  * [13] W. Sun, N. Akashi, Y. Kuniyoshi, and K. Nakajima, “Physics-informed reservoir computing with autonomously switching readouts: a case study in pneumatic artificial muscles,” in _The 32nd 2021 International Symposium on Micro-NanoMechatronics and Human Science_. IEEE, 2021, pp. 1–6.
  * [14] G. Alici, T. Canty, R. Mutlu, W. Hu, and V. Sencadas, “Modeling and experimental evaluation of bending behavior of soft pneumatic actuators made of discrete actuation chambers,” _Soft robotics_ , vol. 5, no. 1, pp. 24–35, 2018.
  * [15] S. Mbakop, G. Tagne, M.-H. Frouin, A. Melingui, and R. Merzouki, “Inverse dynamics model-based shape control of soft continuum finger robot using parametric curve,” _IEEE Robotics and Automation Letters_ , vol. 6, no. 4, pp. 8053–8060, 2021.
  * [16] D. Trivedi, A. Lotfi, and C. D. Rahn, “Geometrically exact models for soft robotic manipulators,” _IEEE Transactions on Robotics_ , vol. 24, no. 4, pp. 773–780, 2008.
  * [17] M. Giorelli, F. Renda, M. Calisti, A. Arienti, G. Ferri, and C. Laschi, “Neural network and jacobian method for solving the inverse statics of a cable-driven soft arm with nonconstant curvature,” _IEEE Transactions on Robotics_ , vol. 31, no. 4, pp. 823–834, 2015.
  * [18] D. P. Finegan, J. Zhu, X. Feng, M. Keyser, M. Ulmefors, W. Li, M. Z. Bazant, and S. J. Cooper, “The application of data-driven methods and physics-based learning for improving battery safety,” _Joule_ , vol. 5, no. 2, pp. 316–329, 2021.
  * [19] L. Zhang, J. Han, H. Wang, R. Car _et al._ , “Deepcg: Constructing coarse-grained models via deep neural networks,” _The Journal of chemical physics_ , vol. 149, no. 3, 2018.
  * [20] J. Bai, H. Jeong, C. Batuwatta-Gamage, S. Xiao, Q. Wang, C. Rathnayaka, L. Alzubaidi, G.-R. Liu, and Y. Gu, “An introduction to programming physics-informed neural network-based computational solid mechanics,” _arXiv preprint arXiv:2210.09060_ , 2022.
  * [21] E. Nava, J. Z. Zhang, M. Y. Michelis, T. Du, P. Ma, B. F. Grewe, W. Matusik, and R. K. Katzschmann, “Fast aquatic swimmer optimization with differentiable projective dynamics and neural network hydrodynamic models,” in _International Conference on Machine Learning_. PMLR, 2022, pp. 16 413–16 427.
  * [22] J. J. Dabrowski, D. E. Pagendam, J. Hilton, C. Sanderson, D. MacKinlay, C. Huston, A. Bolt, and P. Kuhnert, “Bayesian physics informed neural networks for data assimilation and spatio-temporal modelling of wildfires,” _Spatial Statistics_ , vol. 55, p. 100746, 2023.
  * [23] X. Shan and L. Birglen, “Modeling and analysis of soft robotic fingers using the fin ray effect,” _The International journal of robotics research_ , vol. 39, no. 14, pp. 1686–1705, 2020.
  * [24] M. Sadd, _Elasticity: Theory, Applications, and Numerics_ , 4th ed. Elsevier Science, 2009.
  * [25] I. Doghri, _Mechanics of deformable solids: linear, nonlinear, analytical and computational aspects_. Springer Science & Business Media, 2013.
  * [26] S. Timoshenko and J. Goodier, _Theory of Elasticity_ , ser. Engineering societies monographs. McGraw-Hill, 1970.
  * [27] A. G. Baydin, B. A. Pearlmutter, A. A. Radul, and J. M. Siskind, “Automatic differentiation in machine learning: a survey,” _Journal of Machine Learning Research_ , vol. 18, no. 153, pp. 1–43, 2018. [Online]. Available: http://jmlr.org/papers/v18/17-468.html
  * [28] S. Wang, Y. Teng, and P. Perdikaris, “Understanding and mitigating gradient flow pathologies in physics-informed neural networks,” _SIAM Journal on Scientific Computing_ , vol. 43, no. 5, pp. A3055–A3081, 2021.
  * [29] X. Wang, B. Wang, J. Pinskier, Y. Xie, J. Brett, R. Scalzo, and D. Howard, “Fin-bayes: A multi-objective bayesian optimization framework for soft robotic fingers,” _Soft Robotics_ , 2024.
  * [30] Y. Xie, X. Wang, F. Iida, and D. Howard, “Fin-qd: A computational design framework for soft grippers: Integrating map-elites and high-fidelity fem,” _arXiv preprint arXiv:2311.12477_ , 2023.
  * [31] C. Geuzaine and J.-F. Remacle, “Gmsh: A 3-d finite element mesh generator with built-in pre-and post-processing facilities,” _International journal for numerical methods in engineering_ , vol. 79, no. 11, pp. 1309–1331, 2009.
  * [32] J. Pinskier, X. Wang, L. Liow, Y. Xie, P. Kumar, M. Langelaar, and D. Howard, “Diversity-based topology optimization of soft robotic grippers,” _Advanced Intelligent Systems_ , p. 2300505, 2024.