# Language and Dialect Identification of Cuneiform Texts

Tommi Jauhiainen
Department of Digital Humanities
University of Helsinki
<EMAIL_ADDRESS>
&Heidi Jauhiainen
Department of Digital Humanities
University of Helsinki
<EMAIL_ADDRESS>
Tero Alstola
Department of Digital Humanities
University of Helsinki
<EMAIL_ADDRESS>
&Krister Lindén
Department of Digital Humanities
University of Helsinki
<EMAIL_ADDRESS>

###### Abstract

This article introduces a corpus of cuneiform texts from which the dataset for
the use of the Cuneiform Language Identification (CLI) 2019 shared task was
derived as well as some preliminary language identification experiments
conducted using that corpus. We also describe the CLI dataset and how it was
derived from the corpus. In addition, we provide some baseline language
identification results using the CLI dataset. To the best of our knowledge,
the experiments detailed here represent the first time that automatic language
identification methods have been used on cuneiform data.

##  1 Introduction

We have compiled a corpus of cuneiform texts intended to be used in language
identification experiments. As the basis for our corpus, we used the Open
Richly Annotated Cuneiform Corpus (Oracc).111[http://oracc.museum.upenn.edu]
In Oracc, the texts are stored in transliterated form. We created a tool,
Nuolenna, which can transform the transliterations back to the cuneiform
script. Selecting all monolingual lines from Oracc and transforming the
transliterations into cuneiform, we created a new corpus for Sumerian and six
Akkadian dialects.

This corpus was used in the initial experiments where the possibility of
language identification in cuneiform texts was verified. In this paper, we
report some of the results from the initial experiments. To the best of our
knowledge, this is the first time that automatic language identification
methods have been used on cuneiform data. The methods we use for language
identification utilize mainly character _n_ -grams and their observed
probabilities in text.

For the use of the Cuneiform Language Identification (CLI) 2019 shared
task222[https://sites.google.com/view/vardial2019/campaign], we extracted a
dataset from the corpus. The dataset is divided into training, development,
and test portions to be used in the CLI shared task which is part of the third
VarDial Evaluation Campaign. We implemented four baseline language identifiers
and evaluated their performance using the CLI dataset. The results of the
evaluation are presented here.

##  2 Related work

So far, no research into language identification using cuneiform texts has
been openly reported. Language identification studies involving other
contemporary scripts, such as Egyptian hieroglyphs, also seem to be non-
existent.

###  2.1 Cuneiform script and computational methods

In this section, we survey some of the research where computational methods
related to language identification have been used with the cuneiform script.

Kataja and Koskenniemi (1988) discuss the description and computational
implementation of phonology and morphology for Akkadian. They give examples of
the rules in two-level formalism they used with the TWOL rule compiler.

Barthélemy (1998) developed and tested a morphological analyzer for Akkadian
verbal forms. The analyzer works with Akkadian represented in Latin encoding
(transcription).

Tablan et al. (2006) describe their project, which aims to create a tool for
automatic morphological analysis of Sumerian.

Among several languages, Rao et al. (2009) analyzed Sumerian written in
cuneiform using conditional entropy to compare it with the Indus script.
Normalized entropy of sign _n-grams_ between the two scripts was used as
further evidence to indicate the possible linguistic content of the texts
written in the Indus script.

Ponti et al. (2013) used the K-means clustering algorithm to cluster
transliterated cuneiform texts. The texts analyzed were 51 Old Babylonian
letters from Tell Harmal/Šaduppûm. Term frequency (TF) and term frequency -
inverse document frequency (TF-IDF) weighted words were used as features with
the clustering methods. Each document was depicted as a feature vector with
the length of the whole vocabulary. In K-means, the number of clusters has to
be given before the algorithm is applied and Ponti et al. (2013) experimented
with 2 to 15 clusters.

Luo et al. (2015) describe an unsupervised Named-Entity Recognition (NER)
system for transliterated Sumerian. They compared the use of different lengths
of transliterated word _n_ -grams in combination with the Decision List
CoTrain algorithm, and their evaluations show that word bigrams obtain the
highest F1-score. In another article Liu et al. (2015), they describe how they
managed to find unannotated personal names in a corpus and suggest that the
NER system could be used as an automated tool for the annotation task. Liu et
al. (2016) continue the NER research on Sumerian using a variety of supervised
classification methods to detect named entities.

Homburg and Chiarcos (2016) researched automated word segmentation of Akkadian
cuneiform script. They used a sign list to restore CDLI333Cuneiform Digital
Library Initiative [https://cdli.ucla.edu] transliterations back to cuneiform
(represented as UTF-8 characters). This is the only related work we are aware
of in which cuneiform texts encoded in Unicode cuneiform have been processed
previous to our experiments.

Pagé-Perron et al. (2017) present a project dedicated to creating a pipeline
for Sumerian texts. The pipeline is planned to take in transliterated Sumerian
and to produce POS tag annotations and lemmatization as well as machine
translation into English. Chiarcos et al. (2018) further describe the work
done in the project.

In order to measure inter-textual relations, Monroe (2018) calculated the
cosine similarity between word vectors consisting of transliterated Late
Babylonian words.

Svärd et al. (2018) used Pointwise Mutual Information (PMI) to find
collocations and associations between words and word2vec to highlight
paradigmatic relationships of the words of interest. They used transliterated
and lemmatized Akkadian texts from Oracc.

###  2.2 Language identification

Automatic language identification is the task of determining the language of a
piece of text from the clues in the text itself. The computational methods
used in language identification vary from simple wordlists to state-of-the-art
deep learning methods. A recent comprehensive survey on language
identification was conducted by Jauhiainen et al. (2018c). Language
identification for long texts in well-resourced languages is not a difficult
task, but it becomes increasingly more challenging when we target short,
fragmentary, and multilingual texts in languages where the amount of training
material is severely restricted. A separate challenge for language
identification is dealing with closely related languages or with several
dialects of an individual language. The challenge of discriminating between
closely related languages has been investigated in a series of shared tasks
that have been organized as part of VarDial workshops (Zampieri et al., 2014,
2015; Malmasi et al., 2016; Zampieri et al., 2017, 2018).

##  3 Cuneiform texts in Oracc

Our data comes from the Open Richly Annotated Cuneiform Corpus (Oracc). Oracc
is an international cooperative effort containing free online editions of
cuneiform texts from various
projects.444[http://oracc.museum.upenn.edu/doc/about/aboutoracc/ index.html]
Oracc is one of the largest electronic corpora of Sumerian and Akkadian texts,
and it is regularly updated. Our data is a snapshot of Oracc from October 2016
from XML files downloaded with the permission of the site administrators. The
data is comprised of 13,662 separate texts, most of which were originally
written on clay tablets. Some of the texts are duplicates, and before the
language identification we removed the duplicates of texts with identical
Oracc identification numbers. This procedure removes modern duplicates which
have come into existence because a single text has been edited in several
Oracc projects. Those duplicates that have different numbers and are thus
different ancient manuscripts were not removed. Cuneiform writing does not
mark the end of a sentence, and this is not indicated in the XML files either.
Our data can, hence, be divided either into lines or texts with one or more
lines. Oracc also contains some texts or words written in languages other than
Sumerian and Akkadian, such as Hittite, Ugaritic, and Greek, but their number
is so small that they were left out of this research.

The metadata of the texts in Oracc contains information about, for example,
the provenance (the locality where the text was found), the genre, the time
period in which the text was written, and so forth. The basic units in Oracc
XML files are the transliterations of words, which are representations of the
cuneiform signs in Latin script and which are given even if nothing else is
stated about the words. Some of the cuneiform signs have, however, been broken
off or are otherwise unreadable on the original tablets. In those cases, the
word in question, or part of it, is replaced with an ’x’ in the
transliteration. The metadata of a word usually indicates its language, and
some of the projects have also provided the cuneiform signs for each syllable
or word of the transliteration.

The data contains many bilingual documents written in Sumerian and Akkadian.
These documents often have the same text in both languages, sometimes on the
same line.

###  3.1 Sumerian and Akkadian

Sumerian and Akkadian are ancient languages which were spoken and written
primarily in Mesopotamia, present-day Iraq Michalowski (2004); Kouwenberg
(2011). Both languages were written in cuneiform script, but they are not
related, Sumerian being a language isolate and Akkadian an East Semitic
language. The cuneiform script was originally logographic in essence, then
syllabic sign values were introduced to facilitate writing Sumerian, and only
later was the script adapted for Akkadian. Consequently, some features of the
cuneiform writing system are not ideal for Akkadian and many logograms are
used side by side with syllabic spellings of Akkadian words (for further
information see Seri, 2010).

Sumerian was one of the first languages ever written, and the oldest texts
survive from the turn of the fourth and third millennia before the Common Era
(BCE). Akkadian replaced Sumerian as the spoken language during the late third
and early second millennia BCE, but Sumerian was used as a liturgical and
scholarly language until the end of the cuneiform tradition at the beginning
of the Common Era.

Written Akkadian is known from circa 2400 BCE onwards until the first century
CE. The Akkadian language had two main dialects, Assyrian and Babylonian, both
of which are present in our data. Assyrian was used in northern Mesopotamia
and Babylonian in the south. There is written evidence for the simultaneous
use of these dialects for 1,400 years, and both of them changed over time. The
dialects are, hence, further divided into varieties designated as Old, Middle,
and Neo-Assyrian and Old, Middle, and Neo-Babylonian. There was also a
literary variety called Standard Babylonian which was used by both Assyrian
and Babylonian scribes to write texts in certain genres. In Oracc, Middle-
Babylonian is, furthermore, divided into the dialect spoken in Mesopotamia
proper and the one spoken in the peripheries of the Empire. The latter,
referred to as Middle Babylonian Peripheral, is not a coherent dialect but
varies somewhat from site to site. After Assyrian ceased to be a written
language around 600 BCE, a variety called Late Babylonian was written for some
700 years. The differences between the dialects and their varieties are
relatively small, and after learning a variety one can read the other dialect
and varieties as well. In the Oracc metadata, the different dialects and
varieties are given for Akkadian words in most cases.

##  4 Cuneiform representation in Unicode

The effort to provide a standard encoding for cuneiform began in 1999 at Johns
Hopkins University as the Initiative for Cuneiform Encoding (ICE). The
initiative ended up with an approved proposal for cuneiform Unicode in 2004
(officially accepted into Unicode 5.0 in 2006).555The Unicode® Standard
Version 11.0 – Core Specification
[http://www.unicode.org/versions/Unicode11.0.0/appC.pdf] The final list of
cuneiform signs included is a combination of work done earlier at the
University of Chicago, Universität Göttingen, and the University of
California, Los Angeles Cohen et al. (2004).

In the current Unicode standard, there are three blocks of cuneiform signs for
the “Sumero-Akkadian” script. The first one is the block covering the base
cuneiform signs ranging from U+12000 to U+123FF. The second block, from
U+12400 to U+1247F, covers the cuneiform punctuation and numerals and the
third, from U+12480 to U+1254F, is an extension containing additional signs
for the Early Dynastic period. Unicode has only one character for each sign,
even though the signs evolved through the ages. The different ways of writing
the signs could be used to determine the language or dialect used or the time
period of writing.

The cuneiform texts from the Oracc corpus we use in this research were
provided primarily as transliterations using the ASCII Transliteration Format
(ATF). ATF was first defined by CDLI and is a standardized way of
electronically transliterating cuneiform, following the conventions used by
cuneiform scholars in general Koslova and Damerow (2003). The data from the
Oracc corpus is also available as JSON files in an “XCL” format Tinney and
Robson (2018)666[http://oracc.museum.upenn.edu/doc/opendata/], which includes
a similar XML representation of the data as the CDLI archival XML format
Koslova and Damerow (2003). We extracted the individual sign transliterations
in Unicode ATF from the XML representation and recreated the transliteration
for each line.

There was no available software to automatically transform the
transliterations to Unicode cuneiform. As part of Oracc, there is a facility
called “Cuneify,” which can be used online to transform ATF into Unicode
cuneiform.777[http://oracc.museum.upenn.edu/doc/tools/cuneify/ index.html]
However, it is not possible to download the software and it does not handle
the Unicode ATF transliteration. In order to generate the original lines in
cuneiform, we implemented a program called “Nuolenna” which takes in the
transliteration generated from the XML files and re-produces the lines in
Unicode cuneiform.888[https://github.com/tosaja/Nuolenna] The Nuolenna program
uses a list999[https://github.com/tosaja/Nuolenna/blob/master/ sign_list.txt]
of over 11,000 transliteration-sign pairs. As the base for our sign list, we
used a JSON export from the Oracc Global Sign List
(OGSL)101010[http://oracc.museum.upenn.edu/ogsl/] provided by Niek Veldhuis,
to which we added some missing signs. In order to produce the original
cuneiform lines, the program uses ad hoc rules to remove any additional
annotations related to the signs. For example, sometimes an older or more
precise reading can be found within parentheses directly after the reading of
a sign. In such cases, we just remove the parentheses and anything between
them.

##  5 Preliminary language identification experiments

To find out to what extent identifying the language of cuneiform text is
possible, we performed initial language identification experiments using a
state-of-the-art language identification method called HeLI (Jauhiainen et
al., 2016). The HeLI method has recently fared well in VarDial shared tasks
for Swiss-German dialect and Indo-Aryan language identification (Jauhiainen et
al., 2018a, b). The experiments were conducted on individual lines as well as
texts spanning several lines.

###  5.1 Corpus for the preliminary experiments

In Oracc, the transliterated words are separated by whitespaces, which is not
the case in the original documents. In order to mimic the original documents,
we removed all the whitespaces from each line of cuneiform text. We also
ignored any completely broken signs, which were marked with an ’x’.

The individual words in Oracc are tagged with language or dialect information,
and sometimes a single line includes words in different languages or dialects.
As we set out to do language identification on monolingual texts, we used all
those lines which had words in only one language, leaving out multilingual
lines. The language tagging in Oracc is not always precise, and therefore some
lines in our dataset might still include several languages.

In the preliminary experiments, we experimented with the language
identification of both monolingual lines and monolingual texts spanning
several lines with the information about line breaks retained. Mostly, each
text had the lines of one original document, but if the document was
multilingual, it was divided into different texts according to the languages
attested.

We left out the Akkadian varieties of Old and Middle Assyrian as the number of
lines available for those dialects was less than 1,000. We had datasets in the
Sumerian language as well as the Akkadian varieties of Old Babylonian, Middle
Babylonian peripheral, Standard Babylonian, Neo-Babylonian, Late Babylonian,
and Neo-Assyrian. The statistics of the corpus used in the preliminary
experiments are shown in Table 1.

Language or Dialect (abbreviation in the CLI dataset) | Texts | Lines | Signs
---|---|---|---
Sumerian (SUX) | 5,000 | 107,345 |  _c_. 400,000
Old Babylonian (OLB) | 527 | 7,605 |  _c_. 65,000
Middle Babylonian peripheral (MPB) | 365 | 11,015 |  _c_. 95,000
Standard Babylonian (STB) | 1,661 | 35,633 |  _c_. 390,000
Neo-Babylonian (NEB) | 1,212 | 19,414 |  _c_. 200,000
Late Babylonian (LTB) | 671 | 31,893 |  _c_. 260,000
Neo-Assyrian (NEA) | 3,570 | 65,932 |  _c_. 490,000
Table 1:  Number of texts, lines, and signs for each language or variety in
the corpus.

We were interested in experimenting in both in-domain and out-of-domain test
settings as well as with language identification on two different levels:
individual lines and texts. In supervised machine learning, the testing data
is in-domain if it is similar to the training data. For example, if sentences
are from texts that belong to the same genre or collection they are considered
more in-domain than if they are not. An even stronger in-domain case is if the
sentences are from the same text. Classification of test data which is in-
domain with the training data is usually much easier than when it is out-of-
domain. The texts in the Oracc export were in the order111111The projects were
in the alphabetical order by their abbreviations. of “projects,” which are
collections of texts that have some common theme. The texts in different
projects can be considered to be more out-of-domain with each other than those
from the same project. The projects from which the texts were extracted are
listed in Table 2.

Project (abbreviation used in Oracc) | SUX | OLB | MPB | STB | NEB | LTB | NEA
---|---|---|---|---|---|---|---
Bilinguals in Late Mesopotamian Scholarship (blms) | x | x |  | x |  |  |
CAMS/Anzu (cams-anzu) |  |  |  | x |  |  |
CAMS/Barutu (cams-barutu) |  | x |  | x |  |  |
CAMS/The Standard Babylonian Epic of Etana (cams-etana) |  | x |  |  |  |  | x
CAMS/Geography of Knowledge Corpus (cams-gkab) | x |  |  | x |  | x | x
CAMS/Ludlul (cams-ludlul) |  |  |  | x |  |  |
CAMS/Seleucid Building Inscriptions (cams-selbi) |  |  |  | x |  |  |
Cuneiform Commentaries Project on ORACC (ccpo) | x |  |  | x |  |  |
Corpus of Kassite Sumerian Texts (ckst) | x |  |  |  | x |  |
The Amarna Texts (contrib-amarna) |  |  | x | x |  |  |
Cuneiform Texts Mentioning Israelites, Judeans … (ctij) |  |  |  |  | x | x | x
Lexical Texts in the Royal Libraries at Nineveh (dcclt-nineveh) | x |  |  | x |  |  |
Reading the Signs (dcclt-signlists) | x |  |  | x |  |  |
Digital Corpus of Cuneiform Lexical Texts (dcclt) | x | x | x | x |  |  |
Digital Corpus of Cuneiform Mathematical Texts (dccmt) | x | x |  | x |  |  |
Electronic Text Corpus of Sumerian Royal Inscriptions (etcsri) | x |  |  |  |  |  |
Corpus of Glass Technological Texts (glass) |  |  |  | x |  |  |
Hellenistic Babylonia: Texts, Iconography, Names (hbtin) |  |  |  |  |  | x |
Law and Order: Cuneiform Online Sustainable Tool (lacost) | x |  |  |  |  |  |
Old Babylonian Model Contracts (obmc) | x |  |  |  |  |  |
Old Babylonian Tabular Accounts (obta) | x | x |  |  |  |  |
The Inscr. of the Second Dynasty of Isin (ribo-babylon2) | x |  |  |  |  |  |
The Inscr. of the Period of the Uncertain Dynasties (ribo-babylon6) | x |  |  |  |  |  |
Rim-Anum: The House of Prisoners (rimanum) | x | x |  |  |  |  |
The Correspondence of Sargon II, Part I (saao-saa01) |  |  |  |  |  |  | x
Neo-Assyrian Treaties and Loyalty Oaths (saao-saa02) |  |  |  | x | x |  | x
Court Poetry and Literary Miscellanea (saao-saa03) |  |  |  |  | x |  | x
Queries to the Sungod (saao-saa04) |  |  |  |  | x |  |
The Correspondence of Sargon II, Part II (saao-saa05) |  |  |  |  |  |  | x
Legal Trns. of the Royal Court of Nineveh, Part I (saao-saa06) |  |  |  |  |  |  | x
Imperial Administrative Records, Part I (saao-saa07) |  |  |  |  |  |  | x
Astrological Reports to Assyrian Kings (saao-saa08) |  |  |  | x | x |  | x
Assyrian Prophecies (saao-saa09) |  |  |  |  |  |  | x
Letters from Assyrian and Babylonian Scholars (saao-saa10) |  |  |  | x | x |  | x
Imperial Administrative Records, Part II (saao-saa11) |  |  |  |  |  |  | x
Grants, Decres and Gifts of the Neo-Assyrian Period (saao-saa12) |  |  |  |  |  |  | x
Letters from Assyrian and Babylonian Priests to … (saao-saa13) |  |  |  |  | x |  | x
Legal Trns. of the Royal Court of Nineveh, Part II (saao-saa14) |  |  |  |  |  |  | x
The Correspondence of Sargon II, Part III (saao-saa15) |  |  |  |  |  |  | x
The Political Correspondence of Esarhaddon (saao-saa16) |  |  |  |  |  |  | x
The Neo-Babylonian Correspondence of Sargon and … (saao-saa17) |  |  |  |  | x |  |
The Babylonian Correspondence of Esarhaddon and … (saao-saa18) |  |  |  |  | x |  |
The Correspondence of Tiglath-Pileser III and … (saao-saa19) |  |  |  |  | x |  | x
Table 2:  The list of Oracc projects from which the texts in the corpus were
collected.

From this corpus, we generated four different test settings. For the out-of-
domain experiments, we divided the corpus so that we used the first half of
the corpus for training and the second half was divided between development
and testing. For the in-domain experiments, we divided the corpus into parts
of 20 lines or texts and took the 10 first lines or texts from each part for
training, the next 5 for development, and the last 5 for testing. We, thus,
ended up with four different datasets,121212See Table 3. two for lines and two
for texts. Each of the datasets had 50% of the material for training, 25% for
development, and 25% for testing.

###  5.2 Results of the preliminary experiments

The HeLI method is a supervised-learning language identification method where
the language models are created from a correctly tagged training corpus. The
language models consist of words and sign (character) _n_ -grams. When _n_
-grams are extracted from a corpus, the number of unique _n_ -grams is higher
the longer the _n_ -grams are. The actual number of occurrences of the longer
_n_ -grams is lower than the shorter _n_ -grams. The exact optimal value for
_n_ depends on, among many other variables, the size of the training corpus,
the length of the text to be identified, and the repertoire of the languages
considered. Sometimes the longer _n_ -grams could carry important information
even though they are seldom found in the text to be identified. The basic idea
in the HeLI method is to score individual words using the longest length _n_
-grams possible. For each individual language, the words are scored first,
after which the whole text gets the average score of the individual words. In
the case of cuneiform text, as it is not divided into words, we use just sign
_n_ -grams and consider a line of text as a word as far as the HeLI method is
concerned.

The individual words, or in this case lines, are scored by taking the average
score of the found _n_ -grams. Using the notation introduced by Jauhiainen et
al. (2018c), the individual _n_ -grams $f$ found from the line to be tested,
get a score $R$ as in Equation 1:

$R_{HeLI}(g,f)=-\log_{10}\frac{c(C_{g},f)}{l_{C_{g}^{F}}}$ (1)

where $c(C_{g},f)$ is the count of the feature $f$ in the training corpus
$C_{g}$ of the language $g$ and $l_{C_{g}^{F}}$ is the total number of
occurrences of all the _n_ -grams of the same length in the training corpus.
As smoothing, in case the count of a feature is zero in some languages, this
version of the HeLI method uses a score $R_{HeLI}(g,f)$ for the count of one
multiplied by a penalty multiplier.

Using the development sets, we optimized the sign _n_ -gram range and the
penalty multiplier for each setting individually. The results of these
experiments are presented in Table 3. As the performance measure, we use the
F1-score which is the harmonic mean of precision and recall. The results
clearly show how the task of identifying a single line is much harder than
that of a complete text. The task of out-of-domain identification is also
clearly more difficult than that of in-domain, as was expected.

Type of setting |  _n_ -gram range | F1
---|---|---
Lines, out-of-domain | 1–3 | 60
Lines, in-domain | 1–3 | 72
Texts, out-of-domain | 1–4 | 84
Texts, in-domain | 1–3 | 93
Table 3:  The F1-scores attained by the HeLI method in the preliminary
experiments.

Quite many of the misclassified lines were very short; many consisted only of
one sign and were truly ambiguous and often present in different dialects and
even languages. Nevertheless, it was still possible to attain reasonably good
language identification results. The hardest test setting was where the
language of individual out-of-domain lines was to be identified. To us, this
seemed to be the most interesting and relevant setting to be used in the CLI
shared task, especially if we leave out the extremely short and possibly
ambiguous lines.

##  6 The CLI shared task

The CLI shared task 2019, part of the third VarDial Evaluation Campaign,
focused on discriminating between languages and dialects written with
cuneiform signs. The task included two different languages: Sumerian and
Akkadian. Furthermore, the Akkadian language was divided into six dialects:
Old Babylonian, Middle Babylonian peripheral, Standard Babylonian, Neo-
Babylonian, Late Babylonian, and Neo-Assyrian. First, we explain how the
dataset for the shared task was constructed from the corpus described earlier,
and then we present the baseline language identifiers and the results we
attained using them.

###  6.1 The dataset for the shared task

For the CLI task, we created a separate, especially tailored dataset. The
participants were given texts for training and development and separate texts
were given for testing at the end of the campaign. The training set was
exactly the same as the one we used in the preliminary experiments131313In the
out-of-domain individual line identification test setting. and the number of
lines in the training portion for each language or dialect is shown in Table
4.

Language or Dialect | Training
---|---
Sumerian | 53,673
Old Babylonian | 3,803
Middle Babylonian peripheral | 5,508
Standard Babylonian | 17,817
Neo-Babylonian | 9,707
Late Babylonian | 15,947
Neo-Assyrian | 32,966
Table 4:  Number of lines for each language or dialect in the training set
provided during the VarDial 2019 Evaluation Campaign.

For the CLI development and test sets, we performed some further operations.
The original datasets included duplicate lines, so we first removed all
duplicates. Then we filtered out all lines shorter than three characters.
After these operations, the smallest sets were those of Old Babylonian
including 668 lines in the development set and 985 lines in the test set. As
we wanted to make the development and the test sets equal in size between
languages and dialects, we randomly selected the same number of lines from the
other languages. Thus, in the CLI task, the development sets for each language
consisted of 668 lines and the test sets of 985 lines.

###  6.2 Baseline experiments

We used four of the methods described in the survey by Jauhiainen et al.
(2018c) to implement baseline language identifiers for the CLI task. As
features, we used sign _n_ -grams of different lengths.

The first method is called simple scoring. In simple scoring, all the _n_
-grams generated from the line to be identified $M$ are compared to the
language models and for each _n_ -gram found in a language model
$dom(O(C_{g}))$, the score $R$ of the language $g$ is increased by one. The
language gaining the highest score is selected as the predicted language.
Jauhiainen et al. (2018c) formulate the method as in Equation 2:

$R_{\text{\emph{simple}}}(g,M)=\sum_{i=1}^{l_{M^{F}}}\left\\{\begin{array}[]{ll}1&\textbf{,
if }f_{i}\in dom(O(C_{g}))\\\ 0&\textbf{, otherwise}\end{array}\right.$ (2)

where $l_{M^{F}}$ is the number of individual features in the line $M$ and
$f_{i}$ is its $i$th feature.

The second method is the sum of relative frequencies where relative
frequencies are added to the score of the language. Jauhiainen et al. (2018c)
formulate the method as in Equation 3:

$R_{\text{\emph{sum}}}(g,M)=\sum_{i=1}^{l_{M^{F}}}\frac{c(C_{g},f_{i})}{l_{C_{g}^{F}}}$
(3)

where $c(C_{g},f_{i})$ is the count of the feature $f_{i}$ in the training
corpus.

The third method is the product of relative frequencies where the relative
frequencies are multiplied together. Jauhiainen et al. (2018c) formulate the
method as in Equation 4:

$R_{\text{\emph{prod}}}(g,M)=\prod_{i}^{l_{M^{F}}}\frac{c(C_{g},f_{i})}{l_{C_{g}^{F}}}$
(4)

The actual implementation adds together negative logarithms of the relative
frequencies, which produces results with the same ordering. As a smoothing
value, we used the negative logarithm of a comparably small relative
frequency. The actual value was optimized using the development set.

The fourth method is a majority-voting-based ensemble of the three previous
methods.

The parameters and the best possible language models are determined by
training the identifier using the training set and evaluating its performance
on the development set. Once the best parameters are decided, the texts in the
development set can also be added to the training set for the final evaluation
against the test set. We used the macro F1-score as the measure for language
identification performance. For each of the methods, we evaluated all possible
sign _n_ -gram ranges from 1 to 15 using the development set. Table 5 shows
the results for all the methods using parameters optimized with the
development set. In the voting ensemble, we used the best parameters for the
methods from the individual experiments, and in case of a tie, the result from
the product of relative frequencies was used.

Method |  _n_ -gram range | F1 dev | F1 test
---|---|---|---
Prod. of rel. freq. | 1–4 | 0.7263 | 0.7206
Voting Ensemble |  | 0.7222 | 0.7163
HeLI | 1–3 + lines | 0.7171 | 0.7061
Simple scoring | 1–10 | 0.6656 | 0.6554
Sum of rel. freq. | 3–15 | 0.5984 | 0.6016
Table 5:  The macro F1-scores attained by the baseline methods with the CLI
dataset.

The product of relative frequencies method is clearly superior to the other
two basic methods with an F1-score of 0.7206 using 2.0 as the smoothing value
and sign _n_ -grams from one to four. Adding the prediction information from
the other two methods in the form of a voting ensemble also fails to improve
the result. The F1-score achieved when using the HeLI method does not reach
the one from the product of relative frequencies method either. The F1-score
gained by the HeLI method is clearly higher than the score attained in the
preliminary experiments, which was as expected, as we had filtered out some of
the most difficult cases.

Table 5 is a confusion matrix displaying the exact number of identifications.
The diagonal values represent correct identifications. Standard Babylonian and
Neo-Babylonian were the most difficult varieties to distinguish, mostly being
erroneously identified as each other. Late Babylonian was the easiest to
identify, with a recall of over 96%.

Lang. | LTB | MPB | NEA | NEB | OLB | STB | SUX
---|---|---|---|---|---|---|---
LTB | 947 | 6 | 9 | 34 | 13 | 25 | 8
MPB | 3 | 858 | 51 | 94 | 84 | 69 | 55
NEA | 6 | 26 | 780 | 185 | 26 | 148 | 26
NEB | 4 | 19 | 81 | 535 | 30 | 160 | 30
OLB | 3 | 22 | 12 | 16 | 736 | 47 | 110
STB | 17 | 35 | 30 | 113 | 43 | 491 | 101
SUX | 5 | 19 | 22 | 8 | 53 | 45 | 655
Table 6:  Confusion matrix for the product of relative frequencies method. The
rows indicate the actual languages and the columns indicate predicted
languages. Correct identifications are emphasized.

##  7 Conclusions and future work

In this paper, we have shown that it is possible to perform language and
dialect identification in cuneiform texts encoded in Unicode characters. We
have created a dataset to be used in the VarDial Evaluation campaign and
evaluated the performance of four baseline identifiers using the dataset.

Some sizeable Oracc projects were left out of the corpus, for example the
“Royal Inscriptions of the Neo-Assyrian Period” project, as the exact dialect
of the Akkadian language had not been annotated. Furthermore, for the same
reason, only the lines in Sumerian could be used from the “Royal Inscriptions
of Babylonia online (RiBo)” project. We believe that automatic dialect
identification could be useful in making the annotations more detailed and are
planning to provide this kind of automatically deduced information as part of
the Korp version of Oracc.141414http://urn.fi/urn:nbn:fi:lb-2018071121

Some other avenues for further work are language set identification for the
multilingual texts, as well as unsupervised clustering of data without any
predefined languages.

## Acknowledgments

The research was carried out in the context of the “Semantic Domains in
Akkadian Texts” project and the Centre of Excellence in Ancient Near Eastern
Empires at the University of Helsinki (ANEE), both funded by the Academy of
Finland. This work has also been partly funded by the Kone Foundation.

We thank Johannes Bach, Mikko Luukko, Aleksi Sahala, and Niek Veldhuis for
their valuable comments during the experiments and Raija Mattila and Saana
Svärd for their continued support. We are grateful to Robert Whiting for
revising the language of the text and for further insight into the Akkadian
and Sumerian languages.

## References

  * Barthélemy (1998) François Barthélemy. 1998.  A Morphological Analyzer for Akkadian Verbal Forms with a Model of Phonetic Transformations.  In _Proceedings of the Workshop on Computational Approaches to Semitic Languages_ , pages 73–81. Association for Computational Linguistics.
  * Chiarcos et al. (2018) Christian Chiarcos, Ilya Khait, Émilie Pagé-Perron, Niko Schenk, Christian Fäth, Julius Steuer, William Mcgrath, and Jinyan Wang. 2018.  Annotating a Low-Resource Language with LLOD Technology: Sumerian Morphology and Syntax.  _Information_ , 9.
  * Cohen et al. (2004) Jonathan Cohen, Donald Duncan, Dean Snyder, Jerrold Cooper, Subodh Kumar, Daniel Hahn, Yuan Chen, Budirijanto Purnomo, and John Graettinger. 2004.  iClay: Digitizing Cuneiform.  In _The Proceedings of the 5th International Symposium on Virtual Reality, Archaeology and Cultural Heritage (VAST 2004)_ , pages 135–143.
  * Homburg and Chiarcos (2016) Timo Homburg and Christian Chiarcos. 2016.  Akkadian Word Segmentation.  In _Proceedings of the 10th International Conference on Language Resource Evaluation (LREC 2016)_ , pages 4067–4074.
  * Jauhiainen et al. (2018a) Tommi Jauhiainen, Heidi Jauhiainen, and Krister Lindén. 2018a.  HeLI-based Experiments in Swiss German Dialect Identification.  In _Proceedings of the Fifth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial)_ , pages 254–262, Santa Fe, NM.
  * Jauhiainen et al. (2018b) Tommi Jauhiainen, Heidi Jauhiainen, and Krister Lindén. 2018b.  Iterative Language Model Adaptation for Indo-Aryan Language Identification.  In _Proceedings of the Fifth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial)_ , pages 66–75, Santa Fe, NM.
  * Jauhiainen et al. (2016) Tommi Jauhiainen, Krister Lindén, and Heidi Jauhiainen. 2016.  HeLI, a Word-Based Backoff Method for Language Identification.  In _Proceedings of the Third Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial3)_ , pages 153–162, Osaka, Japan.
  * Jauhiainen et al. (2018c) Tommi Jauhiainen, Marco Lui, Marcos Zampieri, Timothy Baldwin, and Krister Lindén. 2018c.  Automatic Language Identification in Texts: A Survey.  _arXiv preprint arXiv:1804.08186_.
  * Kataja and Koskenniemi (1988) Laura Kataja and Kimmo Koskenniemi. 1988.  Finite-state Description of Semitic Morphology: A Case Study of Ancient Akkadian.  In _Proceedings of the 12th International Conference on Computational Linguistics (COLING 1988)_ , volume 1, pages 313–315, Budapest, Hungary.
  * Koslova and Damerow (2003) Natalia Koslova and Peter Damerow. 2003.  From Cuneiform Archives to Digital Libraries: The Hermitage Museum Joins the Cuneiform Digital Library Initiative.  In _Proceedings of the 5th Russian Conference on Digital Libraries (RCDL 2003)_ , St.-Petersburg, Russia.
  * Kouwenberg (2011) Bert Kouwenberg. 2011.  Akkadian in General.  In _The Semitic Languages: An International Handbook_ , pages 330–340. De Gruyter Mouton.
  * Liu et al. (2015) Yudong Liu, Clinton Burkhart, James Hearne, and Liang Luo. 2015.  Enhancing Sumerian Lemmatization by Unsupervised Named-Entity Recognition.  In _Proceedings of the Human Language Technologies: The 2015 Annual Conference of the North American Chapter of the Association for Computational Linguistics (HLT-NAACL 2015)_ , pages 1446–1451, Denver, Colorado.
  * Liu et al. (2016) Yudong Liu, James Hearne, and Bryan Conrad. 2016.  Recognizing Proper Names in UR III Texts through Supervised Learning.  In _Proceedings of the Twenty-Ninth International Florida Artificial Intelligence Research Society Conference (Flairs 2016)_ , pages 535–540.
  * Luo et al. (2015) Liang Luo, Yudong Liu, James Hearne, and Clinton Burkhart. 2015.  Unsupervised Sumerian Personal Name Eecognition.  In _Proceedings of the Twenty-Eighth International Florida Artificial Intelligence Research Society Conference (Flairs 2015)_ , pages 193–198.
  * Malmasi et al. (2016) Shervin Malmasi, Marcos Zampieri, Nikola Ljubešić, Preslav Nakov, Ahmed Ali, and Jörg Tiedemann. 2016.  Discriminating Between Similar Languages and Arabic Dialect Identification: A Report on the Third DSL Shared Task.  In _Proceedings of the Third Workshop on NLP for Similar Languages, Varieties and Dialects_ , pages 1–14, Osaka, Japan.
  * Michalowski (2004) Piotr Michalowski. 2004.  Sumerian.  In _The Cambridge Encyclopedia of the World’s Ancient Languages_ , pages 19–59. Cambridge University Press.
  * Monroe (2018) M Willis Monroe. 2018.  Using Quantitative Methods for Measuring Inter-Textual Relations in Cuneiform.  _Digital Biblical Studies_ , pages 257–280.
  * Pagé-Perron et al. (2017) Émilie Pagé-Perron, Maria Sukhareva, Ilya Khait, and Christian Chiarcos. 2017.  Machine Translation and Automated Analysis of the Sumerian Language.  In _Proceedings of the Joint SIGHUM Workshop on Computational Linguistics for Cultural Heritage, Social Sciences, Humanities and Literature_ , pages 10–16, Vancouver, BC. Association for Computational Linguistics.
  * Ponti et al. (2013) Giovanni Ponti, Daniela Alderuccio, Giorgio Mencuccini, Alessio Rocchi, Silvio Migliori, Giovanni Bracco, and Paola Negri Scafa. 2013.  Data Mining Tools and GRID Infrastructure for Assyriology Text Analysis (an Old-Babylonian Situation Studied Through Text Analysis and Data Mining tools).  http://www.eneagrid.enea.it, Published 2017 in: R. De Boer and J. G. Dercksen, editors, _Private and State in the Ancient Near East - Proceedings of the 58th Rencontre Assyriologique Internationale at Leiden, 16–20 July 2012, Eisenbrauns_.
  * Rao et al. (2009) Rajesh P. N. Rao, Nisha Yadav, Mayank N. Vahia, Hrishikesh Joglekar, R. Adhikari, and Iravatham Mahadevan. 2009.  Entropic Evidence for Linguistic Structure in the Indus script.  _Science_ , 324(5931):1165–1165.
  * Seri (2010) Andrea Seri. 2010.  Adaptation of Cuneiform to Write Akkadian.  In Christopher Woods, editor, _Visible Language. Inventions of Writing in the Ancient middle East and Beyond_ , volume 32 of _Oriental Institute Museum Publications_ , pages 85–98. The Oriental Institute of the University of Chicago.
  * Svärd et al. (2018) Saana Svärd, Heidi Jauhiainen, Aleksi Sahala, and Krister Lindén. 2018.  Semantic Domains in Akkadian Texts.  In Vanessa Bigot Juloux, Amy Rebecca Gansell, and Alessandro Di Ludovico, editors, _CyberResearch on the Ancient Near East and Neighboring Regions. Case Studies on Archaeological Data, Objects, Texts, and Digital Archiving_ , volume 2 of _Digital Biblical Studies_ , pages 224–256. Brill.
  * Tablan et al. (2006) Valentin Tablan, Wim Peters, Diana Maynard, and Hamish Cunningham. 2006.  Creating Tools for Morphological Analysis of Sumerian.  In _Proceedings of the 5th Internation Conference on Language Resources and Evaluation (LREC 2006)_ , pages 1762–1765, Genoa, Italy.
  * Tinney and Robson (2018) Steve Tinney and Eleanor Robson. 2018.  Oracc Open Data: A brief introduction for programmers.  _Oracc: The Open Richly Annotated Cuneiform Corpus_ , Oracc.
  * Zampieri et al. (2017) Marcos Zampieri, Shervin Malmasi, Nikola Ljubešic, Preslav Nakov, Ahmed Ali, Jörg Tiedemann, Yves Scherrer, and Noëmi Aepli. 2017.  Findings of the VarDial Evaluation Campaign 2017.  In _Proceedings of the Fourth Workshop on NLP for Similar Languages, Varieties and Dialects_ , pages 1–15, Valencia, Spain.
  * Zampieri et al. (2018) Marcos Zampieri, Shervin Malmasi, Preslav Nakov, Ahmed Ali, Suwon Shon, James Glass, Yves Scherrer, Tanja Samardžić, Nikola Ljubešić, Jörg Tiedemann, Chris van der Lee, Stefan Grondelaers, Nelleke Oostdijk, Antal van den Bosch, Ritesh Kumar, Bornini Lahiri, and Mayank Jain. 2018.  Language Identification and Morphosyntactic Tagging: The Second VarDial Evaluation Campaign.  In _Proceedings of the Fifth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial)_ , Santa Fe, USA.
  * Zampieri et al. (2014) Marcos Zampieri, Liling Tan, Nikola Ljubešić, and Jörg Tiedemann. 2014\.  A Report on the DSL Shared Task 2014.  In _Proceedings of the First Workshop on Applying NLP Tools to Similar Languages, Varieties and Dialects_ , pages 58–67, Dublin, Ireland.
  * Zampieri et al. (2015) Marcos Zampieri, Liling Tan, Nikola Ljubešić, Jörg Tiedemann, and Preslav Nakov. 2015.  Overview of the DSL Shared Task 2015.  In _Proceedings of the Joint Workshop on Language Technology for Closely Related Languages, Varieties and Dialects (LT4VarDial)_ , pages 1–9, Hissar, Bulgaria.


#  SEED: Series Elastic End Effectors in 6D for Visuotactile Tool Use

H.J. Terry Suh1, Naveen Kuppuswamy2, Tao Pang1, Paul Mitiguy2, Alex Alspach2,
Russ Tedrake12  1Massachusetts Institute of Technology, Cambridge,
Massachusetts 02139
Email: {hjsuh, pangtao<EMAIL_ADDRESS>2Toyota Research Institute, Cambridge,
Massachusetts 02139
Email: {naveen.kuppuswamy, paul.mitiguy<EMAIL_ADDRESS>
###### Abstract

We propose the framework of Series Elastic End Effectors in 6D (SEED), which
combines a spatially compliant element with visuotactile sensing to grasp and
manipulate tools in the wild. Our framework generalizes the benefits of series
elasticity to 6-dof, while providing an abstraction of control using
visuotactile sensing. We propose an algorithm for relative pose estimation
from visuotactile sensing, and a spatial hybrid force-position controller
capable of achieving stable force interaction with the environment. We
demonstrate the effectiveness of our framework on tools that require
regulation of spatial forces. Video link:  https://youtu.be/2-YuIfspDrk

##  I Introduction

Many tasks in robot manipulation require handling of general tools in the
wild; in the future, we believe that robots will be able to grab any tool and
do meaningful control in order to accomplish various tasks and exchange forces
with the environment. To manipulate tools skillfully and robustly, we will
need end effectors that allow controllable hand-tool interaction in hardware,
while having sensing capabilities on this interaction to enable closed-loop
feedback.

Parallel-jaw grippers are sufficient for grasping [6], but quickly meet
limitations when it comes to forceful tool use. Even when sensing is given via
finger attachments [9], the hardware often relies on friction to handle the
forces that arise in hand-tool interaction, and may lack the ability to resist
spatial forces in some of the axes. For example, torques applied perpendicular
to the finger surface are hard to resist, but arise in many tool-use scenarios
[14]. Multi-fingered hands are much more versatile, but existing solutions
(e.g. imposing a full-rank grasp matrix on the tool [18, 7]) also rely on
frictional contacts, which can limit the amount of force they can exert.

More fundamentally, the rigidity of most of our hardware requires non-smooth
contact forces to be used to resist external forces in tool-use. Such forces
can be notoriously hard to control robustly as their behavior changes
instantaneously [29]. In the absence of the challenges brought by rigid
contacts, custom tool changers that are rigidly attached to the robot have
demonstrated impressive capability to achieve finely controlled force
interaction with the environment [24]. However, such solutions require
modifying tools with specialized handles compatible with the tool changer,
which limits the robot’s ability to use unmodified tools.

To alleviate the difficulties coming from rigidity and the non-smooth behavior
that it brings, we ask the following question in this work: can we consider
visuotactile hardware as not only a mechanism for sensing, but also as an
opportunity to provide compliance for control? Indeed, similar ideas have been
proposed in Series Elastic Actuators (SEA)s [21]; by attaching a soft spring
in front of the gearbox whose deformation can be measured, SEAs have been
successful in achieving smooth and stable force control by turning the problem
into that of position control [21, 23, 13].

Figure 1: A visual illustration of our framework.

How can we generalize the benefits of SEAs to the setting of grasping and
using arbitrary tools? We propose an answer that attaches soft, spatially
compliant elements at the end effector right where interaction with the tool
occurs. Mechanically, such a solution can be attached to a low-cost, position-
controlled robot, while still achieving the benefits of SEAs in the
interaction of the end effector and the tool. Through our solution, we aim to
achieve a 6D generalization of SEAs that can be useful for spatial tool use.

Our characterization of spatial series-elastic actuators would not be complete
unless we can measure the deformation of the spatial compliance in real time,
and use the feedback for force control. In order to achieve this, we leverage
recent advances in visuotactile sensing that measure 6D deformation using
vision [35, 9, 15, 2]. In contrast to many works that utilize deep learning to
directly process data from visuotactile sensors in an end-to-end manner, we
propose to measure the pose of the grasped tool relative to the end effector,
abstracting visuotactile sensing as a relative pose estimator.

Our proposed framework of Series Elastic End Effectors in 6D (SEED) consists
of three elements: a manipulator capable of accurate position control, a 6D
spatially compliant stiffness element, and visuotactile sensing that measures
the deformation of the spatial compliance. With these three elements, we show
that we can achieve spatial force control of tools with closed-loop feedback
from visuotactile sensing.

##  II Literature Review

###  II-A Tool-Use in Manipulation

Tool-use has long been one of the hallmarks of intelligence [27], as well as a
practical problem to solve for robotic applications. As such, many existing
works [31, 14] center around how to give robots the ability to use tools.
However, only a few works attempt to perform explicit force control with a
tool that has not been rigidly attached to the robot, but rather, must be
grabbed before it can be used.

Most existing works in this setting focus on planning, where the grabbed tool
must be used to manipulate the pose of another object [31, 14, 10]. Such plans
can be very useful in reaching confined spaces [10] or beyond the workspace of
the manipulator [31]. However, as the focus of these works lie more in
planning, tasks that require force exchange among static objects, such as
using a squeegee, surface grinding, wiping a table, or using torque drivers,
are often not considered.

On the other extreme, classical works in robot force control excel in forceful
manipulation with rigidly attached tools. Strategies such as impedance control
[13] and hybrid force-velocity control [22] have been extensively tested and
applied on problems that require force exchange between the robot and the
environment [24, 1, 33]. However, customized tool changers are quite limited
in terms of versatility in the wild.

Finally, works that attempt to explicitly apply forces with the grasped tool
[32] often run into hardware limitations, as typical parallel jaw grippers
with rigid, flat fingers are unable to resist forces and provide compliance in
certain directions due to their relatively small contact patch.

###  II-B Manipulation using Visuotactile Sensing

Visuotactile sensors [35, 9, 15] consist of a deformable membrane which
interacts with objects, and a camera (color, depth or both) under the membrane
to measure its deformation during interactions. As the measurements from
visuotactile sensors are images, some works have leveraged deep learning
approaches to learn the dynamics [34], or directly learn a map from the input
image or optical flow to the policy [17, 8]. While such approaches can be
effective, we first focus on interpretable abstractions in this work that are
more conducive for understanding, and may provide more inductive bias [28] for
designing deep models in the future.

Other works have taken a more model-based route. In [11], visuotactile sensors
are used to track geometric features of the objects such as lines and points.
These features are utilized to track the pose of the object and the contact
state, which is then used for feedback control. Similarly, [26] tracks the
state of a deformable cable by estimating the contact patch ellipse, and fits
a linear dynamics model which is stabilized by LQR. Although we use a similar
model-based approach, our work is unique in that we generalize the estimator
spatially, then explicitly do force control.

###  II-C Tactile Force and Pose Estimation

Many of the existing works in tactile pose/force estimation attempt to deal
with dense measurements. In [16], ICP is used from dense depth information in
order to estimate the poses of the object. Similarly, [4] uses geometric
contact rendering which is then compared with the dense tactile image. While
such dense information is useful for classification [35], it is unclear if
such dense information is necessary for control.

On the other hand, [11] estimates simpler features such as points and edges,
and [26] estimates ellipsoidal contact patches that are sufficient for
achieving the task. We use similar representation to [26], but estimate the
patch in 3D instead of localizing on the plane. While such approaches are
efficient to implement and is more relevant to the task, we note that they
lack geometric generalizability compared to dense information.

##  III Preliminary: 1D Series Elastic Actuator

In this section, we briefly review the concept of 1D SEAs, their proposed
benefits, and the corresponding control strategies. Although the section will
entirely be a review of previous work on SEAs, the ideas presented here will
have direct correspondences with our generalization.

###  III-A 1D Series Elasticity

Closed-loop force control often requires a motor, a gearbox, and a force
sensor in series. Typically, a relatively stiff sensor based on strain gauges
is used. However, this force-feedback setting can result in instability due to
high contact stiffness [21], as well as non-collocation of sensors and
actuators [13, 12, 25]. This prevents the use of high-gains that are necessary
to overcome undesired effects of the gearbox.

SEAs, initially proposed in [21], can be understood as a special case where
the sensor stiffness is very low. Under this setting, force-feedback enjoys
better stability properties at the expense of controller bandwidth, as the
spring acts like a mechanical low-pass filter [5, 13]. For many household
tool-use tasks such as wiping with a squeegee, the loss of control bandwidth
does not pose a big problem, as such tasks are usually quasistatic. Thus, one
may use high-gain position control to overcome unwanted effects of the
gearbox, while still maintaining stability of the system and achieving greater
force accuracy [21].

###  III-B Force Control of Series Elastic Actuators

We present a simple version of force control with series elastic actuators. In
force control, the user supplies a desired force $f_{d}$, which can be turned
into desired relative position using the sensor stiffness $k$. Then, a high-
gain position controller can be used to achieve this relative position. The
detailed procedure is described in Algorithm 1. In practice, frequency-domain
analysis can be done to carefully choose gains that stabilize the closed-loop
system [23].

1 Given: Desired force $\small~{}^{W}\\!f_{d}$, sensor stiffness $\small k$;

2 Convert desired force to desired deformation
$\small~{}^{T}\\!x^{C}_{d}=f_{d}/k$ ;

3 Convert desired spring deformation to desired position
$\small~{}^{W}\\!x^{T}_{d}=~{}^{W}\\!x^{T}+~{}^{T}\\!x^{C}-~{}^{T}\\!x^{C}_{d}$;

Use position control to regulate to desired position $\small
u=-k_{p}(~{}^{W}\\!x^{T}-~{}^{W}\\!x^{T}_{d})-k_{d}~{}^{W}\\!\dot{x}^{T}$

Algorithm 1 Force Control with SEAs

###  III-C Multi-DOF SEAs for Tool-Use

How can we utilize the benefits of SEAs to multiple degrees-of-freedom? One
straightforward answer might be to connect SEAs serially at the joint-level
[1]. However, achieving accurate end-effector position and force tracking
using joint-level SEAs requires fast and accurate joint-level torque sensing,
which is not available on many position-controlled robots. Instead, we offer
an alternative generalization of SEAs that concentrate the 6D elasticity into
the end effector, while allowing the robot to remain stiff. Our generalization
involves the following three components:

  1. 1.

A 6D deformable element capable of being stiff in multiple directions
simultaneously.

  2. 2.

A mechanism to sense the spatial deformation of the above element.

  3. 3.

A manipulator capable of controlling spatial pose of the deformable element.

##  IV SEED: Series Elastic end effector in 6D

In this section, we present Series Elastic End Effectors in 6D (SEED), a
spatial generalization of 1D SEAs that satisfies the three requirements in
Sec.III-C by using a soft deformable membrane, visuotactile sensing to sense
the spatial deformation of the membrane, and a position-controlled manipulator
to control the pose of the membrane base.

###  IV-A Defining 6D Series Elasticity

One of the challenges of generalizing the 1D SEA using a spatially compliant
element comes from defining an appropriate notion of spatial stiffness [3],
especially for large rotations (rotations up to 30 degrees are common in our
experiments). Rotational stiffness has been traditionally defined on the roll-
pitch-yaw and axis-angle parameterization of rotations [20, 19], which can be
made to work for large rotations.

In this work we have chosen the bushing model, which was initially proposed as
a coordinate-free parameterization of a bushing element in Drake [30]. The
bushing model also works for large rotations, and can be interpreted more
intuitively due to its correspondence to a spring-loaded gimbal (Fig. 1).
Based on the bushing model, we will develop a generalized stiffness map that
relates the relative pose between two frames to a spatial force.

###  IV-B Frame Definition

We give the definition of the frames here in order to better ground our notion
of 6D series elasticity to the setting of a soft and tactile hand grabbing a
tool. At the moment of grasp between the soft hand and the tool, two frames
are initialized: $T$, which is rigidly attached to the gripper at a pre-
defined nominal location (e.g. the center of the gripper), and $C$, which is
rigidly attached to the tool and initialized to be coincident with $T$ (i.e.
identity relative transform).

###  IV-C The Generalized Stiffness Map

Given the definition of these frames, our goal is to characterize the relation
between the relative pose of $C$ with respect to $T$ (denoted as
$\tensor[^{T}]{\mathbf{X}}{{}^{C}}\in\text{SE(3)}$) and the spatial force
(written in frame $T$) applied on $C$, which we denote as
$\tensor[^{T}]{F}{}\in\mathbb{R}^{6}$. We abstractly denote this as a
generalized stiffness map $\mathcal{K}:\text{SE}(3)\rightarrow\mathbb{R}^{6}$
such that Eq.1 holds:

$\tensor[^{T}]{F}{}=\mathcal{K}(\tensor[^{T}]{\mathbf{X}}{{}^{C}}).$ (1)

We expect $\mathcal{K}$ to be a generalized notion of stiffness with
smoothness and monotonicity properties under the following assumption of no-
slip.

######  Assumption 1.

No slip occurs between the contact patch of the gripper and the tool, such
that $\mathcal{K}$ smoothly maps relative transform to spatial force.

We now concretely describe the bushing model $\mathcal{K}$. We denote
${}^{T}\Theta^{C}=[r,p,w]^{\intercal}$ as the roll-pitch-yaw parametrization
(which lives on a gimbal) of
$\tensor[^{T}]{\mathbf{R}}{{}^{C}}\in\text{SO}(3)$, and
$\tensor[^{T}]{x}{{}^{C}}\in\mathbb{R}^{3}$ to be the position component of
$\tensor[^{T}]{\mathbf{X}}{{}^{C}}$. Similarly, the spatial force
$\tensor[^{T}]{F}{}$ is divided into torques
$\tensor[^{T}]{\tau}{}\in\mathbb{R}^{3}$ and forces
$\tensor[^{T}]{f}{}\in\mathbb{R}^{3}$. Then, the bushing model gives spatial
force for a pose using the following relation:

$\small\begin{pmatrix}\tensor[^{T}]{\tau}{}\\\
\tensor[^{T}]{f}{}\end{pmatrix}=\begin{pmatrix}\mathbf{N}^{\intercal}(\tensor[^{T}]{\Theta}{{}^{C}})\mathbf{K}_{\tau}\tensor[^{T}]{\Theta}{{}^{C}}\\\
\mathbf{K}_{f}\tensor[^{T}]{x}{{}^{C}}\end{pmatrix}.$ (2)

where $\mathbf{K}_{\tau}\in\mathbb{R}^{3\times 3}$ is the gimbal stiffness
matrix, and $\mathbf{K}_{f}\in\mathbb{R}^{3\times 3}$ is the standard
translational stiffness matrix.
$\mathbf{N}(\tensor[^{T}]{x}{{}^{C}})\in\mathbb{R}^{3\times 3}$ is the
coordinate transformation matrix necessary to convert gimbal torques to
spatial torques, and is given by

$\small\mathbf{N}(\tensor[^{T}]{\Theta}{{}^{C}})=\begin{pmatrix}\cos w\sec
p&\sin w\sec p&0\\\ -\sin w&\cos w&0\\\ \cos w\tan p&\sin w\tan
p&1\end{pmatrix}.$ (3)

The necessity of the matrix becomes apparent by visualizing
$\mathbf{K}_{\tau}\tensor[^{T}]{\Theta}{{}^{C}}$ as torques that are being
exerted on each axis of the gimbal, while $\tensor[^{T}]{\tau}{}$ is defined
spatially in $\mathbb{R}^{3}$. We obtain the matrix by equating the power on a
spatial representation
$\tensor[^{T}]{\tau}{}\cdot\tensor[^{T}]{\omega}{{}^{C}}$ to power on the
gimbal representation
$\mathbf{K}_{\tau}\tensor[^{T}]{\Theta}{{}^{C}}\cdot\tensor[^{T}]{\dot{\Theta}}{{}^{C}}$,
and using the standard conversion between angular velocities and gimbal rates.
Throughout our work, we make the following assumption on the structure of
$\mathbf{K}_{\tau}$ and $\mathbf{K}_{f}$.

######  Assumption 2.

The gimbal stiffness matrix $\mathbf{K}_{\tau}$ and the translational
stiffness matrix $\mathbf{K}_{f}$ are positive definite diagonal matrices.

Under Assumption 2, we present the following theorem which gives a more
rigorous notion of smoothness mentioned in Assumption 1.

######  Theorem 1.

The bushing model stiffness map
$\mathcal{K}:\text{SE}(3)\rightarrow\mathbb{R}^{6}$ is a diffeomorphism under
Assumption 2 everywhere for $|p|\leq\pi/2$.

###### Proof.

Since there is no coupling between the orientation and translational maps, it
suffices to separately show that each are diffeomorphisms. The translational
map given by ${}^{T}f=\mathbf{K}_{f}\tensor[^{T}]{x}{{}^{C}}$ is trivially a
diffeomorphism under Assumption 2. We use the Inverse Function Theorem to
prove the inverse differentiability of the orientation map. The determinant of
the Jacobian for the orientation map is given by

$\small\det\mathbf{J}=\frac{k_{r}k_{p}k_{y}}{\cos p},$ (4)

where $k_{r},k_{p},k_{y}$ are the diagonal elements of $\mathbf{K}_{\tau}$,
which is well defined everywhere in $|p|<\pi/2$. We complete the proof by
noting that the orientation map is bijective, and show it by providing a well-
defined nonlinear inverse
$\tensor[^{T}]{\Theta}{{}^{C}}=\mathcal{K}^{-1}(^{T}\tau)$:

$\small\begin{pmatrix}r\\\ p\\\
y\end{pmatrix}=\begin{pmatrix}\frac{1}{k_{r}}\big{(}(\tau_{x}\cos
y+\tau_{y}\sin y)\big{)}\cos p-\tau_{z}\sin p\\\
\frac{1}{k_{p}}\big{(}\tau_{y}\cos y-\tau_{x}\sin y\big{)}\\\
\frac{1}{k_{y}}\tau_{z}\\\ \end{pmatrix}.$ (5)

Note that the equation is written in semi-implicit form to save space: one can
easily make it explicit by substituting values starting from the bottom row. ∎

Theorem 1 tells us that our model of $\mathcal{K}$ follows desirable
properties that can smoothly map back and forth between relative pose
deformation and spatial force, which we can effectively use in order to do
force and impedance control in a manner akin to SEAs. We also note that in
hardware, we expect $|p|$ to be confined to $\pi/3$ at most before Assumption
1 is broken and slip occurs.

##  V Force control with SEED

Figure 2: Left: System Identification Setup with the Soft-Bubble gripper [15]
and a 6-axis force/torque sensor. Center: Results of identified stiffness
values for different axis, per change in pressure. Right Top: Change in
pressure as a function of gripper distance command. Right Bottom: Frame
definition.

Now we present our main algorithms for doing control with SEED, which follows
the general philosophy of controllers using SEAs: a force control problem is
turned to a position control problem [21]. Thus, we assume access to a
manipulator that can achieve reliable position commands with high gains and
rates (akin to how SEAs can use high gains to overcome gearbox effects quickly
and achieve accurate positions), which describes most position controlled
manipulators with high mechanical repeatability.

###  V-A Problem Setup - Feedback and Action

To setup the control problem, we note that the position controlled manipulator
can command end-effector pose
$u(t)=\tensor[^{W}]{\mathbf{X}}{{}^{T}}(t)\in\text{SE}(3)$ with high rates
using direct inverse kinematics or integration of differential inverse
kinematics. Our feedback signal will come from the estimation of relative pose
$y(t)=\tensor[^{T}]{\mathbf{X}}{{}^{C}}(t)$, which is measured by the
visuotactile method given in Sec.VII. Then, the goal is to find a policy
$u(t)=\pi(y(t))$ that achieves some desired specification of the user.

Throughout the section, we will assume we have some estimate of parameters
$\hat{\mathbf{K}}_{\tau}$ and $\hat{\mathbf{K}}_{f}$ that define the
generalized stiffness map, and denote the estimated map as
$\hat{\mathcal{K}}(\hat{\mathbf{K}}_{\tau},\hat{\mathbf{K}}_{f})$.

###  V-B 6D Force Control

In force control, the user specifies some desired spatial force
$\tensor[^{W}]{F}{{}_{d}}$, described in the world frame. SEED achieves this
specified spatial force by converting it to some desired relative transform
with the estimated generalized stiffness map $\hat{\mathcal{K}}$. Then, a
position command is sent to the manipulator to achieve this relative pose. We
describe the detailed process in Algorithm 2.

1 Given: Desired wrench $\tensor[^{W}]{F}{{}_{d}}$;

2 Given: Estimated SEED stiffness
$\hat{\mathcal{K}}(\hat{\mathbf{K}}_{\tau},\hat{\mathbf{K}}_{f})$;

3 while  _$t <T$_ do

4 Convert $\tensor[^{W}]{F}{{}_{d}}$ to $\tensor[^{T}]{F}{{}_{d}}(t)$ using
adjoint transform with current position $\tensor[^{W}]{\mathbf{X}}{{}^{T}}(t)$
;

5 Using the estimated SEED stiffness $\hat{\mathcal{K}}$, convert desired
wrench into desired relative pose by
$\tensor[^{T}]{\mathbf{X}}{{}^{C}_{d}}(t)=\hat{\mathcal{K}}^{-1}(\tensor[^{T}]{F}{{}_{d}}(t))$;

6 Convert desired relative pose into desired end effector pose by
$\tensor[^{W}]{\mathbf{X}}{{}^{T}_{d}}(t)=\tensor[^{W}]{\mathbf{X}}{{}^{T}}(t)\cdot\tensor[^{T}]{\mathbf{X}}{{}^{C}}(t)\cdot\tensor[^{C}]{\mathbf{X}}{{}^{T}_{d}}(t)$;

7 Send position command $\tensor[^{W}]{\mathbf{X}}{{}^{T}_{d}}(t)$ to the
position controller.

Algorithm 2 Force Control with SEED

The expression for the orientation part of $\mathcal{K}^{-1}$ has been given
in Eq.5, while inverting the position simply requires $\mathbf{K}^{-1}_{f}$.
We note that, like most force control strategies, the controller will not be
well-behaved if there is no contact with an external environment. In
particular, while position-only force control can move until contact and
maintain some desired force, the orientation torque controller must keep
rotating until contact, which likely runs into workspace limitations of the
manipulator quickly and makes the controller impractical to use in free-space.

###  V-C 6D Hybrid Force/Pose Control

In many tasks involving tools, the goal is to simultaneously control force and
torque in certain directions, while controlling position and orientation in
other directions. We naturally extend spatial force control with SEED to this
setting by defining a partial inverse of the impedance map $\mathcal{K}$ that
attempts to construct the spatial deformation
$\tensor[^{T}]{\mathbf{X}}{{}^{C}_{d}}$ from a subset of specified forces.

####  V-C1 Hybrid Force/Position Control

Given a task-relevant decomposition matrix $\mathbf{P}$ which selects a
subspace for desired position $x_{d}\in\mathbb{R}^{3}$ and desired force
$f_{d}\in\mathbb{R}^{3}$, we can compute the position $x\in\mathbb{R}^{3}$
that achieves the specified positions and forces:

$\small x=\mathbf{P}x_{d}+\mathbf{P}^{\perp}\mathbf{K}_{f}^{-1}f_{d},$ (6)

where $\mathbf{P}^{\perp}$ is the matrix that represents the orthogonal
complement of $\mathbf{P}$.

####  V-C2 Hybrid Torque/Orientation Control

Unlike force / position control, coordinate transform in rotational space does
not happen in a linear manner. Thus, defining hybrid torque/orientation
control for an arbitrary task-relevant coordinate representation is
significantly more difficult. To deal with this problem, we make the following
assumption:

######  Assumption 3.

The decomposition of specified orientation and torques happen in the frame of
$T$.

Such an assumption is not too restrictive under a large class of tools, as
most tools require decomposition of torques and angles in a manner consistent
with its natural task-relevant coordinate frame. Under such assumption, we can
define partial maps from a subset of desired torques to the full orientation
as follows:

  1. 1.

2 torques, 1 angle: The following angles $(r,p,y)$ achieve the given two
desired torques $\tau_{x}^{d},\tau_{y}^{d}$ and one desired angle $y_{d}$,
given the stiffness map $\mathcal{K}$:

$\small\begin{pmatrix}r\\\ p\\\
y\end{pmatrix}=\begin{pmatrix}\frac{1}{k_{r}}((\tau_{x}^{d}\cos
y_{d}+\tau_{y}^{d}\sin y_{d})\cos p-k_{y}y_{d}\sin p)\\\
\frac{1}{k_{p}}(\tau_{y}^{d}\cos y_{d}-\tau_{x}^{d}\sin y_{d})\\\
y_{d}\end{pmatrix}.$ (7)

  2. 2.

1 torque, 2 angles: The following angles $(r,p,y)$ achieves the given torque
$\tau_{z}^{d}$ and the two desired angles $r_{d},p_{d}$, given the stiffness
map $\mathcal{K}$:

$\small r=r_{d}\qquad p=p_{d}\qquad y=\tau_{z}/k_{y}.$ (8)

After recovering the full pose from a subset of desired forces and torques, we
use position control to command this pose, as done in Alg. 2.

##  VI System Identification

In order to apply our framework, we need to do identification on the
parameters of the stiffness map $(\mathbf{K}_{\tau},\mathbf{K}_{f})$, which is
consisted of $6$ stiffness parameters. The stiffness parameters can be
identified by measuring the static sensitivity of wrench with respect to pose.
We achieve this by having a dexterous manipulator grab a 6-axis force/torque
sensor and perturbing the pose to observe responses in wrench.

In addition, to see if squeezing or pressurizing the gripper changes the
stiffness parameters of the hand, we use the pressure sensor on board the
soft-bubble hand [15] to characterize how the gripper distance affects the
pressure, and in turn, how the pressure affects the identified stiffness
values. The results of our experiments are presented in Fig.2.

Along with the quantitative values of stiffness, we summarize our findings
from the identification process:

  1. 1.

The dependence on internal pressure of the hand with respect to the gripper
distance is linear.

  2. 2.

For $x$ direction torque and all the forces, higher pressure near-linearly
corresponds to higher stiffness. The identified stiffness values also have low
standard deviation.

  3. 3.

For $y,z$ direction torque, the measurement is relatively unreliable and the
identified stiffness values are subject to large standard deviations. In
addition, higher pressure does not seem to lead to higher stiffness values
along these directions.

The results of system identification, combined with the monotinicity of the
stiffness map, leads to a very natural interpretation: if stiffer behavior is
desired while controlling the tool, the hardware gives us the means to control
the stiffness by grabbing the tool more firmly or by more pressurization.

##  VII Tactile Relative Pose Estimation

In principle, our framework of control can work well with any tactile end
effector that is compliant enough, and an estimation algorithm that produces a
well-behaved estimate of the relative pose
$\tensor[^{T}]{\mathbf{X}}{{}^{C}}$. In our work, we show an example of such a
relative pose estimator by utilizing the PicoFlexx IR-Depth camera mounted
within the bubble grippers [15].

###  VII-A Contact Patch Estimation

We estimate the position of the contact patch using a simple background
subtraction algorithm. Denote $\mathbf{D}_{k}\in\mathbb{R}^{H\times W}$ as
depth image at time $k$. Then, we simply compare $\mathbf{D}_{k}$ to the
initial depth image $\mathbf{D}_{0}$, taken when the bubble is not in contact.
After performing a thresholding operation to obtain the difference, we perform
a morphological transformation using an elliptical kernel to obtain a binary
mask $\mathbf{M}_{k}\in\\{0,1\\}^{H\times W}$. Finally, we use the calibration
matrix $\mathbf{K}\in\mathbb{R}^{3\times 3}$ to transform the masked depth
image $\mathbf{M}_{k}\odot\mathbf{D}_{k}$ into a set of points
$\\{{}^{L}p_{i}^{P_{L}}\\}$, where $\odot$ denotes element-wise
multiplication, $L$ denotes the left camera frame, and $P_{L}$ denotes the
left contact patch. Finally, we take a mean
${}^{L}p^{P_{L}}=\frac{1}{N}\sum^{N}_{i=1}{}^{L}p_{i}^{P_{L}}\in\mathbb{R}^{3}$
to obtain the 3D coordinate of the contact patch, and repeat this process for
the right camera.

###  VII-B Frame Estimation from Contact Patches

Given the location of the contact patch on the left bubble expressed in the
gripper frame ${}^{G}p^{P_{L}}$ and ${}^{G}p^{P_{R}}$, we average the
positions of the two patches to obtain the position of the contact frame:

${}^{G}p^{C}=({}^{G}p^{P_{L}}+{}^{G}p^{P_{R}})/2.$ (9)

To compute the rotation ${}^{G}\mathbf{R}^{C}$, we introduce an intermediate
frame $C^{\prime}$ such that the $y$ axis of $C^{\prime}$ is aligned with
${}^{G}p^{P_{R}}-{}^{G}p^{P_{L}}$, and the $z$ component of the $x$ axis is
zero. Denote $u,v,w$ as columns of ${}^{G}\mathbf{R}^{C^{\prime}}$ (i.e.
${}^{G}\mathbf{R}^{C^{\prime}}=[u|v|w]$), which individually represent the
components of the unit vector that define ${}^{G}\mathbf{R}^{C^{\prime}}$.
Then, we compute the columns using the following process:

  1. 1.

Set $v$ to be the normalization of ${}^{G}p^{P_{R}}-{}^{G}p^{P_{L}}$.

  2. 2.

Set $u_{z}=0$ to define the zero-pitch frame, and $u_{x}=1$

  3. 3.

Set Compute $u_{y}$ by using $u\cdot v=0$, and normalize $u$.

  4. 4.

Set $w=u\times v$

Figure 3: Frame Definition and example images for Relative Pose Estimation.

###  VII-C Pitch Estimation with Optical Flow

After computing ${}^{C^{\prime}}\mathbf{R}^{C}$, we compute
${}^{C^{\prime}}\mathbf{R}^{C}$, by computing the rotation along the $y$ axis
of $C^{\prime}$. We estimate this quantity by computing the optical flow of
the IR image. We denote as $\mathbf{V}_{0\rightarrow k}$ as the Eulerian flow
of $\mathbf{I}_{k}$ relative to $\mathbf{I}_{0}$. Then, we compute the curl of
$\mathbf{V}_{0\rightarrow k}$:

$\small\theta=k\nabla_{\times}\mathbf{V}_{0\rightarrow
k}=k\bigg{(}\frac{\partial}{\partial x}\mathbf{V}_{0\rightarrow
k}^{y}-\frac{\partial}{\partial y}\mathbf{V}_{0\rightarrow k}^{x}\bigg{)},$
(10)

where the superscript denotes the component of the vector field, and $k$ is
some normalization constant we calibrate for. The gradients are computed using
a Sobel filter with corresponding kernels.

###  VII-D Validation Results

In order to validate the performance of the proposed relative pose estimator,
we use the same setup that was used for system identification (Fig.2). Through
the forward kinematics of the manipulator, and the fact that
${}^{W}\mathbf{X}^{C}$ is a fixed transform for the system identification
setup, we compare the measured values of ${}^{G}\mathbf{X}^{C}$ with the
results of the relative pose estimator ${}^{G}\hat{\mathbf{X}}^{C}$. Our
results, illustrated in Fig.4, show that tracking performance of relative pose
is dependent on which axis is being tracked:

Figure 4: Tracking performance of the Relative Pose Estimation

  1. 1.

The $y$ position, which uses depth information from each camera, can be
tracked reliably.

  2. 2.

On the other hand, the $x$ and $z$ position tracking is not very reliable due
to the large contact patch caused by the cylindrical geometry of the tool.

  3. 3.

The locations of contact patches on both sides give a very good estimate for
roll angle. Optical flow is also successful in tracking pitch.

  4. 4.

While yaw shows reasonable behavior, the estimate tends to underestimate the
true yaw angle as the contact patch lags behind true rotation due to the
softness of the membrane (i.e. perfect roll does not occur).

##  VIII Experiment Methods & Results

###  VIII-A Simulation Methods & Results

To verify the performance of our proposed pipeline, we first set up a
simulation in Drake [30], where the compliance between the tool frame $T$ and
the compliance frame $C$ is simulated using Drake’s 6D compliance element
LinearBushingRollPitchYaw. By assuming a perfect measurement of the relative
pose ${}^{T}\mathbf{X}^{C}$, we aim to decouple the validity of the proposed
controller with the accuracy of the tactile relative pose estimator.

####  VIII-A1 The Squeegee Task

The squeegee is a tool that requires regulation of spatial forces along some
principle axis, while requiring regulation of position along others. We
illustrate the frame definition in Fig.5, and decompose the spatial forces and
positions in the following directions in order to set a task specification of
the hybrid force position controller:

  1. 1.

${}^{W}x^{C},{}^{W}y^{C},{}^{T}\psi^{C}$ are used for position control in
order to specify the trajectory of the tool from a table-top view.

  2. 2.

${}^{T}\tau_{y}$ and ${}^{W}f_{z}$ are used to enforce the magnitude of
pressure between the blade and the table.

  3. 3.

${}^{T}\tau_{x}$ is used to enforce equal pressure distribution.

As a baseline, we include an open-loop trajectory that is tuned such that the
squeegee barely contacts the table, within the mechanical repeatability of the
manipulator (0.1mm). In addition, we modify the controller for the case where
the tool is rigidly fixed (welded) to the end effector in order to simulate
the performance of a custom tool changer. The resulting contact forces are
inspected based on how much force is exerted ($F_{z}=\sum_{i}\lambda^{i}$),
and how much the pressure distribution on the blade is balanced
($\tau_{x}=\sum_{i}r_{i}\times\lambda^{i}$). The resulting trajectory is shown
in Fig.6.

We show that compared to the case where the end effector is rigidly attached,
the compliant hardware allows much better tracking of $\tau_{x}$, such that
equal pressure is applied on both sides of the squeegee. We mainly attribute
this behavior to the built-in compliance, as $\tau_{x}$ behaves well even in
the open-loop setup. By commanding the desired force in closed-loop however,
the 6D hybrid force-position controller adds the ability to exert desired
amount of forces. Finally, we note that there exists offset in the tracking
error due to the unaccounted weight of the tool.

Figure 5: Left: Drake Simulation environment. Right: Frame and contact point
definitions for the squeegee tool. Figure 6: Controller results of tracking
$f_{z}$ and $\tau_{x}$.

###  VIII-B Hardware Methods & Results

Though we have verified the behavior of the controller in simulation assuming
perfect pose tracking, showing the controller on hardware requires coupling
the pose estimator and the controller in all six axes. However, the estimator
is unreliable in certain directions such as yaw or $x$-position, which can
adversely destabilize closed-loop behavior.

In order to overcome these limitations of the estimator, we propose a simple
yet effective strategy: we purposely align the axis that requires force
tracking with the axis that our estimator performs well in. As most tasks
require at most two or three components of force tracking, we show that it is
possible to only estimate well a subset of the relative pose, and still
achieve the underlying task.

####  VIII-B1 Pen Writing Task

We first test the controller on a pen-writing task, where the robot is
commanded to write some characters in the $xy$ plane, while some force is
commanded in the $z$ direction. Our setup is illustrated in Fig.7.D. As the
result of Fig.7.B demonstrates, our controller achieves good tracking
performance of specified force, as observed by the differences in marker
stroke width and darkness.

We also test our controller by writing letters in Fig.7.C. While we are
successful in tracking the characters, the inherent softness of the hardware
sacrifices the bandwidth of the position controller, and frictional
interactions between the marker and the paper (e.g. caused by the Painleve
effect) can compromise the tracking performance of SEED.

####  VIII-B2 Squeegee Task

We test the proposed controller on a real-life task of using a squeegee to
clean some liquid on top of a cutting board. The results of our hardware
experiment are shown in Fig.7.A. While the open-loop baseline fails to exert
much force on the board, the closed-loop controller is successful in pressing
down firmly and clearing all the liquid.

Figure 7: A. Deploying our controller on hardware for using the squeegee (top
row), against the open-loop baseline (bottom row). B. Comparison of open-loop
stroke against closed-loop strokes with different specified downward forces.
C. Performance of character tracking. D. Pen writing setup.

##  IX Conclusion

We have presented SEED, a control and hardware framework that combines the
benefits of hardware compliance with visuotactile sensing. Throughout our
work, we have demonstrated that we can measure the relative pose of a tool
with respect to the gripper using visuotactile sensing. Combined with offline-
identified parameters of our spatial stiffness model, we have shown that we
can achieve closed-loop spatial force control that can be useful for tool-use.
By our demonstration, we aim to alleviate some of the difficulties that rigid
contacts and the associated non-smooth behavior bring in the setting of
grasping and using tools in the wild.

## References

  * Albu-Schäffer et al. [2007] Alin Albu-Schäffer, Christian Ott, and Gerd Hirzinger.  A unified passivity-based control framework for position, torque and impedance control of flexible joint robots.  _The International Journal of Robotics Research_ , 26(1):23–39, 2007.
  * Alspach et al. [2019] Alex Alspach, Kunimatsu Hashimoto, Naveen Kuppuswarny, and Russ Tedrake.  Soft-bubble: A highly compliant dense geometry tactile sensor for robot manipulation.  In _RoboSoft_ , pages 597–604, 04 2019.
  * Angeles [2010] Jorge Angeles.  On the Nature of the Cartesian Stiffness Matrix.  _Ingeniería mecánica, tecnología y desarrollo_ , 3, 09 2010.
  * Bauza et al. [2020] Maria Bauza, Eric Valls, Bryan Lim, Theo Sechopoulos, and Alberto Rodriguez.  Tactile object pose estimation from the first touch with geometric contact rendering, 2020.
  * Buerger [2006] Stephen Buerger.  _Stable, high-force, low-impedance robotic actuators for human-interactive machines_.  PhD thesis, Massachsuetts Institute of Technology, 09 2006.
  * Chen and Burdick [1993] I-Ming Chen and J.W. Burdick.  Finding antipodal point grasps on irregularly shaped objects.  _IEEE Transactions on Robotics and Automation_ , 9(4):507–512, 1993.
  * Cutkosky [1989] M.R. Cutkosky.  On grasp choice, grasp models, and the design of hands for manufacturing tasks.  _IEEE Transactions on Robotics and Automation_ , 5(3):269–279, 1989.
  * Dong et al. [2021] Siyuan Dong, Devesh K. Jha, Diego Romeres, Sangwoon Kim, Daniel Nikovski, and Alberto Rodriguez.  Tactile-rl for insertion: Generalization to objects of unknown geometry, 2021.
  * Donlon et al. [2018] E. Donlon, S. Dong, M. Liu, J. Li, E. Adelson, and A. Rodriguez.  Gelslim: A high-resolution, compact, robust, and calibrated tactile-sensing finger.  In _2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , pages 1927–1934, 2018.
  * Elliott et al. [2016] Sarah Elliott, Michelle Valente, and Maya Cakmak.  Making objects graspable in confined environments through push and pull manipulation with a tool.  In _2016 IEEE International Conference on Robotics and Automation (ICRA)_ , pages 4851–4858, 2016.
  * F.R.Hogan et al. [2020] F.R.Hogan, J.Ballester, S.Dong, and A.Rodriguez.  Tactile dexterity: Manipulation primitives with tactile feedback.  In _2020 International Conference on Robotics and Automation_ , 2020\.
  * GEVARTER [1970] WILLIAM B. GEVARTER.  Basic relations for control of flexible vehicles.  _AIAA Journal_ , 8(4):666–672, 1970.
  * Hogan [1985] Neville Hogan.  Impedance Control: An Approach to Manipulation: Part I—Theory.  _Journal of Dynamic Systems, Measurement, and Control_ , 107(1):1–7, 03 1985.  ISSN 0022-0434.
  * Holladay et al. [2019] Rachel Holladay, Tomás Lozano-Pérez, and Alberto Rodriguez.  Force-and-motion constrained planning for tool use.  In _2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , pages 7409–7416, 2019.
  * Kuppuswamy et al. [2020] N. Kuppuswamy, A. Alspach, A. Uttamchandani, S. Creasy, T. Ikeda, and R. Tedrake.  Soft-bubble grippers for robust and perceptive manipulation.  _International Conference on Intelligent Robots and Systems (IROS)_ , 2020.
  * Kuppuswamy et al. [2020] Naveen Kuppuswamy, Alejandro Castro, Calder Phillips-Grafflin, Alex Alspach, and Russ Tedrake.  Fast model-based contact patch and pose estimation for highly deformable dense-geometry tactile sensors.  _IEEE Robotics and Automation Letters_ , 5(2):1811–1818, 2020.
  * Lee et al. [2019] Michelle A. Lee, Yuke Zhu, Krishnan Srinivasan, Parth Shah, Silvio Savarese, Li Fei-Fei, Animesh Garg, and Jeannette Bohg.  Making sense of vision and touch: Self-supervised learning of multimodal representations for contact-rich tasks.  In _2019 International Conference on Robotics and Automation (ICRA)_ , pages 8943–8950, 2019.
  * Li and Sastry [1988] Z. Li and S.S. Sastry.  Task-oriented optimal grasping by multifingered robot hands.  _IEEE Journal on Robotics and Automation_ , 4(1):32–44, 1988.
  * Natale [2003] C. Natale.  _Interaction Control of Robot Manipulators_.  Springer Tracts in Advanced Robotics, 2003.
  * Natale et al. [1999] C. Natale, B. Siciliano, and L. Villani.  Spatial impedance control of redundant manipulators.  In _Proceedings 1999 IEEE International Conference on Robotics and Automation_ , volume 3, pages 1788–1793 vol.3, 1999.
  * Pratt and Williamson [1995] G. A. Pratt and M. M. Williamson.  Series elastic actuators.  In _Proceedings 1995 IEEE/RSJ International Conference on Intelligent Robots and Systems. Human Robot Interaction and Cooperative Robots_ , volume 1, pages 399–406 vol.1, 1995.
  * Raibert and Craig [1981] M. H. Raibert and J. J. Craig.  Hybrid Position/Force Control of Manipulators.  _Journal of Dynamic Systems, Measurement, and Control_ , 103(2):126–133, 06 1981.  ISSN 0022-0434.  URL https://doi.org/10.1115/1.3139652.
  * Robinson [2000] David William Robinson.  _Design and Analysis of Series Elasticity in Closed-loop Actuator Force Control_.  PhD thesis, Massachsuetts Institute of Technology, 06 2000.
  * Schindlbeck and Haddadin [2015] Christopher Schindlbeck and Sami Haddadin.  Unified passivity-based cartesian force/impedance control for rigid and flexible joint robots via task-energy tanks.  volume 2015, pages 440–447, 06 2015.
  * Sharon et al. [1988] A. Sharon, N. Hogan, and D. E. Hardt.  High bandwidth force regulation and inertia reduction using a macro/micro manipulator system.  In _Proceedings. 1988 IEEE International Conference on Robotics and Automation_ , pages 126–132 vol.1, 1988.
  * She et al. [0] Yu She, Shaoxiong Wang, Siyuan Dong, Neha Sunil, Alberto Rodriguez, and Edward Adelson.  Cable manipulation with a tactile-reactive gripper.  _The International Journal of Robotics Research_ , 0(0):02783649211027233, 0.
  * Shumaker et al. [2011] R.W Shumaker, K.R. Walkup, and B.B. Beck.  _Animal Tool Behavior: The Use and Manufacture of Tools by Animals_.  Johns Hopkins University Press, 2011.
  * Suh and Tedrake [2021] H. Suh and Russ Tedrake.  The surprising effectiveness of linear models for visual foresight in object pile manipulation.  In _WAFR_ , 2021.
  * Suh et al. [2021] H. J. Terry Suh, Tao Pang, and Russ Tedrake.  Bundled gradients through contact via randomized smoothing, 2021.
  * Tedrake and the Drake Development Team [2019] Russ Tedrake and the Drake Development Team.  Drake: Model-based design and verification for robotics, 2019.  URL https://drake.mit.edu.
  * Toussaint et al. [2018] Marc Toussaint, Kelsey R. Allen, Kevin A. Smith, and Joshua B. Tenenbaum.  Differentiable physics and stable modes for tool-use and manipulation planning.  In _Robotics: Science and Systems_ , 2018.
  * Toussaint et al. [2020] Marc Toussaint, Jung-Su Ha, and Danny Driess.  Describing physics for physical reasoning: Force-based sequential manipulation planning.  _IEEE Robotics and Automation Letters_ , PP:1–1, 07 2020\.
  * Villani and Schutter [2008] Luigi Villani and J.D. Schutter.  Force control.  volume 1, pages 161–185, 2008.
  * Wang et al. [2020] Chen Wang, Shaoxiong Wang, Branden Romero, Filipe Veiga, and E. Adelson.  Swingbot: Learning physical features from in-hand tactile exploration for dynamic swing-up manipulation.  _2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , pages 5633–5640, 2020.
  * Yuan et al. [2017] Wenzhen Yuan, Siyuan Dong, and Edward Adelson.  Gelsight: High-resolution robot tactile sensors for estimating geometry and force.  _Sensors_ , 17(12):2762, Nov 2017.  ISSN 1424-8220.  doi: 10.3390/s17122762.