${\mathcal{L}}^{\prime}$. The intention is that we will expand models of $T$
to include as the interpretation of $S$ the product of the sorts $S_{n}$. We
expand every model $\mathcal{M}$ of $T$ by interpreting $S$ as
$\prod_{n\in{\mathbf{N}}}S_{n}^{\mathcal{M}}$. We need a metric for this sort.
The choice of metric is not unique but for definiteness we use, for
$\bar{x},\bar{y}\in\prod_{n\in{\mathbf{N}}}S_{n}^{\mathcal{M}}$,

$d(\bar{x},\bar{y})=\sum_{n\in{\mathbf{N}}}\frac{d_{n}^{\mathcal{M}}(x_{n},y_{n})}{B_{n}2^{n}},$

where $d_{n}$ is the metric symbol for $S_{n}$ and $B_{n}$ is the bound on
that metric. The key property that $d$ has is that for every $\epsilon>0$,
there is an $n\in{\mathbf{N}}$ such that if
$\bar{x},\bar{y}\in\prod_{n\in{\mathbf{N}}}S_{n}^{\mathcal{M}}$ and
$x_{i}=y_{i}$ for every $i<n$, then $d(\bar{x},\bar{y})<\epsilon$. The only
other symbols we add to the language ${\mathcal{L}}^{\prime}$ are maps
$\pi_{n}$ from $S$ to $S_{n}$, and in the canonical expansion of an
${\mathcal{L}}$-structure to an ${\mathcal{L}}^{\prime}$-structure, we
interpret $\pi_{n}$ as the projection onto the $n^{th}$ coordinate. The
uniform continuity modulus for $\pi_{n}$ can be taken to be
$\epsilon/B_{n}2^{n}$.

We record here three families of sentences that hold in all expanded
structures. First, for any $n\in{\mathbf{N}}$, we want to state that

$\sum_{i\leq n}\frac{d_{i}^{\mathcal{M}}(x_{i},y_{i})}{B_{i}2^{i}}\leq
d(x,y)\leq\sum_{i\leq
n}\frac{d_{i}^{\mathcal{M}}(x_{i},y_{i})}{B_{i}2^{i}}+\frac{1}{2^{n}}$

holds for all $x,y\in S$; we can express this fact in continuous logic by
requiring that the following two sets of sentences evaluates to $0$:

$\sup_{x,y\in S}\left(d(x,y)\buildrel\textstyle\ .\over{\hbox{ \vrule
height=3.0pt,depth=0.0pt,width=0.0pt}{\smash{-}}}\ (\sum_{i\leq
n}\frac{d_{i}^{\mathcal{M}}(\pi_{i}(x),\pi_{i}(y))}{B_{i}2^{i}}+\frac{1}{2^{n}})\right).$

and

$\sup_{x,y\in S}\left(\sum_{i\leq
n}\frac{d_{i}^{\mathcal{M}}(\pi_{i}(x),\pi_{i}(y))}{B_{i}2^{i}}\buildrel\textstyle\
.\over{\hbox{ \vrule height=3.0pt,depth=0.0pt,width=0.0pt}{\smash{-}}}\
d(x,y)\right).$

We would also like the finite products to be dense in $S$. We can achieve this
by satisfying the following sentences, one for each $n\in{\mathbf{N}}$:

$\sup_{x_{i}\in S_{i},i<n}\inf_{x\in
S}\max_{i<n}\\{d_{i}(\pi_{i}(x),x_{i})\\}.$

Let $T^{\prime}$ be the ${\mathcal{L}}^{\prime}$-theory consisting of $T$
together with the three preceding sets of sentences. If $\mathcal{M}^{\prime}$
is a model of $T^{\prime}$, then it has the form $(\mathcal{M},Y)$, where
$\mathcal{M}$ is a model of $T$ and $Y$ is the interpretation of the sort $S$.
We wish to see that $S$ is essentially
$\prod_{n\in{\mathbf{N}}}S_{n}^{\mathcal{M}}$. Fix $x_{i}\in
S_{i}^{\mathcal{M}}$. The third set of axioms guarantees that, for every
$n\in{\mathbf{N}}$, there is $y^{n}\in Y$ such that
$d(\pi_{i}(y^{n}),x_{i})<\frac{1}{2^{n}}$ for every $i<n$. The first set of
axioms assures us that the sequence $(y^{n}:n\in{\mathbf{N}})$ converges to
some $y\in Y$ (since $Y$ is complete). For this $y$, we have that
$d(\pi_{n}(y),x_{n})=0$ for all $n\in{\mathbf{N}}$. If there is another
$y^{\prime}\in Y$ with the same property, then using the first set of
sentences again, we have that $d(y,y^{\prime})=0$ and so $y=y^{\prime}$. In
this way we see that $Y$ can be identified with
$\prod_{n\in{\mathbf{N}}}S_{n}^{\mathcal{M}}$ via the projection maps. This
identification is an isometry since the first two sets of axioms guarantees
that the metric on $Y$ is given by

$d(x,y)=\sum_{n}\frac{d_{n}(\pi_{n}(x),\pi_{n}(y))}{B_{n}2^{n}}.$

What we have achieved is the following:

######  Theorem 10.1.

The forgetful functor from $Mod(T^{\prime})$ to $Mod(T)$ is an equivalence of
categories.

###  10.2. Definable Sets

Although the following collection of imaginary elements is not entirely
essential, it is often useful in practice. Suppose that $X$ is a definable
$T$-functor for some ${\mathcal{L}}$-theory $T$. We create a new language
${\mathcal{L}}^{\prime}$ by adding a new sort $S$ and a new function symbol
$i$ with domain $S$ and co-domain equal to the domain of $X$. The intention is
to interpret $S$ as $X(\mathcal{M})$ for $\mathcal{M}$ a model of $T$ and to
interpret $i$ as the inclusion map from $S$ into $\mathcal{M}$. Here are
sentences in ${\mathcal{L}}^{\prime}$ which achieve what we want:

$\sup_{x\in S}d(i(x),X)$

and

$\sup_{x,y\in S}|d(x,y)-d(i(x),i(y))|.$

It is reasonably easy to see that the ${\mathcal{L}}^{\prime}$-theory
$T^{\prime}$ formed from $T$ and these two axioms yields an essentially unique
expansion of models of $T$. Once again we have:

######  Theorem 10.2.

The forgetful functor from $Mod(T^{\prime})$ to $Mod(T)$ is an equivalence of
categories.

###  10.3. Canonical Parameters

The following collection of imaginary elements is the closest analogy to the
classical case. Fix a language ${\mathcal{L}}$ and an ${\mathcal{L}}$-theory
$T$. Choose an ${\mathcal{L}}$-formula $\varphi(\bar{x},\bar{y})$. Suppose
that the sort of $\bar{y}$ is $\bar{S}$. The idea is that we wish to capture
in a single sort all of the functions of the form
$\varphi^{\mathcal{M}}(\bar{x},\bar{a})$ for $\mathcal{M}$ a model of $T$ and
$\bar{a}\in\mathcal{M}$. We will call the elements of this sort the _canonical
parameters_ 555The name comes from the fact that $\bar{a}$ are the parameters
for the function $\varphi(\bar{x},\bar{a})$ but there may be other parameters
which give the same function. If we make any two such sequences equivalent,
then we have a ”canonical” choice for the parameters of this function. of
$\varphi(\bar{x},\bar{y})$. In order to do this, we create a new language
${\mathcal{L}}^{\prime}$ by adding to ${\mathcal{L}}$ a sort $S$ with metric
symbol $d_{S}$ and a function symbol $\pi$ with domain $\bar{S}$ and co-domain
$S$. We add the following two axioms to the theory $T$ to form $T^{\prime}$:

$\sup_{z\in S}\inf_{\bar{y}\in\bar{S}}d(z,\pi(\bar{y}))$

and

$\sup_{\bar{y},\bar{y}^{\prime}\in\bar{S}}\left|d_{S}(\pi(\bar{y}),\pi(\bar{y}^{\prime}))-\sup_{\bar{x}}|\varphi(\bar{x},\bar{y})-\varphi(\bar{x},\bar{y}^{\prime})|\right|.$

The first axiom says that the range of $\pi$ is dense in $S$ and the second
explicitly defines the metric on the range of $\pi$. If $\mathcal{M}$ is a
model of $T$, then it can be expanded to a model of $T^{\prime}$ by letting
$S$ be the completion of $\bar{S}^{\mathcal{M}}$ quotiented by $d^{\prime}$
where

$d^{\prime}(\bar{a},\bar{b})=(\sup_{\bar{x}}|\varphi(\bar{x},\bar{a})-\varphi(\bar{x},\bar{b})|)^{\mathcal{M}}.$

We define $\pi(\bar{a})=\bar{a}/d^{\prime}$ for
$\bar{a}\in\bar{S}^{\mathcal{M}}$. Once again there is essentially only one
way to expand a model of $T$ to a model of $T^{\prime}$ and so we leave it as
an exercise to show that:

######  Theorem 10.3.

The forgetful functor from $Mod(T^{\prime})$ to $Mod(T)$ is an equivalence of
categories.

###  10.4. Conceptual completeness

If one iterates the constructions from the previous three subsections to a
given ${\mathcal{L}}$-theory $T$, one can construct a theory $T^{eq}$
(referred to as $T^{meq}$ in the article by Berenstein and Henson in this
volume) in an expanded language ${\mathcal{L}}^{eq}$ which is closed under
taking finite and countable products of sorts, and adding definable sets and
canonical parameters. (In fact, one can be economical and first close under
products, then add definable sets and finally add canonical parameters.)

In order to state the main theorem regarding $T^{eq}$, we need to say what
interpretation means in continuous logic.

######  Definition 10.4.

Let ${\mathcal{L}}$ and ${\mathcal{L}}^{\prime}$ be continuous languages and
suppose that $T$ and $T^{\prime}$ are theories in ${\mathcal{L}}$ and
${\mathcal{L}}^{\prime}$ respectively. An _interpretation_ $I$ of $T$ in
$T^{\prime}$ assigns:

  1. (1)

to every sort $S$ of ${\mathcal{L}}$ , a sort $I(S)$ of
${\mathcal{L}}^{\prime}$,

  2. (2)

to every function symbol $f\colon\prod S_{i}\rightarrow S$ an
${\mathcal{L}}^{\prime}$-formula $I(f)$ with free variables in $\prod
S_{i}\times S$, such that

$T^{\prime}\models\text{``}I(f)(\bar{x},y)=0\text{ is the graph of a total
function of }\bar{x}\text{''}$

  3. (3)

to every relation symbol $R$ with domain $S_{1}\times\cdots\times S_{n}$ in
${\mathcal{L}}$, an ${\mathcal{L}}^{\prime}$-formula $I(R)$ with domain
$I(S_{1})\times\cdots\times I(S_{n})$,

such that, for any ${\mathcal{L}}$-sentence $\varphi$, $T\models\varphi$ iff
$T^{\prime}\models I(\varphi)$ (where by $I(\varphi)$ one means the result of
applying the interpretation of the language inductively to $\varphi$).

While this definition is not the most general definition of interpretation, it
is sufficient to prove the following theorem, whose proof can be found in [1].

######  Theorem 10.5 (Conceptual completeness).

Suppose that $T$ is an ${\mathcal{L}}$-theory. Then $T^{eq}$ satisfies the
following properties:

  1. (1)

The forgetful functor from $Mod(T^{eq})$ to $Mod(T)$ is an equivalence of
categories.

  2. (2)

If $T^{\prime}$ is an expansion of $T$ in some language
${\mathcal{L}}^{\prime}$ such that the forgetful functor from
$Mod(T^{\prime})$ to $Mod(T)$ is an equivalence of categories, then
$T^{\prime}$ can be interpreted in $T^{eq}$.

##  11\. Omitting types

We say that a structure $\mathcal{M}$ _omits_ a (partial) type $p$ if $p$ is
not realized in $\mathcal{M}$. Criteria in the classical case for the omission
of types is well understood. The situation in the continuous case is more
problematic; see the discussion at the end of this section. We will see here
that the role of definability is pivotal in the omitting types theorem for
continuous logic.

Throughout this section, we work in a theory $T$ in a separable language
${\mathcal{L}}$. As mentioned before, by Proposition 8.8, if $p$ is omitted in
some model, then $p$ is not principal. The converse is the omitting types
theorem for continuous logic:

######  Theorem 11.1 (Omitting types theorem).

For each $n\in\mathbb{N}$, suppose that $p_{n}$ is a complete type which is
not principal. Then there is a separable model of $T$ which omits each
$p_{n}$.

###### Proof.

This proof provides an opportunity to show what a Henkin construction looks
like in the continuous setting. We will sketch the proof where we omit a
single complete type $p$ in one variable and leave it to the reader to handle
the more general case. To this end, we consider the language
${\mathcal{L}}^{\prime}$ which is ${\mathcal{L}}$ together with countably many
new constants symbols $c_{n}$ for $n\in{\mathbf{N}}$. The intention is that we
will construct an ${\mathcal{L}}^{\prime}$-theory $T^{\prime}$ extending $T$
and a model of $T^{\prime}$ in which the interpretations of the $c_{n}$’s are
dense. Recall the notation for open sets in the logic topology: a basic open
set relative to the theory $T$ has the form

$\\{\bar{T}\colon T\subseteq\bar{T}\text{ is a complete
${\mathcal{L}}^{\prime}$-theory and }\bar{T}\models a<\psi<b\\}$

for some ${\mathcal{L}}^{\prime}$-sentence $\psi$ and interval $(a,b)$. We
denote this open set by $O^{T}(\psi,(a,b))$.

We construct $T^{\prime}$ inductively as the intersection of a decreasing
sequence of non-empty basic logical open sets $O_{n}$ for $n\in{\mathbf{N}}$.
We set $O_{0}$ to be the set of all complete ${\mathcal{L}}^{\prime}$-theories
extending $T$. Now, fix a countable dense set of
${\mathcal{L}}^{\prime}$-sentences $\psi_{n}$ for $n\in{\mathbf{N}}$ and a
countable dense set of ${\mathcal{L}}^{\prime}$-formulas in one free variable
$\varphi_{n}(x)$ for $n\in{\mathbf{N}}$. We will want the set $O_{n}$ to
satisfy the following conditions:

First, for every $m$ and every rational number $r$, there is $n\in\mathbb{N}$
and rational interval $(a,b)$ with $b-a<r$ such that

$O_{n}\subseteq O^{T}(\psi_{m},(a,b)).$

Second, for every $m\in\mathbb{N}$ and rational number $r$, if for some $n$ we
have

$O_{n}\subseteq O^{T}(\inf_{x}\varphi_{m}(x),(-\infty,r)),$

then for some $k>n$ and $l\in\mathbb{N}$ we have

$O_{k}\subseteq O^{T}(\varphi(c_{l}),(-\infty,r)).$

It is not hard to see that if we can succeed in obtaining these properties,
then $\bigcap_{n\in{\mathbf{N}}}O_{n}$ is a complete
${\mathcal{L}}^{\prime}$-theory such that in any model, the closure of the
interpretations of the constants $c_{n}$ is the underlying set of a model of
$T$.

In order for the aforementioned model of $T$ to omit $p$, we need to demand
more from the open sets $O_{n}$. In the classical case, it is enough to
require that none of the new constants satisfy $p$. Here, since the new
constants are merely dense in the model of $T$ we have constructed, we will
instead need to keep the constants some fixed distance away from the set of
realizations of $p$. To this end, recall that since $p$ is not principal, by
Proposition 8.9, there is some $\epsilon>0$ such that the open $\epsilon$-ball
around $p$ does not contain any logical open set. Suppose that we are at some
stage of the construction and we have constructed the logical open set
$O_{n}$, which we may assume, after some manipulation, has the form
$O^{T}(\theta(c_{k},\bar{d}),(-\infty,0))$, where $c_{k}$ is the new constant
we are currently interested in keeping away from $p$ and $\bar{d}$ is the rest
of the new constants in $\theta$. Now consider the logical open set
$O^{T}(\inf_{\bar{y}}\theta(x,\bar{y}),(-\infty,0))$. By the assumption that
$p$ is not principal, there is some $q\in S_{x}(T)$ such that $q$ belongs to
$O^{T}(\inf_{\bar{y}}\theta(x,\bar{y}),(-\infty,0))$ and
$d(p,q)>\frac{\epsilon}{2}$. By compactness, we can find some
${\mathcal{L}}$-formula $\psi(x)$ such that:

  * •

for all $\mathcal{M}\models T$ and $a\in\mathcal{M}$ with
$\psi^{\mathcal{M}}(a)<0$, we have
$(\inf_{\bar{y}}\theta(a,\bar{y}))^{\mathcal{M}}<0$, and

  * •

for any $\mathcal{M}\models T$ and $a,b\in\mathcal{M}$ such that $a$ realizes
$p$ and $\psi^{\mathcal{M}}(b)<0$, we have $d(a,b)>\frac{\epsilon}{2}$.

We now let $O_{n+1}=O_{n}\cap O^{T}(\psi(c_{k}),(-\infty,0))$ and note that
this will guarantee that no interpretation of $c_{k}$ will be within
$\frac{\epsilon}{2}$ of a realization of $p$ in any model of $T$. Since we can
achieve this for all of the new constants, no Cauchy sequence of the new
constants will approach a realization of $p$ and so the model we construct
will omit $p$. ∎

The situation regarding omitting partial types in continuous logic is quite
problematic as evidenced by the results of Farah and Magidor from [20]. There
they show the following:

  1. (1)

There is a complete theory $T$ in a separable language and countably many
partial types such that every finite subset is omitted in some model of $T$
but no model of $T$ simultaneously omits all of them.

  2. (2)

There is a complete theory $T$ in a separable language and partial types $s$
and $t$, each one omitted in some model of $T$, but no model omits both.

A variant of the omitting types theorem used to omit certain kinds of partial
types has proven useful in the operator algebra context; for example, see
[18], [24], and Goldbring’s article in this volume.

##  12\. Separable categoricity

For an infinite cardinal $\lambda$, we say that a theory $T$ is _$\lambda$
-categorical_ if $\lambda\geq\chi(T,{\mathcal{L}})$ and whenever $\mathcal{M}$
and $\mathcal{N}$ are models of $T$ such that
$\chi(\mathcal{M})=\chi(\mathcal{N})=\lambda$, then
$\mathcal{M}\cong\mathcal{N}$. We will focus on the case where $\lambda$ is
$\aleph_{0}$ and we will typically say _separably categorical_ instead of
$\aleph_{0}$-categorical. As in the classical case, we have the following
theorem.

######  Theorem 12.1.

Suppose $T$ is an ${\mathcal{L}}$-theory which has only models of density
character at least $\chi(T,{\mathcal{L}})$ and is $\lambda$-categorical. Then
$T$ is complete.

###### Proof.

Suppose that $\mathcal{M}$ and $\mathcal{N}$ are models of $T$. If
$\mathcal{M}$ has density character at least $\lambda$, then by the Downward
Löwenheim-Skolem theorem, we can find $\mathcal{M}_{0}\prec\mathcal{M}$ with
$\chi(\mathcal{M}_{0})=\lambda$. If $\chi(\mathcal{M})<\lambda$, then by the
upward Löwenheim-Skolem theorem applied to $Elem(\mathcal{M})$, we can find
$\mathcal{M}_{0}$ with $\chi(\mathcal{M}_{0})=\lambda$ and
$\mathcal{M}\prec\mathcal{M}_{0}$. Applying the same reasoning to
$\mathcal{N}$, we can find $\mathcal{M}_{0}$ and $\mathcal{N}_{0}$ so that

$\mathcal{M}\equiv\mathcal{M}_{0}\cong\mathcal{N}_{0}\equiv\mathcal{N}.$

From this we conclude that $T$ is complete. ∎

We now state the Ryll-Nardewski Theorem for continuous logic; the proof, done
originally in the positive bounded case, is due to Henson.

######  Theorem 12.2.

Suppose that ${\mathcal{L}}$ is separable and $T$ is an ${\mathcal{L}}$-theory
with no compact models. Then $T$ is separably categorical if and only $T$ is
complete and every complete type is principal.

###### Proof.

If $T$ is separably categorical then it is complete. If there is a complete
type $p$ which is not principal, then by the omitting types theorem, there is
some separable model of $T$ which omits $p$. However, there is also a
separable model of $T$ which realizes $p$; since these two models of $T$
cannot be isomorphic, $T$ is not separably categorical; contradiction.

Now suppose that $T$ is complete and every complete type is principal.
Consider two separable models $\mathcal{M}$ and $\mathcal{N}$ of $T$. We need
to show that $\mathcal{M}\cong\mathcal{N}$.

We construct two sequences
$a_{0}^{0},a^{1}_{0}a^{1}_{1},a^{2}_{0}a^{2}_{1}a^{2}_{2},\ldots$ in
$\mathcal{M}$ and
$b_{0}^{0},b^{1}_{0}b^{1}_{1},b^{2}_{0}b^{2}_{1}b^{2}_{2},\ldots$ in
$\mathcal{N}$ such that all initials segments of the same length have the same
type, that is, for any $k$, there is a fixed type for $a^{n}_{0}\ldots
a^{n}_{k}$ and $b^{n}_{0}\ldots b^{n}_{k}$ independent of $n\geq k$. We will
also want to arrange that for every $k$, $\langle a^{n}_{k}:n\geq k\rangle$
and $\langle b^{n}_{k}:n\geq k\rangle$ form Cauchy sequences converging to
$\bf a_{k}$ and $\bf b_{k}$ respectively so that $\\{\bf
a_{k}:k\in{\mathbf{N}}\\}$ and $\\{\bf b_{k}:k\in{\mathbf{N}}\\}$ are dense in
$\mathcal{M}$ and $\mathcal{N}$ respectively. If we can achieve this, then the
map sending $\bf a_{k}$ to $\bf b_{k}$ extends to an isomorphism from
$\mathcal{M}$ to $\mathcal{N}$.

To start, we enumerate countable dense subsets in $\mathcal{M}$ and
$\mathcal{N}$; call them respectively $\langle c_{k}:k\in N\rangle$ and
$\langle d_{k}:k\in N\rangle$. At stage 0, let $a^{0}_{0}=c_{0}$. By
assumption, $tp^{\mathcal{M}}(c_{0})$ is principal and hence realized in
$\mathcal{N}$ by some $b^{0}_{0}$. In general, at each step we alternate by
either choosing a $c_{k}$ or $d_{k}$ and we revisit each $c_{k}$ and $d_{k}$
infinitely often in the construction. Assume we have chosen $a^{n}_{0}\ldots
a^{n}_{n}$ already and we consider whatever $c_{k}$ is given to us at this
stage. Let $p(x_{0},\ldots,x_{n})$ be the type of $a^{n}_{0}\ldots a^{n}_{n}$
in $\mathcal{M}$ and $q(x_{0},\ldots,x_{n+1})$ be the type of $a^{n}_{0}\ldots
a^{n}_{n}c_{k}$ in $\mathcal{M}$. Since $q$ is principal, we have a formula
$d_{q}(x_{0},\ldots,x_{n+1})$ expressing the distance to realizations of $q$.
Since $d_{q}^{\mathcal{M}}(a^{n}_{0}\ldots a^{n}_{n},c_{k})=0$, we have that
$\inf_{y}d_{q}^{M}(a^{n}_{0}\ldots a^{n}_{n},y)=0$. Since $b^{n}_{0}\ldots
b^{n}_{n}$ satisfies $p$ by assumption, we have
$\inf_{y}d_{q}^{\mathcal{N}}(b^{n}_{0}\ldots b^{n}_{n},y)=0$. This means we
can find $b^{n+1}_{0}\ldots b^{n+1}_{n+1}$ realizing $q$ and such that
$d(b^{n}_{i},b^{n+1}_{i})\leq 1/2^{n}$ for $i=0,\ldots,n$. This guarantees we
have the required Cauchy sequences and we have the required density as well.
To see the latter fact, fix $\epsilon>0$. Choose $N$ large enough so that
$\sum_{n\geq N}1/2^{n}<\epsilon$. If we visit $c_{k}$ at stage $t>N$, then
$\bf a_{t}$ is within $\epsilon$ of $c_{k}$ and so the $\bf a_{k}$’s are dense
in $M$. Similarly, the $\bf b_{k}$’s are dense in $N$, finishing the proof. ∎

Note that the condition “every complete type is principal” in Theorem 12.1 is
equivalent to the statement that the logic and metric topologies on
$S_{\bar{x}}(T)$ agree for all finite tuples $\bar{x}$ of variables.

######  Exercise 12.3.

Prove that a complete theory $T$ in a separable language with no compact
models is separably categorical if and only if $S_{\bar{x}}(T)$ is compact in
the metric topology for all finite tuples $\bar{x}$ of variables.

There are a number of nontrivial examples of separably categorical continuous
theories:

######  Example 12.4.

The following theories are separably categorical:

  1. (1)

The theory of infinite-dimensional Hilbert spaces.

  2. (2)

The theory of the Urysohn sphere.

  3. (3)

The theory of atomless probability algebras.

The fact that the theory of infinite-dimensional Hilbert spaces is separably
categorical follows from the fact that any two separable infinite-dimensional
Hilbert spaces are isomorphic to $\ell^{2}$ while the separable categoricity
of the theory of the Urysohn sphere follows from the argument presented in
Example 5.14. The separable categoricity of the theory of atomless probability
algebras is discussed in Berenstein and Henson’s article in this volume [4].

In the context of operator algebras, there are very few examples of separably
categorical theories. For instance, for any separable II1 factor $N$, there
are continuum many nonisomorphic separable II1 factors elementarily equivalent
to $N$ (see Theorem 4.1 in the author’s article with Goldbring in this
volume). One notable example of a separably categorical
$\mathrm{C}^{*}$-algebra is the algebra $C(X)$ for $X$ a totally disconnected
compact Hausdorff space; in this case, $C(2^{\mathbb{N}})$ is the unique
separable model. The proof of this fact follows from Stone duality together
with the fact that there is a unique countable atomless Boolean algebra.

We end this section with some related considerations. We say that a model
$\mathcal{M}$ of $T$ is _atomic_ if $tp^{\mathcal{M}}(\bar{a})$ is principal
for every finite tuple $\bar{a}\in\mathcal{M}$. Theorem 12.1 (and its proof)
yields the following:

######  Theorem 12.5.

  1. (1)

Any two separable atomic models of $T$ are isomorphic.

  2. (2)

If $T$ is a complete theory in a separable language, then $T$ is separably
categorical if and only if all separable models of $T$ are atomic.

In the remainder of this section, for simplicity, we assume that $T$ is a
complete theory in a separable language.

We say that a model of $T$ is _prime_ if it embeds elementarily into all other
models of $T$.

######  Exercise 12.6.

Prove that a model $\mathcal{M}$ of $T$ is prime if and only if $\mathcal{M}$
is separable and atomic. Moreover, $T$ can have at most one prime model up to
isomorphism. (Hint: To show that a separable atomic model of $T$ is prime, use
a “forth only” version of the back and forth argument appearing in the proof
of Theorem 12.1.)

######  Exercise 12.7.

Prove that $T$ has a prime model if and only if the principal types are dense
in $S_{\bar{x}}(T)$ for all finite tuples $\bar{x}$ of variables.

We say that $T$ is _small_ if $S_{\bar{x}}(T)$ is separable in the metric
topology for all finite tuples $\bar{x}$ of variables. The following is
Proposition 1.17 in [13]:

######  Proposition 12.8.

If $T$ is small, then $T$ has a prime model.

## References

  * [1] J. M. Albert and B. Hart. Metric logical categories and conceptual completeness for first order continuous logic. preprint, 2016.
  * [2] H. Ando, this volume
  * [3] J. Arulseelan, I. Goldbring and B. Hart, The undecidability of having the QWEP, arXiv:2205.07102.
  * [4] A. Berenstein and C.W. Henson, this volume
  * [5] I. Ben Yaacov. On The Expressive Power Of Quantifiers In Continuous Logic, arXiv:2207.01863
  * [6] I. Ben Yaacov. Positive model theory and compact abstract theories. J. Math. Logic, 3, 85–118, 2003.
  * [7] I. Ben Yaacov. Simplicity in compact abstract theories. J. Math. Logic, 3, 163– 191, 2003.
  * [8] I. Ben Yaacov. Uncountable dense categoricity in cats. J. Symb. Logic, 70, 829– 860, 2005.
  * [9] I. Ben Yaacov. Continuous first order logic for unbounded metric structures. J. Math. Logic, 8(2), 225–249, 2008.
  * [10] I. Ben Yaacov, A. Berenstein, C. W. Henson, and A. Usvyatsov, _Model theory for metric structures_ , Model theory with applications to algebra and analysis. Vol. 2, London Math. Soc. Lecture Note Ser., vol. 350, Cambridge Univ. Press, Cambridge, 2008, 315–427.
  * [11] I. Ben Yaacov and A.P. Pederson, A proof of completeness for continuous first order logic, J. of Symb. Log. 75 (2010), no. 1, 168–190.
  * [12] I. Ben Yaacov and A. Usvyatsov. Continuous first order logic and local stability. Trans. Amer. Math. Soc., 362(10):5213?5259, 2010.
  * [13] I. Ben Yaacov and A. Usvyatsov, On d-finiteness in continuous structures, Fundamenta Mathematicae 194 (2007), 67-88.
  * [14] R. Boutonnet, I. Chifan, and A. Ioana, _II 1 factors with non-isomorphic ultrapowers_, Duke Math. J. 166 (2017), 2023-2051.
  * [15] C. C. Chang and H. J. Keisler. Continuous Model Theory. Princeton University Press, 1966.
  * [16] I. Chifan, A. Ioana, and S. Kunnawalkam Elayavalli, _Two non-elementarily equivalent II 1 factors without property Gamma_, preprint.
  * [17] Y. Dabrowski, Continuous model theories for von Neumann algebras, J. of Func. Analysis, 277, 2019.
  * [18] I. Farah, B. Hart, M. Lupini, L. Robert, A. Tikuisis, A. Vignati and W. Winter, _Model Theory of $\mathrm{C}^{*}$-algebras_, Memoirs of the AMS (2021).
  * [19] I. Farah, B. Hart, and D. Sherman, _Model theory of operator algebras II: Model theory_ , Israel Journal of Mathematics 201 (2014), 477-505.
  * [20] I. Farah and M. Magidor, Omitting types in logic of metric structures, J. of Math. Log., 2018.
  * [21] I. Farah, A. Toms and A. Törnquist, Turbulence, orbit equivalence, and the classification of nuclear $\mathrm{C}^{*}$-algebras, Journal für die reine und angewandte Mathematik (Crelle’s Journal), 688 (2014) 101–146.
  * [22] I. Goldbring, Ultrafilters throughout mathematics, American Mathematical Society’s Graduate Studies in Mathematics series, 220 (2022).
  * [23] I. Goldbring, this volume
  * [24] I. Goldbring, Enforceable operator algebras, Journal of the Institute of Mathematics of Jussieu, Volume 20 (2021), 31-63.
  * [25] I. Goldbring and B. Hart, On the theories of McDuff’s II1 factors, IMRN, 27 (2017), 5609–5628.
  * [26] I. Goldbring, B. Hart and T. Sinclair, Correspondences, Ultraproducts and Model Theory, arXiv:1809.00049.
  * [27] I. Goldbring and J. Keisler, Continuous Sentences Preserved under Reduced Products. J. Symb. Log. 87(2), 649–681 (2022).
  * [28] I. Goldbring and T. Sinclair, Games and elementary equivalence of II1 factors, Pacific Journal of Mathematics, 278 (2015), 103–118.
  * [29] B. Hart, EF games, unpublished notes, https://ms.mcmaster.ca/~bradd/EF-games.pdf
  * [30] C. W. Henson. Nonstandard hulls of Banach spaces,. Israel J. Math., 25, 108–144, 1976.
  * [31] B. Hart, B. Kim and A. Pillay, Coordinatisation and canonical bases in simple theories, J. Symb. Logic, 65, 293–309.
  * [32] C. W. Henson and J. Iovino, Ultraproducts in analysis, Analysis and Logic, London Mathematical Society Lecture Notes Series, 262, 2002, 1–113.
  * [33] E. Hewitt. Rings of real-valued continuous functions. Trans. Amer. Math. Soc., 64, 45–99, 1948.
  * [34] J. Keisler, The Ultraproduct Construction. "Ultrafilters Across Mathematics", ed. by V. Bergelson et. al., Contemporary Mathematics 530 (2010), pp. 163–179.
  * [35] H.J. Keisler, _Ultraproducts and saturated models_ , Koninkl. Ned. Akad. Wetetnsch. Proc. Ser. A 67 (= Indag. Math. 26) (1964), 178–186.
  * [36] K. Kunen, _Ultrafilters and independent sets_ , Trans. Amer. Math. Soc. 172, (1972), 199–206.
  * [37] J. Łoś, Quelques remarques, théorèmes et problèmes sur les classes d’efinissables d’algébres. In Mathematical Interpretations of Formal Systems, 98–113. North Holland, 1955.
  * [38] J. Łukasiewicz and A. Tarski, Untersuchungen über den Aussagenkalkül, Comptes rendus de la Société des Sciences et des Lettres de Varsovie, 23 (1930), 1–21.
  * [39] D. McDuff
  * [40] S. Shelah, “Every two elementarily equivalent models have isomorphic ultrapowers”, Israel J. Math., 10 (1971), pp. 224–233.
  * [41] D. Sherman, Notes on automorphisms of ultrapowers of II1 factors. Studia Math., 195, 201–217, 2009.
  * [42] P. S. Urysohn, Sur un espace métrique universel, C. R. Acad. Sci. Paris 180 (1925), 803–806.

## Index

  * approximately finitely satisfiable §6.2
  * atomic diagram item 1
  * atomic formula §4.1
  * atomic model §12
  * axiomatizable §5.1
  * basic formula §4.1
  * canonical parameters §10.3
  * categorical §12
  * complete theory §5.1
  * complete type §7.1
  * definable set §8.1
  * density character §5.2
  * diagonal embedding §2
  * dissection Example 3.9
  * elementary §5.1
  * elementary diagram item 2
  * elementary embedding §5.1
  * elementary equivalent §5.1
  * elementary substructure §5.1
  * embedding 2nd item
  * expansion §3.1
  * filter item 1
  * finitely satisfiable §6.2
  * Fréchet filter item 1
  * homomorphism 1st item
  * interpretation Definition 10.4
  * isomorphism §3.1
  * ${\mathcal{L}}$-formula §5.2
  * logic topology item 1
  * Łoś’ Theorem §6.1
  * metric structure §3.1
  * metric topology §7.1
  * omitting type §11
  * partial type §7.1, §7.3
  * prenex formula Comment 4.2
  * prime model §12
  * principal type §8.3
  * principal ultrafilter item 2
  * quantifier elimination §9
  * quantifier-free formula §4.1
  * real algebra system Definition 5.3
  * reduct §3.1
  * satisfaction §5.1
  * saturated Definition 7.7
  * separably categorical §12
  * small theory §12
  * substructure §3.1
  * $T$-definable §8.1
  * $T$-equivalent §5.2
  * $T$-formula §5.2
  * $T$-functor §8.1
  * theory §5.1
  * type space §7.1
  * ultrafilter item 2
  * ultralimit Definition 2.6
  * ultrapower §2, §6.1
  * ultraproduct §2, §6.1
  * ultraroots §6.4
  * uniform continuity modulus §3.1
  * uniformly bounded §2
  * zero-set §8.1
  * $\kappa$-saturated Definition 7.7


11institutetext: Huazhong University of Science and Technology, School of
Mechanical Science and Engineering, State Key Laboratory of Digital
Manufacturing Equipment and Technology, Wuhan, China 11email:
<EMAIL_ADDRESS>
# GelFlow: Self-supervised Learning of Optical Flow for Vision-Based Tactile
Sensor Displacement Measurement

Zhiyuan Zhang 11 0000-0001-7156-3511 Hua Yang 11 0000-0002-5430-5630 Zhouping
Yin 11

###### Abstract

High-resolution multi-modality information acquired by vision-based tactile
sensors can support more dexterous manipulations for robot fingers. Optical
flow is low-level information directly obtained by vision-based tactile
sensors, which can be transformed into other modalities like force, geometry
and depth. Current vision-tactile sensors employ optical flow methods from
OpenCV to estimate the deformation of markers in gels. However, these methods
need to be more precise for accurately measuring the displacement of markers
during large elastic deformation of the gel, as this can significantly impact
the accuracy of downstream tasks. This study proposes a self-supervised
optical flow method based on deep learning to achieve high accuracy in
displacement measurement for vision-based tactile sensors. The proposed method
employs a coarse-to-fine strategy to handle large deformations by constructing
a multi-scale feature pyramid from the input image. To better deal with the
elastic deformation caused by the gel, the Helmholtz velocity decomposition
constraint combined with the elastic deformation constraint are adopted to
address the distortion rate and area change rate, respectively. A local flow
fusion module is designed to smooth the optical flow, taking into account the
prior knowledge of the blurred effect of gel deformation. We trained the
proposed self-supervised network using an open-source dataset and compared it
with traditional and deep learning-based optical flow methods. The results
show that the proposed method achieved the highest displacement measurement
accuracy, thereby demonstrating its potential for enabling more precise
measurement of downstream tasks using vision-based tactile sensors.

###### Keywords:

Vision-based tactile sensor Optical flow Elastic deformation estimation Deep
learning.

##  1 Introduction

Vision and tactile are crucial sources of information for perceiving and
interacting with the world [1]. With computer vision and robotics advancement,
vision-based tactile sensors fusing both modalities are becoming increasingly
popular for enabling intelligent robots to perceive and manipulate delicate
objects precisely. A typical visual-tactile sensor hardware comprises three
components: a contact module, a camera module, and an illumination module [2].
The contact module requires resilient and optically favorable materials, as
its performance directly affects the accuracy of subsequent optical
measurements. It is often embedded with a marker layer to visualize the
contact material’s deformation. The camera and illumination modules can be
classified into two main categories based on the measurement principle: a
monocular camera system with multi-color illumination systems and multi-camera
systems with a monochromatic illumination system. The integration of these
three modules allows vision-based tactile sensors to capture and measure
various types of information, including force [3], geometry reconstruction
[4], sliding detection [5], and object recognition [1].

The displacement of markers in the contact module provides valuable
information for measuring additional physical properties, such as the shape of
the contacting object, the forces applied, and the roughness of its surface.
This information can be analyzed by the robot for downstream tasks. Accurate
and dense displacement measurements improve the resolution of other modal
information, providing better input for subsequent tasks, thereby enhancing
the accuracy of robot perception and manipulation [6]. However, the contact
module composed of gel material is characterized by large elastic deformation,
which can result in errors in estimating the displacement of existing vision-
based tactile sensors using the optical flow algorithm in OpenCV [7]. These
inaccuracies in displacement measurements can lead to inaccuracies in the
final estimated physical information [8]. Therefore, our motivation is to
develop an accurate pixel-level optical flow estimation method to better deal
with the deformation properties of the gel materials.

In this study, we introduce a self-supervised learning optical flow approach,
named GelFlow, for a more precise measurement of displacement in gel-like
materials. Our proposed method offers two novel loss terms, namely the
Helmholtz velocity decomposition constraint and elastic deformation
constraint, and a practical local flow fusion module to track the movement of
gel materials captured by a monocular camera. These contributions improve
displacement measurement accuracy and enhance vision-based tactile sensors’
capability to estimate physical information. The rest of this paper is
organized as follows. Section 2 provides an introduction to previous works
related to vision-based tactile sensors and dense displacement processing. In
Section 3, the structure and individual modules of the proposed GelFlow method
are elaborated on and discussed in detail. The comparison results with other
optical flow methods and the ablation study are presented in Section 4.
Finally, the conclusions of this work are discussed in Section 5.

##  2 Related Work

The ability to perceive and model the contact surface’s three-dimensional (3D)
geometry is a fundamental feature that distinguishes vision-based tactile
sensors from conventional tactile sensors. Based on different principles of 3D
surface reconstruction, vision-based tactile sensors can be divided into two
categories: sensors based on photometric stereo reconstruction and sensors
based on multi-view geometry reconstruction. Among the first type of sensors,
the GelSight [9] sensor uses an RGB trichromatic light source to illuminate
the contact layer and a monocular camera to capture the image. This algorithm
enables it to obtain the normal gradient of each pixel, resulting in high
accuracy for contact geometry. However, this method requires rigorous
structural design and light source calibration. The GelSlim [10] sensor
improves GelSight’s optical path system by using a mirror-reflective structure
so that the camera no longer has to face the contact body, making the entire
sensor compact. The DIGIT [11] sensor is low-cost, compact, and provides high-
resolution tactile perception, making it more practical for robotic finger
manipulation. Unlike the previous flat structure contact layer, DenseTact [12]
sensor uses a spherical contact layer, making it more suitable for sensing
complex object surfaces. Among the second type of sensors, OmniTact [13]
sensor uses multiple micro-cameras to capture multi-directional high-
resolution deformation information to obtain accurate and reliable measurement
results. GelStereo [14] sensor simplifies the number of cameras required by
using binocular cameras to calculate the depth information of the contact
surface through the disparity map in left and right views. Tac3D [15] further
simplifies the number of cameras by using a monocular camera with a mirror
system to achieve a pseudo-binocular imaging effect, achieving the same 3D
reconstruction purpose.

In addition to 3D reconstruction, other valuable information, such as force
estimation and sliding detection, is also crucial for robot perception. This
information is obtained by measuring the deformation of the contact layer and
then converting it according to specific criteria. Optical flow is an
essential technique for deformation estimation, and accurate optical flow
estimation with high resolution can provide more detailed information for more
precise and dexterous operations. There are two primary approaches to
enhancing the reliability of optical flow estimation: utilizing more precise
optical flow algorithms and designing better marker layers. During the early
stages of vision-based tactile sensor development, the Lucas-Kanada optical
flow method [16] was utilized to track the movement of markers, which could
only produce a sparse vector field. Interpolation methods are used for
upsampling the sparse field, and significant errors occur during this process
[6]. Subsequently, more robust and accurate pixel-level optical flow methods
such as Farneback [17] and Dense Inverse Search (DIS) [18] methods were
adopted to avoid the interpolation error and improve estimation precision [6].
The conventional marker layer consists of an array of sparse black dot
markers, which does not provide rich deformation information. Moreover, a
single color pattern lacked robustness in optical flow estimation due to the
brightness conservation hypothesis. When using a single color pattern, the
similarity of pixels made the optical flow estimation confusing. In order to
overcome the limitations mentioned above, researchers have proposed various
types of marker layers. [19] added high-density internal markers, which
enabled dense optical flow tracking for estimating shear-induced membrane
displacement. [6] replaced the sparse black dot markers with a novel random
color pattern, which achieved more accurate and higher resolution two-
dimensional (2D) deformation estimation.

In this work, our purpose is to propose a pixel-level optical flow method with
high accuracy and robustness. Our method takes advantage of the powerful tools
of deep learning, and we hope it can be helpful in deformation estimation and
other downstream tasks in vision-based tactile sensors.

##  3 Method

###  3.1 Network Architecture

Fig. 1 shows the framework of the GelFlow. First, the encoder module of PWC-
Net [20] is adopted to extracted the multi-scale features for the input image
pairs $I_{1}$ and $I_{2}$ and denoted as $F^{s}_{1}(I_{1})$ and
$F^{s}_{2}(I_{2})$, where the superscript $s$ represents the $s$th scale of
the pyramid. In Optical Flow Estimation Module, apart from basic operators in
PWC-Net like Cost Volume Operator, Coarse Flow Estimator and Refined Flow
Estimator, we add Local Flow Fusion Module (LFFM) for better gel-like
materials deformation estimation. The output flow at the current pyramid scale
$V^{s}_{1{\rightarrow}2}$ is upsampled by a factor of 2 using bilinear
interpolation and then used as the initial flow for the subsequent scale
$V^{s-1}_{1{\rightarrow}2}$. According to the coarse-to-fine strategy, the
feature of the second image $F^{s}_{2}(I_{2})$ are warped [21] using the
output flow (denoted by $\tilde{F}^{s}_{2}(I_{2})$) to reduce the feature
distance between the extracted feature of the first image $F^{s}_{1}(I_{1})$,
enabling better handling of large displacement motion. Note that the input
flow at the top scale is set to $\mathbf{0}$, and the final output flow of
GelFlow $V^{0}_{1{\rightarrow}2}$ is obtained by bilinear upsampling of the
output flow by a factor of 4 at the bottom scale. The multi-scale strategy
allows for the extraction of richer feature information by convolving the
input image at different scales, improving the robustness of optical flow
estimation.

Figure 1: GelFlow, the proposed architecture for deformation measurement for
gel-like materials using coarse-to-fine strategy, local flow fusion module and
Self-supervised losses.
Figure 2: Structure of Local Flow Fusion Module. The feature vector at each
position is dot producted with the feature vectors in the surrounding area.
Then, all of the results within the local range are passed through a softmax
function and used as weights for weighted fusion with the estimated optical
flow. This process results in a locally smooth optical flow field.

###  3.2 Local Flow Fusion Module

As the cross-linking network structure of the gel material results in its
deformation showing overall smoothness [22], thus the generated optical flow
field exhibits a blurred effect. Taking advantage of this property, we
designed a local optical flow fusion module, and Fig. 2 shows the
implementation details of this module. Context features $C^{s}$ extracted by
PWC-Net from each scale are utilized to construct the weight matrix. At each
position, the feature vector is dot producted with the feature vectors of its
neighboring positions (the number of neighboring positions is determined by
the fusion range, usually $3\times 3$ or $5\times 5$) to compute the
similarities. The results are then normalized using a softmax function. The
weight matrix enables the flow to consider not only the position itself but
also its surrounding area, thereby achieving a blurred effect.

###  3.3 Helmholtz Velocity Decomposition Constraint

The deformation of gel materials is complex due to their elastic properties.
Based on the Helmholtz velocity decomposition theorem, compressible motion can
be decomposed into four components: translational motion, linear deformation
motion, shear deformation motion, and rotational motion, given by

$\displaystyle{\mathbf{u}}({\mathbf{x}}+\delta{\mathbf{x}})={\mathbf{u}}({\mathbf{x}})+{{\rm
X}}\delta{\mathbf{x}}+\Theta\delta{\mathbf{x}}+{\rm Z}\delta{\mathbf{x}},$ (1)
$\displaystyle{\rm
X}=\left[{\begin{array}[]{*{20}{c}}{{\varepsilon_{xx}}}&0\\\
0&{{\varepsilon_{yy}}}\end{array}}\right],\Theta=\left[{\begin{array}[]{*{20}{c}}0&{{\varepsilon_{xy}}}\\\
{{\varepsilon_{xy}}}&0\end{array}}\right],{\rm
Z}=\left[{\begin{array}[]{*{20}{c}}0&{-{\omega}}\\\
{{\omega}}&0\end{array}}\right]$ (8)

where ${\rm X},{\Theta}$ and ${\rm Z}$ denote the linear distortion rate
tensor, shear distortion rate tensor, and rotation tensor, respectively;
$\varepsilon_{xx}=\partial u/\partial x$ and $\varepsilon_{yy}=\partial
v/\partial y$ are the linear distortion rates in the $x$ and $y$ directions,
respectively; $\varepsilon_{xy}=(\partial u/\partial y+\partial v/\partial
x)/2$ is the shear distortion rate; $\omega=(\partial v/\partial x-\partial
u/\partial y)/2$ is the rotational angular rate. By decomposing the flow, we
can impose more refined constraints on each component, achieving high-
precision flow estimation. Eq. 1 can be further transformed as

$\frac{{{\mathbf{u}}({\mathbf{x}}+\delta{\mathbf{x}})-{\mathbf{u}}({\mathbf{x}})}}{{\delta{\mathbf{x}}}}={\rm
X}+{\Theta}+{\rm Z}.$ (9)

Thus, the values of linear distortion rate tensor ${\rm X}$, shear distortion
tensor ${\Theta}$, and rotational tensor ${\rm Z}$ are constraint to satisfy
the small motions assumption, i.e.

${\left\|{{\rm{vec}}({\rm
X})}\right\|_{1}}+{\lambda_{\Theta}}{\left\|{{\rm{vec}}(\Theta)}\right\|_{1}}+{\lambda_{\rm
Z}}{\left\|{{\rm{vec}}({\rm Z})}\right\|_{1}},$ (10)

where,

$\displaystyle{\left\|{{\rm{vec}}({\rm X})}\right\|_{1}}$
$\displaystyle=|{\varepsilon_{xx}}|+|{\varepsilon_{yy}}|,$ (11)
$\displaystyle{\left\|{{\rm{vec}}(\Theta)}\right\|_{1}}$
$\displaystyle=2|{\varepsilon_{xy}}|,$ (12)
$\displaystyle{\left\|{{\rm{vec}}({\rm Z})}\right\|_{1}}$
$\displaystyle=2|{\omega}|,$ (13)

${\lambda_{\Theta}}$ and ${\lambda_{\rm Z}}$ are coefficients of distortion
and rotation tensors, respectively, which values will affect the smooth
property of the optical flow; the function $\rm{vec}(\cdot)$ is used to
convert the input into a vector representation. With the help of the Helmholtz
velocity decomposition theorem, the compressible motion can be better
estimated [23].

Furthermore, the optical flow estimated on the edge of the image is more
precise due to the larger gradient. The deformation of the gel is similar
between the edge and other flattened areas. Thus, we adopt an anti-edge-aware
weight for the motion smooth term, and the final decomposition loss is written
as

$\mathcal{L}_{dc}^{s}=\left(1-e^{-\beta\left\|\nabla
I_{1}^{s}\right\|_{1}}\right)\left(\left\|\mathrm{vec}(\mathrm{X})\right\|_{1}+\lambda_{\Theta}\left\|\mathrm{vec}(\Theta)\right\|_{1}+\lambda_{\mathrm{Z}}\left\|\mathrm{vec}(\mathrm{Z})\right\|_{1}\right)$
(14)

In Eq. 14, when an edge is detected (indicated by a larger value in
$\left\|{\nabla I_{1}^{s}}\right\|_{1}$), the weight
$1-{e^{-\beta\left\|{\nabla I_{1}^{s}}\right\|_{1}}}$ increases. Consequently,
the remaining term in $\mathcal{L}_{dc}^{s}$ must decrease during the
minimization process, which enhances the smoothness between the detected edge
and the surrounding area.

Figure 3: Illustration of two typical changes of the gel deformation, with the
red vectors indicating the direction of markers’ movement.

###  3.4 Elastic Deformation Constraint

To further constrain the motion change during the elastic deformation of the
gel materials, we propose a novel regularization term named the deformation
loss term. As shown in Fig. 3, there are two typical deformations in the
motion of gel materials, i.e., shrinking and stretching. To enforce spatial
consistency in gel motion, we can incorporate a constraint that regulates the
change in the area between adjacent pixels before and after deformation.
Firstly, the pixel-level area change ratio is estimated between the input
image pairs. Then, the estimated motion is smooth over the entire gel
materials by constraining the gradient of the ratio. Different from [24], we
calculate the deformation ratio separately for the $x$ and $y$ directions:

$\displaystyle({x^{{}^{\prime}}}-{x_{c}^{{}^{\prime}}})$
$\displaystyle={\mathcal{R}_{x}}(x-{x_{c}}),\quad x\in{\mathcal{N}_{3\times
3}}({x_{c}}),$ (15) $\displaystyle({y^{{}^{\prime}}}-{y_{c}^{{}^{\prime}}})$
$\displaystyle={\mathcal{R}_{y}}(y-{y_{c}}),\quad y\in{\mathcal{N}_{3\times
3}}({y_{c}}),$ (16)

where $x^{\prime}$ and $y^{\prime}$ denote the positions $x$ and $y$ warped by
the optical flow $V_{1{\rightarrow}2}$; the subscript $c$ represents the
center of the local window; ${\mathcal{N}}_{3\times 3}$ represents the local
window size of $3\times 3$. The final deformation ratio is obtained by
multiplying the two ratios together:

$\displaystyle{\mathcal{R}}={\mathcal{R}_{x}}{\mathcal{R}_{y}}.$ (17)

Finally, the combined anti-edge-aware weight can be utilized to define the
deformation loss as follows:

$\displaystyle\mathcal{L}_{df}^{s}=(1-{e^{-\beta{{\left\|{\nabla
I_{1}^{s}}\right\|}_{1}}}}){{\left\|{\nabla\mathcal{R}}\right\|}_{1}}.$ (18)

###  3.5 Loss Function

The widely used photometric loss function in optical flow tasks is adopted for
robust flow estimation, which takes the form:

$\mathcal{L}_{ph}^{s}=\alpha\frac{{1-\text{SSIM}(\tilde{I}_{1}^{s},I_{1}^{s})}}{2}+(1-\alpha){\left\|{\tilde{I}_{1}^{s}-I_{1}^{s}}\right\|_{1}},$
(19)

where SSIM denotes the structural similarity index; $\alpha$ represents the
balance between SSIM and $L_{1}$ distance; $\tilde{I}_{1}^{s}$ indicates the
warped image $I_{2}^{s}$ using the optical flow $V_{1{\rightarrow}2}^{s}$ at
scale $s$. The photometric loss, combined with the proposed decomposition loss
and deformation loss, constructs the loss function at each scale. The multi-
scale loss is defined as the weighted sum of the losses at each scale, denoted
by:

$\mathcal{L}=\sum\limits_{s=0}^{l-2}{{\lambda_{s}}\mathcal{L}_{self}^{s}=}\sum\limits_{s=0}^{l-2}{{\lambda_{s}}(\mathcal{L}_{ph}^{s}+{\lambda_{dc}}\mathcal{L}_{dc}^{s}+{\lambda_{df}}\mathcal{L}_{df}^{s})},$
(20)

where $l$ is the number of total scales created by PWC-Net; $\lambda_{dc}$ and
$\lambda_{df}$ are coefficients that control the balance between each loss;
$\lambda_{s}$ are parameters that weigh the importance of each scale.

##  4 Experimental Analysis

###  4.1 Experiment Setup

The proposed self-supervised learning method does not require labeled training
data. Therefore, we extract 1327 image pairs with a resolution of $480\times
640$ pixels from videos captured by [25]. We reserve 8 image pairs with the
typical motion of gel deformation (large displacement, shrinking and
stretching) for validation and comparison with other optical flow methods. We
train the network for 200 epochs on the training dataset, with a batch size of
4 image pairs per epoch. Subsequently, we fine-tune the network for an
additional 800 epochs using the validation dataset. The number of pyramid
scales, $l$, is set to 8. The fusion region size of the LFFM is set to
$3\times 3$. In the photometric loss term, $\alpha$ is set to 0.85. In the
decomposition loss term, $\beta$ is set to 10, and both $\lambda_{\Theta}$ and
$\lambda_{\rm{Z}}$ are set to 0.01. In the deformation loss term, $\beta$ is
set to 10. In the multi-scale loss term, $\lambda_{s}$ is set to 1.0 for each
scale, while $\lambda_{dc}$ and $\lambda_{df}$ are set to 75 and 0.01,
respectively. The images are initially resized to a resolution of $512\times
640$ pixels before being fed into the network. The output optical flows are
then resized to the original resolution of the images for validation.

###  4.2 Evaluation Metrics

Since there are no ground truth labels in the dataset, we need to warp the
second images into pseudo-first images using the estimated optical flows and
compare the similarity between pseudo-first and authentic-first images. The
higher the similarity, the better the estimation. Two widely used metrics for
evaluating image similarity are PSNR (Peak Signal-to-Noise Ratio) and SSIM.
They are defined as follows:

$\displaystyle\text{PSNR}(I,\tilde{I})$
$\displaystyle=10\times{\log_{10}}\left({\frac{{{{({2^{n}}-1)}^{2}}}}{{\text{MSE}(I,\tilde{I})}}}\right),$
(21) $\displaystyle\text{SSIM}(I,\tilde{I})$
$\displaystyle=\frac{{(2{\mu_{x}}{\mu_{y}}+{c_{1}})(2{\sigma_{xy}}+{c_{2}})}}{{(\mu_{x}^{2}+\mu_{y}^{2}+{c_{1}})(\sigma_{x}^{2}+\sigma_{y}^{2}+{c_{2}})}},$
(22)

where, $n$ represents the bit depth of the pixels; $\text{MSE}(I,\tilde{I})$
is the mean square error between the input image $I$ and the warped image
$\tilde{I}$; $\mu_{x}$ and $\mu_{y}$ are the means of $I$ and $\tilde{I}$,
respectively; $\sigma_{x}$ and $\sigma_{y}$ are the variances of $I$ and
$\tilde{I}$, and $\sigma_{xy}$ represents their covariance; $c_{1}$ and
$c_{2}$ are constants used to maintain stability. Therefore, we will use these
two metrics for comparisons and evaluations in the following subsections.

Table 1: Comparison with traditional and deep learning-based optical flow methods using the validation dataset. ’#’ represents an image pair. The best and the second-best values within each category are marked as bold and underlined, respectively. The best value among all the methods is marked in red. ’ft’ denotes fine-tuning the model on the validation dataset. | Method |  | Metric |  |  #1 |  #2 |  #3 |  #4 |  #5 |  #6 |  #7 |  #8
---|---|---|---|---|---|---|---|---|---|---|---|---
Traditional | Farneback |  | PSNR |  |  38.48 |  35.02 |  32.57 |  32.03 |  31.73 |  32.31 |  33.70 |  32.20
| SSIM |  |  0.96 |  0.91 |  0.91 |  0.89 |  0.89 |  0.91 |  0.91 |  0.90
DIS (Ultra-fast) |  | PSNR |  |  39.64 |  35.12 |  32.89 |  33.23 |  32.63 |  32.92 |  34.93 |  32.39
| SSIM |  |  0.97 |  0.92 |  0.92 |  0.92 |  0.92 |  0.93 |  0.94 |  0.91
DIS (Fast) |  | PSNR |  |  39.75 |  35.26 |  32.89 |  33.28 |  32.60 |  32.97 |  35.11 |  32.49
| SSIM |  |  0.97 |  0.92 |  0.92 |  0.92 |  0.92 |  0.93 |  0.94 |  0.91
DIS (medium) |  | PSNR |  |  $\mathbf{40.25}$ |  $\underline{35.41}$ |  $\underline{33.25}$ |  $\underline{33.86}$ |  $\underline{33.27}$ |  $\underline{33.24}$ |  $\underline{35.41}$ |  $\underline{32.73}$
| SSIM |  |  $\mathbf{0.97}$ |  $\mathbf{0.92}$ |  $\mathbf{0.92}$ |  $\mathbf{0.93}$ |  $\mathbf{0.93}$ |  $\mathbf{0.93}$ |  $\mathbf{0.94}$ |  $\mathbf{0.91}$
TV-L1 |  | PSNR |  |  $\underline{39.98}$ |  $\mathbf{35.44}$ |  $\mathbf{33.31}$ |  $\mathbf{33.95}$ |  $\mathbf{33.68}$ |  $\mathbf{33.42}$ |  $\mathbf{35.64}$ |  $\mathbf{32.88}$
| SSIM |  |  $\underline{0.97}$ |  $\underline{0.92}$ |  $\underline{0.92}$ |  $\underline{0.92}$ |  $\underline{0.93}$ |  $\underline{0.93}$ |  $\underline{0.94}$ |  $\underline{0.91}$
Deep Learning | RAFT |  | PSNR |  |  37.73 |  34.83 |  31.49 |  31.11 |  31.17 |  32.44 |  34.78 |  31.57
| SSIM |  |  0.97 |  0.91 |  0.91 |  0.91 |  0.91 |  0.92 |  0.94 |  0.90
ARFlow |  | PSNR |  |  39.76 |  35.22 |  32.88 |  33.44 |  32.54 |  33.05 |  35.30 |  32.46
| SSIM |  |  0.97 |  0.92 |  0.92 |  0.92 |  0.92 |  0.93 |  0.94 |  0.91
SelfFlow |  | PSNR |  |  40.22 |  35.31 |  33.12 |  33.78 |  32.99 |  33.62 |  35.44 |  32.65
| SSIM |  |  0.97 |  0.92 |  0.92 |  0.93 |  0.93 |  0.93 |  0.94 |  0.91
SelfFlow+ft |  | PSNR |  |  40.36 |  35.74 |  33.35 |  34.30 |  33.64 |  34.22 |  35.90 |  33.02
| SSIM |  |  0.97 |  0.93 |  0.92 |  0.93 |  0.93 |  0.94 |  0.95 |  0.92
GelFlow |  | PSNR |  |  $\underline{40.57}$ |  $\underline{35.90}$ |  $\underline{33.43}$ |  $\underline{34.34}$ |  $\underline{33.65}$ |  $\underline{33.96}$ |  $\underline{35.94}$ |  $\underline{33.11}$
| SSIM |  |  $\underline{0.98}$ |  $\underline{0.93}$ |  $\underline{0.93}$ |  $\underline{0.93}$ |  $\underline{0.93}$ |  $\underline{0.94}$ |  $\underline{0.95}$ |  $\underline{0.93}$
GelFlow+ft |  | PSNR |  |  $\mathbf{40.76}$ |  $\mathbf{36.00}$ |  $\mathbf{33.66}$ |  $\mathbf{34.71}$ |  $\mathbf{34.25}$ |  $\mathbf{35.00}$ |  $\mathbf{36.13}$ |  $\mathbf{33.22}$
| SSIM |  |  $\mathbf{0.98}$ |  $\mathbf{0.93}$ |  $\mathbf{0.93}$ |  $\mathbf{0.94}$ |  $\mathbf{0.94}$ |  $\mathbf{0.95}$ |  $\mathbf{0.95}$ |  $\mathbf{0.93}$

###  4.3 Comparisons with Classical Optical Flow Methods

We compared traditional dense optical flow methods (Farnaback, DIS, TV-L1)
using OpenCV and deep learning-based optical flow methods (RAFT [26], ARFlow
[27], and a self-supervised method named SelfFlow using a photometric loss
mentioned before with a first-order smoothness loss [28]). The results of the
comparison are presented in Table 1. Notably, the self-supervised methods
SelfFlow and GelFlow were fine-tuned using the strategy described in Section
4.1. The validation dataset showed that TV-L1 and DIS (medium) performed
similarly and outperformed other traditional methods. However, the solving
strategy of TV-L1 is time-consuming, making it much slower than the optimized
DIS method. Consequently, the DIS method is widely used as the dense optical
flow estimator in current vision-based tactile sensors.

It is worth mentioning that we directly utilized the pre-trained models of
RAFT and ARFlow, testing them on the vision-based tactile dataset. Therefore,
their performance may not be satisfactory. On the other hand, SelfFlow and
GelFlow were trained on the dataset and further fine-tuned. As a result, they
outperformed the existing traditional methods. The excellent performance can
be attributed to the strong learning ability of convolutional neural networks
and the well-designed loss functions guiding the network output towards the
ground truth. Among all the candidate methods, GelFlow achieved the best
performance with its proposed flow fusion operation and motion decomposition
and deformation loss, which guide the parameters of the network towards global
optimization. In conclusion, the comparisons indicate that the proposed
GelFlow method is particularly adept at handling gel materials’ deformation.

##  5 Conclusion

In this study, we propose the GelFlow method, which incorporates several novel
components to address the challenges posed by gel deformation. Firstly,
GelFlow constructs a multi-scale feature pyramid to extract hidden features
from the input image pairs and handle large displacements effectively. A local
flow fusion module fuses the flow using neighboring flows with appropriate
weights. This fusion process achieves a blurred effect, which is crucial for
capturing the deformations occurring in gel materials. We propose two novel
loss functions to better handle the intricate gel deformations: the velocity
decomposition loss and the elastic deformation loss. A photometric loss
combined with the proposed two novel motion smoothness losses is used to
construct the multi-scale loss to better guide the network from global
optimization. Finally, the network is trained in a self-supervised manner, and
the comparison result with other optical flow methods indicates that the
GelFlow method performs the best due to the superior capacity of the
convolutional neural networks to extract valuable features and the strong
ability of global optimization.

## References

  * [1] Liu, H., Yu, Y., Sun, F., Gu, J.: Visual–tactile fusion for object recognition. IEEE Transactions on Automation Science and Engineering 14(2), 996–1008 (2016)
  * [2] Zhang, S., Chen, Z., Gao, Y., Wan, W., Shan, J., Xue, H., Sun, F., Yang, Y., Fang, B.: Hardware technology of vision-based tactile sensor: A review. IEEE Sensors Journal (2022)
  * [3] Sato, K., Kamiyama, K., Kawakami, N., Tachi, S.: Finger-shaped gelforce: sensor for measuring surface traction fields for robotic hand. IEEE Transactions on Haptics 3(1), 37–47 (2009)
  * [4] Cui, S., Wang, R., Hu, J., Zhang, C., Chen, L., Wang, S.: Self-supervised contact geometry learning by gelstereo visuotactile sensing. IEEE Transactions on Instrumentation and Measurement 71, 1–9 (2021)
  * [5] James, J.W., Lepora, N.F.: Slip detection for grasp stabilization with a multifingered tactile robot hand. IEEE Transactions on Robotics 37(2), 506–519 (2020)
  * [6] Du, Y., Zhang, G., Zhang, Y., Wang, M.Y.: High-resolution 3-dimensional contact deformation tracking for fingervision sensor with dense random color pattern. IEEE Robotics and Automation Letters 6(2), 2147–2154 (2021)
  * [7] Bradski, G.: The opencv library. Dr. Dobb’s Journal: Software Tools for the Professional Programmer 25(11), 120–123 (2000)
  * [8] Zhang, G., Du, Y., Yu, H., Wang, M.Y.: Deltact: A vision-based tactile sensor using a dense color pattern. IEEE Robotics and Automation Letters 7(4), 10778–10785 (2022)
  * [9] Yuan, W., Dong, S., Adelson, E.H.: Gelsight: High-resolution robot tactile sensors for estimating geometry and force. Sensors 17(12), 2762 (2017)
  * [10] Donlon, E., Dong, S., Liu, M., Li, J., Adelson, E., Rodriguez, A.: Gelslim: A high-resolution, compact, robust, and calibrated tactile-sensing finger. In: 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). pp. 1927–1934. IEEE (2018)
  * [11] Lambeta, M., Chou, P.W., Tian, S., Yang, B., Maloon, B., Most, V.R., Stroud, D., Santos, R., Byagowi, A., Kammerer, G., et al.: Digit: A novel design for a low-cost compact high-resolution tactile sensor with application to in-hand manipulation. IEEE Robotics and Automation Letters 5(3), 3838–3845 (2020)
  * [12] Do, W.K., Kennedy, M.: Densetact: Optical tactile sensor for dense shape reconstruction. In: 2022 International Conference on Robotics and Automation (ICRA). pp. 6188–6194. IEEE (2022)
  * [13] Padmanabha, A., Ebert, F., Tian, S., Calandra, R., Finn, C., Levine, S.: Omnitact: A multi-directional high-resolution touch sensor. In: 2020 IEEE International Conference on Robotics and Automation (ICRA). pp. 618–624. IEEE (2020)
  * [14] Cui, S., Wang, R., Hu, J., Wei, J., Wang, S., Lou, Z.: In-hand object localization using a novel high-resolution visuotactile sensor. IEEE Transactions on Industrial Electronics 69(6), 6015–6025 (2021)
  * [15] Zhang, L., Wang, Y., Jiang, Y.: Tac3d: A novel vision-based tactile sensor for measuring forces distribution and estimating friction coefficient distribution. arXiv preprint arXiv:2202.06211 (2022)
  * [16] Bouguet, J.Y., et al.: Pyramidal implementation of the affine lucas kanade feature tracker description of the algorithm. Intel corporation 5(1-10), 4 (2001)
  * [17] Farnebäck, G.: Two-frame motion estimation based on polynomial expansion. In: Image Analysis: 13th Scandinavian Conference, SCIA 2003 Halmstad, Sweden, June 29–July 2, 2003 Proceedings 13. pp. 363–370. Springer (2003)
  * [18] Kroeger, T., Timofte, R., Dai, D., Van Gool, L.: Fast optical flow using dense inverse search. In: Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part IV 14. pp. 471–488. Springer (2016)
  * [19] Kuppuswamy, N., Alspach, A., Uttamchandani, A., Creasey, S., Ikeda, T., Tedrake, R.: Soft-bubble grippers for robust and perceptive manipulation. In: 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). pp. 9917–9924. IEEE (2020)
  * [20] Sun, D., Yang, X., Liu, M.Y., Kautz, J.: Pwc-net: Cnns for optical flow using pyramid, warping, and cost volume. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 8934–8943 (2018)
  * [21] Brox, T., Bruhn, A., Papenberg, N., Weickert, J.: High accuracy optical flow estimation based on a theory for warping. In: European conference on computer vision. pp. 25–36. Springer (2004)
  * [22] Du, Y., Zhang, G., Wang, M.Y.: 3d contact point cloud reconstruction from vision-based tactile flow. IEEE Robotics and Automation Letters 7(4), 12177–12184 (2022)
  * [23] Lu, J., Yang, H., Zhang, Q., Yin, Z.: An accurate optical flow estimation of piv using fluid velocity decomposition. Experiments in Fluids 62, 1–16 (2021)
  * [24] Yang, G., Ramanan, D.: Upgrading optical flow to 3d scene flow through optical expansion. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 1334–1343 (2020)
  * [25] Yang, F., Ma, C., Zhang, J., Zhu, J., Yuan, W., Owens, A.: Touch and go: Learning from human-collected vision and touch. In: Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (2022), https://openreview.net/forum?id=ZZ3FeSSPPblo
  * [26] Teed, Z., Deng, J.: Raft: Recurrent all-pairs field transforms for optical flow. In: Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16. pp. 402–419. Springer (2020)
  * [27] Liu, L., Zhang, J., He, R., Liu, Y., Wang, Y., Tai, Y., Luo, D., Wang, C., Li, J., Huang, F.: Learning by analogy: Reliable supervision from transformations for unsupervised optical flow estimation. In: IEEE Conference on Computer Vision and Pattern Recognition(CVPR) (2020)
  * [28] Jonschkowski, R., Stone, A., Barron, J.T., Gordon, A., Konolige, K., Angelova, A.: What matters in unsupervised optical flow. In: Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16. pp. 557–572. Springer (2020)