# Generation, augmentation, and alignment: A pseudo-source domain based method
for source-free domain adaptation 111Under review

Yuntao Du, Haiyang Yang, Mingcai Chen, Juan Jiang, Hongtao Luo, Chongjun Wang

###### Abstract

Conventional unsupervised domain adaptation (UDA) methods need to access both
labeled source samples and unlabeled target samples simultaneously to train
the model. While in some scenarios, the source samples are not available for
the target domain due to data privacy and safety. To overcome this challenge,
recently, source-free domain adaptation (SFDA) has attracted the attention of
researchers, where both a trained source model and unlabeled target samples
are given. Existing SFDA methods either adopt a pseudo-label based strategy or
generate more samples. However, these methods do not explicitly reduce the
distribution shift across domains, which is the key to a good adaptation.
Although there are no source samples available, fortunately, we find that some
target samples are very similar to the source domain and can be used to
approximate the source domain. This approximated domain is denoted as the
pseudo-source domain. In this paper, inspired by this observation, we propose
a novel method based on the pseudo-source domain. The proposed method firstly
generates and augments the pseudo-source domain, and then employs distribution
alignment with four novel losses based on pseudo-label based strategy. Among
them, a domain adversarial loss is introduced between the pseudo-source domain
the remaining target domain to reduce the distribution shift. The results on
three real-world datasets verify the effectiveness of the proposed method.

## Introduction

Deep neural networks need large labeled samples to train the model (He, Zhang
et al. 2016; Ren et al. 2015) and the performance will drop seriously due to
the domain shift when applying the trained model directly to a new domain. As
a promising learning paradigm, unsupervised domain adaptation (UDA), a sub-
field of transfer learning, is able to transfer knowledge from the labeled
source domain to the unlabeled target domain to overcome this challenge (Pan
and Yang 2010).

Conventional deep unsupervised domain adaptation methods are built on two main
strategies: moment matching (Long et al. 2015; Kang, Jiang et al. 2019; Chen
et al. 2020a) and adversarial domain adaptation (Ganin et al. 2016; Saito,
Watanabe et al. 2018; Zhang, Liu et al. 2019). The former minimizes the
statistical distribution discrepancy across domains and the latter reduces the
domain discrepancy in an adversarial manner. And existing UDA methods need to
access both the source samples and target samples simultaneously to train the
model. However, in some scenarios, the source model instead of the source
samples is able to be obtained due to data privacy and safety. Such setting is
denoted as source-free domain adaptation (SFDA) (Liang, Hu et al. 2020; Li,
Jiao et al. 2020). Previous UDA methods cannot be applied to SFDA directly due
to the absence of source samples.

Generally, the goal of SFDA is to train a well-performed model in the target
domain based on a trained source model and unlabeled target samples. There are
some attempts to tackle this problem. Both SHOT (Liang, Hu et al. 2020) and
CPGA (Qiu et al. 2021) adopt pseudo-label based strategy together with
information maximization and prototypes adaptation respectively. Besides, both
MA (Li, Jiao et al. 2020) and SDDA (Kurmi, Subramanian, and Namboodiri 2021)
adopt a generative adversarial net to generate either target-like or source-
like samples to train the model. However, these methods do not explicitly
reduce the distribution shift across domains. According to classical UDA
theories (Ben-David et al. 2009; Zhang, Liu et al. 2019), it is important to
reduce the distribution discrepancy across domains to achieve a good
adaptation and such property should also be meet in SFDA. So the large domain
shift is a limitation for existing SFDA methods and needs to be addressed.

Figure 1: The motivation of the proposed method. (a) Although there are no
source samples, we find that the target samples near the source prototypes can
be used to represent the source domain. Thus, (b) PS firstly generates a
pseudo-source domain by selecting such target samples. Then, (c) PS enlarges
this pseudo-source domain using data augmentation. Lastly, (d) PS performs
distribution alignment with four losses. Among them, the domain adversarial
loss is introduced to reduce the distribution shift.

Based on the above analysis, we aim to reduce domain shift to further improve
the performance of SFDA. Although there are no source samples, the trained
model can retain and reflect original data distribution information. As shown
in Figure 1(a), assuming there are some source samples, it is obvious that
they are spread around the corresponding prototypes, where the class centers
or weight vectors of the classifier can be regarded as the prototypes (Saito,
Watanabe et al. 2018). Although there exists domain shift, we obverse that
some target samples are also spread around the corresponding source prototypes
and they are very similar to the source domain. Thus, these target samples
could be used to approximate the source domain. We denote these samples as
pseudo-source samples and the corresponding new domain as the pseudo-source
domain. In other words, the target samples could be split into two disjoint
parts, i.e., the pseudo-source samples and the remaining target samples. In
such cases, reducing the distribution discrepancy across domains can be
approximately achieved by minimizing the discrepancy between the pseudo-source
samples and the remaining target samples. Although some UDA methods also split
the target samples into two parts, such as easy samples and hard samples in
(Pan et al. 2020), the goal is different. The method in (Pan et al. 2020) aims
to reduce the intra-domain gap within the target domain while our method aims
to generate pseudo-source samples for reducing the shift across domains.

Following the above idea, in this paper, we propose a method named _Pseudo-
Source domain based source-free domain adaptation_ (PS). Our method is
composed of three steps. The first step is to generate a pseudo-source domain
by selecting the target samples that are similar to the source domain. As the
number of the samples in the pseudo-source domain is smaller than the number
of the remaining target samples, the second step is to augment the pseudo-
source domain to enlarge this new domain for better adaptation. The third step
is to perform distribution alignment with four proposed losses. During
distribution alignment, PS also adopts pseudo-label based strategy to train
the model. Notably, among four losses, a domain adversarial loss is introduced
between the pseudo-source domain and the remaining target domain to decrease
the distribution discrepancy across domains. To sum up, our principal
contributions are summarized as follows:

  * •

We are the first to generate a pseudo-source domain from the target samples
and perform distribution alignment for SFDA, such that the domain discrepancy
across domains can be reduced even without source samples.

  * •

We propose a novel method, which generates and augments the pseudo-source
domain and then performs distribution alignment for explicit adaptation.

  * •

Extensive experiments are conducted and the results show that the proposed
method achieves better performance than state-of-the-art methods.

## Related Work

### Unsupervised Domain Adaptation

A classical domain adaptation theory (Ben-David et al. 2009) indicates that it
is crucial to reduce the distribution discrepancy across domains to achieve
better adaptation. Based on this theory, many domain adaptation methods have
been proposed and they are divided into moment matching and adversarial domain
adaptation. The goal of the former is to reduce the statistical distribution
discrepancy across domains. The widely used statistical measurements include
the first-order moment (Long et al. 2015; Kang, Jiang et al. 2019), the
second-order moment(Sun, Feng, and Saenko 2016), high-order moment(Chen et al.
2020a; Zellinger et al. 2017) and other statistical measurements (Long et al.
2017; Li et al. 2020; Shen et al. 2018).

Adversarial domain adaptation, which is inspired by generative adversarial
network (Goodfellow, Pouget-Abadie et al. 2014), reduces the distribution
discrepancy in an adversarial manner. DANN (Ganin et al. 2016) introduces a
domain discriminator which plays a min-max game with the feature extractor by
the domain adversarial loss. MCD (Saito, Watanabe et al. 2018) introduces two
classifiers as a discriminator to play a min-max game with the feature
extractor such that the source samples are pushed into the support of the
target domain. Considering practical multi-class problem, MDD (Zhang, Liu et
al. 2019) proposes a margin-based theory, and a new method based on this
theory is proposed. Following methods (Cicek and Soatto 2019; Chen et al.
2020b) adopt the multi-class discriminator which considers the domain and
class information simultaneously. However, previous UDA methods cannot be
applied to SFDA directly due to the absence of source samples.

### Source-Free Domain Adaptation

The source samples are unavailable in source-free domain adaptation due to
data privacy and security. However, a model trained by the source samples is
given and the goal is to adapt the trained model to the target domain. Some
methods adopt pseudo-label based strategy. SHOT (Liang, Hu et al. 2020) learns
a target-specific feature extraction module by self-supervised pseudo-labeling
together with information maximization and implicitly aligns representations
of the target domain to the source model. CPGA (Qiu et al. 2021) proposes to
utilize the hidden knowledge in the source model and exploits it to generate
source avatar prototypes as well as target pseudo-labels for domain alignment.
PrDA (Kim, Cho et al. 2020) progressively updates the target model in a self-
learning manner by leveraging a pre-trained model from the source domain, such
that the pseudo-labels would become more accurate for better adaptation.

Some methods adopt generative adversarial nets (GAN) to generate more samples.
MA (Li, Jiao et al. 2020) proposes to generate target-style data with a class
conditional generative adversarial net. SDDA (Kurmi, Subramanian, and
Namboodiri 2021) treats the trained classifier as an energy-based model to
learn the data distribution along with a generative adversarial net for
generating source-like samples. Besides, some works explore source-free domain
adaptation under different settings such as universal SFDA (Kundu et al. 2020)
and multi-source SFDA (Ahmed et al. 2021; Feng et al. 2021). Moreover, Some
works focus on different applications such as semantic segmentation (Liu,
Zhang, and Wang 2021; Zhao et al. 2021) and object detection (Li et al. 2021).
Although achieving remarkable progress, these methods do not explicitly reduce
the distribution shift across domains.

Figure 2: The model structure of the proposed method. $g_{s}$, $f_{s}$ and
$g_{t}$, $f_{t}$ are the feature extractors and classifiers of the source
model and target model, respectively. GRL is the gradient reversal layer and
$D$ is the domain discriminator.

## Problem Definition

In a vanilla UDA task, there are a labeled source domain
$\mathcal{D}_{s}=\\{x_{s}^{i},y_{s}^{i}\\}_{i=1}^{n_{s}}$ and an unlabeled
target domain $\mathcal{D}_{t}=\\{x_{t}^{i}\\}_{i=1}^{n_{t}}$, where
$x_{s}^{i}\in\mathcal{X}_{s}$, $y_{s}^{i}\in\mathcal{Y}_{s}$, and
$x_{t}^{i}\in\mathcal{X}_{t}$. The feature space and the label space are the
same across domains, i.e., $\mathcal{X}_{s}=\mathcal{X}_{t}=\mathcal{R}^{d}$
and $\mathcal{Y}_{s}=\mathcal{Y}_{t}=\\{1,2,...,K\\}$. While the source
samples and target samples are drawn from different distributions $P_{s}(x,y)$
and $P_{t}(x,y)$, i.e., $P_{s}(x,y)\neq P_{t}(x,y)$. In UDA, the goal is to
train a model that can perform well in the target domain using both labeled
source samples of $\mathcal{D}_{s}$ and unlabeled target samples of
$\mathcal{D}_{t}$.

While in SFDA, the target domain can not access the source samples, but a
model $F_{s}$ trained by the source samples is given. The goal of SFDA is to
use the trained source model $F_{s}$ and unlabeled target samples of
$\mathcal{D}_{t}$ to train a new model $F_{t}$, such that $F_{t}$ performs
well in the target domain.

## Method

The model structure of the proposed method is shown in Figure 2(b). The source
trained model $F_{s}$ is composed of a feature extractor $g_{s}$ and a
classifier $f_{s}$, i.e., $F_{s}(x)=f_{s}(g_{s}(x))$. The target model also
contains a feature extractor $g_{t}$ and a classifier $f_{t}$, i.e.,
$F_{t}(x)=f_{t}(g_{t}(x))$. Besides, a domain discriminator $D$ is also
introduced. To better transfer knowledge from the source domain to the target
domain, the feature extractor $g_{t}$ of target model is initialized by the
feature extractor $g_{s}$ of source model, and the target classifier $f_{t}$
is also initialized by the source classifier $f_{s}$. Note that the source
classifier $f_{s}$ is fixed during training, while the target classifier
$f_{t}$ would be updated. In the following subsections, we firstly introduce
the training process of source model and then describe these training steps of
the proposed method.

### Training of Source Model

In SFDA setting, as is shown in Figure 2(a), a model
$F_{s}:\mathcal{X}_{s}\rightarrow\mathcal{Y}_{s}$ is trained by the source
samples and the trained model $F_{s}$ is delivered to the target domain. To
further increase the discriminability of the source model, following previous
work (Liang, Hu et al. 2020), PS adopts the label-smoothing technique to train
the model as it encourages samples to lie in tight evenly separated clusters
(Müller, Kornblith et al. 2019). The objective function is,

$\displaystyle\mathcal{L}_{cls}^{s}(g_{s},f_{s};\mathcal{D}_{s})=$ (1)
$\displaystyle-\mathbb{E}_{(x_{s},y_{s})\in\mathcal{D}_{s}}$
$\displaystyle\sum\nolimits_{k=1}^{K}q_{k}^{ls}\log\delta_{k}(f_{s}(g_{s}(x_{s})))$

where $f_{s}(g_{s}(x_{s}))$ is the $K$-dimensional output of the source sample
$x_{s}$. $q^{ls}_{k}=(1-\gamma)q_{k}^{s}+\gamma/K$ is the smoothed label and
$\gamma$ is the smoothing parameter which is empirically set to 0.1.
$\delta_{k}(\mathbf{a})=\frac{exp(a_{k})}{\sum_{i}exp(a_{i})}$ denotes the
$k$-th element in the softmax output of a $K$-dimensional vector $\mathbf{a}$.
$q^{s}$ is the one-of-$K$ encoding of $y_{s}$ where $q_{k}^{s}$ is ‘1’ for the
correct class and ‘0’ for the rest.

### Generation of Pseudo-Source Domain

As the source samples are not available in the target domain, conventional UDA
methods can not be directly used to reduce the domain shift. Fortunately, we
find that some target samples can be used to represent the source domain. Such
samples are denoted as pseudo-source samples and constitute a new domain
denoted as pseudo-source domain. This process can be regarded as dividing the
samples in the target domain into two disjoint parts, where the first part is
the pseudo-source domain, denoted as $\hat{\mathcal{D}}_{s}$, and the second
part is the remaining target domain, denoted as $\hat{\mathcal{D}}_{t}$. It is
obvious that $\hat{\mathcal{D}}_{s}\cap\hat{\mathcal{D}}_{t}=\varnothing$ and
$\hat{\mathcal{D}}_{s}\cup\hat{\mathcal{D}}_{t}=\mathcal{D}_{t}$.

As is shown in Figure 1, the target samples near the source prototypes are
similar to the source domain. The entropy criterion is widely used in UDA
(Long, Cao et al. 2018; Pan et al. 2020) to evaluate the uncertainty of
classifier prediction for a given sample. The lower the entropy is, the closer
the sample is to the prototype. Thus, in this paper, PS adopts the entropy
criterion to select the target samples that can be used to simulate the source
domain. Given a trained source model $F_{s}$, all the target samples firstly
pass the source model and then the entropy $H(x_{t})$ is calculated,

$H(x_{t})=-\sum\nolimits_{k=1}^{K}\delta_{k}(f_{s}(g_{s}(x_{t})))\log\delta_{k}(f_{s}(g_{s}(x_{t})))$
(2)

To generate a class-balanced pseudo-source domain, all the target samples are
sorted in ascending order of entropy and PS selects samples from each class
separately, i.e., the top $\alpha$ proportion of samples in each class are
selected. As the samples in the target domain are unlabeled, the pseudo-labels
are used instead. For the sample ${x}_{t}\in\mathcal{D}_{t}$, the pseudo-label
$\hat{y}_{t}$ is obtained by

$\hat{y}_{t}=\arg\max_{y\in\\{1,...,K\\}}f_{s}(g_{s}(x_{t}))$ (3)

By this strategy, PS generates a class-balanced new domain and we
experimentally find that $\alpha=0.1$ works well.

### Pseudo-Source Domain Augmentation

Previous works have shown that the larger the source domain is, the better
performance the model could get (Ganin et al. 2016; Kumar et al. 2018).
However, the number of the samples in the pseudo-source domain
$\hat{\mathcal{D}}_{s}$ is smaller than that in the remaining target domain
$\hat{\mathcal{D}}_{t}$ (namely,
$|\hat{\mathcal{D}}_{s}|:|\hat{\mathcal{D}}_{t}|\approx$ 1:9). Thus, PS
enlarges the pseudo-source domain using data augmentation. In this work, mixup
(Zhang et al. 2018) is choosen, as it is a simple yet effective supervised
data augmentation method. Given two samples from the pseudo-source domain,
e.g., $\hat{x}_{s}^{i},\hat{x}_{s}^{j}\in\hat{\mathcal{D}}_{s}$, whose pseudo-
labels are obtained by Equ 3 and denoted as $\hat{y}_{s}^{i}$ and
$\hat{y}_{s}^{j}$, PS augments the pseudo-source domain by mixing these two
samples,

$\begin{split}&\hat{x}_{s,aug}=\lambda\hat{x}_{s}^{i}+(1-\lambda)\hat{x}_{s}^{j}\\\
&\hat{y}_{s,aug}=\lambda\hat{y}_{s}^{i}+(1-\lambda)\hat{y}_{s}^{j}\end{split}$
(4)

where $\lambda\sim$ Beta($\beta$, $\beta$), for $\beta\in(0,\infty)$. The new
domain which is composed of augmented pseudo-source samples is denoted as
augmented domain
$\hat{\mathcal{D}}_{aug}=\\{\hat{x}_{s,aug},\hat{y}_{s,aug}\\}_{i=1}^{n}$. And
the domain that contains the original pseudo-source domain
$\hat{\mathcal{D}}_{s}$ and the augmented source domain
$\hat{\mathcal{D}}_{aug}$ is denoted as augmented pseudo-source domain
$\hat{\mathcal{D}}_{s}^{aug}$, i.e.
$\hat{\mathcal{D}}_{s}^{aug}=\hat{\mathcal{D}}_{s}\cup\hat{\mathcal{D}}_{aug}$.
After adopting mixup, the dataset can be effectively enlarged.

### Distribution Alignment

After generating and augmenting the pseudo-source domain, PS performs
distribution alignment where pseudo-label based strategy is also used to train
the model. In distribution alignment, four complementary losses are proposed
for better adaptation in the target domain.

#### Classification loss

PS firstly trains the new model $F_{t}(x)$ to classify the samples in both the
augmented pseudo-source domain and the remaining target domain correctly.
Similar to previous work (Saito, Ushiku, and Harada 2017), training the model
to classify the target samples with pseudo-labels could help the model capture
the characteristics of the target domain. The objective function is

$\displaystyle\mathcal{L}_{cls}(g_{t},f_{t};\hat{\mathcal{D}}_{s}^{aug},\hat{\mathcal{D}}_{t})$
$\displaystyle=\mathbb{E}_{(\hat{x}_{s},\hat{y}_{s})\sim\hat{\mathcal{D}}_{s}^{aug}}L(g_{t}(f_{t}(\hat{x}_{s})),\hat{y}_{s})$
(5)
$\displaystyle+\mathbb{E}_{(\hat{x}_{t},\hat{y}_{t})\sim\hat{\mathcal{D}}_{t}}$
$\displaystyle L(f_{t}(g_{t}(\hat{x}_{t})),\hat{y}_{t})$

where $L(\cdot,\cdot)$ is the cross-entropy loss. For the samples in
$\hat{\mathcal{D}}_{t}$, the pseudo-labels obtained by Equ 3 may be incorrect
as the entropy is very high. We incorporate the method in SHOT (Liang, Hu et
al. 2020) to regenerate the pseudo-labels and the detail steps are shown in
the supplementary materials.

#### Diversity loss

As there are only unlabeled samples in the remaining target domain, to avoid
the collapse mode, PS adopts the information maximization (Gomes, Krause, and
Perona 2010) to make the target outputs globally diverse,

$\displaystyle\mathcal{L}_{div}(g_{t},f_{t};\hat{\mathcal{D}}_{t})$
$\displaystyle=\sum\nolimits_{k=1}^{K}p_{1,k}logp_{1,k}+\sum\nolimits_{k=1}^{K}p_{2,k}logp_{2,k}$
(6) $\displaystyle=[D_{KL}$
$\displaystyle(p_{1},\frac{1}{K}\mathbf{1}_{K})+D_{KL}(p_{2},\frac{1}{K}\mathbf{1}_{K})]-2logK,$

where $\mathbf{1}_{K}$ is a $K$-dimensional vector with all ones, and
$p_{1}=\mathbb{E}_{\hat{x}_{t}\in\hat{\mathcal{D}}_{t}}[\delta(f_{s}(g_{t}(\hat{x}_{t})))]$
and
$p_{2}=\mathbb{E}_{\hat{x}_{t}\in\hat{\mathcal{D}}_{t}}[\delta(f_{t}(g_{t}(\hat{x}_{t})))]$
are the mean output embeddings of the whole target domain by classifiers
$f_{s}$ and $f_{t}$, respectively. $p_{1,k}$ and $p_{2,k}$ are the $k$-th
element of the outputs of $f_{s}$ and $f_{t}$, respectively. With the fair
diversity-promoting objective $\mathcal{L}_{div}$, the learned model can
circumvent the trivial solution where all unlabeled data have the same one-hot
encoding.

Algorithm 1 Training of PS

0: Trained source model $F_{s}$ composed of $g_{s}$ and $f_{s}$, unlabeled
target samples of $\mathcal{D}_{t}$, batchsize $B$, and proportion $\alpha$.

0: Trained target model $F_{t}$ composed of $g_{t}$ and $f_{t}$.

1: Initialization: Initialize $g_{t}$ with $g_{s}$, initialize $f_{t}$ with
$f_{s}$.

2: repeat

3: Sample a minibatch of size $B$ from $\mathcal{D}_{t}$.

4: Generation: Sort all the target samples with the entropy calculated by Equ
2 in each class, and split samples of the top $\alpha$ proportion as
$\hat{\mathcal{D}}_{s}$ and the remaining samples as $\hat{\mathcal{D}}_{t}$.

5: Augmentation: Augment the samples in pseudo-source domains
$\hat{\mathcal{D}}_{s}$ by Equ 4 to get augmented pseudo-source domain
$\hat{\mathcal{D}}_{s}^{aug}$.

6: Alignment:

7: 1) Train the feature extractor $g_{t}$ and domain discriminator $D$ by Equ
9.

8: 2) Train the feature extractor $g_{t}$ and the classifier $f_{t}$ by Equ
10.

9: until Maximum iteration or loss convergence

#### Constrain loss

the source classifier $f_{s}$ is kept unchanged during training. As the source
classifier $f_{s}$ could retain the learned knowledge from the source domain,
PS aims to force the learned target classifier $f_{t}$ to be not far away from
the source classifier $f_{s}$. Such strategy is also used in continuous
learning and incremental learning (Li and Hoiem 2018; Parisi et al. 2019).
Specially, PS trains the features extractor $g_{t}$ to make the outputs of
these two classifiers similar,

$\begin{split}\mathcal{L}_{\mathrm{cons}}(g_{t};\hat{\mathcal{D}}_{t})=&\mathbb{E}_{\hat{x}_{t}\sim\hat{\mathcal{D}}_{t}}[L(f_{s}(g_{t}(\hat{x}_{t})),f_{t}(g_{t}(\hat{x}_{t})))\\\
&+L(f_{t}(g_{t}(\hat{x_{t}})),f_{s}(g_{t}(\hat{x}_{t})))]\end{split}$ (7)

#### Domain adversarial loss

Though achieving remarkable progress (Liang, Hu et al. 2020; Li, Jiao et al.
2020), previous SFDA methods do not explicitly reduce the distribution shift
across domains. So in this work, PS aims to learn domain-invariant features in
an adversarial manner. Similar to previous work (Ganin and Lempitsky 2015), a
domain discriminator $D$ is introduced to distinguish the augmented pseudo-
source domain $\hat{\mathcal{D}}_{s}^{aug}$ from the remaining target domain
$\hat{\mathcal{D}}_{t}$. While the feature extractor $g_{t}$ is trained to
confuse the domain discriminator $D$. The adversarial objective function is,

$\displaystyle\mathcal{L}_{g}(g_{t},D;\hat{\mathcal{D}}_{s}^{aug},\hat{\mathcal{D}}_{t})$
$\displaystyle=\mathbb{E}_{\hat{x}_{s}\sim\hat{\mathcal{D}}_{s}^{aug}}[\log
D(g_{t}(\hat{x}_{s}))]$ (8)
$\displaystyle+\mathbb{E}_{\hat{x}_{t}\sim\hat{\mathcal{D}}_{t}}[\log(1-D(g_{t}(\hat{x}_{t})))]$

### Training Steps

In this subsection, we summarize the training steps of PS. Combining the above
objectives discussed together, the overall objective is:

$\min_{g_{t}}\max_{D}\mathcal{L}_{cons}(g_{t})+\lambda_{g}\mathcal{L}_{g}(g_{t},D)\vspace{-2mm}$
(9)
$\min_{g_{t},f_{t}}\mathcal{L}_{div}(g_{t},f_{t})+\lambda_{c}\mathcal{L}_{cls}(g_{t},f_{t})$
(10)

Following the previous method (Ganin and Lempitsky 2015), the min-max training
procedure in Equ 9 is accomplished by applying a Gradient Reversal Layer
(GRL). GRL behaves as the identity function during the forward propagation and
inverts the gradient sign during the backward propagation, hence driving the
parameters to maximize the output loss. During the distribution alignment, the
model is trained by the losses in Equ 9 and Equ 10 alternatively.

Remark: In our method, PS performs pseudo-source domain generation, pseudo-
source domain augmentation and distribution alignment in an alternative
manner. All three steps are employed in a minibatch of size $B$. The pseudo-
source code is shown in Algorithm 1.

Method (Source$\to$Target) | Source-free | S$\to$M | U$\to$M | M$\to$U | Avg.
---|---|---|---|---|---
Source-only | ✗ | 67.1$\pm$0.6 | 69.6$\pm$3.8 | 82.2$\pm$0.8 | 73.0
ADDA | ✗ | 76.0$\pm$1.8 | 90.1$\pm$0.8 | 89.4$\pm$0.2 | 85.2
ADR | ✗ | 95.0$\pm$1.9 | 93.1$\pm$1.3 | 93.2$\pm$2.5 | 93.8
CDAN+E | ✗ | 89.2 | 98.0 | 95.6 | 94.3
CyCADA | ✗ | 90.4$\pm$0.4 | 96.5$\pm$0.1 | 95.6$\pm$0.4 | 94.2
rRevGrad+CAT | ✗ | 98.8$\pm$0.0 | 96.0$\pm$0.9 | 94.0$\pm$0.7 | 96.3
SWD | ✗ | 98.9$\pm$0.1 | 97.1$\pm$0.1 | 98.1$\pm$0.1 | 98.0
SHOT-IM | ✓ | 89.6$\pm$5.0 | 96.8$\pm$0.4 | 91.9$\pm$0.4 | 92.8
SHOT | ✓ | 98.9$\pm$0.0 | 98.4$\pm$0.6 | 98.0$\pm$0.2 | 98.4
PS (ours) | ✓ |  99.0$\pm$0.1 |  98.6$\pm$0.0 |  98.2$\pm$0.2 | 98.6
Target-supervised (Oracle) |  | 99.4$\pm$0.1 | 99.4$\pm$0.1 | 98.0$\pm$0.1 | 98.8

Table 1: Classification accuracies (%) on Digits dataset.

## Experiments

Method | Source-free | Ar$\rightarrow$Cl | Ar$\rightarrow$Pr | Ar$\rightarrow$Rw | Cl$\rightarrow$Ar | Cl$\rightarrow$Pr | Cl$\rightarrow$Rw | Pr$\rightarrow$Ar | Pr$\rightarrow$Cl | Pr$\rightarrow$Rw | Rw$\rightarrow$Ar | Rw$\rightarrow$Cl | Rw$\rightarrow$Pr | Avg.
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
ResNet-50 | ✗ | 34.9 | 50.0 | 58.0 | 37.4 | 41.9 | 46.2 | 38.5 | 31.2 | 60.4 | 53.9 | 41.2 | 59.9 | 46.1
MCD | ✗ | 48.9 | 68.3 | 74.6 | 61.3 | 67.6 | 68.8 | 57.0 | 47.1 | 75.1 | 69.1 | 52.2 | 79.6 | 64.1
CDAN | ✗ | 50.7 | 70.6 | 76.0 | 57.6 | 70.0 | 70.0 | 57.4 | 50.9 | 77.3 | 70.9 | 56.7 | 81.6 | 65.8
MDD | ✗ | 54.9 | 73.7 | 77.8 | 60.0 | 71.4 | 71.8 | 61.2 | 53.6 | 78.1 | 72.5 | 60.2 | 82.3 | 68.1
BNM | ✗ | 52.3 | 73.9 | 80.0 | 63.3 | 72.9 | 74.9 | 61.7 | 49.5 | 79.7 | 70.5 | 53.6 | 82.2 | 67.9
BDG | ✗ | 51.5 | 73.4 | 78.7 | 65.3 | 71.5 | 73.7 | 65.1 | 49.7 | 81.1 | 74.6 | 55.1 | 84.8 | 68.7
SRDC | ✗ | 52.3 | 76.3 | 81.0 | 69.5 | 76.2 | 78.0 | 68.7 | 53.8 | 81.7 | 76.3 | 57.1 | 85.0 | 71.3
PrDA | ✓ | 48.4 | 73.4 | 76.9 | 64.3 | 69.8 | 71.7 | 62.7 | 45.3 | 76.6 | 69.8 | 50.5 | 79.0 | 65.7
SHOT-IM | ✓ | 55.4 | 76.6 | 80.4 | 66.9 | 74.3 | 75.4 | 65.6 | 54.8 | 80.7 | 73.7 | 58.4 | 83.4 | 70.5
BAIT | ✓ | 57.4 | 77.5 | 82.4 | 68.0 | 77.2 | 75.1 | 67.1 | 55.5 | 81.9 | 73.9 | 59.5 | 84.2 | 71.6
CPGA | ✓ | 59.3 | 78.1 | 79.8 | 65.4 | 75.5 | 76.4 | 65.7 | 58.0 | 81.0 | 72.0 | 64.4 | 83.3 | 71.6
SHOT | ✓ | 57.1 | 78.1 | 81.5 | 68.0 | 78.2 | 78.1 | 67.4 | 54.9 | 82.2 | 73.3 | 58.8 | 84.3 | 71.8
PS (ours) | ✓ | 57.8 | 77.3 | 81.2 | 68.4 | 76.9 | 78.1 | 67.8 | 57.3 | 82.1 | 75.2 | 59.1 | 83.4 | 72.1

Table 2: Classification accuracies (%) on the Office-Home dataset (ResNet-50).

Method | Source-free | plane | bicycle | bus | car | horse | knife | mcycl | person | plant | sktbrd | train | truck | Per-class
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
ResNet-101 | ✗ | 55.1 | 53.3 | 61.9 | 59.1 | 80.6 | 17.9 | 79.7 | 31.2 | 81.0 | 26.5 | 73.5 | 8.5 | 52.4
CDAN | ✗ | 85.2 | 66.9 | 83.0 | 50.8 | 84.2 | 74.9 | 88.1 | 74.5 | 83.4 | 76.0 | 81.9 | 38.0 | 73.9
SAFN | ✗ | 93.6 | 61.3 | 84.1 | 70.6 | 94.1 | 79.0 | 91.8 | 79.6 | 89.9 | 55.6 | 89.0 | 24.4 | 76.1
SWD | ✗ | 90.8 | 82.5 | 81.7 | 70.5 | 91.7 | 69.5 | 86.3 | 77.5 | 87.4 | 63.6 | 85.6 | 29.2 | 76.4
TPN | ✗ | 93.7 | 85.1 | 69.2 | 81.6 | 93.5 | 61.9 | 89.3 | 81.4 | 93.5 | 81.6 | 84.5 | 49.9 | 80.4
PAL | ✗ | 90.9 | 50.5 | 72.3 | 82.7 | 88.3 | 88.3 | 90.3 | 79.8 | 89.7 | 79.2 | 88.1 | 39.4 | 78.3
MCC | ✗ | 88.7 | 80.3 | 80.5 | 71.5 | 90.1 | 93.2 | 85.0 | 71.6 | 89.4 | 73.8 | 85.0 | 36.9 | 78.8
CoSCA | ✗ | 95.7 | 87.4 | 85.7 | 73.5 | 95.3 | 72.8 | 91.5 | 84.8 | 94.6 | 87.9 | 87.9 | 36.8 | 82.9
PrDA | ✓ | 86.9 | 81.7 | 84.6 | 63.9 | 93.1 | 91.4 | 86.6 | 71.9 | 84.5 | 58.2 | 74.5 | 42.7 | 76.7
SHOT-IM | ✓ | 93.7 | 86.4 | 78.7 | 50.7 | 91.0 | 93.5 | 79.0 | 78.3 | 89.2 | 85.4 | 87.9 | 51.1 | 80.4
MA | ✓ | 94.8 | 73.4 | 68.8 | 74.8 | 93.1 | 95.4 | 88.6 | 84.7 | 89.1 | 84.7 | 83.5 | 48.1 | 81.6
BAIT | ✓ | 93.7 | 83.2 | 84.5 | 65.0 | 92.9 | 95.4 | 88.1 | 80.8 | 90.0 | 89.0 | 84.0 | 45.3 | 82.7
SHOT | ✓ | 94.3 | 88.5 | 80.1 | 57.3 | 93.1 | 94.9 | 80.7 | 80.3 | 91.5 | 89.1 | 86.3 | 58.2 | 82.9
PS (ours) | ✓ | 95.3 | 86.2 | 82.3 | 61.6 | 93.3 | 95.7 | 86.7 | 80.4 | 91.6 | 90.9 | 86.0 | 59.5 | 84.1

Table 3:  Classification accuracies (%) on the large-scale VisDA dataset
(ResNet-101).

### Datasets

We conduct experiments on three benchmark datasets: (1) Digits is a standard
dataset that focuses on digit recognition. We follow the protocol of (Hoffman,
Tzeng et al. 2018) and choose three subsets: SVHN (S), MNIST (M), and USPS
(U). (2) Office-Home (Venkateswara, Eusebio et al. 2017) is a widely used
dataset, which contains four domains: Artistic images (Ar), Clip Art (Cl),
Product images (Pr) and Real-world images (Rw). Each domain has 65 categories.
(3) VisDA (Peng, Usman et al. 2017) is a challenging large-scale dataset that
concentrates on the synthesis-to-real object recognition task. The source
domain contains 152k synthetic images while the target domain has 55k real
object images with 12 classes.

### Baselines

We compare PS with three types of baselines: (1) Source-only: ResNet (He,
Zhang et al. 2016) or LeNet (LeCun et al. 1998); (2)  UDA methods: ADDA (Tzeng
et al. 2017), ADR (Saito et al. 2018), MCD (Saito, Watanabe et al. 2018), CDAN
(Long, Cao et al. 2018), CyDADA (Hoffman, Tzeng et al. 2018), SAFN (Xu, Li et
al. 2019), SWD (Lee et al. 2019a), TPN (Pan, Yao et al. 2019), CAT (Deng, Luo,
and Zhu 2019), MDD (Zhang, Liu et al. 2019), SWD (Lee et al. 2019b), BDG
(Yang, Xia et al. 2020), PAL (Hu, Liang et al. 2020), MCC (Jin, Wang et al.
2020), BNM (Cui, Wang et al. 2020), CoSCA (Dai, Cheng et al. 2020) and SRDC
(Tang, Chen, and Jia 2020); (3)  SFDA methods: PrDA (Kim, Cho et al. 2020),
SHOT (Liang, Hu et al. 2020), MA (Li, Jiao et al. 2020), BAIT (Yang, Wang et
al. 2020) and CPGA (Qiu et al. 2021).

### Implementation Details

Our method is implemented based on PyTorch. For a fair comparison, we report
the results of all baselines in the corresponding papers. For the network
architecture, a ResNet (He, Zhang et al. 2016) pre-trained on ImageNet is
adopted as the feature extractor of all methods for the object recognition
task. For the digit recognition task, we use the same architectures with CDAN
(Long, Cao et al. 2018), namely, the classical LeNet-5 (LeCun et al. 1998)
network is utilized for USPS$\leftrightarrow$ MNIST and a variant of LeNet is
utilized for SVHN$\rightarrow$ MNIST. Following (Liang, Hu et al. 2020), we
replace the original fully connected (FC) layer with a task-specific FC layer
followed by a weight normalization layer. We train the model using SGD
optimizer with momentum 0.9 and weight decay $5e^{-4}$. The learning rate of
the domain discriminator and the classifier is 10 times that of the feature
extractor, where the former is set to be $1e^{-4}$ for Digits and $1e^{-5}$
for Office-Home and VisDA. Besides, the number of epoch is set to be 200, 800,
and 60 for Digits, Office-Home, and VisDA, respectively. For hyper-parameters,
we set $\alpha$ to be 0.1 for all datasets and batchsize $B$ to be 500, 658,
and 128 for Digits, Office-Home, and VisDA, respectively. We set
$\lambda_{g}=0.5$ and $\lambda_{c}=1$ for all datasets except
$\lambda_{c}=0.3$ for Office-Home.

### Results

For Digits recognition, as shown in Table 1, PS obtains the best mean
accuracies for each task and outperforms prior work in terms of the average
accuracy. The proposed method performs better than SHOT and SHOT-IM, as PS
could take the distribution discrepancy into consideration. It is noticed that
both PS and SHOT achieve better results than the supervised learning for task
M$\rightarrow$U, which is because that the large source domain MNIST contains
more useful information suitable for the small target domain USPS. For Office-
Home dataset, as shown in Table 2, the proposed PS achieves the best
performance compared with other SFDA methods w.r.t. the average accuracy over
four transfer tasks. Moreover, our method shows its superiority in the task of
Cl$\rightarrow$Ar and Rw$\rightarrow$Ar and comparable results on the other
tasks. Moreover, from Table 3, PS outperforms all the state-of-the-art methods
on the more challenging large-scale dataset VisDA. Specifically, PS gets the
best accuracy in the six categories and obtains comparable results in others.
Note that for these three datasets, even compared with the state-of-the-art
methods using source data (e.g.,SWD, SRDC, CoSCA), our PS is able to obtain a
competitive result as well, which shows that PS could make a better adaptation
across domains.

### Insight Analysis

#### Ablation of selection criterion of pseudo-source samples

To study the selection criterion of pseudo-source samples, we compare the
model with two criterion, i.e., entropy criterion (our method) and random
selection (with the same proportion in each class). As shown in Table 4, PS
with entropy criterion (line 3) outperforms random selection (line 1) by 0.5%
in VisDA dataset. We also find that random selection achieves better results
than some SFDA method, as such split strategy can be regraded as reducing the
intra-domain discrepancy within the target domain (Pan et al. 2020).

Method | Digits | VisDA
---|---|---
Random selection with mixup | 98.4 | 83.6
Entropy criterion without mixup | 98.4 | 83.4
Entropy criterion with mixup (ours) | 98.6 | 84.1

Table 4: Ablation study of generation and augmentation on Digits and VisDA
dataset.

#### Ablation of pseudo-source domain augmentation

To explore the impact of data augmentation, we conduct experiments with and
without mixup. From Table 4, compared with other SFDA method, both settings
achieve better performance. In other words, even without mixup (line 2), our
method could also get comparable results. With mixup (line 3), the model gets
a better performance.

#### Ablation of loss functions

To investigate the losses in distribution alignment, the quantitative results
of the model optimized by different losses are shown in Table 5. When only
using classification loss $\mathcal{L}_{cls}$, the model gets better
performance than source-only model as the self-training based strategy could
perform implicit alignment to the source model. Besides, our model is able to
further improve the performance when introducing the losses
$\mathcal{L}_{div}$ and $\mathcal{L}_{con}$. Such a result verifies the
benefit of encouraging diverse outputs and retaining knowledge of source
domain. Finally, a combination of all losses can achieve the best results.

#### Distribution distance

The $\mathcal{A}$-distance is a measure of domain discrepancy (Ben-David et
al. 2009), defined as $d_{A}=2(1-2\epsilon)$, where $\epsilon$ is the error
rate of a domain classifier trained to discriminate source domain and target
domain. To show the true distribution distance across domains, we compute the
$\mathcal{A}$-distance using both true source samples and target samples. Note
that the source samples are only used for computing $\mathcal{A}$-distance,
not used for training. Results on tasks M$\rightarrow$U are shown in Figure 3.
SHOT implicitly aligns representations of the target domain to the source
model, thus getting a smaller distance than source-only method. PS could get
smaller $\mathcal{A}$-distance than SHOT by further reducing the domain shift
explicitly.

Figure 3: $\mathcal{A}$-distance.

Method | Digits | VisDA
---|---|---
Source-only | 79.3 | 46.6
$\mathcal{L}_{cls}$ | 98.3 | 77.6
$\mathcal{L}_{cls}$ \+ $\mathcal{L}_{div}$ | 98.3 | 78.4
$\mathcal{L}_{cls}$ \+ $\mathcal{L}_{div}$ \+ $\mathcal{L}_{cons}$ | 98.4 | 83.0
Ours (all) | 98.6 | 84.1

Table 5: Ablation of losses on Digits and VisDA dataset.

#### Parameter sensitivity

In this subsection, we evaluate the sensitivity of two hyper-parameters
batchsize $B$ and proportion $\alpha$ on Digits. As shown in Figure 4(a), the
results demonstrate that our method is non-sensitive to the batchsize.
Although PS generates and augments the pseudo-source domain within a minibatch
instead of the whole domain, it could achieve stable results because many
random minibatches can be seen as an approximation of the whole domain. The
results in Figure 4(b) show that with the increasing of $\alpha$, the
performance grows firstly and then drops. The reason is that a small
proportion would lead to a small number of pseudo-source samples and a large
proportion would select some samples not similar to the source domain into the
pseudo-source domain. We experimentally find $\alpha=0.1$ works well.

(a) Sensitivity of batchsize $B$ (b) Sensitivity of proportion $\alpha$

Figure 4: Results of parameter sensitivity.

#### Embedding visualization

Figure 5(a) and (b) show the t-SNE visualization of the features from the
pseudo-source domain and the remaining target domain for task M$\rightarrow$U
(10 classes) before and after alignment, respectively. Before alignment, there
exists large distribution shift between the pseudo-source domain and the
remaining target domain. While after alignment the distribution shift is
reduced and the features of target samples have become discriminative, thus,
the samples can be easily classified by the classifier.

Figure 5: t-SNE visualization for features from the pseudo-source domain and
the remaining target domain.

## Conclusions

In this paper, we focus on source-free domain adaptation, where only a trained
source model and unlabeled target samples are given. Considering that previous
SFDA methods do not explicitly reduce the distribution shift across domains,
we propose a novel method to reduce the shift explicitly. To be specific, we
find that some target samples can be used to represent the source domain and
the new domain is denoted as pseudo-source domain. The proposed method firstly
generates and augments the pseudo-source domain and then employs distribution
alignment between the pseudo-source domain and the remaining target domain.
Thus, the domain shift can be reduced. Extensive experiments on three datasets
show that our method outperforms the state-of-the-art methods.

## References

  * Ahmed et al. (2021) Ahmed, S. M.; Raychaudhuri, D. S.; Paul, S.; Oymak, S.; and Roy-Chowdhury, A. 2021\.  Unsupervised Multi-source Domain Adaptation Without Access to Source Data.  _CVPR_ .
  * Ben-David et al. (2009) Ben-David, S.; Blitzer, J.; Crammer, K.; Kulesza, A.; Pereira, F. C.; and Vaughan, J. W. 2009.  A theory of learning from different domains.  _Machine Learning_ 79: 151–175.
  * Chen et al. (2020a) Chen, C.; Fu, Z.; Chen, Z.; Jin, S.; Cheng, Z.; Jin, X.; and Hua, X. 2020a.  HoMM: Higher-order Moment Matching for Unsupervised Domain Adaptation.  In _AAAI_.
  * Chen et al. (2020b) Chen, Q.; Du, Y.; Tan, Z.; Zhang, Y.; and Wang, C.-J. 2020b.  Unsupervised Domain Adaptation with Joint Domain-Adversarial Reconstruction Networks.  In _ECML/PKDD_.
  * Cicek and Soatto (2019) Cicek, S.; and Soatto, S. 2019.  Unsupervised Domain Adaptation via Regularized Conditional Alignment.  _2019 IEEE/CVF International Conference on Computer Vision (ICCV)_ 1416–1425.
  * Cui, Wang et al. (2020) Cui, S.; Wang, S.; et al. 2020.  Towards Discriminability and Diversity: Batch Nuclear-Norm Maximization Under Label Insufficient Situations.  _CVPR_ .
  * Dai, Cheng et al. (2020) Dai, S.; Cheng, Y.; et al. 2020.  Contrastively Smoothed Class Alignment for Unsupervised Domain Adaptation.  In _ACCV_.
  * Deng, Luo, and Zhu (2019) Deng, Z.; Luo, Y.; and Zhu, J. 2019.  Cluster Alignment With a Teacher for Unsupervised Domain Adaptation.  _2019 IEEE/CVF International Conference on Computer Vision (ICCV)_ 9943–9952.
  * Feng et al. (2021) Feng, H.; You, Z.; Chen, M.; Zhang, T.-Y.; Zhu, M.; Wu, F.; Wu, C.; and Chen, W. 2021.  KD3A: Unsupervised Multi-Source Decentralized Domain Adaptation via Knowledge Distillation.  _ICML_ .
  * Ganin and Lempitsky (2015) Ganin, Y.; and Lempitsky, V. 2015.  Unsupervised domain adaptation by backpropagation.  In _ICML_.
  * Ganin et al. (2016) Ganin, Y.; Ustinova, E.; Ajakan, H.; Germain, P.; Larochelle, H.; Laviolette, F.; Marchand, M.; and Lempitsky, V. S. 2016.  Domain-Adversarial Training of Neural Networks.  _J. Mach. Learn. Res._ 17: 59:1–59:35.
  * Gomes, Krause, and Perona (2010) Gomes, R.; Krause, A.; and Perona, P. 2010.  Discriminative Clustering by Regularized Information Maximization.  In _NIPS_.
  * Goodfellow, Pouget-Abadie et al. (2014) Goodfellow, I. J.; Pouget-Abadie, J.; et al. 2014.  Generative Adversarial Networks.  In _NeruIPS_.
  * He, Zhang et al. (2016) He, K.; Zhang, X.; et al. 2016.  Deep Residual Learning for Image Recognition.  _CVPR_ .
  * Hoffman, Tzeng et al. (2018) Hoffman, J.; Tzeng, E.; et al. 2018.  CyCADA: Cycle-Consistent Adversarial Domain Adaptation.  In _ICML_.
  * Hu, Liang et al. (2020) Hu, D.; Liang, J.; et al. 2020.  PANDA: Prototypical Unsupervised Domain Adaptation.  _ArXiv_ .
  * Jin, Wang et al. (2020) Jin, Y.; Wang, X.; et al. 2020.  Minimum Class Confusion for Versatile Domain Adaptation.  In _ECCV_.
  * Kang, Jiang et al. (2019) Kang, G.; Jiang, L.; et al. 2019.  Contrastive Adaptation Network for Unsupervised Domain Adaptation.  _CVPR_ .
  * Kim, Cho et al. (2020) Kim, Y.; Cho, D.; et al. 2020.  Progressive Domain Adaptation from a Source Pre-trained Model.  _ArXiv_ .
  * Kumar et al. (2018) Kumar, A.; Sattigeri, P.; Wadhawan, K.; Karlinsky, L.; Feris, R.; Freeman, W.; and Wornell, G. 2018.  Co-regularized Alignment for Unsupervised Domain Adaptation.  _NeurIPS_ .
  * Kundu et al. (2020) Kundu, J. N.; Venkat, N.; RahulM., V.; and Babu, R. V. 2020.  Universal Source-Free Domain Adaptation.  _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ 4543–4552.
  * Kurmi, Subramanian, and Namboodiri (2021) Kurmi, V.; Subramanian, V. K.; and Namboodiri, V. P. 2021.  Domain Impression: A Source Data Free Domain Adaptation Method.  _2021 IEEE Winter Conference on Applications of Computer Vision (WACV)_ 615–625.
  * LeCun et al. (1998) LeCun, Y.; Bottou, L.; Bengio, Y.; and Haffner, P. 1998.  Gradient-based learning applied to document recognition.
  * Lee et al. (2019a) Lee, C.-Y.; Batra, T.; Baig, M. H.; and Ulbricht, D. 2019a.  Sliced wasserstein discrepancy for unsupervised domain adaptation.  In _CVPR_.
  * Lee et al. (2019b) Lee, C.-Y.; Batra, T.; Baig, M. H.; and Ulbricht, D. 2019b.  Sliced Wasserstein Discrepancy for Unsupervised Domain Adaptation.  _2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ 10277–10287.
  * Li et al. (2020) Li, J.; Chen, E.; Ding, Z.; Zhu, L.; Lu, K.; and Shen, H. T. 2020.  Maximum Density Divergence for Domain Adaptation.  _IEEE transactions on pattern analysis and machine intelligence_ .
  * Li, Jiao et al. (2020) Li, R.; Jiao, Q.; et al. 2020.  Model adaptation: Unsupervised domain adaptation without source data.  In _CVPR_.
  * Li et al. (2021) Li, X.; Chen, W.; Xie, D.; Yang, S.; Yuan, P.; Pu, S.; and Zhuang, Y. 2021.  A Free Lunch for Unsupervised Domain Adaptive Object Detection without Source Data.  In _AAAI_.
  * Li and Hoiem (2018) Li, Z.; and Hoiem, D. 2018.  Learning without Forgetting.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ 40: 2935–2947.
  * Liang, Hu et al. (2020) Liang, J.; Hu, D.; et al. 2020.  Do We Really Need to Access the Source Data? Source Hypothesis Transfer for Unsupervised Domain Adaptation.  In _ICML_.
  * Liu, Zhang, and Wang (2021) Liu, Y.; Zhang, W.; and Wang, J. 2021.  Source-Free Domain Adaptation for Semantic Segmentation.  _CVPR_ .
  * Long et al. (2015) Long, M.; Cao, Y.; Wang, J.; and Jordan, M. I. 2015.  Learning Transferable Features with Deep Adaptation Networks.  _ICML_ .
  * Long, Cao et al. (2018) Long, M.; Cao, Z.; et al. 2018.  Conditional adversarial domain adaptation.  In _NeruIPS_.
  * Long et al. (2017) Long, M.; Zhu, H.; Wang, J.; and Jordan, M. I. 2017.  Deep Transfer Learning with Joint Adaptation Networks.  In _ICML_.
  * Müller, Kornblith et al. (2019) Müller, R.; Kornblith, S.; et al. 2019.  When Does Label Smoothing Help?  In _NeurIPS_.
  * Pan et al. (2020) Pan, F.; Shin, I.; Rameau, F.; Lee, S.; and Kweon, I. 2020.  Unsupervised Intra-Domain Adaptation for Semantic Segmentation Through Self-Supervision.  _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ 3763–3772.
  * Pan and Yang (2010) Pan, S. J.; and Yang, Q. 2010.  A Survey on Transfer Learning.  _IEEE Transactions on Knowledge and Data Engineering_ 22: 1345–1359.
  * Pan, Yao et al. (2019) Pan, Y.; Yao, T.; et al. 2019.  Transferrable prototypical networks for unsupervised domain adaptation.  In _CVPR_.
  * Parisi et al. (2019) Parisi, G. I.; Kemker, R.; Part, J. L.; Kanan, C.; and Wermter, S. 2019.  Continual Lifelong Learning with Neural Networks: A Review.  _Neural networks_ 113: 54–71.
  * Peng, Usman et al. (2017) Peng, X.; Usman, B.; et al. 2017.  VisDA: The Visual Domain Adaptation Challenge.  _ArXiv_ .
  * Qiu et al. (2021) Qiu, Z.; Zhang, Y.; Lin, H.; Niu, S.; Liu, Y.; Du, Q.; and Tan, M. 2021.  Source-free Domain Adaptation via Avatar Prototype Generation and Adaptation.  _IJCAI_ .
  * Ren et al. (2015) Ren, S.; He, K.; Girshick, R. B.; and Sun, J. 2015.  Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ 39: 1137–1149.
  * Saito, Ushiku, and Harada (2017) Saito, K.; Ushiku, Y.; and Harada, T. 2017.  Asymmetric Tri-training for Unsupervised Domain Adaptation.  In _ICML_.
  * Saito et al. (2018) Saito, K.; Ushiku, Y.; Harada, T.; and Saenko, K. 2018.  Adversarial Dropout Regularization.  _ICLR_ .
  * Saito, Watanabe et al. (2018) Saito, K.; Watanabe, K.; et al. 2018.  Maximum classifier discrepancy for unsupervised domain adaptation.  In _CVPR_.
  * Shen et al. (2018) Shen, J.; Qu, Y.; Zhang, W.; and Yu, Y. 2018.  Wasserstein Distance Guided Representation Learning for Domain Adaptation.  In _AAAI_.
  * Sun, Feng, and Saenko (2016) Sun, B.; Feng, J.; and Saenko, K. 2016.  Return of Frustratingly Easy Domain Adaptation.  In _AAAI_.
  * Tang, Chen, and Jia (2020) Tang, H.; Chen, K.; and Jia, K. 2020.  Unsupervised Domain Adaptation via Structurally Regularized Deep Clustering.  In _CVPR_.
  * Tzeng et al. (2017) Tzeng, E.; Hoffman, J.; Saenko, K.; and Darrell, T. 2017.  Adversarial Discriminative Domain Adaptation.  _CVPR_ .
  * Venkateswara, Eusebio et al. (2017) Venkateswara, H.; Eusebio, J.; et al. 2017.  Deep Hashing Network for Unsupervised Domain Adaptation.  _CVPR_ .
  * Xu, Li et al. (2019) Xu, R.; Li, G.; et al. 2019.  Larger norm more transferable: An adaptive feature norm approach for unsupervised domain adaptation.  In _ICCV_.
  * Yang, Xia et al. (2020) Yang, G.; Xia, H.; et al. 2020.  Bi-Directional Generation for Unsupervised Domain Adaptation.  In _AAAI_.
  * Yang, Wang et al. (2020) Yang, S.; Wang, Y.; et al. 2020.  Unsupervised Domain Adaptation without Source Data by Casting a BAIT.  _ArXiv_ .
  * Zellinger et al. (2017) Zellinger, W.; Grubinger, T.; Lughofer, E.; Natschläger, T.; and Saminger-Platz, S. 2017.  Central Moment Discrepancy (CMD) for Domain-Invariant Representation Learning.  _ICLR_ .
  * Zhang et al. (2018) Zhang, H.; Cissé, M.; Dauphin, Y.; and Lopez-Paz, D. 2018.  mixup: Beyond Empirical Risk Minimization.  _ICLR_ .
  * Zhang, Liu et al. (2019) Zhang, Y.; Liu, T.; et al. 2019.  Bridging Theory and Algorithm for Domain Adaptation.  In _ICML_.
  * Zhao et al. (2021) Zhao, Y.; Zhong, Z.; Luo, Z.; Lee, G. H.; and Sebe, N. 2021.  Source-Free Open Compound Domain Adaptation in Semantic Segmentation.  _ArXiv_ abs/2106.03422.


# Visibility-Inspired Models of Touch Sensors for Navigation

Kshitij Tiwari1, Basak Sakcak1, Prasanna Routray2, Manivannan M.2, and Steven
M. LaValle1 1Center of Ubiquitous Computing, Faculty of Information Technology
and Electrical Engineering, University of Oulu, Oulu, Finland,
<EMAIL_ADDRESS>Lab, Center for Virtual Reality and Haptics,
Indian Institute of Technology Madras, IndiaThis work was supported by a
European Research Council Advanced Grant (ERC AdG, ILLUSIVE: Foundations of
Perception Engineering, 101020977), Academy of Finland (projects PERCEPT
322637, CHiMP 342556), and Business Finland (project HUMOR 3656/31/2019).

###### Abstract

This paper introduces mathematical models of touch sensors for mobile robots
based on visibility. Serving a purpose similar to the pinhole camera model for
computer vision, the introduced models are expected to provide a useful,
idealized characterization of task-relevant information that can be inferred
from their outputs or observations. Possible tasks include navigation,
localization and mapping when a mobile robot is deployed in an unknown
environment. These models allow direct comparisons to be made between
traditional depth sensors, highlighting cases in which touch sensing may be
interchangeable with time of flight or vision sensors, and characterizing
unique advantages provided by touch sensing. The models include contact
detection, compression, load bearing, and deflection. The results could serve
as a basic building block for innovative touch sensor designs for mobile robot
sensor fusion systems.

##  I Introduction

(a) Roller lever switch

(b) Bumper switch

(c) Pogo pin

(d) LVDT

(e) Rat vibrissae

Figure 1: Various types of touch sensors. 1(a) and 1(b) are common contact
detectors; 1(c) and 1(d) are typical compressible touch sensors (Images from
e-ptnk.co.kr/eng/pogo-pin/ and hoffmann-krippner.com/types-of-lvdts/,
respectively). LVDT stands for Linear Variable Differential Transducer; 1(e)
shows macro- and microvibrissae in rats [1].

Touch sensor designs have evolved over the past four decades yet the touch
modality is not as deeply explored as vision and auditory. The existing touch
sensors are often inspired by touch receptors and are aimed at robotic and
industrial applications which typically require human-like prehensile
manipulation capabilities for the robots to work safely around humans. Beyond
manipulation, in nature, mammals (like rats, shrews, pinnipeds etc.) and
fishes use the touch modality for navigation utilizing pre- and post-contact
touch feedback. Thus, in this work, we look further to explore the possibility
of using the touch modality for navigation with mobile robots.

For navigation in unknown and unstructured environments researchers typically
resort to distal sensors such as cameras and Lidars. The utility of such
sensors may seem limiting when it comes to cases where a robot needs to
navigate in dimly lit environments. Similarly, soft terrains present wheel
slippage and sinkage challenges for ground robots and the space exploration
community can benefit from such solutions [2]. Although vision-based solutions
such as [3] exist, these approaches are often prone to failure under sudden
illumination changes and dust. Similarly, when inspecting underwater mines
using an autonomous marine vehicle, the murky and rough waters may even harm
the optical sensors [4]. Few researchers have started looking into potential
use cases of vibrissae-enabled robots [5]. Rodents use their vibrissae, a form
of touch sensor, to detect both tactile and kinesthetic features [6]. Although
tactile features help infer the surface profile through textural perception
[1], kinesthetic features are perceived through proprioception [7], which can
be thought of as self-localization.

In this work, we investigate the utility of touch sensors mounted on a mobile
robot for proprioception. Some touch sensors provide very limited information
such as the bumper sensors (see Figs. 1(a), 1(b)) whereas others could capture
a lot more information about the objects such as their surface profile. As
surface profile provides a qualitative idea about the environment, profile
sensing with touch sensors can augment the path planning capability for mobile
robots. We consider two types of touch sensors- rigid and compliant where
compliance is limited to their ability to compress (change its volume under
applied transverse load, as in Figs. 1(c), 1(d)) or bend (similar to the
vibrissae shown in Fig. 1(e)). Compressible touch sensors can be used for
motion planning for collision resilient robots as was shown in [8] and hence,
we consider compressibility as a desirable form of compliance for touch
sensors. Also, we are interested in contrasting the quality of information
between actuated and unactuated touch sensors. For instance, in [9], an
actuated probe (rigid link) was used for terrain identification to guide the
otherwise blind robot through an environment while avoiding dangerous patches.
In [10], the researchers showed how actuation directly impacts object
localization. They considered two variants of a whisker-like touch sensor
array- one actuated like the motile macrovibrissae and the other static like
the immotile microvibrissae. They showed that the static touch sensor had
relatively poor object localization accuracy, and in this work we analyze such
claims in terms of preimages.

To contrast various touch sensors, we resort to virtual sensor models from
[11]. The aim here is to compare the sensors in terms of their preimages, and
develop mathematical models independent of their physical realization. This
could elucidate various aspects of touch sensors to increase the task success
rate when deploying mobile robots in unstructured environments for tasks such
as navigation, localization or mapping.

##  II Virtual touch sensor models

There is a wide spectrum of touch sensors available in the literature, some of
which were discussed above. But the question remains, given any two touch
sensors, how does one know which is superior compared to the other? A coarse
elimination is possible given some prior knowledge of the downstream task, but
then given a family of sensors suited to the task, how does one narrow down to
an optimal sensor with respect to a particular relevant criterion? Every
transducer and design has strengths and weaknesses making it difficult to do a
fair comparison. To facilitate this discussion, here we discuss the concept of
virtual sensors from [11]. A virtual sensor is a mathematical abstraction of
the sensor, different from the models that explain its physics, and
independent of its physical realization thereby allowing a fair comparison. In
what follows, we will first describe the concept of sensor mapping followed by
formally defining the state spaces. Then, we present various virtual touch
sensors followed by a discussion on compositions of such models and their
comparison with conventional visibility-based sensors.

###  II-A Sensor Mapping

Even though the mathematical model that explains the physics of a sensor
varies with its design we can define a virtual sensor [11] as a mapping,

$h:X\rightarrow Y,$ (1)

from the robot’s state space $X$ to the observation space $Y$. The state space
$X$ refers to the set of all the states that the robot can be in and the
observation space $Y$ is the set of all possible observations (measurements)
that a sensor can make. In the following, we will refer to $h$ as the sensor
mapping. In order to lighten the notation, in the following sections, when
describing some sensors, we will use $h(x;m)$ to describe a sensor mapping for
which $m$ are the parameters.

Typically a sensor mapping is not bijective; whereas each $x\in X$ maps to a
single $y\in Y$, multiple states can give out the same observation. Therefore,
in the general case, the sensor mapping is not invertible and the preimage of
$y$ under $h$, written as $h^{-1}(y)$, is defined as the set,

$h^{-1}(y)=\\{x\in X\ |\ y=h(x)\\}.$ (2)

Note that if the sensor mapping is bijective, then Eq. (2) corresponds to the
inverse of the function $h$. As $h$ is defined over the state space $X$, the
subsets of $X$ corresponding to the preimages of $h$ form a partition of $X$
denoted by $\Pi(h)$.

Consider two sensors, $h_{1}$ and $h_{2}$, defined over the same fixed state
space $X$ that has different observation spaces $Y_{1}$ and $Y_{2}$. If the
partition $\Pi(h_{1})$ is a refinement of $\Pi(h_{2})$ then $h_{1}$ can
simulate $h_{2}$ which means that there exists a function $g:Y_{1}\rightarrow
Y_{2}$ such that $h_{2}(x)=g(h_{1}(x))$, written as $h_{2}=g\circ h_{1}$ [11].
Two sensors are equivalent, denoted by $h_{1}\cong h_{2}$, if the partition
induced by the sensors, $\Pi(h_{1})$ and $\Pi(h_{2})$, respectively, satisfies
$\Pi(h_{1})=\Pi(h_{2})$. If two sensors are equivalent they can simulate each
other.

###  II-B State Space

We consider a mobile robot moving in a 2D planar environment $W$ which is the
closure of the open set that is piecewise diffeomorphic to a finite disjoint
union of circles. The boundary of $W$ is denoted as $\partial W$. The robot is
equipped with a touch sensor which can be seen as a link attached to the robot
base whose characteristics vary across different models. However, the
discussion in this work is limited to touch sensors that are either rigid or
compliant that too are either compressible or bendable.

We will refer to three coordinate frames and the corresponding homogeneous
transformations between them as shown in Fig 2a. We express the configurations
of the touch sensor and the robot as $q_{s}$ and $q_{r}$, respectively. The
robot configuration $q_{r}$ is defined by $q_{r}=(q_{x},q_{y},q_{\theta})$, in
which $(q_{x},q_{y})\in W$ and $q_{\theta}\in S^{1}$ are the position and the
orientation of the robot with respect to the world frame, respectively. The
world frame corresponds to an absolute reference frame with respect to which
the robot configuration is defined. The robot frame and the sensor frame are
the reference frames attached to the robot and the touch sensor, respectively.
The homogeneous transform between the robot frame and the world frame is given
by ${T}_{rw}(q_{r})$ and varies with the configuration of the robot $q_{r}$.
As for the touch sensor in its nominal form, it is aligned with the x-axis of
the sensor frame (see Figs. 2b and c) which will be referred as the sensor
axis from hereon. For unactuated touch sensors, the sensor frame is fixed with
respect to the robot frame and ${T}_{sr}$ is the homogeneous transform between
two coordinate frames. To simplify the notation, we will assume that the robot
frame and the sensor frame coincide for unactuated touch sensors, such that
${T}_{sr}$ is an identity matrix. In cases that the touch sensor is actuated,
the homogeneous transform between the sensor and robot frame is given by
${T}_{sr}(\psi)$ in which $\psi\in S^{1}$ is the amount of rotation of the
sensor frame with respect to the robot frame.

We now define the configurations of compliant touch sensors. Compressible
touch sensors can compress along the sensor axis, similar to a linear spring
or a link attached to a prismatic joint (like Figs. 1(c) and 1(d)). The sensor
configuration is expressed by the variable $q_{s}=\delta$ and the robot state
is the tuple $x=(q_{r},\delta)$ in which $\delta\in[0,\delta_{max}]$ is the
amount of compression along the sensor axis (as shown in Fig. 2b). For the
bendable touch sensors (like Figs. 1(b) and 1(e)) we assume that the
deflection from the nominal form is parameterized by
$q_{s}=\alpha\in[\alpha_{min},\alpha_{max}]$ in which
$\alpha_{min}<0<\alpha_{max}$. If $\alpha=0$ then the touch sensor is in its
nominal form (see Fig. 2c). Thus, each $\alpha$ corresponds to a
diffeomorphism that maps the points along the touch sensor at its nominal
shape to the ones corresponding to $\alpha\not=0$. The robot state $x$ is the
robot configuration and the sensor configuration, i.e., $x=(q_{r},\alpha)$.

If the sensor can not change its configuration by itself but conforms to the
environment, $\delta$ and $\alpha$ are not free variables. Consequently, their
value can be derived from the robot configuration $q_{r}$ and the environment
description $W$, similar to closed kinematic chains. To this end, for each
type of sensor, we can define a contact-constraint equation. For a
compressible touch sensor,

$\delta=f_{comp}(q_{r},\partial W)$ (3)

is the amount of compression for a given environment and robot configuration.
Similarly,

$\alpha=f_{bend}(q_{r},\partial W)$ (4)

describes the amount of deflection. The solution to Eq. (3) is unique, meaning
that for each $q_{r}$ we can uniquely determine the amount of compression.
However, Eq. (4) can admit multiple solutions.

Finally, let $\mathcal{R}(x)\subset\mathbb{R}^{2}$ denote the points occupied
by the robot in the world frame, equipped with the touch sensor at state $x$.
The robot state space $X$ is the set of all the states that the robot can be
in without collisions. The robot is in collision if $\mathcal{R}(x)\setminus
W\not=\emptyset$.

Figure 2: (a): Transformation between the world (fixed), robot and sensor
frames. The function $T$ represents a homogeneous transformation. The
transformation from robot frame to world frame is given by ${T}_{rw}(q_{r})$,
and the static transform from the sensor frame to robot frame is given by
${T}_{sr}$. For actuated touch sensors the sensor to robot frame transform is
given by ${T}_{sr}(\psi)$ in which $\psi$ represents the rotation of the
sensor frame with respect to the robot frame. (b) Various configurations of
compressible and (c) bendable touch sensor.

###  II-C Sensor models

We now introduce a family of touch sensors spanning rigid and compliant touch
sensors. Although the discussion focuses on a single unactuated touch sensor,
the models presented herein can easily be extended to actuated touch sensors
as will be discussed in Sec. III.

We first consider contact detectors such that the sensor is not compliant with
the environment and it is fixed with respect to the robot (see, for example,
Fig. 1(a)). The state in this case can be expressed solely by the robot
configuration, i.e., $x=q_{r}$.

######  Model 1 (Contact detector)

Let $p\in\mathbb{R}^{2}$ be the position of the contact detector whose
coordinates are expressed in the sensor frame. The sensor mapping for the
contact detector is

$\displaystyle h_{cd}(x;p)$ $\displaystyle=\begin{cases}1&\text{if
}{T}_{rw}(q_{r}){T}_{sr}\,\begin{pmatrix}p&1\end{pmatrix}^{T}\in\partial W\\\
0&\text{otherwise}.\end{cases}$ (5)

The preimages corresponding to $h_{cd}$ result in a partition $\Pi(h_{cd})$ of
two classes.

The next model describes a touch sensor when the contact can be detected along
the touch sensor whose geometry is defined as the curve
$\sigma:[0,1]\rightarrow\mathbb{R}^{2}$ which maps to coordinates in the
sensor frame.

######  Model 2 (Contact detector strip)

Suppose there are $N$ interior disjoint intervals, $I_{i}$, $i=1,\dots,N$ such
that $\bigcup_{i=1}^{N}I_{i}=[0,1]$. Let the end points of $N$ intervals
correspond to a uniform discretization of $[0,1]$ imposing a particular
resolution. Let $\mathcal{I}$ denote the set of $N$ intervals and let
$\mathcal{I}_{C}$ be the set of intervals that are in contact with the
boundary such that $\mathcal{I}_{C}=\\{I\in\mathcal{I}\mid\exists
p\in\sigma(I)\text{ s.t. }\,h_{cd}(x,p)=1\\}\subseteq\mathcal{I}$. Then, the
contact detector strip is described by

$h_{cds}(x,\sigma)=\mathcal{I}_{C}.$ (6)

For the limiting case, as $N\rightarrow\infty$, the observation corresponds to
the exact location (or locations) of contact along the strip. At the other
extreme, when $N=1$, the whole strip reduces to just one interval reporting
whether a contact is made anywhere along the strip.

Next, we discuss a touch sensor that complies in response to touch by
compressing along the sensor axis. The robot state is given by
$x=(q_{r},\delta)$ in which $\delta\in[0,\delta_{max}]$ is the amount of
compression.

######  Model 3 (Compression sensor)

The sensor which can detect the amount of compression in the touch sensor is
described by

$h_{cs}(x)=\delta.$ (7)

The preimage of an observation $y=h_{cs}(x)>0$ is typically a union of
disjoint two-dimensional subsets of the state space. Suppose the length of the
uncompressed touch sensor is $\ell$. Let $P_{\theta}\subset W$ be the set of
all points $u\in W$ for which there exists a $v\in\partial W$ such that a line
segment $\overline{uv}$ of length $l-\delta$ and slope $\theta$ satisfies
$\overline{uv}\setminus W=\emptyset$. Then $h_{cs}^{-1}(y)=\bigcup_{\theta\in
S^{1}}P_{\theta}\times\\{\theta\\}\times\\{y\\}$. The preimage of $y=0$ is all
the states such that the touch sensor of length $\ell$ is not in collision.
Thus, the states for which the touch sensor is in contact with the boundary
and the ones such that the touch sensor lies completely in the interior of $W$
belong to the same equivalence class and are indistinguishable.

Yet another form of compliance is the ability to bend under load. For this,
consider the bendable touch sensors for which the state is $x=(q_{r},\alpha)$.

######  Model 4 (Load sensor)

This sensor can detect the amount of shear force applied to the compliant
bendable touch sensor. This model addresses the transverse point load, a force
applied at a single point along the touch sensor . The sensor mapping is
described by

$h_{ls}(x)=F,$ (8)

and $F\in[-F_{max},F_{max}]$, in which $F_{max}$ is the maximum bearable load.

The next model describes the state of the robot in terms of
$x=(q_{r},\alpha)$, in which $\alpha$ is the deflection of the touch sensor
upon contact. The load applied, physical dimension, and material property
together determine the deflection.

######  Model 5 (Deflection sensor)

The sensor which can detect the amount of deflection in the touch sensor is
described by

$h_{ds}(x)=\alpha.$ (9)

For each $\alpha\in[\alpha_{min},\alpha_{max}]$, the touch sensor has the
shape described by the curve $\sigma_{\alpha}:[0,1]\rightarrow\mathbb{R}^{2}$
which maps to the coordinates expressed in the sensor frame. The preimages for
$y=h_{ds}(x)\neq 0$ is all states such that intersection of $\partial W$ and
the image of $\sigma_{\alpha}$ (coordinates mapped to the world frame) is not
an empty set and that $\mathcal{R}(x)\setminus W=\emptyset$. This corresponds
to a union of disjoint 2D or 3D (in case multiple orientations are possible to
touch the boundary at some point along the sensor) subsets of the state space.
Also for Model 5, for $y=0$, the states for which the touch sensor is in
contact with the boundary and the ones such that the touch sensor lies
completely in the interior of $W$ belong to the same equivalence class and are
indistinguishable.

###  II-D Composition of virtual touch sensors

Recall that the preimage corresponding to the observation $y=0$ for Models 3
and 5 include also the states at which the sensor is not in contact with the
environment which creates ambiguity. This can simply be alleviated by adding a
contact detector at the touch sensor tip. To this end, Model 3 can be extended
by combining it with Model 1 such that

$h^{\prime}_{cs}(q_{r},\delta)=\begin{cases}\delta&\text{if
}\,h_{cd}(q_{r};p_{t})=1\\\ \\#&\text{otherwise},\end{cases}$ (10)

in which $\\#$ refers to no value and $p_{t}$ is the point corresponding to
the tip of the touch sensor whose exact location in sensor frame is
$p_{t}=(\ell-\delta,0)$. The same strategy can be followed also for bendable
touch sensor corresponding to a deflection sensor (Model 5) in which case
$p_{t}=\sigma_{\alpha}(1)$ and the respective sensor mapping is

$h^{\prime}_{ds}(q_{r},\alpha)=\begin{cases}(\alpha,1)&\text{if
}\,h_{ds}(q_{r};p_{t})=1\\\ (\alpha,0)&\text{otherwise}.\end{cases}$ (11)

On the other hand, the main issue with bendable touch sensors is the ambiguity
regarding the point of contact. Unlike compressible touch sensors for which
the contact needs to be at the tip to get a non-zero observation, bendable
touch sensors can be in contact with the environment at any point along the
sensor and still achieve that. To refine the preimages, we can enhance
$h_{ds}$ (Model 5) with a contact detector strip (Model 2). Suppose that the
contact detector strip can conform to the shape assumed by the bendable touch
sensor (like an elastic strip). The new sensor model is described as

$h_{bc}(q_{r},\alpha)=h_{cds}(q_{r};\sigma_{\alpha}),$ (12)

in which $\sigma_{\alpha}$ is the curve corresponding to the deflection
determined by $\alpha=h_{ds}(x)$. The observation vector $y=h(x)$ would then
correspond to the intervals (or points if $K=\infty$) such that the bendable
touch sensor is in contact with $\partial W$. The partition of $X$ resulting
from the preimages of $h$, i.e., $\Pi(h_{bc})$, will be a refinement of
$\Pi(h_{ds})$ such that each element of $\Pi(h_{bc})$ is a union of disjoint
two-dimensional subsets of $X$.

Figure 3: A bendable touch sensor can be modeled as a simply supported
cantilever beam of length $\ell$. An intermediate load $F$ is applied at a
distance $a\leq\ell$ from the base in the positive direction of sensor axis.
Then, $\gamma(s)$ is the deflection at any point $s$ along the sensor such
that $\gamma(0)=0$ and maximum deflection is attained at $\ell$.

Suppose the bendable sensor is in contact with the environment at a single
point and let $h_{bc}$ be the contact detector strip defined in Eq. (12) that
reports the exact point of contact ($N=\infty$). Let $a=h_{bc}(x)$ be the
point that is in contact with the boundary (coordinates expressed in the
sensor frame), and let $F=h_{ls}(x)$ be the load acting on the sensor. If we
consider only small deflections that satisfy the assumptions of classical beam
theory, we can model the deflection from the nominal sensor shape as a
function of the distance along the sensor length. Then, deflection $\gamma(s)$
(see Fig 3) at any $s\in[0,1]$ under load $F=h_{ls}(x)$ can be calculated as
[12]

$\displaystyle\gamma(s)=\begin{cases}\frac{-F(s\ell)^{2}(3a-s\ell)}{6EI}&\text{if
}\,0\leq s<\frac{a}{\ell}\\\ \frac{-Fa^{2}(3s\ell-a)}{6EI}&\text{if
}\,\frac{a}{\ell}\leq s<1\\\ \frac{Fa^{2}(3\ell-a)}{6EI}&\text{if }\,s=1,\\\
\end{cases}$ (13)

in which $E\,,I$ stand for the Young’s modulus and moment of inertia,
respectively, and $\ell$ is the sensor length. Note that the maximum
deflection happens at the tip of the touch sensor. By considering small
deflections, we also assume that the projection of the sensor tip on the
horizontal axis of the sensor frame corresponds to $\ell$ and that the length
of the sensor under load, that is, $\ell^{\prime}$ satisfies
$\ell^{\prime}\approx\ell$. Consequently, we can define the curve
corresponding to the bendable touch sensor as
$\sigma_{\alpha}(s)=(s\ell,\gamma(s))$ in which $\alpha=h_{ds}(x)$. This
provides an example when the composition of a load sensor and a contact
detector sensor in Eq. (12) is equivalent to a deflection sensor (Model 5).

##  III Comparison to visibility-based models

In this section, we compare touch sensors from Sec. II-C with traditional
depth sensors from [11] and evaluate conditions under which these are
equivalent with respect to the definitions introduced in Sec. II-B.

###  III-A Virtual depth sensors

Relevant depth sensor models are presented here for completeness. Let
$Q=W\times S^{1}$ be the robot configuration space, that is, the set of all
possible configurations of a point robot111Note that we do not need to assume
a point robot, and $Q$ can as well be the set of all robot configurations such
that the robot footprint is contained in $W$., over which a depth sensor is
defined.

######  Model 6 (Depth-limited directional depth sensor)

Let $b(q_{r})\in\partial W$ denote the point struck by the ray emanating from
$(q_{x},q_{y})$ in the direction of $q_{\theta}$. The sensor model is given by

$h_{dds}(q_{r})=\begin{cases}d(q_{r})&\text{if }d_{min}\leq d(q_{r})\leq
d_{max}\\\ \\#&\text{otherwise,}\end{cases}$ (14)

in which $d(q_{r})=||(q_{x},q_{y})-b(q_{r})||$ and $[d_{min},d_{max}]$ with
$d_{min}<d_{max}$ is the sensor detection range.

Similar to Model 6, we can define a K-directional depth sensor. Suppose there
is a set of offset angles $\phi_{1},\ldots,\phi_{K}$, which are oftentimes
evenly spaced. The sensor mapping for such a sensor is then given by
$h_{kdd}(q_{r})=(y_{1},\dots,y_{K})$ in which
$y_{k}=||(q_{x},q_{y})-b(q_{x},q_{y},q_{\theta}+\phi_{k})||$ for every offset
angle $\phi_{k\in\\{1,\ldots,K\\}}$.

######  Model 7 (Depth-limited omnidirectional depth sensor)

In the limit case, as $K\rightarrow\infty$, we obtain an omnidirectional depth
sensor. In this case, the observation $h_{od}(q_{r})=y$ is an entire function
$y:[0,2\pi)\rightarrow[d_{min},d_{max}]\cup\\{\\#\\}$ and

$y(\phi)=h_{dds}(q_{x},q_{y},q_{\theta}+\phi).$ (15)

###  III-B Unactuated touch sensors

For compressible and bendable touch sensors, the state space $X$ is a subset
of the Cartesian product of the robot configuration space and possible touch
sensor configurations. As the two families of sensors (depth sensors and touch
sensors) do not share the same state space, they cannot be compared directly
based on the previously introduced definitions of equivalence and sensor
simulation. However, in the following, we will use a map from the state space
to the robot configuration space to argue about these relations. Let
$\texttt{proj}:X\rightarrow Q$ be a function that maps the state to its
elements corresponding to the robot configuration. Consider a touch sensor
$h_{1}$ and denote the partition induced by its preimages as $\Pi(h_{1})$.
Suppose proj is bijective. Then, the projection of the elements of each set in
$\Pi(h_{1})$ onto $Q$ forms a partition of $Q$, denoted as $\Pi_{Q}(h_{1})$. A
touch sensor, $h_{1}$, defined over $X$ and a depth sensor, $h_{2}$, defined
over $Q$ are equivalent if the partitioning of the robot configuration space
induced by the two sensors, $\Pi_{Q}(h_{1})$ and $\Pi(h_{2})$, respectively,
satisfy $\Pi_{Q}(h_{1})=\Pi(h_{2})$. Similarly, $h_{1}$ can simulate $h_{2}$
if they are equivalent or if $\Pi_{Q}(h_{1})$ is a refinement of $\Pi(h_{2})$.

Since the contact constraint in Eq. (3) admits a unique solution, the mapping
proj from the state space to robot configuration space is bijective for a
compressible touch sensor, that is, there is a single state that the robot can
be in for each robot configuration. Hence, if the compressible touch sensor is
fully compressible, i.e., $\delta_{max}=\ell$, in which $\delta_{max}$ is the
maximum attainable compression and $\ell$ is the sensor length, the projection
of the elements in $\Pi(h_{cs})$ to $Q$ forms a partition $\Pi_{Q}(h_{cs})$ of
$Q$. Note that, if the touch sensor is not fully compressible, that is,
$\delta_{max}<\ell$, then $\texttt{proj}(X)\subset Q$, meaning that there are
some configurations that are not collision-free for a robot carrying a
compressible touch sensor.

We now consider a fully compressible sensor of length $\ell$ together with a
contact detector at the sensor tip, that is, $h^{\prime}_{cs}$ as given in Eq.
(10), and a depth-limited directional depth sensor $h_{dds}$ (Model 6) with
$d_{min}=0$ and $d_{max}=\ell$. The observation spaces $Y_{cs}$ and $Y_{dds}$
corresponding to $h^{\prime}_{cs}$ and $h_{dds}$ are
$Y_{cs}=Y_{dds}=[0,\ell]$. Suppose the touch sensor frame is aligned with the
robot frame such that $\psi=0$ and ${T}_{sr}(\psi)$ is the identity matrix
(see Fig. 2). Then, we can define a function $g:Y_{cs}\rightarrow Y_{dds}$
such that $h_{dds}(q_{r})=(g\circ h_{cs})(q_{r},\delta)=\ell-
h_{cs}(q_{r},\delta)$ which implies that
$\Pi_{Q}(h^{\prime}_{cs})=\Pi(h_{dds})$ and proves that the following
proposition holds.

###### Proposition 1

A fully compressible touch sensor $h^{\prime}_{cs}$ of length $\ell$ as
described in Eq. (10) can simulate (in the sense from Section II-A) a depth-
limited directional depth sensor with $d_{min}=0$ and $d_{max}=\ell$.

For the limiting case, as $\ell\rightarrow\infty$, a fully retractable
compression sensor of length $\ell$ can simulate a directional depth sensor
($d_{max}=\infty)$, which returns the distance to the closest point on the
environment boundary along the direction of $q_{\theta}$. At the other
extreme, for $\ell=0$, $h_{cs}$ is merely a contact detector $h_{cd}$ (Model
1) with $p=(0,0)$. Furthermore, it follows from Proposition 1 that an array of
of $K$ fully compressible touch sensors with length $\ell$, arranged at angles
$\psi_{1},\dots,\psi_{K}$ with respect to the robot reference frame such that
simultaneous observations yield the vector $(y_{1},\dots,y_{K})$ can simulate
a depth-limited K-directional depth sensor $h_{kdd}$ described in Sec. III-A.

For a bendable touch sensor, the solution to the contact constraint in Eq. (4)
may not be unique. For instance, consider the scenario as shown in Fig. 4 in
which touch sensor can be in multiple deflection modes (corresponding to
different $\alpha$) at the same configuration
$q_{r}=(q_{x},q_{y},q_{\theta})$. Thus, proj is a many to one mapping and the
projection of the elements of $\Pi(h_{ds})$ to $Q$ corresponds to a cover of
$Q$. This implies that we can not compare $h_{ds}$ with another sensor defined
over $Q$ in terms of previously introduced definitions of equivalence and
sensor simulation.

Figure 4: A bendable touch sensor can be in multiple different modes at the
same robot configuration. Therefore, the solution to the contact constraint is
not unique.

###  III-C Actuated touch sensors

Suppose that the touch sensor is attached to a revolute joint that can be
controlled to change its configuration. The state of the system $x$ now
includes the robot configuration $q_{r}$ and the joint configuration $\psi\in
S^{1}$ such that $x=(q_{r},\psi,\dots)$. A measurement is obtained by changing
the configuration of the revolute joint while the robot configuration $q_{r}$
is kept constant. Let a state trajectory up to time $t$ be denoted as
$\tilde{x}:[0,t]\rightarrow X$. The set of all state-trajectories such that
$\texttt{proj}(x)$ is constant for all $x\in\tilde{x}$ for any possible $t>0$
is denoted by $\tilde{X}$. The sensor mapping for an actuated sensor is
$h:\tilde{X}\rightarrow Y$. Since the state is now augmented with the joint
configuration, in the following, with a slight abuse of previously introduced
definitions, we consider the sensor models in Section II-C to be defined over
this extended state space.

We begin by considering a compressible touch sensor described as
$h^{\prime}_{cs}$ in Eq. 10 attached to a revolute joint. The state of the
system is expressed by $x=(q_{r},\psi,\delta)$. Suppose that for each
measurement, the revolute joint trajectory corresponds to a full rotation of
the sensor that is defined as $\tilde{\psi}:[0,t]\rightarrow S^{1}$ such that
$\tilde{\psi}$ is a monotonically increasing function with $\tilde{\psi}(0)=0$
and $\tilde{\psi}(t)=2\pi$ while $\texttt{proj}(\tilde{x}(\tau))=q_{r}$,
$\forall\tau\in[0,t]$.

Let $\tilde{X}_{fr}\subseteq\tilde{X}$ be the set of all trajectories that
satisfy this motion. It follows from Proposition 1 that $h^{\prime}_{cs}$ can
be used to obtain the distance to the boundary (up to $\ell$) along the sensor
axis, which corresponds to the direction of $q_{\theta}+\psi$ for an actuated
touch sensor. Then, $\forall\psi\in[0,2\pi)$ the shifted distance
$d(q_{x},q_{y},q_{\theta}+\psi)=\ell-h^{\prime}_{cs}(x)$. Considering the full
motion that spans all $\psi\in[0,2\pi)$, $h_{cs}$ combined with motion results
in the observation $y:S^{1}\rightarrow[0,\ell]$ such that
$y(\psi)=d(q_{x},q_{y},q_{\theta}+\psi)$ if
$d(q_{x},q_{y},q_{\theta}+\psi)\leq\ell$ and $y(\psi)=\\#$, otherwise. Then,
the following proposition holds.

###### Proposition 2

Suppose $\tilde{X}_{fr}$ corresponding to the full rotation of the revolute
joint is not empty. Let $Q^{\prime}\subseteq Q$ be defined as
$Q^{\prime}=\bigcup_{\tilde{x}\in\tilde{X}_{fr}}\texttt{proj}(\tilde{x}(0)).$
Then, an actuated compressible sensor as defined above can simulate (in the
sense from Section II-A) a depth-limited omnidirectional depth sensor (Model
7) defined over $Q^{\prime}$.

The main reason why an actuated compressible touch sensor (even if it is fully
compressible) may not simulate a depth-limited omnidirectional depth sensor
defined over the entire $Q$ is that for certain $q_{r}$ a full rotation of the
revolute joint may not be possible due to the sensor characteristics and the
environment (see for example, Fig. 6). This issue could be alleviated to some
extent (although not fully) by defining a slightly more complicated motion
(for example, a full (or partial) counterclockwise (ccw) followed by a
rotation in the clockwise (cw) direction) or with the sensor design. To this
end, configuration space decomposition of a robot carrying a compressible
sensor and determining its connectivity remains as an interesting open
problem.

We now consider a mobile robot equipped with a bendable touch sensor composed
of a bendable link together with a deflection sensor (Model 5) attached to a
revolute joint. The state of the system is expressed as
$x=(q_{r},\psi,\alpha)$. Considering the same motion corresponding to a full
rotation of the revolute joint, let $P\subset W$ be part of the environment
swept by the touch sensor such that

$P=\bigcup_{x\in\tilde{x}}\bigcup_{s\in[0,1]}{T}_{rw}(q_{r}){T}_{sr}(\psi)\sigma_{\alpha}(s).$
(16)

The observation is the function $y(\psi)=h_{ds}(x)$. The preimage of $y$
contains all possible robot configurations $q_{r}$ such that same profile
information is obtained, i.e., all $q_{r}$ such that $P\in W$ and $\partial
P\cap\partial W\not=\emptyset$, in which $\partial P$ is the boundary of the
area swept by the touch sensor. Note that the preimages corresponding to the
combination of rotary motion and deflection sensor still result in a cover of
the robot configuration space since different $P$ can be obtained for the same
$q_{r}$ based on the configuration of the bendable touch sensor at the
beginning of the motion.

We now consider a motion similar to actuated whisking (see for example [13,
10]) which corresponds to a full ccw rotation followed by a full cw rotation.
In the following, we will show that with this type of motion, under certain
conditions, a bendable touch sensor with a deflection sensor can simulate a
depth sensor. Let $\mathcal{D}_{\ell}(q_{x},q_{y})$ denote the disk of radius
$\ell$ centered at $(q_{x},q_{y})$ as shown in Fig. 5. Suppose that
$\mathcal{D}_{\ell}(q_{x},q_{y})\setminus W$ (light gray area in Fig. 5) is
convex and that at the beginning of the motion, the touch sensor is in its
nominal form, that is, $\alpha=0$. Denote the intersection point of the touch
sensor at the beginning of the motion and the boundary of
$\mathcal{D}_{\ell}(q_{x},q_{y})$ as $p_{0}$. The instances at which $\alpha$
changes from zero to non-zero or non-zero to zero are called critical
instances and they happen when the touch sensor starts bending or regains its
nominal form. Since $\mathcal{D}_{\ell}(q_{x},q_{y})\setminus W$ is convex, a
rotation in single direction results in at most two critical instances: when
the sensor gets in contact with the boundary of $W$ so it starts bending and
if possible, when it regains its nominal shape. Suppose, during a full
rotation in the ccw direction, the nominal shape is achieved after bending and
the swept area is denoted by $P_{ccw}$. Let
$p_{1},p_{2}\in\partial\mathcal{D}_{\ell}(q_{x},q_{y})$ be the intersection
points of the touch sensor and $\partial\mathcal{D}_{\ell}(q_{x},q_{y})$ at
the beginning and at the end of the bending, respectively (see Fig. 5). We
assume that $P_{ccw}\cap\partial W$ is a connected curve. Let $u_{1}$ be an
endpoint of this curve which corresponds to the last point along the $\partial
W$ that the touch sensor is in contact with during a ccw motion. For cw
rotation, denote the swept area by $P_{cw}$ and define $p_{3},p_{4}$, similar
to $p_{1},p_{2}$, and define $u_{2}$ similar to $u_{1}$. We again assume that
$P_{cw}\cap\partial W$ is connected. We say that $p^{\prime}$ lies to the left
of $p$ if $p^{\prime}$ lies on the left hand side of the ray starting at
$(q_{x},q_{y})$ and passing through $p$, the relation is denoted by
$p^{\prime}>_{\ell}p$.

Let $V(q_{x},q_{y})\subseteq W$ be the set of points visible from
$(q_{x},q_{y})$. A point $p^{\prime}\in W$ is visible from $p\in W$ if the
line segment $\overline{uv}$ is contained in $W$. The following lemma
establishes a relation between the visibility of $(q_{x},q_{y})$ and a
bendable sensor at $(q_{x},q_{y})$ executing a whisking motion as described
above which satisfies the assumptions.

###### Lemma 1

Suppose $\mathcal{D}_{\ell}(q_{x},q_{y})\setminus W$ is convex, and suppose
that $u_{1}>_{\ell}u_{2}$ and $p_{0}>_{\ell}p_{2}$. Then,
$P=\mathcal{D}_{\ell}(q_{x},q_{y})\cap V(q_{x},q_{y})$, in which
$P=P_{ccw}\cup P_{cw}$.

###### Proof:

Since $\mathcal{D}_{\ell}(q_{x},q_{y})\setminus W$ is convex, there are two
tangents from $(q_{x},q_{y})$. Let $\vec{r}_{r}$ and $\vec{r}_{\ell}$ be two
rays starting from $(q_{x},q_{y})$ and representing the right and left
tangents, respectively (see Fig. 5). Rays $\vec{r}_{r}$ and $\vec{r}_{\ell}$
intersect the boundary of $\mathcal{D}_{\ell}(q_{x},q_{y})$ at two distinct
points, denoted by $p_{r}$ and $p_{\ell}$, respectively. Let
$p_{\ell}{}^{\frown}p_{r}$ be the circular arc from $p_{\ell}$ to $p_{r}$. A
circular arc $p{}^{\frown}p^{\prime}$ satisfies
$p^{\prime\prime}>_{\ell}p,\forall p^{\prime\prime}\in
p{}^{\frown}p^{\prime}$. Let $A$ (green area in Fig. 5) be the circular sector
corresponding to the arc $p_{\ell}{}^{\frown}p_{r}$ and let $B$ (blue area in
Fig. 5) be the connected region bounded by $\vec{r}_{r}$ (corresponding to the
left half-plane determined by $\vec{r}_{r}$), $\vec{r}_{\ell}$ (corresponding
to the right half-plane determined by $\vec{r}_{\ell}$) and $\partial W$ that
includes $(q_{x},q_{y})$. Then, $\mathcal{D}_{\ell}(q_{x},q_{y})\cap
V(q_{x},q_{y})=A\cup B$. Consider the ccw motion. The area swept before
bending is the circular sector determined by the arc $p_{0}{}^{\frown}p_{1}$.
Since $p_{0}>_{\ell}p_{2}$, sensor achieves its nominal form before it reaches
back to $p_{0}$. Then, the total area swept with $\alpha=0$ is the circular
sector determined by $p_{0}{}^{\frown}p_{1}$. Furthermore, since $p_{1}$ is
the intersection point when the bending begins, the ray starting at
$(q_{x},q_{y})$ and passing through $p_{1}$ is the right tangent
$\vec{r}_{r}$. Similarly, consider the cw motion. The area swept before the
bending begins is the circular sector determined by the arc
$p_{3}{}^{\frown}p_{0}$. Since $p_{3}$ is before the sensor begins to bend,
the ray starting at $(q_{x},q_{y})$ and passing through $p_{3}$ corresponds
the left tangent $\vec{r}_{\ell}$. Hence, $p_{3}=p_{\ell}$. Consequently, the
union of the two circular sectors corresponding to area swept with $\alpha=0$
resulting from ccw and cw motion equals to $A$. Since $P_{cw}\cap\partial W$
and $P_{ccw}\cap\partial W$ are two connected curves and $u_{1}>_{\ell}u_{2}$,
part of $\partial W$ visible from $(q_{x},q_{y})$ is swept. Furthermore,
bending is a continuous deformation which implies that the swept area does not
have any holes. Then, the area swept for $\alpha\neq 0$ equals to $B$. This
shows that $A\cup B=\mathcal{D}_{\ell}(q_{x},q_{y})\cap V(q_{x},q_{y})=P$, in
which $P=P_{cw}\cup P_{ccw}$. ∎

Figure 5: Critical points for an actuated bendable touch sensor mounted on a
revolute joint capable of cw and ccw motions. Light gray area corresponds to
$\mathcal{D}_{\ell}(q_{x},q_{y})\setminus W$, green area is $A$ and blue area
is $B$.

Let $\tilde{X}_{w}$ be the set of all trajectories corresponding to the
whisking motion and let $\tilde{X}^{\prime}_{w}\subseteq\tilde{X}_{w}$ be the
ones such that for each $\tilde{x}\in\tilde{X}^{\prime}_{w}$ it is true for
the corresponding swept area $P$ that $u_{1}>_{\ell}u_{2}$ and
$p_{0}>_{\ell}p_{2}$ and that $\mathcal{D}_{\ell}(q_{x},q_{y})\setminus W$
with respect to the robot configuration $\texttt{proj}(\tilde{x}(0))=q_{r}$ is
convex. We can now compare a bendable touch sensor with a depth sensor owing
to the inclusion of motion.

###### Proposition 3

Suppose $\tilde{X}^{\prime}_{w}$ is not empty. Let $Q^{\prime}\subseteq Q$ be
defined as
$Q^{\prime}=\bigcup_{\tilde{x}\in\tilde{X}_{fr}}\texttt{proj}(\tilde{x}(0)).$
Then, an actuated deflection touch sensor with a contact detector at the tip
executing a whisking motion as defined above can simulate (in the sense from
Section II-A) $h_{od}$ (Model 7) defined over $Q^{\prime}$.

###### Proof:

It follows from Lemma 1 that for each $q_{r}\in Q^{\prime}$, the swept area
$P$ with respect to the trajectory $\tilde{x}\in\tilde{X}^{\prime}_{w}$ such
that $\texttt{proj}(\tilde{x}(0))=q_{r}$ satisfies
$P=\mathcal{D}_{\ell}(q_{x},q_{y})\cap V(q_{x},q_{y})$. Then we can define
function $y:[0,2\pi)\rightarrow[0,\ell]$ such that
$y(\psi)=||(q_{x},q_{y})-b_{P}(q_{x},q_{y},q_{\theta}+\psi)||$, in which
$b_{P}(q_{x},q_{y},q_{\theta}+\psi)$ is the point struck on the boundary of
$\partial P$ by the ray in the direction of $q_{\theta}+\psi$ starting from
$(q_{x},q_{y})$ if no $x=(q_{x},q_{y},q_{\theta},\psi,\alpha)\in\tilde{x}$
satisfies $h^{\prime}_{ds}(x)=(0,0)$ (see Eq. (11) and recall that all the
sensor mappings are now defined over the augmented state space), and $\\#$
otherwise. Finally $P$ is defined as Eq. (16) for a whisking motion using
$h^{\prime}_{ds}$ (Model 5) to determine $\sigma_{\alpha}$ for each
$x\in\tilde{x}$. ∎

Similar to rotary motion, one can consider a translational motion too, in
which case the touch sensor is swept (if in contact) along the boundary of the
environment. This introduces an additional complexity since the robot
configuration is no longer fixed over the course of sensor movement. We expect
translational motion together with a rotational motion (similar to whisking)
to be the most useful for touch based navigation. However, we will investigate
such touch behaviors in future works.

##  IV Discussion

In this work, we introduced several noise-free virtual touch sensor models of
a touch sensor mounted on a mobile robot. These models were then used to show
under which conditions these types of sensors are equivalent to conventional
visibility-based models. The motivation behind this study was to elevate the
scattered use of touch sensors up to the level of a broad sensor category that
warrants systematic modeling to better understand their applications for
navigation. This would then inspire designing more navigation-centric touch
sensors which could be mounted on a mobile robot for safely exploring unknown
environments. This is especially useful in vision devoid scenarios like
underwater inspections in murky waters or exploration in areas with suspended
caustic chemicals that would otherwise damage sensors like cameras and Lidars.
Mobile robots endowed with touch sensing abilities, when deployed in
unstructured environments, can benefit from our study in various domains such
as health care, sports, ergonomics, logistics, and service robotics.

Existing touch sensor designs are primarily geared towards furthering our
understanding of touch receptors such as vibrissae in rats [14] and human-like
grasping [15]. However, touch sensors such as the contact strip (Model 2) and
the compression sensor (Model 3) are also useful in reducing the uncertainty
in state estimate while ensuring the safety of the touch sensor and the robot.

When comparing virtual sensor models, the main difficulty was the lack of
shared state space across various sensors under consideration (as certain
configurations are not feasible for touch sensors). This can be alleviated by
designing touch sensors which are fully compressible, retractable, and/or
bendable. To illustrate, consider the scenario shown in Fig. 6 where the touch
sensor disrupts the connectivity of the robot configuration space. In such
cases, using a different touch sensor (such as a bendable touch sensor) can be
the solution. Additionally, for real world deployment, touch sensors need to
be robust to wear and tear and the abilities such as telescopic extensions or
fully retractable compression can help the robot maneuver even in tight spaces
by modulating its footprint which is otherwise not feasible when using
conventional sensors like a camera.

Figure 6: Navigation limitations with compression sensor

The next step in our study would be to consider actuated touch sensors on a
mobile base such that the system would have both rotational and translation
degrees of freedom as opposed to only rotation as considered above. This would
open new avenues for developing impact-resilient robots by utilizing touch
sensor feedback-based motion primitives. The models presented here were for an
ideal touch sensor however, the abstract models independent of their physical
realization could very well serve a purpose similar to the pinhole camera
model for computer vision. For the real world, sensor noise needs to be
accounted for as in such cases the preimages result in a cover as opposed to a
partition of the state space. To address the stochastic nature of the sensors,
sensor fusion and effective data representation algorithms are critical
aspects to consider when using touch sensors for mobile robot navigation.
Sensor fusion techniques could be used for combining spatial, temporal, or
spatiotemporal sensor data from multiple modules of the same sensor (as in a
vibrissal array) or heterogeneous sensors (as in fusing touch sensor data with
other on board sensors). As a result, developing efficient filtering and
sensor fusion algorithms for processing touch data becomes a priority,
especially tactile SLAM. Recent attempts at sensor fusion (see [16, 17]) show
promising performance improvements when touch sensor information is fused with
other sensors on board such as cameras which have longer detection range and
dense area coverage. Touch sensors, despite their short range, can complement
such sensors and dedicated filters for fusing information need to be
developed. Aside from accounting for noise, the ideal models considered in
this work can also be extended to more practical situations such as those
involving multiple forces simultaneously acting on the cantilever, and
simultaneous compression and deflecting of the beam.

## References

  * [1] R. A. Grant, A. L. Sperber, and T. J. Prescott, “The role of orienting in vibrissal touch sensing,” _Frontiers in behavioral neuroscience_ , vol. 6, p. 39, 2012.
  * [2] G. Reina, L. Ojeda, A. Milella, and J. Borenstein, “Wheel slippage and sinkage detection for planetary rovers,” _IEEE/Asme Transactions on Mechatronics_ , vol. 11, no. 2, pp. 185–195, 2006.
  * [3] G. M. Hegde, C. Ye, C. A. Robinson, A. Stroupe, and E. Tunstel, “Computer-vision-based wheel sinkage estimation for robot navigation on lunar terrain,” _IEEE/ASME Transactions on Mechatronics_ , vol. 18, no. 4, pp. 1346–1356, 2013.
  * [4] G. R. Scholz and C. D. Rahn, “Profile sensing with an actuated whisker,” _IEEE Transactions on Robotics and Automation_ , vol. 20, no. 1, pp. 124–127, 2004.
  * [5] J. An, P. Chen, Z. Wang, A. Berbille, H. Pang, Y. Jiang, T. Jiang, and Z. L. Wang, “Biomimetic hairy whiskers for robotic skin tactility,” _Advanced Materials_ , p. 2101891, 2021.
  * [6] D. I. McCloskey, “Kinesthesia, kinesthetic perception,” in _Sensory Systems: II_. Springer, 1988, pp. 36–38.
  * [7] J. C. Tuthill and E. Azim, “Proprioception,” _Current Biology_ , vol. 28, no. 5, pp. R194–R203, 2018.
  * [8] Z. Lu, Z. Liu, and K. Karydis, “Deformation recovery control and post-impact trajectory replanning for collision-resilient mobile robots,” in _2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2021, pp. 2030–2037.
  * [9] P. Giguere and G. Dudek, “A simple tactile probe for surface identification by mobile robots,” _IEEE Transactions on Robotics_ , vol. 27, no. 3, pp. 534–544, 2011.
  * [10] N. F. Lepora, M. Pearson, and L. Cramphorn, “Tacwhiskers: Biomimetic optical tactile whiskered robots,” in _2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2018, pp. 7628–7634.
  * [11] S. M. LaValle, _Sensing and filtering: A fresh perspective based on preimages and information spaces_. Now Publishers, 2012.
  * [12] J. E. Shigley, _Shigley’s mechanical engineering design_. Tata McGraw-Hill Education, 2011, ch. 4.
  * [13] T. J. Prescott, M. J. Pearson, B. Mitchinson, J. C. W. Sullivan, and A. G. Pipe, “Whisking with robots,” _IEEE robotics & automation magazine_, vol. 16, no. 3, pp. 42–50, 2009.
  * [14] M. J. Pearson, B. Mitchinson, J. C. Sullivan, A. G. Pipe, and T. J. Prescott, “Biomimetic vibrissal sensing for robots,” _Philosophical Transactions of the Royal Society B: Biological Sciences_ , vol. 366, no. 1581, pp. 3085–3096, 2011.
  * [15] W. Chen, H. Khamis, I. Birznieks, N. F. Lepora, and S. J. Redmond, “Tactile sensors for friction estimation and incipient slip detection—toward dexterous robotic manipulation: A review,” _IEEE Sensors Journal_ , vol. 18, no. 22, pp. 9049–9064, 2018.
  * [16] O. Struckmeier, K. Tiwari, M. Salman, M. J. Pearson, and V. Kyrki, “ViTa-SLAM: A bio-inspired visuo-tactile slam for navigation while interacting with aliased environments,” in _2019 IEEE International Conference on Cyborg and Bionic Systems (CBS)_. IEEE, 2019, pp. 97–103.
  * [17] M. J. Pearson, S. Dora, O. Struckmeier, T. C. Knowles, B. Mitchinson, K. Tiwari, V. Kyrki, S. Bohte, and C. M. Pennartz, “Multimodal representation learning for place recognition using deep Hebbian predictive coding,” _Frontiers in Robotics and AI_ , vol. 8, 2021.

  *[ccw]: counterclockwise
  *[cw]: clockwise