Copyright for this paper by its authors. Use permitted under Creative Commons
License Attribution 4.0 International (CC BY 4.0).

ISWC’22: The 21st International Semantic Web Conference, October 23–27, 2022,
Hangzhou, China

<EMAIL_ADDRESS>]

<EMAIL_ADDRESS>]

<EMAIL_ADDRESS>]

<EMAIL_ADDRESS>][1]

[1]Corresponding author.

# VGStore: A Multimodal Extension to SPARQL for Querying RDF Scene Graph

Yanzeng Li  Wangxuan Institute of Computer Technology (WICT), Peking
University, China  Zilong Zheng  Beijing Institute for General Artificial
Intelligence (BIGAI), Beijing, China  Wenjuan Han  Lei Zou

(2022)

###### Abstract

Semantic Web technology has successfully facilitated many RDF models with rich
data representation methods. It also has the potential ability to represent
and store multimodal knowledge bases such as multimodal scene graphs. However,
most existing query languages, especially SPARQL, barely explore the implicit
multimodal relationships like semantic similarity, spatial relations, etc. We
first explored this issue by organizing a large-scale scene graph dataset,
namely Visual Genome, in the RDF graph database. Based on the proposed RDF-
stored multimodal scene graph, we extended SPARQL queries to answer questions
containing relational reasoning about color, spatial, etc. Further demo (i.e.,
VGStore) shows the effectiveness of customized queries and displaying
multimodal data.

###### keywords:

SPARQL RDF Multimodal KBQA

##  1 Introduction

Over the recent years, we have witnessed an explosive growing trend on
multimodal models due to the increasing computing power and massive multimodal
datasets. Despite of inspiring performance that keeps updating on various
multimodal benchmarks, the interpretability and reasonability have recently
been challenged by researchers, namely, models are memorizing multimodal
statistical features rather than understanding the joint information among
them. For example, Visual Question Answering (VQA), a representative
multimodal testbed for vision understanding, requires model to reason over
images and answer questions. However, the current mainstream models still
depend on end-to-end training by fitting input signals to ground truth
answers, while neglecting the underlying visual relations and semantics.

To address these issues, we leverage an intrinsically explainable task,
Knowledge Base Question Answering (KBQA), which aims to answer Natural
Language Questions by referring to external Knowledge Base (KB). Semantic
Parsing (SP)-based KBQA is a mainstream technique to solve the such QA problem
via parsing a question into a KB query (such as SPARQL) [1]. The latest works
are devoted to improving the performance of natural language understanding and
parsing, while neglecting the expressiveness of KB queries, limiting the
application of SP-based methods in multimodal datasets. Although RDF has
sufficient representation capability to describe multimodal data, the lack of
multimodal semantic relationships in standard SPARQL has become a major
challenge in applying SP-based KBQA methods to multimodal domains. Researchers
have attempted to extend SPARQL for this purpose. For example, SPARQL-MM [2]
proposed to use custom aggregation functions to access media fragments.
However, previous works still suffer from limitations in extensibility and
vague semantic representation, resulting in rare applications. Specifically,
the custom aggregation functions introduced in SPARQL-MM are easy to
understand by human beings but hard to understand and be expressed by the
machine. This is because it brings significantly extra complexity to the query
statement, e.g., if multimodal query statements in SPARQL-MM are used as query
conditions, it inevitably leads to the union or nested query; however, the SP-
based methods only support simple queries in the foreseeable future.

In this demo, we designed an ontology to organize the multimodal scene graph
and store it with RDF. Furthermore, we implemented semantic multimodal SPARQL
queries by extending the SPARQL engine, enabling the ability to answer
questions related to multimodal information such as visual and spatial
reasoning.

##  2 Storing Visual Genome with RDF

Visual Genome (VG) [3] is a large-scale dataset for fine-grained scene graphs,
with rich annotations of images, regions, objects, as well as their
relations111http://visualgenome.org/VGViz/explore demonstrates the dataset.. A
synset from WordNet [4] is introduced to link different scene graphs via the
lexical relations between literals of the object relations. In addition, VG
provided 1,445,332 relevant questions for 108,077 images, which are difficult
to be answered by traditional SP-based KBQA methods because the SPARQL engine
does not support the arithmetic opeartions needed to answer these multimodal
questions.

Figure 1: (a) The example of relations between classes in the ontology of
visual genome. (b) The example of data properties of the class. (c) The
implicit semantic relations between visual blocks.

For querying convenience, we formalize the elements of VG in RDF. Fig. 1(a)
shows the designed ontology of RDF-stored VG (RDF-VG)222Due to space
limitations, the detail of data processing and ontology organization are
attached to the code repository.. Fig. 1(b) demonstrates properties of the
defined classes in RDF-VG. The properties $(x,y,h,w)$ determine the visual
block of region or object by tailoring image. We store RDF-VG in gStore [5],
which is a graph-oriented RDF data management system supporting complex SPARQL
queries on graph data.

##  3 Querying Multimodal Information via SPARQL

Traditional SPARQL engines (such as our backbone - gStore) cannot perform
queries involving multimodality and thus cannot directly answer such questions
(e.g. what color is this cat?). Therefore, we developed a VGStore extension
based on standard SPARQL grammar and pyparsing [6] to parse custom predicates
(as Fig. 1(c) shows) for arbitrary query patterns, enabling the ability of
traditional graph databases to perform multimodal queries by passing through
the extra computing requirements to third-party tools (e.g. OpenCV, Torch,
etc.). The architecture of VGStore is shown in Fig. 2 (Left).

Figure 2: Left: The diagram of VGStore. Right: An example of VGStore handling the multimodal SPARQL query (Python-like pseudocode). Table 1: Part of the supported non-standard querying clauses. The ?a and ?b indicate the regular query variables. Customized Triple Pattern | Description
---|---
?a ?mm.sim ?b. |  Represent the semantic similarity between ?a and ?b.
?a $\langle$mm.color$\rangle$ ?color. |  Output the primary color of ?a to variable ?color.
?a ?mm.pos ?b. |  Represent the relative position relation between ?a and ?b.
?a ?mm.count ?b. |  Count the number of component ?b in image ?a.
FILTER(...) |  Filter the results via customized variables,
e.g., FILTER(?mm.sim > 0.5).
ORDER BY ... |  Sort the results via customized variables,
e.g., ORDER BY DESC(?mm.count).

VGStore analyzes, matches, and replaces the clauses in the original SPARQL
query that contain custom predicates. It recursively replaces all non-standard
query patterns with the standard SPARQL syntax, and stores the replacement
process in an operation stack temporarily. The standard SPARQL query can be
handed over to the backbone graph database for execution. After getting the
result, the inverse operation is successively performed according to the
staged replacement operations in the stack. Finally, the intent of the
original SPARQL query would be restored. Fig. 2 (Right) demonstrated how a
multimodal query containing the non-standard custom predicate variable is to
be executed. Table 2 illustrates part of the supported query patterns in
VGStore, covering questions involving color, counting, and relative position
in the VG question-answer dataset, which account for 15.0%, 11.4%, and 7.0% of
all questions, respectively. Other simple questions (e.g., “What is this?”)
can be expressed and queried by native SPARQL directly, and the remaining non-
factual questions (e.g., “What is this man’s motivation?”) or inference
questions (e.g., “When was the picture taken?”) are out of scope in this
demonstration.

##  4 Discussion and Next Step

Although VGStore successfully supports multimodal queries by extending virtual
predicates, it still has some limitations. VGStore is written in Python, which
brings additional latency in runtime, and it is possible to reduce the
performance loss by native support in the SPARQL engine. In addition, when the
VGStore queries large data, and there are multiple extended query statements,
it would cause severe performance problems. This drives us to schedule and
parallelize the third-party tool calls.

VGStore currently only supports several basic query patterns (as listed in
Table 1) specialized for the RDF-VG dataset, and does not adapt to other VQA
datasets, nor does it support richer query patterns. Therefore, our next steps
for improvements include supporting more query patterns and extending the
applicability of VGStore to more graph databases with large-scale multimodal
graphs.

Figure 3: The interface in use for querying the RDF-stored visual genome scene
graph with customized query pattern.

##  5 Demonstration

This paper presented VGStore, an extension to SPARQL for querying multimodal
information on RDF. The demo showcased the web user interface of VGStore for
querying multimodal SPARQL on RDF-VG, as shown in Fig. 3. With this demo, we
provided a potential solution to the main challenge of SP-based multimodal
KBQA and laid the foundation of multimodal knowledge graph. Our code of RDF-VG
builder, preliminary parser and frontend of demonstration are available at
https://github.com/pkumod/VGStore.

###### Acknowledgements.

This work was supported by NSFC under grant 61932001, U20A20174. The
corresponding author of this paper is Lei Zou (zoulei@pku.edu.cn). We
sincerely thank reviewers for their valuable comments and advises.

## References

  * Zhang et al. [2022] M. Zhang, R. Zhang, Y. Li, L. Zou,  Crake: Causal-enhanced table-filler for question answering over large scale knowledge base,  in: Findings of the Association for Computational Linguistics: NAACL 2022, 2022, pp. 1787–1798.
  * Kurz et al. [2014] T. Kurz, S. Schaffert, K. Schlegel, F. Stegmaier, H. Kosch,  Sparql-mm-extending sparql to media fragments,  in: European Semantic Web Conference, Springer, 2014, pp. 236–240.
  * Krishna et al. [2017] R. Krishna, Y. Zhu, O. Groth, J. Johnson, K. Hata, J. Kravitz, S. Chen, Y. Kalantidis, L.-J. Li, D. A. Shamma, et al.,  Visual genome: Connecting language and vision using crowdsourced dense image annotations,  International journal of computer vision 123 (2017) 32–73.
  * Miller [1995] G. A. Miller,  Wordnet: a lexical database for english,  Communications of the ACM 38 (1995) 39–41.
  * Zeng and Zou [2018] L. Zeng, L. Zou,  Redesign of the gstore system,  Frontiers of Computer science 12 (2018) 623–641.
  * McGuire [2007] P. McGuire, Getting started with pyparsing, " O’Reilly Media, Inc.", 2007.

  *[VQA]: Visual Question Answering


# Image-to-Joint Inverse Kinematic of a Supportive Continuum Arm Using Deep
Learning ††thanks:

Shayan Sepahvand  Department of Mechanical,
Industrial, and Mechatronics Engineering
Toronto Metropolitan University
Toronto, Canada
Email<EMAIL_ADDRESS>Guanghui Wang  Department of Computer
Science
Toronto Metropolitan University
Toronto, Canada
Email<EMAIL_ADDRESS>Farrokh Janabi-Sharifi  Department of Mechanical,
Industrial, and Mechatronics Engineering
Toronto Metropolitan University
Toronto, Canada
Email<EMAIL_ADDRESS>
###### Abstract

In this work, a deep learning-based technique is used to study the image-to-
joint inverse kinematics of a tendon-driven supportive continuum arm. An eye-
off-hand configuration is considered by mounting a camera at a fixed pose with
respect to the inertial frame attached at the arm base. This camera captures
an image for each distinct joint variable at each sampling time to construct
the training dataset. This dataset is then employed to adapt a feed-forward
deep convolutional neural network, namely the modified VGG-16 model, to
estimate the joint variable. One thousand images are recorded to train the
deep network, and transfer learning and fine-tuning techniques are applied to
the modified VGG-16 to further improve the training. Finally, training is also
completed with a larger dataset of images that are affected by various types
of noises, changes in illumination, and partial occlusion. The main
contribution of this research is the development of an image-to-joint network
that can estimate the joint variable given an image of the arm, even if the
image is not captured in an ideal condition. The key benefits of this research
are twofold: 1) image-to-joint mapping can offer a real-time alternative to
computationally complex inverse kinematic mapping through analytical models;
and 2) the proposed technique can provide robustness against noise, occlusion,
and changes in illumination. The dataset is publicly available on Kaggle.

###### Index Terms:

Supportive Cooperative Continuum Robot (SCCR), Deep Learning (DL), Transfer
Learning, Inverse Kinematics (IK), Convolutional Neural Network (CNN).

##  I Introduction

A continuum robot (CR) is characterized as a manipulable structure made of
constituent material capable of assuming curved shapes with uninterrupted
tangent vectors [1]. Taking inspiration from the flexible appendages of
animals with flexible bodies, continuum robots employ a sequence of continuous
curves as the structural basis for generating bending motion, as opposed to
using rigid-link robots [2]. Continuum robots (CRs) differ from hyper-
redundant and pseudo-continuum robots due to the presence of continuous
backbones, allowing their structure to be conceptualized as infinite degrees
of freedom (DoF). This design offers several advantages, including dexterity,
lightweight construction, simplicity [3], compliance, and scalability that
allows small-scale construction to enable their operation in cluttered and
constrained environments, which in turn leads to superior performance in
interaction with humans and tasks where collaboration is required[4].

Leveraging their soft characteristics, they have gained widespread adoption in
the field of medical surgery [5], industry operation [6], subsea applications
[7], repair of gas turbine engines in on-wing scenarios[8], and search and
rescue operations [9]. Despite these advancements, CRs are required to be
further developed to improve their maneuverability, payload, and speeds for
potentially complex tasks such as manipulation and grasping. Consequently,
more accurate modeling and real-time control systems are required to deal with
these problems. Efficient kinetic-dynamic mapping techniques are hence at the
center of attention of research on CRs. Among those, kinematic mapping takes
priority due to the quasi-static nature of motions in CRs. Particularly, the
inverse kinematic (IK) solution is often needed for position control of CRs
for a given end-point trajectory. The IK problem comprises obtaining the joint
variables corresponding to a given end-effector pose [10]. There are two
general approaches to solving the IK problem. The commonly used IK method
based on differential kinematics requires the calculation of the analytical
Jacobian to solve the problem. However, analytical Jacobian is not readily
available in exact CR models (using Coserrat rod theory), which require the
solution of partial differential equations (PDEs), not leading to an efficient
and closed-form solution. A vision-based learning approach seems to offer an
alternative approach.

Recently, learning-based methods have been proposed to deal with the problem
of robots’ forward and inverse kinematics. For instance, in [11], the IK of a
manipulator with seven DoF employing Artificial Neural Networks (ANNs) is
solved. The CNN takes the end-effector’s desired pose as the input and
generates the joint variable of each joint. Grassmann et al. suggested an
approach to solving the FK and the IK of concentric tube CRs [12]. Here, two
ANNs with Rectified Linear Unit (ReLU) activation functions are suggested to
estimate the kinematics [12]. The problem is further formulated for
reinforcement learning (RL) to provide a model-free RL-based approach for IK
of concentric tube CRs [13]. The most analogous work to ours is presented in
[14], where a methodology is used to obtain the IK for concentric tube CRs.
They showed that the learning method outperforms the vanilla numerical
technique in terms of accuracy.

All of the above methods addressed the learning-based modeling of the IK for
rigid-link arms or concentric tube CRs. However, the problem of learning-based
IK for the supportive tendon-driven cooperative CRs (SCCRs) [15] has not been
investigated so far to the best of the author’s knowledge. The existing gap
motivated us to develop a vision-based learning technique for the IK of SCCRs.
The approach exploits the capability of deep CNNs, transfer learning, and
fine-tuning to detect the features more efficiently and precisely estimate the
joint variables. This is the main contribution of this paper.

The content of this paper is as follows. In section II, the continuum arm’s
dynamic model is introduced briefly to justify the use of the learning-based
method. The prototype specifications are explained in detail. The process of
creating the dataset, the CNN structure, and its training are discussed in
section II. Section III explains the experiments and the analysis in general.
The learning curves are investigated, and the result of the prediction for the
test dataset is also examined. We also studied the robustness of the method
against noise and illumination variations and used visual inspection of
features to analyze the cons and pros of the convolutions. Finally, the
concluding remarks are presented in section IV.

##  II Methodology

In this section, a methodology is suggested to approximate the IK for a
supportive tendon-driven continuum robot from an image using a deep learning-
based approach. A deep CNN is employed to learn the joint variables using the
visual information. A simple webcam is used to acquire the frames by a camera
to construct the training dataset. Experimental tests are used to verify the
performance of the proposed method by evaluating the precision of the
estimations and robustness against factors such as noise and change in
illumination. The model is shown in Fig. 1.

Figure 1: Architecture of modified VGG-16 model

###  II-A Dynamics of the supportive tendon-driven continuum robot

In this paper, a supportive tendon-driven CCR is considered. The following
system (1) of nonlinear PDEs describe the general dynamics of the system using
the Cosserat rod modeling approach [4]:

$\begin{split}\frac{\partial\xi}{\partial
s}&=f\left(\frac{\partial\xi}{\partial t},\xi,\tau,t\right)\\\
\frac{\partial\xi}{\partial t}&=g\left(\frac{\partial\xi}{\partial
t},\xi,\tau,t\right)\end{split}$ (1)

where $\tau$ is the vector of the tendon’s tension usually applied at the tip,
changing the shape of the CR, $\xi$ is the vector of states, $s$ denotes the
spatial variable, $t$ is the time, and $f(.)$ and $g(.)$ are two nonlinear,
complex functions. The details of Cosserat rod modeling are provided in the
literature [4]. It is worth highlighting that the exerted tension is applied
through DC motors. Even though the displacement command of the DC motors does
not appear in (1), a simple look-up table can be used to map the DC motor
variables to the applied tension at the tip. This also omits the need for
precise and expensive force sensors to measure the tension.

###  II-B Development and Design of the Prototype

As illustrated in Fig. 2, the SCCR comprises the operative (the arm on the
left) and supportive arms (the arm on the right). Each arm consists of several
elements: additional spacer disks for transmitting motor tensions to the tip
flexible spring steel backbone, spacer disks made with PLA filament to guide
the tendons, and four braided Kevlar lines serving as tendons. These tendons
align along a circle with a radius of $18.55$ (mm), running parallel to the
backbone. The manipulation of the tendons is done using Dynamixel servomotors
AX-12A (Robotis, Lakeforest CA). A pneumatic gripper is mounted on the top of
the supportive arm to grip the operative arm. The specification of the
prototype is summarized in Table I.

Figure 2: The experimental setup used to create the dataset TABLE I: The experimental setup and its components’ specifications. Component | Specification | Value
---|---|---
Camera | Image Size |  $1080\times 720$ (pixels)
Spatial Resolution | 96 (dpi)
Intensity Resolution | 256
Channels | RGB
Autofocus | Off
Frame Rate |  $30$ (fps)
Field of View | $78^{\circ}$
Tendon | Breaking strength | 31.75 (Kg)
Backbone | Density | 7800 $\mathrm{(Kg/m^{3})}$
Young’s modulus (E) | 207 $\mathrm{(GPa)}$
Length (L) | 0.4 $\mathrm{(m)}$
Radius (r) | 0.9 $\mathrm{(mm)}$

###  II-C Data Acquisition

Data acquisition involves several steps. Firstly, an ON/OFF command is sent to
the Arduino; accordingly, the pneumatic actuators direct the compressed air to
the pneumatic gripper so the supportive arm grips the operative arm. Then, a
random joint variable command is transmitted to the Dynamixel motors using
MATLAB Software Development Kit (SDK). This joint variable serves as the label
used to train the CNN, which is subsequently stored in a text file once it is
sent to the servo motor After that, the Logitech C615 HD 1080p camera captures
an image and stores it in the dataset directory; this process is repeated
until the pre-specified number of images is acquired. However, there is a
delay (about two seconds) between each iteration to allow the SCR to reach its
final configuration as well as the PC to communicate with the control boards
and camera. Fig. 3, indicates six training images and their corresponding
joint variables $q$ shown in Fig. 2.

(a) $q=2.063\mathrm{(rad)}$

(b) $q=2.902\mathrm{(rad)}$

(c) $q=1.688\mathrm{(rad)}$

(d) $q=-2.386\mathrm{(rad)}$

(e) $q=2.675\mathrm{(rad)}$

(f) $q=0.265\mathrm{(rad)}$

Figure 3: Six images of the training dataset and their labels

###  II-D Image-to-joint CNN Training

The input of the CNN is an RGB image, and it outputs the angle of the
actuator. The model used is a modified VGG-16, and the base model is trained
using the ImageNet dataset [16]. The last block containing two 2D
convolutional, a ReLu, and a 2D max pooling layer, is discarded from the
conventional VGG-16 to preserve more features since the pre-trained kernels do
not suit our application due to the fact that these filters indicate their
best performance for the ImageNet dataset, not ours. Next, using transfer
learning techniques, the convolutional layer weights are frozen, and the
prediction layers are omitted and replaced with a dense layer with a linear
activation function. However, to further enhance the results, after adjusting
the weights of the prediction layer, its weights can be frozen. In contrast,
the weights of the convolutional layers are unfrozen and trained with a lower
learning rate for fewer epochs compared to the initial training, as the
performance of the CNN is drastically impacted by changes in the weights of
the kernels. The last layer is a single neuron with a linear activation
function, which outputs the joint angle $q$. The model structure is summarized
in Table II.

TABLE II: The structure of the model for a $600\times 600$ RGB input image Layer | Type | Input Shape | Output Shape
---|---|---|---
1 | Input | $(\textsc{BS}^{*},300,300,3)$ | $(\textsc{BS},300,300,3)$
2 | 2D Convolution | $(\textsc{BS},300,300,3)$ | $(\textsc{BS},300,300,64)$
3 | ReLu | $(\textsc{BS},300,300,64)$ | $(\textsc{BS},300,300,64)$
4 | 2D Max Pooling | $(\textsc{BS},300,300,64)$ | $(\textsc{BS},150,150,64)$
5 | 2D Convolution | $(\textsc{BS},150,150,64)$ | $(\textsc{BS},150,150,128)$
6 | ReLu | $(\textsc{BS},150,150,128)$ | $(\textsc{BS},150,150,128)$
7 | 2D Max Pooling | $(\textsc{BS},150,150,128)$ | $(\textsc{BS},75,75,128)$
8 | 2D Convolution | $(\textsc{BS},75,75,128)$ | $(\textsc{BS},75,75,256)$
9 | 2D Convolution | $(\textsc{BS},75,75,256)$ | $(\textsc{BS},75,75,256)$
10 | ReLu | $(\textsc{BS},75,75,256)$ | $(\textsc{BS},75,75,256)$
11 | 2D Max Pooling | $(\textsc{BS},75,75,256)$ | $(\textsc{BS},37,37,256)$
12 | 2D Convolution | $(\textsc{BS},37,37,256)$ | $(\textsc{BS},37,37,512)$
13 | 2D Convolution | $(\textsc{BS},37,37,5)$ | $(\textsc{BS},37,37,512)$
14 | ReLu | $(\textsc{BS},37,37,512)$ | $(\textsc{BS},37,37,512)$
15 | 2D Max Pooling | $(\textsc{BS},37,37,512)$ | $(\textsc{BS},18,18,512)$
16 | Flatten | $(\textsc{BS},18,18,512)$ | $(\textsc{BS},165888)$
17 | Dense (Linear) | $(\textsc{BS},165888)$ | $(\textsc{BS},1)$
*BS denotes the batch size.

##  III Results

The model implemented in TensorFlow 2.13.1 and is trained on a dataset
$\\{\mathbb{I},\mathbb{Q}\\}$ where $\mathbb{I}$ denotes the collection of
1000 images of the SCCR setup, and $q_{i}\in\mathbb{Q}$ with
$q_{i}\in\left[\frac{-7\pi}{2},\frac{7\pi}{2}\right]$ represents a scalar
angle sample. The data split ratio for learning, validation, and testing is
8:1:1. During transfer learning and fine-tuning stages, the loss function,
which is the mean squared error (MSE), is minimized. The performance of the
system against common disturbing factors such as different types of noise,
partial occlusion, and change in illumination is studied. A statistical
analysis of the results of each stage is conducted to give a general view of
the performance of the system. Eventually, to validate the results further, a
K-fold cross-validation technique is established.

###  III-A Learning Curves

In this section, the learning curves of the modified VGG-16 for $800$ training
images are presented. Nonetheless, before using these images for training,
they were resized to the resolution of $300\times 300$ pixels using nearest
neighborhood interpolation to avoid memory limitations. In addition, as the
intensity resolution is $256$, the image is divided by $255$ for
normalization. The Adam optimizer with learning rate $\lambda=10^{-6}$, decay
rates $0.9$, and $0.999$, a mini-batch size of $16$, and the epoch number
$150$ are selected as the hyper-parameters. Figs. 4 (a) and (b) show how
training and validation losses change with epochs for the two stages of
training. During the first step, the training and validation losses undergo a
similar descending trend, dropping from approximately $2$ to less than 0.25
$\mathrm{rad}^{2}$ in 20 epochs. Likewise, the pattern is descending for the
second stage; nonetheless, the change in loss is infinitesimal. The training
loss decreased from 0.005 to 0.002 $\mathrm{rad}^{2}$; nonetheless, the
validation loss exhibits fewer variations and lowered from $0.01$ to
approximately 0.008 $\mathrm{rad}^{2}$.

###  III-B Prediction results for the test images

The error distributions for $100$ unforeseen test images using the CNN are
illustrated in Fig. 5. The histogram is left-skewed, suggesting that most of
the error values lie on the lower end, but there are a few that are larger.
Just above $40\%$ of the errors are less than $0.05$ rad, whereas about $10\%$
are larger than $0.15$ rad. In terms of the estimation time for the test
images, the average time was $0.7$ s when computation was performed on the
GeForce RTX 3060.

(a) The transfer learning stage (b) The fine-tuning stage

Figure 4: The learning curves  Figure 5: Prediction error distribution for the
image-to-joint CNN

###  III-C Comparison between different CNNs

In this section, the test results are investigated for various state-of-the-
art CNN models in terms of the loss value of test data after undergoing $150$
epochs of learning. The results of this experiment, along with the number of
layers of each CNN, are listed in Table III. The best estimation error is
provided by DenseNet201 with a loss of $0.0106$, whereas EfficientNetB0
indicates poor estimation results. The loss values for DenseNet121 and
InceptionV3 stand at $0.0198$ and $0.0286$, respectively. Overall, when the
number of layers is considered as a deciding factor affecting the computation
time, VGG16 outperforms the other CNNs in our experiment in terms of the loss
value and simplicity.

TABLE III: A comparison between different CNNs CNN | MSE | MAE | Layers
---|---|---|---
InceptionV3 | $0.0286$ | $0.1332$ | $189$
DenseNet201 | $0.0106$ | $0.0846$ | $402$
DenseNet121 | $0.0198$ | $0.121$ | $242$
EfficientNetB0 | $4.1208$ | $1.7535$ | $132$
MobileNet | $0.0592$ | $0.1942$ | $55$

###  III-D Robustness tests on training images

In real-world applications, the input images captured by the camera are
affected by various disturbing factors such as noise and variation in
illumination. In this section, the CNN is first trained with $7000$ images (in
addition to the original images) that were affected by undesirable factors,
including Gaussian with zero mean and standard deviation $0.01$, speckle noise
with the same mean and standard deviation as the Gaussian noise, impulse
noises with probability $0.01$ when its distribution is normal, low and high
illumination, and partial occlusion. The gamma transformation with
$\gamma=0.2$ and $\gamma=2$ was used to create dark and bright images to study
the effect of illumination, as shown in Figs. 6 (a) and (b), respectively. The
images were partially occluded by squares with sides of $200$ pixels located
at random coordinates in the image. The trained CNN was tested six times
against 100 unforeseen images. Each test was designed to evaluate the
performance of the trained CNN for one of the aforementioned factors. The same
test images were employed to test the prediction of the CNN-trained original
compared to the one that was adapted using the augmented dataset. The
evaluation loss of the CNN trained with augmented images was lower than that
of the original CNN, indicating the improvement in robustness against all of
the factors. For example, in the case of the occurrence of Gaussian noise, the
MAE significantly decreased from above $1.4$ to just above $0.2$ rad. The
smallest change was for the occluded images, where the MAE dropped from $1$ to
$0.4$ rad.

(a) Gamma transformed image with $\gamma=0.5$

(b) Gamma transformed image with $\gamma=2$

Figure 6: Sample of training images affected by change in illumination Figure
7: The bar plot of estimation MAE for different tests

###  III-E K-fold cross-validation

To further evaluate the performance of the CNN, the dataset is divided into
ten groups of $100$ images. As a result, the model is trained ten times using
nine subsets and tested against the remaining images. It is worth highlighting
that as the images captured exhibit the random movement of the arm, the
shuffling of the images is skipped. The distribution of the MAE is illustrated
in Fig. 8. The distribution of the error in all groups is similar, suggesting
the lack of over-fitting. Most of the errors lie between $0.03$ to $0.04$ rad,
and the variance of the MAE for all groups is $9.440\times 10^{-6}$ rad2,
confirming the consistency of the results for all of the groups.

Figure 8: The bar plot of the MAE for all of the groups

###  III-F Visual inspection of feature maps

One of the methods to analyze the performance of the deep CNN is to extract
the feature maps that each block of the modified VGG-16 outputs. To this end,
the output of the first block is shown in Fig. 9. Some of the kernels have a
satisfactory performance in extracting the overall shape of the continuum arm.
Nevertheless, some of these filters have a negative impact on the angle
estimation. For example, several filters are trained to detect the background
instead of the robot. Likewise, some filters detect the lines, like the line
on the right side of the scene. The base of the robot is also detected in
several convolutions, which is not desirable. Interestingly, there are filters
that can be used to detect the point that supportive arm and operative arm
connect.

Figure 9: The output of the first block of the CNN (4th layer)

##  IV Conclusion

In this paper, the problem of IK modeling of a supportive cooperative
continuum robot is investigated. The image dataset is created using a USB
webcam fixed in the room. Then, for each random joint variable, we captured an
image and stored it along with the joint angle. After creating the dataset, we
examined different techniques for choosing the appropriate deep CNN. As a
result, a modified VGG-16 trained with the ImageNet dataset was selected. The
last block and prediction layer are omitted since they did not serve as
beneficial for our application. After the transfer learning stage, we utilized
fine-tuning to further reduce the loss function final value. The distribution
of the estimation error shows that about $40\%$ of the errors are less than
$0.05$ rad. The robustness tests showed that the CNN is highly robust against
different types of noises, changes in illumination, and partial occlusion as
the mean absolute value of prediction error was less than $0.4$ rad for all
cases. In addition, the K-fold cross-validation verified the lack of over-
fitting for all groups. Lastly, we analyzed one of the feature maps to see
whether all of the filters are useful, and we noticed that some of them do not
contribute to the detection of the CR body, which is crucial for the
estimation of IK.

## References

  * [1] J. Burgner-Kahrs, D. C. Rucker, and H. Choset, “Continuum robots for medical applications: A survey,” IEEE Transactions on Robotics, vol. 31, no. 6, pp. 1261–1280, 2015.
  * [2] J. c, Q. Fang, P. Xiang, D. Sun, Y. Xue, R. Jin, K. Qiu, R. Xiong, Y. Wang, and H. Lu, “A survey on design, actuation, modeling, and control of continuum robot,” Cyborg and Bionic Systems, vol. 2022, 2022.
  * [3] B. A. Jones and I. D. Walker, “Practical kinematics for real-time implementation of continuum robots,” IEEE Transactions on Robotics, vol. 22, no. 6, pp. 1087–1099, 2006.
  * [4] F. Janabi-Sharifi, A. Jalali, and I. D. Walker, “Cosserat rod-based dynamic modeling of tendon-driven continuum robots: A tutorial,” IEEE Access, vol. 9, pp. 68703–68719, 2021.
  * [5] Z. Yang, H. Yang, Y. Cao, Y. Cui, and L. Zhang, “Magnetically actuated continuum medical robots: A review,” Advanced Intelligent Systems, vol. 5, no. 6, p. 2200416, 2023.
  * [6] Y. Wu, T. Marks, A. Cherian, S. Chen, C. Feng, G. Wang, and A. Sullivan, “Unsupervised joint 3d object model learning and 6d pose estimation for depth-based instance segmentation,” in Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, pp. 0–0, 2019.
  * [7] J. Davies, D. Lane, G. Robinson, D. O’Brien, M. Pickett, M. Sfakiotakis, and B. Deacon, “Subsea applications of continuum robots,” in Proceedings of 1998 International Symposium on Underwater Technology, pp. 363–369, 1998.
  * [8] X. Dong, D. Axinte, D. Palmer, S. Cobos, M. Raffles, A. Rabani, and J. Kell, “Development of a slender continuum robotic system for on-wing inspection/repair of gas turbine engines,” Robotics and Computer-Integrated Manufacturing, vol. 44, pp. 218–229, 2017.
  * [9] H. Tsukagoshi, A. Kitagawa, and M. Segawa, “Active hose: an artificial elephant’s nose with maneuverability for rescue operation,” in Proceedings 2001 ICRA. IEEE International Conference on Robotics and Automation (Cat. No.01CH37164), vol. 3, pp. 2454–2459, 2001.
  * [10] B. Siciliano, L. Sciavicco, L. Villani, and G. Oriolo, Robotics, Modelling, Planning and Control.  London: Springer, 2009.
  * [11] H. A. Elkholy, A. S. Shahin, A. W. Shaarawy, H. Marzouk, and M. Elsamanty, “Solving inverse kinematics of a 7-dof manipulator using convolutional neural network,” in Proceedings of the International Conference on Artificial Intelligence and Computer Vision (AICV2020) (A.-E. Hassanien, A. T. Azar, T. Gaber, D. Oliva, and F. M. Tolba, eds.), (Cham), pp. 343–352, Springer International Publishing, 2020.
  * [12] R. Grassmann, V. Modes, and J. Burgner-Kahrs, “Learning the forward and inverse kinematics of a 6-dof concentric tube continuum robot in se(3),” in Proceedings 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 5125–5132, 2018.
  * [13] K. Iyengar, G. Dwyer, and D. Stoyanov, “Investigating exploration for deep reinforcement learning of concentric tube robot control,” International Journal of Computer Assisted Radiology and Surgery, vol. 15, pp. 1157–1165, Jul 2020.
  * [14] N. Liang, R. M. Grassmann, S. Lilge, and J. Burgner-Kahrs, “Learning-based inverse kinematics from shape as input for concentric tube continuum robots,” in Proceedings 2021 IEEE International Conference on Robotics and Automation (ICRA), pp. 1387–1393, 2021.
  * [15] A. Lotfavar, S. Hasanzadeh, and F. Janabi-Sharifi, “Cooperative continuum robots: Concept, modeling, and workspace analysis,” IEEE Robotics and Automation Letters, vol. 3, no. 1, pp. 426–433, 2018.
  * [16] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in Proceedings 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248–255, 2009.