Learning Mobile Manipulation

David Watkins

Submitted in partial fulfillment of the

requirements for the degree of

Doctor of Philosophy

under the Executive Committee

of the Graduate School of Arts and Sciences

COLUMBIA UNIVERSITY

2024

© 2024

David Watkins

All Rights Reserved

Abstract

Learning Mobile Manipulation

David Watkins

Providing mobile robots with the ability to manipulate objects has, despite
decades of research, remained a challenging problem. The problem is
approachable in constrained environments where there is ample prior knowledge
of the environment layout and manipulatable objects. The challenge is in
building systems that scale beyond specific situational instances and
gracefully operate in novel conditions. In the past, researchers used
heuristic and simple rule-based strategies to accomplish tasks such as scene
segmentation or reasoning about occlusion. These heuristic strategies work in
constrained environments where a roboticist can make simplifying assumptions
about everything from the geometries of the objects to be interacted with,
level of clutter, camera position, lighting, and a myriad of other relevant
variables. The work in this thesis will demonstrate how to build a system for
robotic mobile manipulation that is robust to changes in these variables. This
robustness will be enabled by recent simultaneous advances in the fields of
big data, deep learning, and simulation. The ability of simulators to create
realistic sensory data enables the generation of massive corpora of labeled
training data for various grasping and navigation-based tasks. It is now
possible to build systems that work in the real world trained using deep
learning entirely on synthetic data. The ability to train and test on
synthetic data allows for quick iterative development of new perception,
planning and grasp execution algorithms that work in many environments.

To build a robust system, this thesis introduces a novel multiple-view shape
reconstruction architecture that leverages unregistered views of the object.
To navigate to objects without localizing the agent, this thesis introduces a
novel panoramic target goal architecture that takes previous views of the
agent to inform a policy to navigate through an environment. Additionally, a
novel next-best-view methodology is introduced to allow the agent to move
around the object and refine its initial understanding of the object. The
results show that this deep learned sim-to-real approach performs best when
compared to heuristic-based methods in terms of reconstruction quality and
success-weighted-by-path-length (SPL). This approach is also adaptable to the
environment and robot chosen due to its modular design.

###### Table of Contents

  1. 1 Introduction
    1. 1.1 Mobile Manipulation
    2. 1.2 Problem Statement
    3. 1.3 Approach
    4. 1.4 Procedure
  2. 2 Related Work
    1. 2.1 Robotic Navigation
    2. 2.2 Robotic Visual Shape Understanding
    3. 2.3 Mobile Manipulation
  3. 3 Learning Visual Navigation
    1. 3.1 Introduction
    2. 3.2 Method
      1. 3.2.1 Formulation
      2. 3.2.2 Navigation Pipeline
    3. 3.3 Experiments
      1. 3.3.1 Network Training Setup
      2. 3.3.2 Comparison Methods
      3. 3.3.3 Evaluation Criteria
      4. 3.3.4 Navigation Results
    4. 3.4 Experiment: MineRL Basalt Competition
      1. 3.4.1 Problem Setup
      2. 3.4.2 Methodology
      3. 3.4.3 Results
    5. 3.5 Conclusion
  4. 4 Visual Tactile Manipulation
    1. 4.1 Introduction
    2. 4.2 Visual-Tactile Geometric Reasoning Method
    3. 4.3 Completion of Simulated Geometric Shapes
    4. 4.4 Completion of YCB/Grasp Dataset Objects
      1. 4.4.1 Mesh Generation
    5. 4.5 Comparison to Other Completion Methods
      1. 4.5.1 Geometric Comparison Metrics
      2. 4.5.2 Grasp Comparison in Simulation
      3. 4.5.3 Live Grasping Results
    6. 4.6 Conclusion
  5. 5 Two-View Shape Understanding
    1. 5.1 Introduction
    2. 5.2 Methodology
    3. 5.3 Experiments
      1. 5.3.1 Dataset Generation
      2. 5.3.2 Two-View Ablation
      3. 5.3.3 Evaluation
    4. 5.4 Results
    5. 5.5 Conclusion
  6. 6 Mobile Manipulation
    1. 6.1 Introduction
    2. 6.2 Method
      1. 6.2.1 Navigation to the object
      2. 6.2.2 Isolate Object Geometry
      3. 6.2.3 Completion
      4. 6.2.4 Next-Best-View
      5. 6.2.5 Panoramic Prediction
      6. 6.2.6 Two-View Completion
      7. 6.2.7 Manipulation
    3. 6.3 Experiments
      1. 6.3.1 Reconstruction Quality Tests
      2. 6.3.2 Navigation Tests
      3. 6.3.3 End-to-End Mobile Manipulation Testing
    4. 6.4 Conclusion
  7. 7 Multiple View Shape Understanding
    1. 7.1 Introduction
    2. 7.2 Methodology
      1. 7.2.1 CNN Architecture
    3. 7.3 Experiments
      1. 7.3.1 Performer Tests
      2. 7.3.2 Evaluation
    4. 7.4 Results
    5. 7.5 Conclusion
  8. 8 Discussion and Conclusion
    1. 8.1 Introduction
    2. 8.2 Summary
    3. 8.3 Contributions
    4. 8.4 Current Limitations and Future Work
      1. 8.4.1 Learned Visual Navigation Future Work
      2. 8.4.2 Visual-Tactile Shape Understanding Future Work
      3. 8.4.3 Multi-View Shape Understanding Future Work
      4. 8.4.4 Mobile Manipulation Future Work
    5. 8.5 Learning Mobile Manipulation: A Crucial Step in the Future of Robotics
  9. A MNIST Digit Completion
    1. A.1 Introduction
    2. A.2 Methodology
      1. A.2.1 Autoencoder Reconstruction
      2. A.2.2 Single-View Reconstruction
      3. A.2.3 Two-View Reconstruction
      4. A.2.4 Multiple-View Reconstruction
    3. A.3 Experiments
      1. A.3.1 Dataset Modification
      2. A.3.2 Metrics
    4. A.4 Results
    5. A.5 Conclusion
  10. B Combining Learning from Human Feedback and Knowledge Engineering to Solve Hierarchical Tasks in Minecraft
    1. B.1 Introduction
    2. B.2 Problem Setup
    3. B.3 Methods
      1. B.3.1 State Classification
      2. B.3.2 Estimated Odometry
      3. B.3.3 Learning and Engineering Subtasks and the State-Machine
      4. B.3.4 Evaluation Methods
    4. B.4 State Classifier Labeling GUI
    5. B.5 State-Machine Definition for each Task
    6. B.6 Human Evaluation Interface
    7. B.7 TrueSkillTM Score per Match
    8. B.8 Pairwise Comparison per Performance Metric and Task
    9. B.9 Samples of Hybrid Agent Solving the Tasks
    10. B.10 Results and Discussion
    11. B.11 Conclusions

Acknowledgements

I would like to acknowledge all members of the Columbia Robotics Lab (CRLab)
for making my stay there an enjoyable and rewarding experience. In particular,
I would like to thank Adam Richardson, Bohan Wu, Caroline Weinberg, Chaiwen
Chou, Feng Xu, Gavi Rawson, Ian Huang, Jack Shi, Jenny Li, Jiaheng Hu,
Jonathan Sanabria, Lucas Schuermann, Madhavan Seshadri, Neil Chen, Shriya
Balaji Palsamudram, Vaibhav Vavilala, Wenhao Li, Xuelong Mu, and Zizhao Wang
for their support and collaborative effort during my Ph.D. I would like to
acknowledge Amy Xu, Ashley Kling, Dr. Boyan Penkov, Chris Mulligan, Dave
Chisholm, Emily Chen, Howon Byun, Katy Gero, Kevin Kwan, Khaled Atef, Levi
Oliver, Lynne Weber, and Oriana Fuentes for their collaboration during my
tenure at Columbia University. All the work they have done has broadened my
understanding of the field of computer science.

My research colleagues have provided me with a wealth of support and knowledge
throughout my studies. Dr. Iretiayo Akinola and Dr. Jacob Varley both
collaborated and taught me about the world of robotics and research while
making our work a joy. Dr. Carmine Elvezio and Henrique Maia provided me with
plenty of jokes, guidance, and support during the difficult years of the
pandemic while also providing great feedback about my research. To my
collaborators Jingxi Xu, John Hui, and Dr. Vinicius Goecks, whose research
expertise and excellent attitude towards computer science helped guide me
through a variety of projects. I would like to acknowledge Dr. Caroline Yu,
Dr. Cassie Meeker, Chad Dechant, Dr. Richard Townsend, and Dr. Travis Riddle,
for collaborating with me on various research projects.

To my teachers and mentors, Ann Bozdogan, Anne Fleming, Bill Andersen, Dr.
Buck Weaver, Cammy Morteo, Dr. Carole Srinivasan, Sister Catherine Clifford,
Colm McGarry, Cynthia Daigle Xenakis, Dana Gurwitch, Edward Barry, Edward
Kern, George Peterson, Gerald Herlihy, Hipolito Rivera, James Hyland, Dr.
Janet Kayfetz, Jayant Srinivasan, Joyce Cavanaugh, Judy Landis, Kathleen
Germaine, Kelly Naughton, Michael Alger, Dr. Pierluigi Miraglia, Shreyas Shah,
Stephen Flynn, Walter Johnson, and Prof. Wayne Snyder, whose guidance
throughout my life has molded me into the person I am today. I am grateful to
each educator who has guided me to this point in my life.

The faculty at Columbia University have provided me with years of experience
that helped me throughout my research experience. Prof. Jae Woo Lee, Prof.
John Kender, and Prof. Paul Blaer, have all helped me through their anecdotes
and mentoring. Thank you to Prof. Daniel Rubenstein, Prof. John Kymissis,
Prof. Martha Kim, and Prof. Valerie Purdie-Greenaway, for their collaboration
during my Ph.D. The members of my committee, Prof. Matei Ciocarlie, Dr.
Michael Reed, and Prof. Shuran Song, deserve special thanks for the guidance
and support provided to me throughout this process. I also extend thanks to my
fellowship advisor, Dr. Nicholas Waytowich, who collaborated and advised me on
novel research methods, and exposed me to a wealth of information.

My advisor, Prof. Peter Allen, without whom none of this would have been
possible. Peter’s knowledge and guidance gave me the tools to become the
researcher I am now. He has given me the opportunity to improve the world
through robotics. I am eternally grateful.

To my siblings, Jonathan Watkins, Emelie Watkins, and Matthew Watkins, whose
humor and support over the course of my life have been invaluable resources to
me. Our many moments of joy and fun together have helped to keep me engaged
and youthful during my life and especially during my Ph.D.

To my grandparents, $\dagger$Rafael Valls, Lillian Emanuelli, Henry Watkins,
and Jacqueline Watkins, whose infinite wisdom and support throughout my life
has fueled my curiosity for science and discovery. I am extremely fortunate to
have been able to share this part of my life with them.

Rarely does one get to extend as much gratitude as I have for both my parents,
David Watkins, and Lillian Watkins. Without them, I would not have been able
to start or finish my Ph.D. program. The wealth of support, validation,
expertise, and solutions they have provided me over the 27 years of my life
has been invaluable. I could not and would not ask for better parents than
them.

Funding Sources

Graduate study was supported by a fellowship from the U.S. Army Research
Laboratory through the Oak Ridge Associated Universities and by NSF Grant CMMI
1734557.

This research was sponsored by the Army Research Laboratory and was
accomplished under Cooperative Agreement Number W911NF-18-2-0244 and
W911NF-20-2-0114. The views and conclusions contained in this document are
those of the authors and should not be interpreted as representing the
official policies, either expressed or implied, of the Army Research
Laboratory or the U.S. Government. The U.S. Government is authorized to
reproduce and distribute reprints for Government purposes notwithstanding any
copyright notation herein.

Dedication

I dedicate this thesis to my parents, David Vincent Watkins and Lillian Valls
Watkins.

Preface

Before you lie the dissertation "Learning Mobile Manipulation", the basis of
which is a solution to the problem of robotic mobile manipulation. This
document has been written to fulfill the graduation requirements of the
Computer Science graduate program at Columbia University in the city of New
York. I was engaged in researching and authoring this dissertation from
January 2020 to May 2022.

My initial background before starting my Ph.D. was specifically in software
design principles and programming languages. When I started working in my
advisor’s, Peter Allen’s, lab, I was able to leverage my understanding of
parallel programming to optimize algorithmic approaches to robotics. Over the
course of my Ph.D., I have leveraged a software-oriented approach to robotics
that has allowed me to create robust solutions that work in both real and
simulated contexts. I have gained skills in robotic navigation, manipulation,
and deep learning through the many different projects I have worked on.

This project is a culmination of work done over the course of my Ph.D. in
robotic manipulation, shape understanding, robotic navigation, and next-best-
view planning. The research questions were formulated together with my
advisor, Peter Allen. This research was challenging, but through this
investigation I was able to produce a unique solution to mobile manipulation.

My motivation for this work is to provide a unique perspective on mobile
manipulation. The field of robotics is currently undergoing a transition
between algorithmic approaches from the past 60 years of robotics research and
new deep learned approaches. These approaches are both valid, but which
approach, or a hybrid of the two, will perform best remains to be seen. In my
thesis I have put effort into building a hybrid approach to solve the problem
of mobile manipulation.

The audience of this dissertation is aspiring roboticists, veterans of the
field of robotics, or people looking to implement mobile manipulation systems.
. I constructed this thesis to document my research conducted during my Ph.D.
first, but second to illustrate the challenges a new researcher may face when
trying to build complicated end-to-end systems that require deep learning and
control methods.

I would like to acknowledge the editors of this thesis, Dr. Janet Kayfetz and
Prof. Peter Allen. The chapters of this thesis are based on previously
published work. Each of these chapters have shared authors who have
contributed knowledge and writing to help me accomplish this research. The co-
contributors are as follows:

  1. 1.

Chapter 3 Jingxi Xu, Nicholas Waytowich, and Peter Allen

  2. 2.

Chapter 4 Jacob Varley and Peter Allen

  3. 3.

Chapter 5 Jacob Varley and Peter Allen

  4. 4.

Chapter 6 Peter Allen, Henrique Maia, Madhavan Seshadri, Jonathan Sanabria,
Nicholas Waytowich, and Jacob Varley

  5. 5.

Chapter 7 Jacob Varley, Krzysztof Choromanski, and Peter Allen

  6. 6.

Appendix B Vinicius G. Goecks, Nicholas Waytowich, and Bharat Prakash

Thank you to all my co-contributors for your help and expertise over the
course of my graduate studies.

I hope you enjoy your reading.

David Watkins

## Introduction

### 1.1 Mobile Manipulation

The field of mobile manipulation derives its direction from the set of skills
used by humans as they interact with their environment. As toddlers learn to
walk by 11-12 months of age, they explore and reason about their environment,
and they also learn to manipulate objects within that space, for example
stacking, sorting, and counting. These same skills encompass the multiple
facets of robotic mobile manipulation research. A robot that can successfully
manipulate its environment must move throughout an environment or space to
reason about its environment and to manipulate objects within that space. It
is these skills and abilities translated into a complexity of multiple systems
operating in concert that roboticists have worked to address for decades.

While researchers are currently able to build robotic mobile manipulation
systems to manipulate objects within an environment relying on specific
constraints on the problem, building a general-purpose mobile manipulation
system remains a challenging engineering problem. The engineering difficulty
increases exponentially as the number of intertwined systems increases. More
specifically, mobile manipulation requires navigation, manipulation, and
robotic control systems where each requires its own set of pipelines, with
overall complexity additionally impacted as the environment becomes cluttered,
messy, or as dynamic elements like people in the space move around. Robots,
unlike people, face three critical issues when interacting in a space: 1) they
are not localized in the environment, 2) they do not know the objects they
will manipulate beforehand; and 3) they do not know how to leverage tactile
information or multiple views of an object. This thesis addresses these issues
by proposing a solution to localization-free robotic mobile manipulation to
manipulate unseen objects leveraging multiple object views.

Figure 1.1: The PR2 is emblematic of mobile manipulation. It has a set of arms
for interacting with its environment. It uses a set of wheels for locomotion.
For sensory information, it has a LIDAR sensor, a set of cameras, as well as
optional force torque and tactile sensing. It is important to highlight these
traits as they describe what the target robot for this research is.

A robotic mobile manipulation pipeline can be expressed as a state machine
where the agent is trying to reach a goal state via a sequence of transitions.
There are many goal configurations that a robotic agent may want to achieve,
such as lifting an object or opening a door. Important attributes of a mobile
manipulator include that it can move around its environment, capable of
acquiring sensory information about its surroundings, and has a manipulator
that can interact with objects in a workspace. There have been many examples
of mobile manipulators, but one is particularly emblematic of the title
"Mobile Manipulator": the Willow Garage PR2. The PR2 has two arms each with a
parallel jaw gripper, a LIDAR sensor, a series of cameras, and wheels for
locomotion. Each of these traits are highlighted in Figure 1.1. Highlighting
these specific set of traits is important as it establishes what kind of robot
would be relevant for this thesis. It also helps illustrate how each state in
the proposed mobile manipulation state machine maps onto a real-world example:

Figure 1.2: The state transition diagram of a mobile manipulator. The agent
starts from some starting configuration eventually accomplishing a target
task. Simplifying a mobile manipulation pipeline in this paradigm helps to
illustrate what is required to innovate research in this field.

  1. 1.

Locomotion The robotic agent will move through its environment to a target
position or configuration of its joints to allow itself to begin executing a
manipulation task. This task can be done by a variety of robotic agents
including humanoid, drone, quadrupedal, and aquatic. Once at the target
location, the agent will then begin a data acquisition stage.

  2. 2.

Data Acquisition The agent needs to collect information about its target
object using cameras, tactile sensors, temperature sensors, and any modality
that provides information about the target. This stage informs the agent as to
whether it should continue searching for additional information from another
position relative to the object to increase the success of a manipulation
step. If the agent decides that it has received enough information, it will
then move to a manipulation planning step.

  3. 3.

Manipulation The agent will utilize the information it collected to plan a
series of moves allowing it to execute a manipulation task on the environment.
This manipulation task could be picking up a piece of fruit in a kitchen,
opening a cabinet, or loading a dishwasher. Manipulation planning and
execution is a combination of robotic control systems and algorithmic
approaches to utilize information about the agent’s environment to better
accomplish a task.

The state transition diagram is shown in Figure 1.2.

Mobile manipulation encompasses such a wide variety of robotic fields and
disciplines that it can be challenging to address in a single work. But each
time researchers address robotic mobile manipulation, the work brings
roboticists closer to a general-purpose mobile manipulation platform: one that
will be able to aid humans in their homes, workplaces, and lives to accomplish
more than they could do on their own. This dissertation is no different.

### 1.2 Problem Statement

The impetus for mobile manipulation research can be stated as follows: There
is no clear data-driven approach for localization free robotic mobile
manipulation. Such manipulation systems are not yet generalized to arbitrary
locations or unseen objects in a real-world environment considering sensor
noise, task location, workspace configuration, and other environmental
variables. Current systems constrain the problem to conform to global
positions in the environment or restricting the problem to a known object
database. The lack of a natural basis for coordinate systems in the real-world
is preventing these agents from handling more complex scenarios outside of
controlled lab and factory settings where researchers can choose and identify
experimental constraints. Constraining the set of objects also prevents
generalization to new domains of problems where knowing object geometry
beforehand is not possible.

### 1.3 Approach

This dissertation presents a novel approach to mobile manipulation. The scope
of mobile manipulation is restricted to manipulating an object indoors, in an
environment the agent knows beforehand, with an object that the robotic agent
can grasp. The agent is not aware of the object beforehand, nor is it allowed
to localize itself at runtime. The agent can navigate to an object, understand
the shape of that object, and manipulate the object without requiring it to
know its position in the environment at runtime via transitioning from
heuristic methods to data driven approaches. This system is described in five
stages: (1) learned visual panoramic-target navigation, (2) single-view shape
understanding, (3) next-best-view planning, (4) multiple-view shape
understanding, and (5) manipulation. The agent utilizes deep-learned methods
to determine a near optimal path to the object location without requiring a
global position to the goal location, but instead a novel panoramic target
goal image. Because the agent is a robotic mobile manipulator, the agent’s
initial understanding of the object can be updated using multiple views to
refine the predicted shape of the object. Additionally, the system can utilize
tactile and visual information for multi-modal refinement of the shape
geometry. An overview of this pipeline is shown in Figure 1.3.

Figure 1.3: Two-View Robotic Mobile Manipulation Pipeline Given an image of
the goal object and a panoramic target goal, the agent navigates to the region
and aligns itself with the object. It then uses a single-view completion of
the object to predict a next-best-view and utilizes a predicted panoramic
image to navigate to that next-best-view. Upon capturing the next-best-view it
refines its prediction of the object to calculate and perform a grasp on the
object.

A key idea to explore is the application of data driven techniques to enable
robotic mobile manipulation. The various navigation, grasping, and shape
understanding sub-tasks are approached using heuristics, or static rule-based
approaches. These heuristic methods work if the underlying assumptions hold.
For example, shape completion by symmetry works very well for symmetric
objects, or navigation relying on odometry and lidar for localization assumes
no sensor error in ideal circumstances. Unfortunately, these heuristic methods
fail when utilized in situations where the underlying assumptions do not hold.
In the presented framework, it employs data driven techniques making the
systems more robust in comparison to heuristic approaches. This data driven
approach enables this system to adapt to new situations when presented new
training data. Training data can then be created at scale using simulation
enabling the system to be exposed to a wide variety of conditions, making this
approach flexible and useful to new robotics technologies.

### 1.4 Procedure

This thesis presents a data driven approach for localization-free mobile
manipulation of novel objects. Because the mobile manipulation pipeline
follows a data-driven approach, it is generalizable to a variety of different
environments. The procedure is as follows:

  1. 1.

The agent will receive an 8-image RGBD panoramic target goal where the target
object is located. Using a learned navigation system which uses only visual
information to navigate to the target location, the agent successfully
navigates to the position and aligns itself with the object. This navigation
pipeline allows the agent to move through an environment to arbitrary goals
without localizing itself at runtime. The details of this system are addressed
in Chapter 3.

  2. 2.

Upon reaching the target location, the agent aligns itself with the object. It
can utilize visual-tactile information to get a better understanding of the
object. The agent can also utilize tactile sensors on its end-effector to
capture information about the occluded side of the object. With the current-
view and tactile information, the robotic agent utilizes a visual-tactile
fusion convolutional-neural-network (CNN) to predict the geometry of the
object more accurately than a single-view alone. This stage is addressed in
Chapter 4.

  3. 3.

The agent may instead capture two views of the object. To utilize two views of
the object without localizing the agent, a CNN model is needed that can
leverage these two in-frame 2.5D images. The design and implementation of this
network is described in Chapter 5.

  4. 4.

The agent requires an intelligent methodology for predicting a next-best-view
and navigating to it. The agent can utilize a single-view shape completion CNN
to estimate the geometry of the object. This initial shape understanding is
then used to predict a next-best-view of the object by considering only the
parts of the predicted shape that the network was uncertain about. Using this
prediction, the agent then navigates to this next-best-view by predicting the
panoramic view at the new target location and reusing the visual navigation
system to navigate there. Upon reaching the target location, the agent
captures a second view of the object and uses the two-view shape completion
architecture that uses two unregistered views to create a more accurate shape
estimation. Once the shape is finalized the agent can manipulate the object.
The mobile manipulation system is described in Chapter 6.

  5. 5.

There is a lot of information potentially gathered by the agent while moving
around the object in the form of multiple shape views. To leverage these
views, a CNN model is needed to utilize an arbitrary number of in-frame 2.5D
images. The use of performers, a form of attention layers, is used to refine
the initial shape completion of an object with many views. The design and
implementation of this architecture is described in Chapter 7.

The contributions of the work described in this thesis are:

  * •

An end-to-end system for mobile manipulation of household graspable objects
utilizing novel learning algorithms

  * •

An algorithm that takes an initial shape completion estimate of the
manipulation target using voxel grid occupancy thresholding to plan the next-
best-view

  * •

An algorithm that uses a predicted panoramic goal and reuses the long-range
learned image navigation system to navigate to the next-best-view locally

  * •

A learned two-view shape completion method that creates a more accurate
reconstruction for robotic manipulation using an initial view and a next-best-
view

  * •

A learned visual-tactile shape completion method that given the initial and
tactile views creates a more accurate reconstruction for robotic manipulation

  * •

A learned multiple-view shape completion method that can take an arbitrary
number of views to refine its understanding of the object geometry

  * •

A series of metrics and benchmarks that can be used to evaluate mobile
manipulation systems in future work

  * •

Ablation studies demonstrating the utility of system subcomponents as well as
overall system performance

  * •

An open-source dataset of trajectories, object point clouds, and object
placements in real-world scanned environments to reproduce results

## Related Work

Mobile manipulation is composed of multiple subcomponents all working together
to manipulate a robot’s environment. This process has been studied and
developed for decades, with one of the first informative work from Joshi et
al. describing the process of modeling a robot with a mobile base while
accounting for noise in the environment [81]. This thesis describes how
multiple subcomponents can come together to address mobile manipulation
holistically. These subcomponents cover different subfields of robotics,
including shape understanding, navigation, next-best-view planning, and
manipulation.

### 2.1 Robotic Navigation

###### Reinforcement learning methods for navigation

Previous work in visual navigation [188] provides a target-driven
reinforcement learning framework for robotic visual navigation. Our method
shares the same objective of navigating to the goal position using the goal
image, the current image, and a sequence of history images. [6, 147] trains a
reinforcement learning (RL) agent to navigate in realistic cluttered
environments using a PointGoal (e.g., a specific location of the goal target).
They assume an idealized GPS which constantly provides the relative goal
position of the target and use this information to train their agents. Both
[147] and [188] claim that their learned policy generalizes across targets and
environments. [188] only evaluates their method on new targets that are
several steps away from the targets that the agent is trained on, and the
scene-specific layer must be retrained for the policy to work in a new
environment. [147] relies on an idealized GPS and the specific location of the
goal and it generalizes to new environments by learning to imitate a bug-
algorithm to follow the boundaries of its environment. Imagine a scenario
where a person is placed into a building they have not seen before with
nothing but an image of the place they need to get to. It would be unfair for
us to expect this person to navigate to the target location in any efficient
manner. Therefore, a robotic agent would be unable generalize to new untrained
environments using vision alone. [118, 116] evaluate reinforcement learning
strategies and associated deep learning architectures. Both works presented a
novel approach to navigating through structured environments and set a
baseline for navigation research. The experiments in [118] use a synthetic 3D
maze environment with a single goal which does not exhibit the complexity of
real-world settings. [116] trains an agent to navigate a long-range path to
the target goal using real-world Google map street views. However, they
provide the agent with the coordinate of the goal rather than images. The
street view also has extra information brought by signs and building
characteristics. [48, 53, 31] present hierarchical robot navigation methods
using reinforcement learning to learn local and short-range obstacle avoidance
tasks. They propose sampling-based path planning algorithms as global map
planners. These methods use 1D lidar sensor data and a dynamic goal position
as input. [61, 89] use value iteration networks [160] to learn navigation
strategies in simplistic synthetic simulated environments. [123] evaluates
different representations for target-driven visual navigation using a semantic
target and an off-the-shelf segmenter. [23] presents a method to navigate to a
fixed goal in a known environment.

###### Supervised learning methods for navigation

While most work in using learning methods for robotic navigation relies on
deep RL because of its self-supervised convenience, supervised methods in
navigation are less explored. [141, 106] apply CNNs to help robotic navigation
but they reply on odometry and need the goal location specified. They train
models to only execute simple tasks such as collision detection and position
evaluation.

###### Datasets and simulators for navigation environments

The broader area of active and embodied perception has received increased
interest focusing on robotics navigation, task planning, and manipulation. New
datasets for navigation environments have been created which feature fully
scanned 3D homes and buildings such as the Stanford2D3DS [10] and Matterport3D
[27] datasets. Additionally, a large synthetic dataset of homes is offered by
both the SUNCG [155] and Gibson environments [180]. These new environments
enable researcher to train an agent in simulation using real-world data and
obtain training data much faster than would be possible in the real-world
alone. Since the advent of these datasets, the MINOS [146], Gibson [180],
Habitat [147], and AI2THOR [95] simulators all offer simulation for real-world
navigation. These simulators allow agents to be trained using ground truth
positioning, fast rendering, and training RL agents at scale. The Gibson
simulator uses PyBullet [37] to simulate collisions with the environment as
well as dynamic environment tasks. Additionally, it offers a framework,
Goggles, which takes RGB images from the simulator and uses a learned transfer
model to render the image photo-realistically.

###### End-to-end machine learning

The term “end-to-end machine learning” is used for algorithms that learn
purely from data with minimal bias or constraints added by human designers,
besides the ones that are already inherently built-in to the learning
algorithm. For example, deep reinforcement learning algorithms learning
directly from raw pixels [121, 120] and algorithms that automatically
decompose tasks in hierarchies with different time scales [41, 168]. This
often includes massive parallelization and distributed computation [46, 144]
to fulfill the data requirement of these algorithms. Other works train robotic
agents to navigate through their environment using RGBD information to
determine optimal discrete steps to navigate to a visual goal [173].

###### Human-in-the-loop machine learning

Learning from human feedback can take different forms depending on how human
interaction is used in the learning-loop [174, 57]. A learning agent can be
trained based on human demonstrations of a task [132, 9, 20]. Agents can learn
from suboptimal demonstrations [22], end goals [136], or directly from
successful examples instead of a reward function [47]. Human operators can
augment the human demonstrations with online interventions [2, 1, 145, 58] or
offline labeling [143, 142] while still maintaining successful at the proposed
task. Agents can learn the reward or cost function used by the demonstrator
[127, 50] through sparse interactions in the form of evaluative feedback [92,
109, 171] or human preferences given a pair of trajectories [34].
Additionally, agents can learn from natural language-defined goals [185].
Finally, agents can learn from combining human data with reinforcement
learning [137, 59, 139].

### 2.2 Robotic Visual Shape Understanding

###### Algorithmic shape understanding

Several recent uses of tactile information to improve estimates of object
geometry have focused on the use of Gaussian Process Implicit Surfaces (GPIS)
[176]. Several examples along this line of work include [24, 184, 16, 44, 78,
154, 111]. This approach can quickly incorporate additional tactile
information and improve the estimate of the object’s geometry local to the
tactile contact or observed sensor readings. There have additionally been
several works that incorporate tactile information to better fit planes of
symmetry and superquadrics to observed point clouds [75, 74, 15]. These
approaches work well when interacting with objects that conform to the
heuristic of having clear detectable planes of symmetry or are easily modeled
as superquadrics.

###### Visual fusion reconstruction

There has been much successful research in utilizing continuous streams of
visual information like Kinect Fusion [126] or SLAM [161] to improve models of
3D objects for manipulation, an example being [98, 97]. In these works, the
authors develop an approach to building 3D models of unknown objects based on
a depth camera observing the robot’s hand while moving an object. The approach
integrates both shape and appearance information into an articulated ICP
approach to track the robot’s manipulator and the object while improving the
3D model of the object. Similarly, another work [70] attaches a depth sensor
to a robotic hand and plans grasps directly in the sensed voxel grid. These
approaches improve their models of the object using only a single sensory
modality but from multiple points in time.

###### Visual deep-learning-based reconstruction

Modern work in shape reconstruction has centered around pose estimation of
known objects [134, 135]. These works utilize multi-layer-perceptrons (MLP) to
encode points into a dense representation to then perform a classification
task. This can be useful for scene segmentation, part identification, or pose
estimation. Other work has divided the reconstruction and pose estimation into
two different steps to allow for a more refined mesh prediction of a set of
objects on a table [76]. Additional work with point cloud reconstruction has
taken each layer of points to further refine the prediction of the object
[105]. Other shape completion systems utilize voxel representations to perform
shape understanding by voxelizing the input point cloud into a fixed size
voxel representation [183, 11].

###### Visual-tactile shape understanding

The idea of incorporating sensory information from vision, tactile and force
sensors is not new [4]. Despite the intuitiveness of using multi-modal data,
there is still no consensus on which framework best integrates multi-modal
sensory information in a way that is useful for robotic manipulation tasks.
While prior work has been done to complete geometry using depth alone [39,
175], none of these works consider tactile information. More recent works have
looked at informing decision making using partial observations from tactile
information [182].

Varley et al. created a shape completion method using single depth images
[163]. The work provides an architecture to enable robotic grasp planning via
shape completion, which was accomplished using a 3D CNN. The network was
trained on an open-source dataset of over 440,000 3D exemplars captured from
varying viewpoints. At runtime, a 2.5D point cloud captured from a single
point of view was fed into the CNN, which fills in the occluded regions of the
scene, allowing grasps to be planned and executed on the completed object. The
runtime of shape completion is rapid because most of the computational costs
of shape completion are borne during offline training. This prior work
explored how the quality of completions vary based on several factors. These
include whether the object being completed existed in the training data, how
many object models were used to train the network, and the ability of the
network to generalize to novel objects, allowing the system to complete
previously unseen objects at runtime. The completions are still limited by the
training datasets and occluded views that give no clue to the unseen portions
of the object. From a human perspective, this problem is often alleviated by
using the sense of touch.

###### Implicit shape reconstruction

New work in rendering unseen images using implicit representations have been a
potential source for new research. Work done by Sitzmann et al. analyzed the
effectiveness of generating meshes, images, and even sound from input data via
implicit representations and novel use of sine as an activation function
[153]. Other work looked at generating novel views of an environment using
only a single view [45]. Other researchers have utilized text-based mesh
generation using these new implicit representations [88].

###### Object datasets

Many different object datasets exist for performing reconstruction. An early
attempt at reconciling the disconnect between simulation and real is the YCB
household object dataset [25]. This dataset consisted of over seventy
purchasable real-world objects which could be used by researchers to perform
experiments but also came with high resolution scanned meshes to perform
simulation experiments. Additional datasets, such as Thingi10K, aim to provide
a 3D printable set of meshes that can be used for a variety of mesh generation
tasks [187]. The Grasp database dataset incorporated data from a variety of
different graspable meshes that have well defined geometry [19]. The ShapeNet
dataset consists of labeled high resolution meshes of different objects that
are useful for creating environments or visual object analysis [29].

### 2.3 Mobile Manipulation

###### Autonomous mobile manipulation

Fully autonomous mobile manipulation has long been an important goal in
robotics, with particular focus on such wide-ranging applications as
manufacturing, warehousing, construction, and household assistance [83, 110,
77]. Mobile manipulation encompasses a sequence of robot navigation, object
detection, view planning, grasp planning, and grasp execution which makes it a
challenging task. This is particularly evident when the task environment is
dangerous to be explored by a human. Increasing efforts in this problem aim to
map, traverse, and grasp in unknown and partially observable environments
[148, 129, 169]. Active perception to set up a goal based on some current
belief to achieve an action is a good model for how to potentially solve this
problem [12].

Navigating accurately and efficiently in an environment is a crucial first
step to achieving autonomous mobile manipulation. Traditional position and
mapping focused algorithms include a Simultaneous Localization and Mapping
(SLAM) [43] technique to plan a collision free path. Such techniques, while
effective in mapping and localizing in an unknown environment, are sensitive
to odometry errors and noise. Increasingly, reinforcement learning-based
techniques have also been used to solve navigation in complex environments
[188, 52, 117]. However, reinforcement learning uses sparse rewards and
requires extremely large amount of training episodes to achieve a good
navigation model.

Mobile manipulation has been explored in a variety of contexts, including
household mobile manipulation. One of the first examples of mobile
manipulation is HERB [156] allowing a robotic agent to select grasping targets
and navigate through an environment. However, it required checkerboard
localization and precise sensors to plan tasks. Nevertheless, this work helped
form the basis for robotic mobile manipulators, including the separation of
manipulation and navigation tasks. Some works have come out more recently
advancing on this initial vision that utilize global localization and point
cloud reconstruction in household environments [17, 38, 178].

Indoor map exploration provides a mobile robot with many opportunities to
navigate around the environment but becomes increasingly difficult when
deprived of sensory information. Several authors have proposed reinforcement
learning techniques for this task [188, 7, 13]. We share a common goal as Zhu
et al. [188] where the task is to navigate using images. Some assume the
presence of an idealized global localization system where they train the bot
to reach the goal location [7, 106, 141, 179]. Another aims to solve the
problem of traversing unknown environments by combining model-based control
with learning-based perception where the task is to produce a set of waypoints
leading up to the goal [13].

###### Grasp planning using visual information

On approaching the goal, the mobile robot captures images nearby containing
the object of interest. However, this image only represents the raw sensory
data, which is incomplete due to the field of view and the approach angle.
Grasp planning with such minimal and incomplete information is a challenging
task prone to failures. Accurate shapes of objects improve grasp planning and
execution success rate. Several works have proposed a deep learning approach
to produce predictive mesh representations of partial views for unseen objects
through either 3D convolutions or graph convolutions for robotic grasping
[164, 40, 107, 172]. Several geometric solutions to object 3D modeling have
been proposed as well [176, 98, 97, 70].

###### Next-best-view planning

A mobile manipulation system also needs to leverage the ability to acquire
additional information by moving a vision system. To find candidate next-best-
views, an algorithm to evaluate the quality of the current geometric
understanding of an object is needed while considering the cost of acquiring
additional views. Foundational work from Connolly did this by identifying
positions of the sensor that will maximize data collected [36]. Much work came
after Connolly et al. utilizing heuristic approaches to determine obstructions
that would block future collected data [131, 26, 30, 166]. A recent survey of
next-best-view algorithms showed that Chen et al. had the best next-best-view
performance [86]. Another work from McGreavy et al. [113] looked at next-best-
view planning using a cylindrical model to analyze the visibility of an
initial object candidate and find an optimal view. Their model does not use a
learned shape completion system and relies on a priori knowledge of objects.
Some newer work in shape segmentation is also applicable for next-best-view
work as they allow for better segmentation for unseen object geometry [181].
Other work in occlusion-based grasping has shown success in mapping voxels
using registered views [82]. One method to capture a next-best-view is to use
an eye-in-hand camera as shown by Potthast et al. [133], however there are
kinematic restrictions with respect to the workspace of the robot that prevent
these views from being captured. Other works have presented object
reconstruction under uncertainty that utilize an algorithmic approach to
estimate the object’s geometry while utilizing the odometry of the mobile
robot [165].

Several mobile manipulation benchmarks exist to evaluate the performance of a
system [21, 149, 158]. These benchmarks evaluate the performance of a mobile
manipulator in a variety of contexts but fail to take advantage of modern
simulators [180] that utilize real-world scanned data of household objects
[25] and environments [27, 10].

## Learning Visual Navigation

To build a mobile manipulation system, a robot must be able to perform
locomotion within its own environment. There are many ways to enable mobility
in a robot, whether its driving itself via wheels, moving quadrupedal legs,
flying via drone rotors, or even rolling its whole body. The goal of
navigation can similarly be general. A robot may navigate through its
environment to collect additional information about potential obstacles. A
robot may be moving through an environment to map its surroundings. To refine
the scope of this work, a solution to localization-free robotic navigation
using visual information only is provided. Using visual information alone
means the agent does not have to worry about odometry at runtime or perform
localization computations using noisy RGBD data. The content in this chapter
will explore how to build a system that can effectively navigate its own
environment via training only in simulation.

The basis for navigating through an environment is inspired by a human’s
ability to reason about their environment to plan navigation tasks and is
fundamental to intelligent behavior. Therefore, it has been a focus of
research in robotics for many years. Traditionally, robotic navigation is
solved using model-based methods with an explicit focus on position inference
and mapping, such as Simultaneous Localization and Mapping (SLAM) [43]. These
models use path planning algorithms, such as Probabilistic Roadmaps (PRM) [87]
and Rapidly Exploring Random Trees (RRT) [101, 99] to plan a collision-free
path. These methods ignore the rich information from visual input and are
overly sensitive to robot odometry and noise in sensor data. For example, a
robot navigating through a room may lose track of its position due to the
navigation software not properly modeling friction.

The exploration of localization free robotic navigation will show that a
history buffer of previously seen images, a panoramic target goal, and use of
a custom autoencoder are all beneficial to the success rate of an agent
navigating to a goal position. A robot’s perception of its nearby environment
can include many errors that are common to human beings. A history buffer
helps prevent the robot from getting stuck in a loop of repeating actions
between time steps. A panoramic target goal allows the agent to better reason
about the goal location without having to worry about whether the goal image
contains too little information for it. A custom autoencoder ensures that the
embedding of the image contains relevant information and not features learned
from an unrelated dataset. Each of these individually contribute and impact to
the overall performance of the agent. Additionally, this framework can be
mapped onto problems in other domains. This chapter will explore an
experimental result using a similar design to build an agent to play
Minecraft.

### 3.1 Introduction

This work in navigation focuses on exploring supervised methods (in
particular, imitation learning) to bring better performance to robotic visual
navigation, while taking advantage of the current progress in robotic
simulators and datasets to efficiently collect training data. A navigation
pipeline is presented where the agent learns to navigate to unseen targets
using the current RGBD view and a novel 8-image panoramic goal without using
GPS, compass, map, or relative position of goals at runtime. Panoramic images
are chosen as they allows for easier acquisition of training data for a policy
model and they allow the agent to generalize better to unseen targets. A
framework to efficiently generate expert trajectories in the Gibson [180]
simulator using a 3D scan of the environment of interest is also shown.
Additionally, a methodology is provided for discretizing a continuous
trajectory into a series of {forward, right, left} commands. Shown in Figure
3.1 is an example successful trajectory of this system.

Figure 3.1: A successful trajectory executed in house17 from the Matterport3D
dataset. The history buffer and current view are the state of the pipeline.
The panoramic goal is 8 RGBD images each taken at a $None$ turn. The top-down
view is the agent moving through the trajectory with the blue sphere as the
start position and the green sphere as the goal position. Smaller blue spheres
are positions that the agent has been to, and the orange spheres are the
remaining positions. The images are taken at the current position of the
robotic agent.

Why use imitation learning instead of a commonly used deep learning method:
reinforcement learning (RL)? Model-free RL agents have performed well on many
robotic tasks [93, 124, 122, 8], leading researchers to rely on RL for robotic
navigation tasks [188, 53, 118, 48]. Recent work in robotic visual navigation
uses reinforcement learning which trains an agent to navigate to a goal using
only the current and goal RGB images [188]. While reinforcement learning has
the convenience of being weakly supervised, it suffers from sparse rewards in
navigation, requires a substantial number of training episodes to converge,
and struggles to generalize to unseen targets. The problem is further
exacerbated when the navigation environment becomes large and complex (across
multiple rooms and scenes with various obstacles), leading to difficult long-
range path solutions. Two issues for RL approaches yet to be addressed are (1)
lack of generalization capability to unseen target goals, and (2) data
inefficiency, i.e., the model requires an enormous number of episodes of trial
and error to converge. These problems become more pronounced in robotic visual
navigation when training an agent in a complex and large environment across
multiple rooms and distinctive styles of scenes and where long-range solutions
are required.

When using an imitation learning methodology, one needs to acquire copious
amounts of labeled training data to train the agent. Issues such as
distributional drift, domain randomization, and gradient decay become more
relevant when switching to this methodology. Thankfully, new advancements in
annotated 3D maps of real-world data, such as Stanford2D3DS [10] and
Matterport3D [27], enable the collection of substantial amounts of indoor
trajectory data. Training an agent in more of an environment is a way to
increase robustness and decrease distributional drift. These kinds of data
acquisition methods can be time-consuming or impossible using a real-world
robot. It does require having a real-world scan of the environment, but
companies such as Matterport [27] or software such as RTABMap [100] make
indoor mapping much easier and affordable than they were previously. One such
example indoor environment from the Matterport3D [27] dataset is shown in
Figure 3.2.

Figure 3.2: An example environment in the Matterport3D [27] dataset showing
the mesh of the environment on the left, some of the panoramas used to
generate the mesh, and the resulting semantic mesh of the environment with
object categories. The dataset contains 80 individually scanned homes each
semantically annotated with relevant labels.

Once the necessary environmental scans are captured, software to collect
images and simulate a robot is needed. Simulation of robotic sensors and
execution of their actions have existed in some form since the beginning of
robotics research on a computer [112]. To capture visual images along a
trajectory, a modern simulator is needed that is capable of rendering images
in color (RGB) and depth. RGB and depth are captured in simulation via
rendering 1) meshes of these environments and 2) the robot via OpenGL.
Simulators capable of collecting this data have arisen in the past few years
in the form of MINOS [146], Gibson [180], Habitat [147], and THOR [188]. These
systems enable simultaneous use of real and simulated environments for
training, without the need for visual domain adaptation. While each has their
own pros and cons, Gibson [180] is used as it features real-world and photo-
realistic data generated from fully scanned 3D homes and buildings that allow
for easier collection of demonstration data for supervised learning. At the
time of this research, Gibson was the only simulator which allowed for us to
simulate robotic grasping in addition to collecting visual information. Since
the publication, Habitat [147] simulator from Facebook has added robotic
grasping to their features as well as faster rendering times. Future work may
want to utilize the features of this newer simulator to collect more data,
however this system performs well despite not having collected data as quickly
as other alternatives could have.

### 3.2 Method

#### 3.2.1Formulation

The goal of this work is to enable the robot to autonomously navigate to a
target position, described by a set of panoramic images taken at the goal,
without providing any odometry, GPS or relative location of the target but
only RGBD input from the robot’s point of view. The agent is not required to
achieve a particular orientation relative to the goal image at runtime. The
problem is referred to as target-driven visual navigation in the literature
[188], where the task objective (i.e., navigation destination) is specified as
input to the model. Traditional learning-based visual navigation methods have
focused on learning goal-specific models that tackle individual tasks in
isolation, where the goal information is hard-coded in the neural network
representations, leading to poor generalization to unseen / unexplored
targets. Target-driven approaches learn to navigate to new targets without re-
training, using a single navigation pipeline.

The navigation pipeline, denoted as $\Pi$, takes as input the observation of
the current state $s_{i}$ at time step $i$, the target information $g$, and
outputs an action $a_{i}\in\\{forward,right,left,done\\}$.

$a_{i}=\Pi\left(s_{i},g\right)$ (3.1)

The left / right action indicates turning the agent in place left / right 10
degrees and the forward action moves the agent 0.1$m$ ahead. The unknown
transition model $\Gamma$ of the environment updates the state, denoted by
$s_{i+1}=\Gamma(s_{i},a_{i})$, when an action $a_{i}$ is executed. The
objective is that given any goal $g$ in the map, a maximum number of steps
$T$, and a success threshold $\zeta$, the navigation pipeline $\Pi$ can
generate a sequence of actions $\\{a_{i}\\},i\in[t]$, which satisfies (1)
$t<T$, (2) $a_{t}=done$, (3) the final location of the robot is within $\zeta$
meters of the target location, and (4) the length of the path should be as
short as possible. The navigation pipeline is fully automated as it learns to
stop at the goal and does not require human intervention.

The state $s_{i}$ is the current RGBD visual observation and a history buffer
of 4 concatenated past RGBD images, both of which are from the agent’s
viewpoint. The goal information $g$ is a set of 8 panoramic RGBD images. An
example of the state and the goal information is shown in Figure 3.1.

#### 3.2.2Navigation Pipeline

The navigation pipeline $\Pi$ consists of three separately trained models
using neural networks, the autoencoder model $A$, the policy model $E$ and the
goal checking model $G$.

The autoencoder model generates latent representations (i.e., embeddings) for
both the state $s_{i}$ and the goal $g$, denoted $A(s_{i})$ and $A(g)$. The
policy model takes two inputs, the embeddings of the current state and the
embeddings of the target, and produces a probability distribution over three
actions, $\displaystyle a_{i}\in\\{\textrm{{forward}, {right}, {left}}\\}\sim
E\left(A(s_{i}),A(g)\right))$. It then picks the action with highest
probability from this distribution. The policy model is responsible for
leading the agent towards the goal with as little exploration as possible. The
goal checking model is a binary function which takes the same input as the
policy model, and decides if the agent has reached the target or not, denoted
by $G(A(s_{i}),A(g))\in\\{1,0\\}$, where 1 corresponds to done and 0
corresponds to not done. An overview of the navigation pipeline is shown in
Figure 3.3.

##### Autoencoder Model

Because the input into the neural network models is RGBD images, the training
is more efficient if embeddings are used instead of the raw input. Instead of
extracting features from an intermediate layer of a pre-trained classifier
such as ResNet-50 [188, 67], an autoencoder is trained from images captured
from the same environment.

Figure 3.3: Overview of the navigation pipeline. The flow starts when a new
navigation task is received. The agent encodes each new image captured from
the environment using a trained Autoencoder $A$. The encoded images are passed
into the Goal Checker $G$ to determine if the agent is $done$. If the $G$ is
not $done$, the agent then passes the current image, the previous set of
images, and the panoramic target goal into a policy model to determine the
next action to take. This action then updates the environment, and the process
repeats until $done$. Figure 3.4: An example of reconstructed images from the
autoencoder model trained in the house2 environment. The top row is three
predicted output images (RGB image appended by depth image); the bottom row is
the original images.

Like RedNet [79], the autoencoder network is based on a 6-layer CNN with batch
normalization on every layer during training. The reconstruction half of the
network is made up of an additional 6 transposed convolutional layers with
batch normalization applied before each transposed convolution. Rectified
linear unit (ReLU) is used as the activation function. The Adam optimizer [91]
is used to minimize the mean squared error between the reconstructed and the
original images. The autoencoder can compress a $256\times 256\times 4$ RGBD
image into the $4096D$ latent space ($\times 64$ space savings). It is then
used to encode each image of the state and each image of the panoramic goal. A
detailed topology of the encoder section is pictured in 3.6(a). An example of
the autoencoder performance is shown in Figure 3.4.

##### Policy Model

The policy model takes as input the embeddings of stacked observations and the
panoramic goal images to generate the next action
$a_{i}\in\\{\textrm{{forward}, {right}, {left}}\\}$.

The policy model is a fully-connected multilayer perceptron (MLP) as shown in
Figure 3.5. Also evaluated is the performance of a variety of other deep
learning architectures including convolution along the temporal dimension and
long short-term memory (LSTM) [72], with different numbers of past images in
the state and a different number of panoramic goal images, but the MLP
architecture using 4 previous images and 1 current image outperforms other
methods. Its larger number of parameters increases its ability to model
complex functions.

The embeddings of the state and the panoramic goal are first concatenated to
form a $13\times 4096$ matrix and then progress through 3 fully-connected
layers followed by batch normalization (during training) and ReLU activation
after each layer, to generate a $16D$ vector. The $16D$ vector passes through
the last fully-connected layer to generate three logits. A SoftMax activation
then outputs a distribution over three actions {forward, right, left}. This
SoftMax activation means the policy model always chooses the most probable
choice of action. The Adam optimizer is used on the cross-entropy loss for
back propagation. At testing, the action with highest probability is chosen
deterministically.

As an ablation study, the performance of several deep learning architectures
is evaluated on a subset of the area1 environment. For this subarea, the model
which performs best is picked from all experiments described in section 3.3.
The models to be evaluated include 1) fully-connected network with a 5-image
history buffer, 2) fully-connected network with 5-image history buffer keeping
every third image, 3) LSTM with a 15-image history buffer, 4) LSTM with a
25-image history buffer, 5) dual branch temporal convolutional network with a
15-image history buffer, 6) dual branch temporal convolutional network with a
25-image history buffer, 7) a fully connected single-image single-target
model, and 8) a fully connected single-image panoramic-target model. A fully
connected model with more images on the history buffer is not assessed because
the VRAM on a 1080Ti does not provide sufficient space to train a model of
this size. The LSTM and convolutional architectures are evaluated in lieu of a
larger fully connected model as they are more scalable. The proposed model,
using a 5-image history buffer, outperforms all the above architectures on the
testing subset of area1.

Figure 3.5: The layout of the policy model network which takes the embeddings
of a sequence of past observations, a set of panoramic goal images, and the
current observation of the robot. Numbers inside each rectangle correspond to
the input dimensions of each layer. Conv1D represent 1D convolution and
Flatten represents the flatten operator on a matrix. The SoftMax is taken of
the final logits from MLP to generate a distribution over 3 actions.

Figure 3.5 illustrates the network architecture of the dual branch temporal
convolutional network. There are two separate branches that deal with the
state space (current observations stacked with history observations) and goal
embeddings, respectively. The model applies convolution along the temporal
dimension of the sequence of images. This is like a 3D convolution [162] on
image sequences but uses 1D convolution because of the latent space input
embeddings. Compared to a fully connected multi-layer perceptron (MLP), it has
the advantages of 1) being much more lightweight due to parameter-sharing in
the convolution process, 2) scalable to longer history buffer, and 3) is
better at capturing spatiotemporal features from the data. LSTM is also known
to perform well on sequential data and has been widely used in the computer
vision community for action recognition and video preprocessing, but it
consumes a large amount of time and resources to train and does not perform
well as shown in section 3.3.

The $1024D$ output vectors from both branches are then fused to a $2048D$
vector. It passes through 2 fully connected layers and the SoftMax activation
function to generate a distribution over three actions. The policy model
chooses the one with the highest probability for the agent to take.

During the training phase, images (and in the case of the history policy,
image histories) and commands were randomly selected from a training set of
700 trajectories. The cross-entropy loss was computed between the output of
the policy network and the ground-truth commands.

(a)

(b)

Figure 3.6: (a) Encoder architecture of the autoencoder Progression through
each layer consists of a convolution with a stride of 2 followed by batch
normalization and ReLU activation. (b) Goal checking model architecture Conv1D
is the 1D convolution operation.

##### Goal Checking Model

The goal checking model takes in the embeddings of the current observation
concatenated with the panoramic goal images and predicts whether the agent is
at the target position, as shown in 3.6(b).

This model is created in response to an optimization on the original
architecture which had the policy model output a done action when the robotic
agent arrives at a goal position. The policy training data is too sparse for
the agent to effectively learn identifying a goal location because it only has
one positive example of done at the end of each trajectory. All the other
steps are negative examples for not done. There is a significant imbalance in
the number of positive and negative examples. In addition, at runtime the
robot is likely to arrive at the target position from a different viewpoint
than those in the panoramic goal images, but during training the policy model
receives a view that is one of the panoramic goal images. An additional binary
classifier is implemented to identify whether the agent has arrived at the
goal location. When this model predicts a done action the navigation pipeline
terminates.

The goal checking model is a dual-branch network with 1D convolution over the
panoramic goal branch. The $1024D$ vector from the goal-branch is then
concatenated with the current embedding branch to form a $5120D$ fused vector,
which then passes through an MLP with a hidden layer of 512 units to output
the probability of the goal being reached. Weights are then updated using an
Adam optimizer on the cross-entropy loss. While using the learned goal checker
at runtime, to reduce noise, the agent does a $None$ rotation when its belief
of reaching the goal is over 0.99. It calls the learned goal checker after
each $None$ turn. The agent outputs done only if the average probability is
over 0.9.

### 3.3 Experiments

Figure 3.7: Eight randomly selected non-overlapping successful trajectories in
area1. Blue dots are start positions and green dots are goal positions.
Trajectory $5$ and $8$ show recovery behavior which leads to a successful
trajectory.

The navigation pipeline is evaluated in 2 environments selected from the
Stanford2D3DS dataset and 3 environments selected from the Matterport3D
dataset. Metadata (including number of rooms and area) are shown in Table 3.2.
The Gibson simulator is used with a Fetch [177] robot and focus on how the
navigation pipeline generalizes to unseen targets under the same trained
environment. All experiments are conducted using an NVIDIA 1080Ti GPU.
Examples of planned paths, recovery behavior, and experimental environments
can be found in the attached video.

#### 3.3.1Network Training Setup

For each submodule (autoencoder, policy, and goal checker) in the navigation
pipeline, the corresponding dataset needs to be generated for training and
testing using the colored meshes loaded in the Gibson simulator. The model
with least loss (autoencoder) or highest accuracy (policy, goal checker) on
the test set is selected. Unless otherwise specified, a learning rate of
$0.001$ is used.

##### 2D Environment Map

Each environment is represented as a 3D mesh in both the Matterport3D and
Stanford 2D3DS datasets. To plan expert trajectories in the simulator, a 2D
map of the environment which contains obstacles the robot would encounter
during execution of a trajectory is needed. The environment is discretized
into a grid of $nxm$ cells where $n$ is the number of cells in the y-axis and
$m$ is the number of cells in the x-axis. Each cell is a fixed dimension that
can be changed depending on how granular the positions for the robot need to
be for generating data. A cell resolution of $1cm\times 1cm$ was used during
testing. At each position, the robot is modeled as a bounding box to check for
a collision with the mesh of the environment. For a Fetch robot the collision
bounding box is $0.6m\times 0.6m\times 1.6m$ where $0.6m$ is the diameter of
the robot and $1.6m$ is the height of the robot. After checking a cell for a
collision with the environment, an occupancy map of valid positions for the
robot to navigate to is obtained. Due to the uneven nature of tthe target
embeddinghe floor of the environment, collisions with faces of the environment
mesh close to the ground level are ignored, which was $0.05m$ for each
environment. Five maps of the scenes house1, house2, house17 are generated
from Matterport3D and area1 and area2 from Stanford 2D3DS datasets. An example
of the occupancy map for house1 is shown in Figure 3.8. With these positions
calculated, the expert trajectories could then be produced.

Figure 3.8: 2D map of the house1 environment in the Matterport3D environment.
Each pixel is 0.01m x 0.01m. The collision-free locations for the robot are
shown in green, obstacles are shown in black and in-collision positions are
shown in white. Some regions are non-contiguous and if trajectories were too
short within their subregion, they were not used for navigation planning.

##### Autoencoder

For each environment’s autoencoder dataset, 120$K$ collision-free locations
are randomly sampled from the generated map and capture RGBD images at those
locations in the simulator. A 0.9 / 0.1 train / test split is used. The
autoencoder model is trained for 200 epochs which on average takes 12 GPU
hours for each environment.

##### Policy

Using Dijkstra’s algorithm with costmap optimization, collision-free expert
trajectories can be generated. To map the trajectory into to a sequence of
{${forward,right,left}$} commands, each waypoint position is rendered. A novel
discretization strategy used to turn waypoints into discretized commands. The
agent looks ahead 25 steps to determine whether to turn left or right. The
robot turns left or right whether the look-ahead position is turned at an
offset greater than $None$ and outputs turns in increments of $None$. Using
only the next position would result in the robot constantly recalculating its
direction and the robot would turn at every step to face a new direction.
Forward commands are given if the robot is farther from the next position than
$0.1m$, and if so, moves the robot forward in increments of $0.1m$ until under
the threshold. The algorithm for this process is shown in Algorithm 1.

Algorithm 1 Trajectory Discretization

trajectories $T$ = {  $pose_{1}$ , …,  $pose_{N}$ } ;

start_pose = T[0];

goal_pose = T[0];

angle_threshold = $20^{\circ}$;

commands = [];

distance_step = $0.1m$;

for $t_{i},t_{i+1},t_{i+25}$ in $T$ do

heading_direction = $t_{i}$ \- $t_{i+25}$;

current_robot_angle = start_pose.yaw;

if current_robot_angle - heading_direction $>$ angle_threshold then

command = $LEFT$ if angle_difference $>$ 0 else $RIGHT$;

commands.append($command$ for every 10 degree difference);

final_angle = heading_direction;

end if

unit_pose_increment = [cos(final_angle), sign(final_angle), 0] *
distance_step;

distance_to_goal = position_distance(current_position, next_position);

if distance_to_goal $>$ distance_tolerance then

commands.append($FORWARD$ for every $distance\\_step$ in
$distance\\_to\\_goal$;

end if

return commands, poses;

end for

Figure 3.9: An algorithm for converting a sequence of poses from a path
planning algorithm into a sequence of discrete commands to be executed by the
agent. The values used in this script, such as $distance\\_step$ are the same
as what were used in the training data. This method uses a look ahead
algorithm to check several steps ahead when picking a correct orientation.
During testing, the agent would swerve too frequently if the look-ahead
position were not sufficiently far in the future.

Once these commands are generated, the trajectory is executed in the Gibson
simulator capturing the RGBD view along every step. This results in a large,
supervised learning dataset of varying trajectory lengths that could be used
for training the agent to learn the policy. The trained autoencoder is used to
generate embeddings of RGBD images at each step in the expert trajectories.
The average number of steps per trajectory varies from 41 (house2) to 332
(area1). Each training / testing example is constructed by taking the
embeddings from past 4 steps concatenated with the embedding at the current
step. Because larger environments tend to have longer trajectories, 3000
trajectories are generated for area1 and area2, 5000 trajectories for house17,
and 7000 trajectories for house1 and house2 to keep the total number of
individual steps the same. $80\%$ of the trajectories are used for training
and the rest for testing. The policy network is trained for 200 epochs which
takes around 90 GPU hours. The average accuracy for the policy model is 0.91.

##### Goal Checker

For each environment, positive training examples are collected by randomly
sampling 150$K$ positions in the environment for the panoramic goal images and
then sample another position within a 0.1$m$ radius for the current image.
Negative examples are collected by randomly sampling 150$K$ positions for the
panoramic goal images and then sample another position at least $1m$ away from
the current image. A 0.9 / 0.1 train / test split is used. The network is
trained for 300 epochs for on average 36 GPU hours. The average training
accuracy is 0.95 across all environments.

##### Training / Testing Split for Trajectory Data

Validation of the navigation policy model was done using a test dataset
containing 400 trajectories consisting of varying numbers of commands/images
captured from the trajectory execution rendering. Upon every epoch during
training for both the autoencoder and policy networks, each network is
evaluated using a test dataset of 20% of the overall dataset for each to make
sure the network was still learning.

The training and test data both contained varied length episodes, ranging from
200 to 300 timesteps per episode. Each timestep corresponds with a single
action, which can be a turn in either direction, a forward movement, or a
terminus. Each of the three possible movements (left, right, forward) were
controlled by discretely moving the robot in a particular direction, which in
the configuration of PyBullet produced deterministic results. Figure 3.7 shows
sample trajectories used in the training set.

See Table 3.1 for statistics about each of the datasets generated. Each cell
of the table corresponds to the number of training elements captured for each
of the environments and datasets used. The Goal Checker had 9 images captured
for every training example, 1 for the current view and 8 for the panoramic
goal. The autoencoder and navigation policy had one image captured per
position. Overall, $3,026,827$ images were used for training.

Environment | Autoencoder | Goal Checker |  $\\#$ Trajectories | Navigation Policy
---|---|---|---|---
area1 | 107668 | 268971 | 2998 | 995720
area2 | 119532 | 239305 | 2998 | 695515
house1 | 113780 | 233341 | 5582 | 299612
house2 | 112095 | 232583 | 5599 | 189789
house17 | 119852 | 239150 | 5600 | 846191

Table 3.1: Each environment contained different numbers of images for each
environment based on the size of the environment and how many valid positions
were available for the robot. Each environment’s dataset took 100GB of
storage, totaling 500GB of data.

#### 3.3.2Comparison Methods

The proposed navigation pipeline is compared with an RGBD SLAM [100] approach
and the target-driven deep RL method from [188]. The methods to be examined
are described below.

  1. a)

SLAM is the Real-Time Appearance-Based Mapping (RTABMap) library provided by
[100], which is an RGBD Graph-Based SLAM approach based on an incremental
appearance-based loop closure detector. The robot is not given the map
beforehand in the implementation and builds it up as it moves along.

  2. b)

Siamese Actor-Critic (SAC) is the method proposed by [188]. A scene specific
layer is kept for each environment. The network is then trained for each
environment to generalize to unseen targets within the same house. A goal-
reaching reward of $10.0$ is provided upon task completion and a small penalty
of $-0.01$ at each time step. The network is trained on 100 targets with a
maximum step size of 10000 for each episode. Each environment is given a
budget of 20$M$ frames (steps).

  3. c)

Navigation Pipeline (GPS) is a variant of the proposed navigation pipeline.
Instead of using the learned goal checking model, it uses the GPS information
and the provided goal coordinate to check whether the agent is at the goal.

  4. d)

Navigation Pipeline (no GPS) is the proposed navigation pipeline with the
learned goal checker, without GPS, odometry or goal coordinate provided.

a), b) and c) assume the agent has an idealized GPS and is provided with the
static goal coordinate as in [147]. As a result, the agent can compute the
relative position of the target at each time step and can use this information
to check if the goal has been reached.

#### 3.3.3Evaluation Criteria

The performance of the navigation pipeline is evaluated using 400 randomly
sampled start-goal pairs for which a valid path exists. The start and goal
locations have never appeared in the training examples. The agent is started
at the starting position and provide it with 8 panoramic goal images taken at
the goal location. The objective is to navigate to the goal position (no
requirements on the robot’s final orientation) autonomously with the shortest
path possible using only visual input. The success tolerance is a $0.5m$
radius within the target position. Unlike previous works which do not penalize
collision through training and allow collision at runtime [188, 147], physics
are simulated, and the trial is considered a failure when collision occurs.

Like many previous works on navigation benchmarks [188, 147, 6, 119], three
evaluation metrics are used:

  1. 1.

Success Rate is the number of successful trials over the total number of
trials.

  2. 2.

Success Weighted by Path Length ($SPL$) [6] metric is shown in Formula 3.2,
where $l_{i}$ is the length of the shortest path between start and goal
position, $p_{i}$ is the length of the observed path taken by an agent, and
$S_{i}$ is a binary indicator of success in trial $i$. This metric weighs each
success by the quality of path and thus is always $\leq$ Success Rate.

$SPL=\frac{1}{N}\sum_{i=1}^{N}S_{i}\frac{l_{i}}{\max(p_{i},l_{i})}$ (3.2)

  3. 3.

Observed over Optimal Ratio ($OOR$) measures the average ratio of observed
path length over optimal path length for successful trials.

#### 3.3.4Navigation Results

The proposed navigation pipeline significantly outperforms RGBD SLAM and the
state-of-the-art deep RL method in terms of path quality and success rate, as
shown in Table 3.2. See Figure 3.7 for several example trajectories generated
by the method in the area1 environment.

SLAM [100] struggles to localize itself using RGBD alone, due to the high
complexity of the testing environments. It succeeds only when the start
position is close to the goal position. SAC [188] performs much worse than the
navigation pipeline due to the sparse reward and limited number of training
frames. In THOR [188], each environment is a single room, and the researchers
use synthetic images, but the environments used here can be up to 1031$m^{2}$
with 40 rooms with real-world images. These environments have higher
complexity with more obstacles and the entrances to the rooms can be extremely
narrow resulting in a difficult solution. SAC needs millions of frames to
converge in the environments provided, which is not practical. THOR [188]
claims they can generalize to new targets by evaluating only on 10 targets
that are several steps away from the training targets. In the experimentation,
the targets can be anywhere on the map. Most of the successes for SAC is when
the target location happens to be in a same room as the start location. The
proposed method also requires much fewer simulation steps / training frames
($\sim 700K$ compared to 20$M$) and less training time (90 GPU hours compared
to 300 GPU hours). While the proposed approach does not appear to be saturated
in learning a policy over the largest environment, area1, it is unlikely that
given current methods of generating data and training that the model would
generalize further to larger environments. These models also struggle to
generalize between environments when trained over multiple environment
datasets.

The proposed method with no GPS achieves similar performance to the variant
with GPS. As an ablation study, instead of having a separate goal checking
model, a done action is generated directly from the policy model. Using a
separate goal checking model increases the success rate by $0.2\sim 0.5$. In
the cases where the policy model incorrectly identifies done, it either
outputs done prematurely or passes the goal without terminating. The amount of
training data is intentionally kept the same across all environments to
evaluate how the performance changes with the complexity of the environment.
When the number of rooms is over 30, the proposed navigation pipeline starts
to struggle to get to the goal. Despite the reduction in performance due to
environmental complexity, the method performs on average $0.556$ and $0.442$
better in success rate than SLAM and SAC, respectively. Given that high
navigation accuracy is achieved on smaller environments, the performance in
area1 and area2 will go up if trained on more expert trajectories. A future
direction is to analyze the amount of training data needed for a given
environmental complexity.

Environment | Model | Success Rate | SPL | OOR
---|---|---|---|---
house2 (66$m^{2}$, 6 rooms) | SLAM | 0.7575 | 0.5208 | 2.726
SAC | 0.8400 | 0.5620 | 3.538
Proposed (GPS) | 0.9950 | 0.9810 | 1.066
Proposed (no GPS) | 0.9875 | 0.9724 | 1.053
house1 (89$m^{2}$, 10 rooms) | SLAM | 0.3575 | 0.2380 | 2.809
SAC | 0.5200 | 0.3648 | 2.961
Proposed (GPS) | 0.9975 | 0.9811 | 1.064
Proposed (no GPS) | 0.9225 | 0.8748 | 1.252
house17 (220$m^{2}$, 14 rooms) | SLAM | 0.0900 | 0.0642 | 2.354
SAC | 0.2800 | 0.1281 | 5.919
Proposed (GPS) | 0.9800 | 0.7853 | 2.020
Proposed (no GPS) | 0.9150 | 0.7179 | 2.389
area2 (1031$m^{2}$, 31 rooms) | SLAM | 0.0700 | 0.0500 | 4.447
SAC | 0.1700 | 0.1073 | 3.997
Proposed (GPS) | 0.7250 | 0.5536 | 2.504
Proposed (no GPS) | 0.6625 | 0.4714 | 2.948
area1 (786$m^{2}$, 40 rooms) | SLAM | 0.0100 | 0.0002 | 117.1
SAC | 0.0425 | 0.0195 | 6.967
Proposed (GPS) | 0.6600 | 0.3954 | 4.504
Proposed (no GPS) | 0.5750 | 0.2705 | 5.896
Average (483.4$m^{2}$, 20.2 rooms) | SLAM | 0.2570 | 0.1746 | 25.89
SAC | 0.3705 | 0.2363 | 4.076
Proposed (GPS) | 0.8715 | 0.7393 | 2.232
Proposed (no GPS) | 0.8125 | 0.6614 | 2.707
Table 3.2: Different method results over 5 environments, with the best values
in bold. For SPL higher values are better. For OOR lower values are better.
The proposed method with no GPS achieves similar performance to the variant
with GPS. The proposed methods perform best across all environments and
metrics.

### 3.4 Experiment: MineRL Basalt Competition

The learned visual navigation system is not limited to robotic visual
navigation. The paradigm can be applied to other domains where information is
limited and the benefit of classification of state and a policy to reproduce
actions are clearly defined. A solution like the proposed learned visual
navigation method was submitted to the MineRL Basalt competition in 2021 by me
and collaborators and won first prize for best performance as well as the most
human like agent [60, 150]. This section will show how this method worked and
show the similarities between robotic visual navigation and the proposed
Minecraft solution. Additional information about the competition and solution
is provided in Appendix B.

#### 3.4.1Problem Setup

The MineRL Basalt competition was a competition held in 2021 to foster
creativity in replicating human-demonstration data via learned agents in a
Minecraft simulation [151]. The BASALT competition environments did not
include reward functions, as the goal was to build solutions to potential
real-world problems. Humans judged the results to evaluate the effectiveness
of each agent of completing a given task. The tasks were instead defined by a
human-readable description, which was given both to the competitors and to the
site visitors and workers doing the evaluation of the videos that trained
agents generate. All competition information was provided at
https://minerl.io/basalt/.

The four tasks in the competition were 1) finding a cave (FindCave), 2)
placing a waterfall and taking a picture of it (MakeWaterfall), 3) building an
animal pen and containing an animal in it (BuildAnimalPen), and 4) replicating
the architecture of a village by building a new house (BuildVillageHouse).
Each task was designed to be incrementally more challenging than the last.
Each of these tasks required the agent to "throw a snowball" when finished
with the task. The competition organizers also provided each participant team
with a dataset with 40 to 80 human demonstrations for each task, not all of
them completing the task, and the starter codebase to train a behavior cloning
agent, to evaluate the agent, and to make the solution submission.

#### 3.4.2Methodology

Figure 3.10: Diagram illustrating the approach. Using data from the available
human demonstration dataset, humans provide additional binary labels to image
frames to be used to train a state classifier that can detect relevant
features in the environment such as caves and mountains. The available human
demonstration dataset is also used to train a navigation policy via imitation
learning to replicate how humans traverse the environment. A separate odometry
module estimates the current agent’s position and heading solely based on the
action taken by the end. During test time, the agent uses the learned state
classifier to provide useful information to an engineered state-machine that
controls which subtask the agent should execute at every time-step.

Since no reward signal was given by the competition organizers and compute
time was limited, direct deep reinforcement learning approaches were not
feasible [121, 104, 120]. With the limited human demonstration dataset, end-
to-end behavior cloning also did not result in high-performing policies,
because imitation learning requires a substantial number of high-quality data
[20, 125]. Also attempted was to address the tasks using adversarial imitation
learning approaches such as Generative Adversarial Imitation Learning (GAIL)
[71], however, the large-observation space and limited compute time also made
this approach infeasible.

Hence, to solve the four tasks of the MineRL BASALT competition, a combined
machine learning and knowledge engineering approach is used, also known as
_hybrid intelligence_ [84, 42]. As seen in the main diagram of the approach
shown in Figure 3.10, the machine learning part of this method is seen in two
different modules: first, a state classifier is learned using additional human
feedback to identify relevant states in the environment; second, a navigation
subtask is learned separately for each task via imitation learning using the
human demonstration dataset provided by the competition. The knowledge
engineering part is seen in three different modules: first, given the relevant
states classified by the machine learning model and knowledge of the tasks, a
state-machine is designed that defines a hierarchy of subtasks and controls
which one should be executed at every time-step; second, solutions for the
more challenging subtasks were designed that were not able to learn directly
from data; and third, an estimated odometry module is engineered that provides
additional information to the state-machine and enables the execution of the
more complex engineered subtasks.

Like the learned visual navigation pipeline, the state classifier is analogous
to the goal checker module where at each step the state classifier is
determining the next state of the pipeline. The behavior cloning policy is
analogous to the policy model from before, except with more states that are
possible due to the more complex action space in Minecraft. This solution does
estimate the odometry to enable certain steps in the policy where it is
critical that the agent navigate back to a previous state, however the policy
itself does not consider this estimated odometry at runtime.

#### 3.4.3Results

Four different approaches are considered to solve the four tasks proposed in
the MineRL Basalt competition:

  * •

Hybrid: the main proposed agent, which combines both learned and engineered
modules. The learned modules are the navigation subtask policy (learns how to
navigate using the human demonstration data provided by the competition) and
the state classifier (learns how to identify relevant states using additional
human-labeled data). The engineered modules are the multiple subtasks, hand-
designed to solve subtasks that were not able to be learned from data. These
engineered modules are the estimated odometry and the state-machine, which
uses the output of the state classifier and engineered task structure to
select which subtask should be followed at each time-step.

  * •

Engineered: almost identical to the Hybrid agent described above, however, the
navigation subtask policy that was learned from human demonstrations is now
replaced by a hand-designed module that randomly selects movement and camera
commands to explore the environment.

  * •

Behavior Cloning: end-to-end imitation learning agent that learns solely from
the human demonstration data provided during the competition. This agent does
not use any other learned or engineered module, which includes the state
classifier, the estimated odometry, and the state-machine.

  * •

Human: human-generated trajectories provided by the competition. They are
neither guaranteed to solve the task nor solve it optimally because they
depend on the level of expertise of each human controlling the agent.

Each combination of condition (behavior cloning, engineered, hybrid, human)
and performance metric (best performer, fastest performer, most human-like
performer) is treated as a separate participant of a one-versus-one
competition where skill rating is computed using the TrueSkillTM Bayesian
ranking system [69]. The main proposed “Hybrid” agent, which combines
engineered and learned modules, outperforms both pure hand-designed
(“Engineered”) and pure learned (“Behavior Cloning”) agents in the “Best
Performer” category, achieving $5.3\%$ and $25.6\%$ higher mean skill rating
when compared to the “Engineered” and “Behavior Cloning” baselines,
respectively. However, when compared to the “Human” scores, our main proposed
agent achieves $21.7\%$ lower mean skill rating, illustrating that even the
best approach is still not able to outperform a human player with respect to
best performing the task. More results and analysis are shown in Appendix B.

### 3.5 Conclusion

This chapter proposed a navigation pipeline which does not rely on odometry,
map, compass or indoor position at runtime and is purely based on the visual
input and a novel 8-image panoramic goal. This method learns from expert
trajectories generated using RGBD maps of several real environments. Using
robotic simulators with real data and photo-realistic rendering, efficient
collection of numerous expert trajectories can be collected. This enables the
training of agents in simulation with real-world data, thereby bridging the
sim-to-real domain adaptation. Experiments show that the proposed method 1)
achieves better performance than state-of-the-art baselines, especially in
complex environments with difficult and long-range path solutions; 2) requires
fewer training samples and less training time; and 3) can work across
different environments given an RGBD map. Additionally, this methodology can
be extended to other domains, such as imitation-learned Minecraft agents.

## Visual Tactile Manipulation

Once the agent reaches the object, how can it reason about the object’s
geometry? What sensory modalities can a robot work with to make an estimate of
the object geometry. In previous work, researchers used a single 2.5D image to
reconstruct an object [164]. This estimation can be further refined by
utilizing tactile sensors on a robotic end-effector to refine a shape
estimation better than using depth information alone. By increasing the
quality of the shape estimation, it allows for a grasp planner to plan a grasp
more accurately for manipulation tasks. This chapter explores the benefits of
a shape completion system that leverage visual-tactile information and how the
performance is improved over a single-view prediction.

### 4.1 Introduction

Robotic grasp planning based on raw sensory data is difficult due to occlusion
and incomplete information regarding scene geometry. Often, one sensory
modality does not provide enough context to enable reliable planning. For
example, a single depth sensor image cannot provide information about occluded
regions of an object, and tactile information is incredibly sparse. To solve
this, a 3D convolutional neural network is used to enable stable robotic grasp
planning by incorporating both tactile and depth information to infer occluded
geometries. This multi-modal system uses both tactile and depth information to
form a more complete model of the space the robot can interact with and to
provide a complete object model for grasp planning.

At runtime, a point cloud of the visible portion of the object is captured,
and multiple guarded moves are executed in which the hand is moved towards the
object, stopping when contact with the object occurs. The newly acquired
tactile information is combined with the original partial view, voxelized, and
sent through the CNN to create a hypothesis of the object’s geometry.

Figure 4.1: Completion example from tactile and depth data. A few samples of
tactile data can significantly improve the system’s ability to reason about 3D
geometry. The Depth Only Completion for the pitcher does not capture the
handle well, whereas the tactile information gives a better geometric
understanding. The additional tactile data allowed the CNN to correctly
identify a handle in the completion mesh and similar completion improvement
was found for the novel rubber duck not in the training set.

Depth information from a single point of view often does not provide enough
information to accurately predict object geometry. There is often unresolved
uncertainty about the geometry of the occluded regions of the object. To
alleviate this uncertainty, tactile information is utilized to generate a new,
more accurate hypothesis of the object’s 3D geometry, incorporating both
visual and tactile information. Figure 4.1 demonstrates an example where the
understanding of the object’s 3D geometry is significantly improved by the
additional sparse tactile data collected via this framework. An overview of
the sensory fusion architecture is shown in Figure 4.2.

This method is differentiated from others [170] in that this CNN is acting on
both the depth and tactile as input information fed directly into the model
rather than using the tactile information to update the output of a CNN not
explicitly trained on tactile information. This enables the tactile
information to produce non-local changes in the resulting mesh. In many cases,
depth information alone is insufficient to differentiate between two potential
completions, for example a pitcher vs a rubber duckie. In these cases, the CNN
utilizes sparse tactile information to affect the entire completion, not just
the regions near the tactile glance. If the tactile sensor senses the occluded
portion of a drill, the CNN can turn the entire completion into a drill, not
just the local portion of the drill that was touched.

Figure 4.2: Both tactile and depth information are independently captured and
voxelized into $40^{3}$ grids. These are merged into a shared occupancy map
which is fed into a CNN to produce a hypothesis of the object’s geometry.

### 4.2 Visual-Tactile Geometric Reasoning Method

The framework utilizes a trained CNN to produce a mesh of the target object,
incorporating both depth and tactile information in a single input channel.
The same architecture as found in [163] is used. The model was implemented
using the Keras [32] deep learning library. Each layer used rectified linear
units as nonlinearities except the final fully connected (output) layer which
used a sigmoid activation to restrict the output to the range $[0,1]$. Cross-
entropy error $E(y,y^{\prime})$ is used as the cost function with target $y$
and output $y^{\prime}$:

$E(y,y^{\prime})=-\left(y\log(y^{\prime})+(1-y)\log(1-y^{\prime})\right)$

Figure 4.3: Example training pair from the geometric shape dataset. For the
left, red dots represent tactile readings and blue dots represent the depth
image. The blue points on the right are the ground truth 3D geometry.
Algorithm 2 Simulated YCB/Grasp Tactile Data Generation

1: grid_dim = 40 // resolution of voxel grid

2: npts = 40 // num locations to check for contact

3: vox_gt_cf = align_gt_to_depth_frame(vox_gt)

4: xs = rand_ints(start=0, end=grid_dim-1, size=npts)

5: ys = rand_ints(start=0, end=grid_dim-1, size=npts)

6: tactile_vox = []

7: for x, y in xs, ys do

8: for z in range(grid_dim-1, -1, -1) do

9: if vox_gt_cf[x, y, z] == 1 then

10: tactile_vox.append(x, y, z)

11: continue

12: end if

13: end for

14: end for

15: tactile_points = vox2point cloud(tactile_vox)

16: return tactile_points

Figure 4.4: The sampling of tactile points in simulation is treated as a two-
dimensional problem of trying to recreate a Barrett fingertip capturing
tactile data of the object. By sampling 40 random points on the occluded side
of the object the robot can capture as much data as captured in 6 guided
tactile measurements in a physical experiment. Given the resolution of the
voxel grid is $40^{3}$, the robot is sampling $2.5\%$ of all positions on the
occluded side of the object. This sparsity of data worked to advantages both
in terms of reducing data required for training and forcing the network to
utilize tactile data as much as possible.

This cost function encourages each output to be close to either 0 for
unoccupied target voxels or 1 for occupied target voxels. The optimization
algorithm Adam [90], which computes adaptive learning rates for each network
parameter, is used with default hyperparameters ($\beta_{1}=0.9$,
$\beta_{2}=0.999$, $\epsilon=10^{-8}$) except for the learning rate, which is
set to 0.0001. Weights were initialized following the recommendations of [68]
for rectified linear units and [56] for the logistic activation layer (batch
size=32).

### 4.3 Completion of Simulated Geometric Shapes

(a) Training Views

(b) Holdout Views

(c) Holdout Meshes

Figure 4.5: Several examples from different data splits for evaluating
completion quality. The bottom row is ground truth $40^{3}$ occupancy, and the
top row is a voxelization of the depth and tactile information. Training Views
were used to train the CNN, and Holdout Views are views of meshes used to
train the CNN but from views not used during training. Holdout Meshes are
views of meshes that the network never saw during training.

(a) Hand approach

(b) Finger contact

(c) Finger curl

Figure 4.6: Barrett hand showing contact with a fixed object. (a) The hand is
manually brought to an approach position, (b) approaches the object, and (c)
the fingers are curled to contact the object and collect tactile information.
This process is repeated 6 times over the occluded surface of the object.

To evaluate the system’s ability to utilize additional tactile sensory
information to reason about 3D geometry, initial experimentation was done on a
toy geometric shape dataset. This dataset consisted of conjoined half-shapes.
Both front and back halves of the objects were randomly chosen to be either a
sphere, cube, or diamond of varying sizes. The front and back halves do match
in size. An example shape is shown in Figure 4.3(b), a half-cube half-sphere.
Next, synthetic sensory data was generated for these example shapes and
embedded in a $40^{3}$ voxel grid. Depth information was captured from a fixed
camera location, and tactile data was generated by intersecting 3 rays with
the object. The rays originated at (13, 20, 40), (20, 20, 40) and (26, 20,
40), and traveled in the $-z$ direction until either contact occurred with the
object or the ray left the voxelized volume. Sensory data for a shape is shown
in Figure 4.3.

Input Data | Jaccard similarity
---|---
Tactile Only | 0.863
Depth Only | 0.890
Depth & Tactile | 0.986
Table 4.1: Jaccard similarity for test set of geometric shapes produced from
the same CNN architecture trained with various input data. When trained using
both tactile and depth information the CNN can complete the object with a near
perfect Jaccard similarity. Depth or tactile alone is not sufficient to reason
about object geometry in these problems.

Three networks with the architecture from [163] were trained on a simulated
dataset of geometric shapes (Figure 4.3) where the front and back were
composed of two differing shapes. Sparse tactile data was generated by
randomly sampling voxels along the occluded side of the voxel grid. A network
is trained that only utilized tactile information. This performed poorly due
to the sparsity of information. A second network was given only the depth
information during training and performed better than the tactile-only network
did. It still encountered many situations where it did not have enough
information to accurately complete the obstructed half of the object. A third
network was given depth and tactile information which successfully utilized
the tactile information to differentiate between plausible geometries of
occluded regions. The network architecture is shown in Figure 4.7.

Figure 4.7: Visual-Tactile CNN Architecture The CNN architecture takes a
single $40^{3}$ voxelization of the input point cloud, embeds it, then
performs a dense $64000$ reconstruction of the object. This system works well
via convolutional layers abstracting the geometry of the input, and the
embedding using information from the entire input region.

Results for this geometric test set are shown in Table 4.1. The Jaccard
similarity improved from 0.890 in the depth only network to 0.986 in the depth
and tactile network. This task demonstrated that a CNN can be trained to
leverage sparse tactile information to decide between multiple object geometry
hypotheses. When the object geometry had sharp edges in its occluded region,
the system would use tactile information to generate a completion that
contained similar sharp edges in the occluded region. This completion is more
accurate not just in the observed region of the object but also in the
unobserved portion of the object. It was also found that tactile only was not
enough information to get a complete understanding of the object’s geometry
with a Jaccard score of 0.863.

Figure 4.8:  As the difficulty of the data splits increase, the delta between the Depth Only CNN completion accuracy and the Tactile and Depth CNN completion accuracy increases. The additional tactile information is more useful on more difficult completion problems.  | Completion
---
Method
| Train
---
View(Sim)
| Holdout
---
View(Sim)
| Holdout
---
Model(Sim)
| Holdout
---
(Live)
Partial | 0.01 | 0.02 | 0.01 | 0.01
Convex Hull | 0.50 | 0.51 | 0.46 | 0.43
GPIS | 0.47 | 0.45 | 0.35 | 0.48
Depth CNN | 0.68 | 0.65 | 0.65 | 0.37
Depth Tactile | 0.69 | 0.66 | 0.65 | 0.64
Table 4.2: Jaccard similarity results measuring the intersection over union of two voxelized meshes, as described in section 4.5. (Larger is better) | Completion
---
Method
| Train
---
View(Sim)
| Holdout
---
View(Sim)
| Holdout
---
Model(Sim)
| Holdout
---
(Live)
Partial | 7.8 | 7.0 | 7.6 | 11.9
Convex Hull | 32.7 | 45.1 | 49.1 | 11.6
GPIS | 59.9 | 79.2 | 118.0 | 17.9
Depth CNN | 6.5 | 6.9 | 6.5 | 16.5
Depth Tactile | 5.8 | 5.8 | 6.2 | 7.4
Table 4.3: Hausdorff distance results measuring the mean distance in millimeters from points on one mesh to points on another mesh, as described in section 4.5. (Smaller is better) | Completion
---
Method
| Train
---
View(Sim)
| Holdout
---
View(Sim)
| Holdout
---
Model(Sim)
| Holdout
---
(Live)
Partial | 19.9mm | 21.1mm | 16.6mm | 18.6mm
Convex Hull | 13.9mm | 16.1mm | 14.1mm | 10.5mm
GPIS | 17.1mm | 16.0mm | 21.3mm | 20.8mm
Depth CNN | 12.1mm | 13.7mm | 12.4mm | 22.9mm
Depth Tactile | 7.7mm | 13.9mm | 13.6mm | 6.2mm
Table 4.4: Pose error results from simulated grasping experiments. This is the average L2 difference between planned and realized grasp pose averaged over the 3 fingertips and the palm of the hand, in millimeters. (Smaller is better)  | Completion
---
Method
| Train
---
View(Sim)
| Holdout
---
View(Sim)
| Holdout
---
Model(Sim)
| Holdout
---
(Live)
Partial | 8.19∘ | 6.71∘ | 8.78∘ | 7.67∘
Convex Hull | 3.53∘ | 4.01∘ | 4.59∘ | 3.77∘
GPIS | 4.65∘ | 4.79∘ | 4.95∘ | 5.92∘
Depth CNN | 3.09∘ | 3.56∘ | 4.52∘ | 6.83∘
Depth Tactile | 2.48∘ | 3.41∘ | 4.95∘ | 2.43∘
Table 4.5: Joint error results from simulated grasping experiments. This is the mean L2 distance between planned and realized grasps in degrees averaged over the hand’s 7 joints. The Depth and Tactile method’ smaller error demonstrates a more accurate geometry reconstruction. (Smaller is better) | Completion
---
Method
| Partial
---
| Convex
---
Hull
| GPIS
---
| Depth
---
CNN
| Depth Tactile
---
Time (s) | 1.533s | 0.198s | 45.536s | 3.308s | 3.391s
Table 4.6: Algorithmic methods like partial and convex hull have faster
runtimes than using a neural network, however all methods are significantly
faster than the GPIS timings. This means faster planning and execution of a
grasp than using alternative methods.

### 4.4 Completion of YCB/Grasp Dataset Objects

The dataset from [163] is used to create a new dataset consisting of half a
million triplets of oriented voxel grids: depth, tactile, and ground truth.
Depth voxels are marked as occupied if visible to the camera. Tactile voxels
are marked occupied if tactile contact occurs within the voxel. Ground truth
voxels are marked as occupied if the object intersects a given voxel,
independent of perspective. The point clouds for the depth information were
synthetically rendered in the Gazebo [94] simulator. This dataset consists of
608 meshes from both the Grasp [85] and YCB [25] datasets. 486 of these meshes
were randomly selected and used for a training set and the remaining 122
meshes were kept for a holdout set.

The synthetic tactile information was generated according to Algorithm 2. To
generate tactile data, the voxelization of the ground truth high resolution
mesh (vox_gt) (Alg.2:L1) was aligned with the captured depth image (Alg.2:L3).
40 random $(x,y)$ points were sampled to generate synthetic tactile data
(Alg.2:L4-5). For each of these points (Alg.2:L6), a ray was traced in the
$-z$, direction and the first occupied voxel was stored as a tactile
observation (Alg.2:L10). Finally this set of tactile observations was
converted back to a point cloud (Alg.2:L15).

Two identical CNNs were trained where one CNN was provided only depth
information (Depth Only) and a second was provided both tactile and depth
information (Tactile and Depth). During training, performance was evaluated on
simulated views of meshes within the training data (Training Views), novel
simulated views of meshes in the training data (Holdout Views), novel
simulated views of meshes not in the training data (Holdout Meshes), and real
non-simulated views of 8 meshes from the YCB dataset (Holdout Live).

The Holdout Live examples consist of depth information captured from a real
Kinect and tactile information captured from a real Barrett Hand attached to a
Staubli Arm. Depth filtering is used to mask out the background of the
captured depth cloud. The object was fixed in place during the tactile data
collection process. While collecting the tactile data, the arm was manually
moved to place the end effector behind the object and 6 exploratory guarded
motions were made where the fingers closed towards the object. Each finger
stopped independently when contact was made with the object, as shown in
Figure 4.6.

Figure 4.8 demonstrates that the difference between the Depth Only CNN
completion and the Tactile and Depth CNN completion becomes larger on more
difficult completion problems. The performance of the Depth Only CNN nearly
matches the performance of the Tactile and Depth CNN on the training views.
Because these views are used during training, the network can generate
reasonable completions. Moving from Holdout Views to Holdout Meshes to Holdout
Live, the completion problems move further away from the examples experienced
during training. As the problems become harder, the Tactile and Depth network
outperforms the Depth Only network by a greater margin, as it can utilize the
sparse tactile information to differentiate between various completions. This
trend shows that the network can make more use of the tactile information when
the depth information alone is insufficient to generate a quality completion.
Meshes are generated from the output of the combined tactile and depth CNN
using a marching cubes algorithm. The density of the rich visual information
and the coarse tactile information is preserved by utilizing the post-
processing from [163].

#### 4.4.1Mesh Generation

Algorithm 3 shows how the dense partial view and tactile information are
merged into a $40^{3}$ voxel grid. More information about this method is
available from [163].

Algorithm 3 Visual-Tactile Shape Completion

1: //cnn_out: $40^{3}$ voxel output from CNN

2: //observed_pc: captured point cloud of object

3: d_ratio $\leftarrow$ densityRatio(observed_pc, cnn_out)

4: upsampled_cnn $\leftarrow$ upsample(cnn_out, d_ratio)

5: vox $\leftarrow$ merge(upsampled_cnn, observed_pc)

6: vox_no_gap $\leftarrow$ fillGaps(vox)

7: vox_weighted $\leftarrow$ CUDA_QP(vox_no_gap)

8: mesh $\leftarrow$ mCubes(vox_weighted)

9: return mesh

To merge with the partial view, the output of the CNN is converted to a point
cloud, and its density is compared to the density of the partial view point
cloud (Alg.3:L3). The CNN output is up-sampled by $d\\_ratio$ to match the
density of the observed point cloud (Alg.3:L4). The upsampled output from the
CNN is then merged with the observed point cloud of the partial view, and the
combined cloud is voxelized at the new higher resolution of
$(40*d\\_ratio)^{3}$ (Alg.3:L5). Any gaps in the voxel grid between the
upsampled CNN output and the observed partial view cloud are filled
(Alg.3:L6). The voxel grid is smoothed using a CUDA implementation of the
convex quadratic optimization problem from [103] (Alg.3:L7). The weighted
voxel grid is then run through marching cubes (Alg.3:L8).

Figure 4.9: The entire Holdout Live dataset. These completions were all created from data captured from a real Kinect and a real Barrett Hand attached to a Staubli Arm. The Depth and Tactile Clouds have the points captured from a Kinect in red and points captured from tactile data in blue. Notice many of the Depth Only completions do not extend far enough back but instead look like other objects that were in the training data (ex: cell phone, banana). The Depth and Tactile method outperforms the Depth Only, Partial, and Convex Hull methods in terms of Hausdorff distance and Jaccard similarity. Note that the GPIS completions form large and inaccurate completions for the Black and Decker box and the Rubbermaid Pitcher, whereas the Depth and Tactile method correctly bounds the end of the box and finds the handle of the pitcher.  | Completion
---
Method
| Partial
---
| Convex
---
Hull
| GPIS
---
| Depth
---
CNN
| Depth Tactile
---
Lift Success (%) | 62.5% | 62.5% | 87.5% | 75.0% | 87.5%
Joint Error (∘) | 6.37∘ | 6.05∘ | 10.61∘ | 5.42∘ | 4.67∘
Time (s) | 1.533s | 0.198s | 45.536s | 3.308s | 3.391s
Table 4.7: Lift Success is the percentage of successful lift executions. Joint
Error is the average error per joint in degrees between the planned and
executed grasp joint values. While GPIS and the Depth Tactile method have the
same lift success, the Depth Tactile method is 1340% faster and has 41% of the
joint error, making the process more dependable. (Smaller is better). Average
time to complete a mesh using each completion method. While the convex hull
completion method is fastest, Depth and Tactile has a superior tradeoff
between speed and quality.

### 4.5 Comparison to Other Completion Methods

This framework is benchmarked against the following general visual tactile
completion methods.
Partial Completion: The set of points captured from the Kinect is concatenated
with the tactile data points. The combined cloud is run through marching
cubes, and the resulting mesh is then smoothed using MeshLab’s [35]
implementation of Laplacian smoothing. These completions are accurate where
the object is directly observed but make no predictions in unobserved areas of
the scene.
Convex Hull Completion: The set of points captured from the Kinect is
concatenated with the tactile data points. The combined cloud is run through
QHull to create a convex hull. The hull is then run through MeshLab’s
implementation of Laplacian smoothing. These completions are reasonably
accurate near observed regions. However, a convex hull will fill regions of
unobserved space.
Gaussian Process Implicit Surface Completion (GPIS): Approximated depth cloud
normals were calculated using PCL’s KDTree normal estimation. Approximated
tactile cloud normals were computed to point towards the camera origin. The
depth point cloud was downsampled to size $M$ and appended to the tactile
point cloud. A distance offset $d$ is used to add positive and negative
observation points along the direction of the surface normal. A sample of the
Gaussian process is captured using [54] with a $n^{3}$ voxel grid and a noise
parameter $s$ to create meshes from the point cloud. An exhaustive parameter
search to determine the values of $M,s,n,d$ is conducted by sampling the
Jaccard similarity of GPIS completions where $M=[200,300,400]$,
$s=[0.001,0.005]$, $n=[40,64,100]$, and $d=[0.005,0.0005]$. An optimal value
of $M=300$ was found to be a good tradeoff between speed and completion
quality. Additionally values of $s=0.001$, $d=0.0005$, and $n=100$ were used.

In prior work [163], the Depth Only CNN completion method was compared to both
a RANSAC based approach [130] and a mirroring approach [18]. These approaches
make assumptions about the visibility of observed points and do not work with
data from tactile contacts that occur in unobserved regions of the workspace.

#### 4.5.1Geometric Comparison Metrics

Jaccard similarity results are shown in Table 4.2. The Jaccard similarity was
used to compare $40^{3}$ CNN outputs with the ground truth. This metric is
also used to compare the final resulting meshes from several completion
strategies. The completed meshes were voxelized at $80^{3}$ and compared with
the ground truth mesh. The proposed Depth and Tactile method results in higher
similarity to the ground truth meshes than do all other described approaches.

Table 4.3 shows the mean values of the symmetric Hausdorff distance for each
completion method. The Hausdorff distance metric computes the average distance
from the surface of one mesh to the surface of another. A symmetric Hausdorff
distance was computed with MeshLab’s Hausdorff distance filter in both
directions. In this metric, the proposed tactile and depth CNN mesh
completions are significantly closer to the ground truth compared to the other
approaches’ completions.

Both the partial and Gaussian process completion methods are accurate when
close to the observed points but fail to approximate geometry in occluded
regions. Through training, the Gaussian Process completion method would often
create a large and unruly object if the observed points were only a small
portion of the entire object or if no tactile points were observed in
simulation. Using a neural network has the added benefit of abstracting object
geometries, whereas the alternative completion methods fail to approximate the
geometry of objects which do not have points bounding their geometry.

#### 4.5.2Grasp Comparison in Simulation

To evaluate the framework’s ability to enable grasp planning, the system was
evaluated in simulation using the same set of completions. The use of
simulation allowed for the quick planning and evaluation of 7900 grasps.
GraspIt! was used to plan grasps on all the completions of the objects by
uniformly sampling different approach directions. These grasps were then
executed not on the completed object but on the ground truth meshes in
GraspIt!. To simulate a real-world grasp execution, the completion was removed
from GraspIt! and the ground truth object was inserted in its place. Then the
hand was placed 20 cm away from the ground truth object along the approach
direction of the grasp. The spread angle of the fingers was set, and the hand
was moved along the approach direction of the planned grasp either until
contact was made or a maximum approach distance was traveled. Then fingers
closed to the planned joint values and each finger continued to close until
either contact was made with the object, or the joint limits were reached.

Table 4.4 shows the average difference between the planned and realized
Cartesian fingertip and palm poses. Table 4.5 shows the difference in pose of
the end effector between the planned and realized grasps averaged over the 7
joints of the hand. Using the proposed Depth and Tactile method, the end
effector ended up closer to its intended location in both joint space and the
palm’s Cartesian position versus other completion methods’ grasps.

#### 4.5.3Live Grasping Results

To further evaluate the network’s efficacy, the grasps were planned and
executed on the Holdout Live views using a Staubli arm with a Barrett Hand.
The grasps were planned using meshes from the different completion methods
described above. For each of the 8 objects, the arm was used once using each
completion method. The results are shown in Figure 4.9 and Table 4.7. The
proposed Depth and Tactile method enabled an improvement over the other
visual-tactile shape completion methods in terms of grasp success rate and
resulted in executed grasps closer to the planned grasps, as shown by the
lower average joint error (and much faster than GPIS). While the success rate
of lifting the tactile and depth CNN was equal to the success rate of the
Gaussian Process completion, the proposed Depth and Tactile method constructed
an object geometry significantly faster, as shown in Table 4.6, and had a much
lower average joint error, as shown in Table 4.7. After performing lifts on
these 8 objects, this visual-tactile completion method performs better in
terms of timing, completion quality, and grasp quality for objects the network
has not observed.

### 4.6 Conclusion

This chapter explored how to leverage visual-tactile information to predict
object geometry better than a single-view. A mobile manipulator or stationary
robot with both visual information and tactile information would be able to
leverage the aforementioned methodology to improve performance without the
need to move upon reaching the object. Utilizing even a few tactile contacts
was useful for improvement in grasp posturing and completion accuracy.
Additionally, this system was experimentally validated on a dataset
representative of household and tabletop objects in simulation and in real-
world testing. However, not every robot has access to tactile information. In
the next chapter, a robot will utilize capturing multiple views of an object
to refine its initial object hypothesis. This will offer more rich information
than tactile and take advantage of the mobile based.

## Two-View Shape Understanding

Upon reaching an object conveyed through a panoramic target goal, a robotic
agent can leverage its mobile based to capture an additional view of the
object. This offers uniquely richer data to provide additional information
about an object’s geometry. However, it can be difficult to register these
views due to odometry error and noise. This chapter explores a novel two-view
object completion method that allows the robot to use two unregistered views
of the object to improve its shape estimation.

### 5.1 Introduction

Shape understanding based on single images is difficult. This was explored
previously when utilizing visual and tactile information in Chapter 4 to
complete an object but utilizing two images to complete an object is even more
challenging. While a single depth sensor can be moved to capture two views of
an object, aligning those views is challenging when odometry noise is high. To
solve this, a 3D convolutional neural network is used to enable robust shape
estimation by leveraging two unregistered views. This means each image of an
object is kept in its respective image frame. This methodology can be used to
complete objects with only two views. Providing more accurate reconstructions
of objects helps to enable a variety of robotic tasks such as manipulation,
collision checking, sorting, and cataloging.

Figure 5.1: Two views of an object can help refine the prediction. Shown in
red are partial views of the target object. A single-view reconstruction of
the object is shown in green, and a two-view reconstruction of the object is
shown in purple. The two-view prediction correctly captures the handle of the
pitcher while the single-view reconstruction misses this critical part of the
object geometry, in this case a pitcher from the YCB object dataset [25].

At runtime, a partial 2.5D image of an object is captured. This first view is
passed through a shape completion network to produce a shape estimation of the
target object. A second 2.5D view is then captured of the object. These two
views are passed into a dual-encoder CNN, called two-view split conv, which
encodes each view separately. These encodings are then added together and
passed through a CNN decoder to produce a final voxel reconstruction. An
example of this reconstruction is shown in Figure 5.1.

Training a neural network to extract registration information from two
disparate views allows for a variety of mobile robots to estimate object
geometry, such as a drone or two robots within the same environment.
Additionally, the utilization of a richer decoder architecture inspired by
work from Yang et al. [183] has improved reconstruction quality over a single-
view [164]. An application of this methodology for image reconstruction is
shown in Appendix A.

### 5.2 Methodology

The CNN architecture in Chapter 4 [172] does not address how to combine
unregistered information. The tactile information captured on the occluded
side of the object was superimposed onto the input. To make sure the mobile
robot does not need to localize at runtime, a new CNN architecture is designed
that takes two views, the current and previous, that are both kept in their
original image frame. As input, it takes two voxelized partial views and
outputs a voxelized reconstruction of the original object geometry. This model
outputs a voxel grid of occupancy scores. The occupancy scores are thresholded
at $0.5$ and perform marching cubes [108] to mesh the resultant voxel grid.
Example completions and their improvements are shown in Figure 5.2 which
figure demonstrates that a second view can offer significant improvement over
a single view.

Figure 5.2: Two-view reconstructions showing that two views are better than
one. Current input (red), a single view reconstruction (green), the previous
view of the object (blue), the two-view split conv completion (purple), and
finally the ground truth mesh (yellow). These meshes were not observed during
training for either network. A higher Jaccard score is better. All meshes are
from the Grasp dataset [19].

##### Prediction Refinement

An additional improvement over previous work is that known empty voxels are
marked as empty at the reconstruction step. This ensures full utilization of
the input image. This empty check can only occur for the current view, as the
previous image is not aligned with the frame the object is being completed in.
The point cloud from the previous capture in its image frame and the current
view of the object in its image frame are both voxelized into $40^{3}$ voxel
grids. A hypothesis about the object’s geometry is generated in the current
view image frame and produce a $40^{3}$ voxel grid of occupancy scores given
the two views of the object where $1$ is filled and $0$ is unfilled. Voxel
grid certainties are turned into a mesh by thresholding occupancy scores at a
decision boundary of $0.5$. Then any voxels that are known to be empty are
marked by tracing a ray from the camera vector to points in the voxel grid and
any voxels along that line that are not occluded are marked as empty. This
fixes any erroneous shapes on the visible side of the object.

##### CNN Architecture

The single view architecture used in Chapter 4 [172] had two shortcomings. The
first was that the reconstruction did not take advantage of deconvolutional
layers. The second is that it did not leverage two encoders to incorporate the
disparate information coming from multiple views. A proposed improved single-
view architecture is shown in Figure 5.3 that addresses the first problem.