#  Octopi: Object Property Reasoning with Large Tactile-Language Models

Samson Yu2, Kelvin Lin2, Anxing Xiao2, Jiafei Duan4, and Harold Soh23  2Dept.
of Computer Science, National University of Singapore  4University of
Washington, 3NUS Smart Systems Institute

###### Abstract

Physical reasoning is important for effective robot manipulation. Recent work
has investigated both vision and language modalities for physical reasoning;
vision can reveal information about objects in the environment and language
serves as an abstraction and communication medium for additional context.
Although these works have demonstrated success on a variety of physical
reasoning tasks, they are limited to physical properties that can be inferred
from visual or language inputs. In this work, we investigate combining tactile
perception with language, which enables embodied systems to obtain physical
properties through interaction and apply common-sense reasoning. We contribute
a new dataset PhysiCLeAR, which comprises both physical/property reasoning
tasks and annotated tactile videos obtained using a GelSight tactile sensor.
We then introduce Octopi, a system that leverages both tactile representation
learning and large vision-language models to predict and reason about tactile
inputs with minimal language fine-tuning. Our evaluations on PhysiCLeAR show
that Octopi is able to effectively use intermediate physical property
predictions to improve physical reasoning in both trained tasks and for zero-
shot reasoning. PhysiCLeAR and Octopi are available on
https://github.com/clear-nus/octopi.

##  I Introduction

For humans, touch is a crucial sense that provides physical information beyond
what vision can provide (e.g., material properties, texture information,
temperature) especially during occlusion. This in turn improves our ability to
perform physical reasoning [36, 4] and act in our world. Here, we are
interested in enabling general purpose robots, specifically those empowered
with large-language models (LLM), to perform similar physical reasoning. While
recent work has demonstrated that LLMs and large vision-language models
(LVLMs) can provide an impressive level of common-sense and physical reasoning
[45, 54, 3, 52], they are limited to either visual or text input modalities,
and thus have limited performance in scenarios with visual ambiguity [16].

Figure 1: Zero-shot avocado ripeness selection by combining tactile
information with commonsense knowledge. Using inputs from its tactile sensor,
Octopi identifies the left avocado as softer. Using commonsense reasoning,
Octopi infers that it is ripe and fulfils the user’s request.

This paper extends LVLMs to have the sense of touch. We posit that
incorporating a tactile modality into LVLMs will enable better physical
reasoning in real-world environments. As an example, Fig. 1 illustrates how
commonsense knowledge is applied together with tactile information to complete
a novel physical task in a zero-shot manner. Here, the robot leverages its
tactile inputs together with the LLM’s commonsense knowledge (that ripe
avocados are soft) to correctly select the ripe avocado (despite not having
been trained with tactile inputs from avocados). Here, we use visual-tactile
sensors, i.e., the GelSight [58], that provides image frames that reveal
physical object properties such as texture and hardness [57]. However, there
remains a significant domain gap between natural images that typical LVLMs are
trained with and the tactile data.

To bridge this gap, we created the PhysiCLeAR dataset, which comprises
GelSight images on a variety of real world objects, along with object labels
and part annotations. PhysiCLeAR complements existing tactile datasets [57,
59, 17, 55, 18] as it provides three physical property annotations,
specifically hardness, roughness, and bumpiness, that have been used in prior
research [42, 19, 37, 10, 5, 25] and can be potentially inferred from the
GelSight data. PhysiCLeAR includes an training and evaluation suite comprising
five reasoning tasks, which can serve as a benchmark for the research
community.

Using PhysiCLeAR, we developed Octopi (Object Comprehension with Tactile
Observations for Physical Intelligence). Octopi is a LLaMA-based [48, 49] LVLM
(Vicuna [11]) equipped with a CLIP-based [38] tactile encoder, whose
representations have been aligned via projection. In experiments, we show that
Octopi is able to use its tactile modality to predict project properties,
reason about scenarios, and even predict avocado ripeness.

Figure 2: PhysiCLeAR and Octopi. The full pipeline of our system with key contributions starred. We collect tactile videos for everyday household objects by hand with two exploratory procedures: pressing and rotation. The videos are annotated by three annotators for three physical properties: hardness, roughness and bumpiness. PhysiCLeAR leverages the videos and annotations for five language-driven physical description and understanding tasks. Octopi is a LVLM fine-tuned on PhysiCLeAR for tactile-grounded physical understanding and reasoning. TABLE I: GelSight Dataset Comparisons. PhysiCLeAR provides physical property labels for tactile descriptions and physical reasoning across three physical properties, as compared to five existing GelSight datasets. We further compare them with PhysiCLeAR across three diversity measures. Property diversity refers to whether there are objects in the dataset that vary across the three properties we selected: hardness, roughness and bumpiness. Object diversity indicates whether there is more than one type of object in the dataset. Material diversity indicates the number of different materials in the dataset. Dataset | Property Label Availability | Property Diversity | Object Diversity | Material Diversity
---|---|---|---|---
Hardness Dataset (2016) [57] | Yes (only hardness) | Yes | Yes | Medium
Clothing Dataset (2018) [59] | Yes | Yes | No (only clothing) | Low
ObjectFolder 2.0 (2022) [17] | No | No (only hard objects) | Yes | Medium
Touch and Go (2022) [55] | No | Yes | Yes | High
ObjectFolder-Real (2023) [18] | No | No (only hard objects) | Yes | Medium
PhysiCLeAR | Yes | Yes | Yes | Medium

Contributions. In summary, this paper makes the following key contributions:

  * •

A new GelSight dataset, PhysiCLeAR, that exhibits property diversity, object
diversity, and material diversity for selected physical properties.

  * •

Octopi, a framework for physical reasoning that leverages vision-based optical
tactile sensors, pre-trained vision models, and the reasoning capabilities of
LLMs.

  * •

An accompanying training and evaluation suite spanning five tasks and baseline
results using Octopi.

We hope that PhysiCLeAR and Octopi will spur research in tactile-enabled
physical reasoning for embodied AI systems [14].

##  II Related Work

In this section, we briefly review prior work on tactile representation
learning with the GelSight sensor, large vision-language models (LVLMs) and
language/vision-guided physical reasoning. There has been significant work in
tactile-based manipulation and physical reasoning, and we refer readers
desiring information on these topics to relevant survey papers [36, 14, 64,
46, 27].

Tactile Representation Learning with GelSight. Tactile representation learning
has advanced significantly in recent years as robotic manipulation requires
more precision beyond what can be provided by vision alone [43]. Among the
available tactile sensors, vision-based sensors have gained popularity due to
their high-resolution image outputs and versatility. In particular, the
GelSight sensor has been used in recent work [31, 57, 58, 23, 59] for
inferring physical properties (e.g. hardness, texture and liquid volume) and
to manipulate objects [47]. A key benefit of the GelSight is that its image
outputs can be easily processed by modern deep learning methods [23]. As a
result, popular vision algorithms have been used for tactile representation
learning with GelSight [60, 8]. In our work, we exploit recent advances in
tactile representation learning to extend the capabilities of LVLMs to reason
about vision-based tactile input.

Large Vision-Language Models. Recent advancements in Large Language Models
(LLMs) have spurred a significant increase in efforts to integrate vision
models with LLMs, exemplified by Flamingo [1], BLIP-2 [28], and MiniGPT 4 V2
[9]. These Large Vision-Language Models (LVLMs) have shown remarkable
effectiveness in utilizing web-scale image-text data for image-based
reasoning, benefiting a range of applications from robotics [7, 15] to medical
imaging [41]. Very recent work involves developing LVLMs that can process
video content [29, 35], enabling reasoning over dynamic visual information, or
integrating multi-modal sensory data [62].

TABLE II: Physical Property Details. We select three physical object properties and list their descriptions and semantic categories.  Property | Description | Categories
---|---|---
Hardness | how easily an object’s surface is deformed when pressed | soft, moderately hard, hard
Roughness | how rough an object’s surface is | smooth, slightly rough, rough
Bumpiness | the size of bumps on an object’s surface | no bumps, small bumps, big bumps

Physical Reasoning with Language and Vision as Context. The exploration of
physical reasoning in conjunction with language predates the emergence of
large language models (LLMs). Early studies focused on assessing model
proficiency in physical reasoning. For example, the PIQA [6] benchmark
evaluates models on physical common sense, whereas PROST[2] examines their
understanding of physical reasoning concepts. Subsequent advancements in
language grounding have led to works such CLEVRER [56], PIP [13], SPACE [12]
and Phys101 [53], which investigate the acquisition of physical reasoning
skills from visual contexts.

In the emerging LLM era, research has focused on object-centric physical
reasoning in LLMs. This involves evaluating various LLMs for their physical
reasoning capabilities, e.g., NEWTON [51], and employing Vision-Language
Models (VLMs) to predict physical properties that are then used to facilitate
reasoning, as demonstrated in physically-grounded VLMs [16]. Unlike previous
studies that primarily address physical reasoning through the integration of
vision and language, Octopi stands out as the first model capable of
processing tactile images alongside language instructions to enable physical
reasoning. There has been recent work [21] that uses simulated tactile inputs
with LLMs, but we focus on real-world tactile data.

##  III PhysiCLeAR - Tactile and Physical Understanding Training & Evaluation
Suite

This section describes PhysiCLeAR, which comprises a tactile dataset with
physical property and object-part annotations, along with a training and
evaluation suite.

###  III-A Physical Property Selection

In this work, we focus on three categorical object properties: hardness,
roughness and bumpiness. We list each property’s description and categories in
Table II. Briefly, hardness is characterized by the extent of surface
deformation when subjected to pressure; _roughness_ pertains to the texture of
the surface; and _bumpiness_ describes the prominence of surface protrusions.
The _hardness_ of an object correlates with its compliance and thermal
characteristics. In contrast, _roughness_ and _bumpiness_ are attributes
influenced by the surface’s friction coefficient [10].

The selection of _hardness_ , _roughness_ , and _bumpiness_ as physical
attributes in our research is grounded in their relevance for physical
reasoning [42, 19, 37, 10, 5, 25]. Generally, static physical properties of
objects are categorized into geometric (e.g., size), material (e.g.,
hardness), and affective (e.g., comfort) [40]. Our study predominantly
addresses material properties, as we deemed geometric and affective properties
too challenging to ascertain using the GelSight. The choice of these specific
properties was also informed by the data collection methodology [26], tailored
to the limitations and strengths of the GelSight sensor, including
considerations for its sensitivity and durability.

###  III-B Dataset Collection & Annotation

To facilitate the grounding of our physical reasoning on tactile inputs, we
collected a dataset of 74 everyday objects, totalling 408 tactile videos and
corresponding videos showing the object as the data was collected. These
objects were selected to span across our three selected properties, with
variations across object types and materials. Detailed comparisons between
PhysiCLeAR and existing GelSight datasets can be found in Table I.

The GelSight data was collected by-hand to mitigate risk of damaging the
sensors and due to the challenge of securing different parts of irregularly-
shaped objects while performing the required sampling motions. For each
selected object, we captured up to seven tactile videos for each distinct
region identified by a human evaluator. This process involved a two-step
procedure: initially pressing the GelSight sensor against the object to
capture pressure readings, followed by rotating the sensor to acquire shear
readings. Each video generated from a single GelSight sensor reading
constitutes an individual data point within our dataset.

Annotations of the physical properties were carried out by three independent
annotators, with the average score used as the final annotation for each data
point. Annotators were provided with both the tactile videos and the objects.
Each property has three categories, and annotators were given the following
guidelines for labeling each property:

  * •

Hardness: The label soft is for objects that are compressible with little
force, moderately hard for objects that are compressible with moderate force,
and hard for objects that are incompressible even with a large pressing force.

  * •

Roughness: smooth is for objects that present very minimal or no resistance
when we slide our finger across its surface, slightly rough for objects with
slight resistance, and rough for objects with significant resistance.

  * •

Bumpiness: no bumps is for objects with no visible protrusions on its surface,
small bumps for objects with protrusions smaller than $\approx 1/4$ of the
tactile image upon contact, and big bumps for objects with protrusions larger
than $1/4$.

This process yielded over 1,200 annotations and we observed high inter-
annotator agreement scores (ICC3k of 0.894 (hardness), 0.979 (roughness), and
0.792 (bumpiness)). For reference, a score above 0.75 is considered good or
excellent reliability. The dataset was subsequently divided into three
distinct subsets (training, validation, and testing) following an 80-10-10
split. This division resulted in 60 objects for training, and 7 objects each
for validation and testing.

###  III-C Training & Evaluation Suite

TABLE III: PhysiCLeAR Task Details. PhysiCLeAR contains five physical description and understanding tasks. We give each task’s motivation and indicate whether they are used for Octopi’s training and/or evaluation. Task | Motivation | Training [T] | Evaluation [E]
---|---|---|---
Object Property Description (OPD) | LLM learns to describe tactile videos using physical properties. | ✓ | ✓
Property Comparison (PC) | Align LLM physical understanding with physical property descriptions. | ✓ | ✓
Property Superlative Selection (PSS) | Align LLM physical understanding with physical property descriptions. | ✓ | ✓
Property-object Matching (POM) | Align LLM object understanding with physical property descriptions. | ✓ | ✓
Property Scenario Reasoning (PSR) | Evaluate LLM’s ability to resolve scenarios using physical properties. | ✗ | ✓
TABLE IV: PhysiCLeAR Task Prompt Setup. We provide the language prompts for each task in PhysiCLeAR. The LLM (i.e. ASSISTANT) responses for each task are ground-truth tactile descriptions, except the last response in italics for PSR. This italicized response is generated by Octopi-13b, since PSR is not used for training and hence there is no ground-truth response. Task | Prompt Setup
---|---
OPD |  USER: Describe the physical properties of ¡tact_start¿$T_{1},...,T_{N}$ ¡tact_end¿.
|  ASSISTANT: The surface feels somewhat hard, lacks noticeable roughness and
there are no textural variations or fine ridges. Overall, it
| presents a moderately hard and smooth surface with no bumps.
PC |  USER: Is the object in the tactile video ¡tact_start¿$T_{1,1},...,T_{1,N}$¡tact_end¿ covered with smaller bumps than the one in ¡tact_start¿
|  $T_{2,1},...,T_{2,N}$¡tact_end¿? Describe both objects before answering.
|  ASSISTANT: First object: The surface feels solid and sturdy, lacks
noticeable roughness, and there are no prominent bumps or ridges.
| Overall, it presents a hard and smooth surface with no bumps. Second object:
The surface feels yielding, exhibits significant roughness and
| there are small bumps and ridges present. Overall, it presents a soft and
rough surface with small bumps. Conclusion: Yes, the first object is
| covered with smaller bumps.
PSS |  USER: Given three tactile videos: a) ¡tact_start¿$T_{1,1},...,T_{1,N}$¡tact_end¿, b) ¡tact_start¿$T_{2,1},...,T_{2,N}$¡tact_end¿, c) ¡tact_start¿
|  $T_{3,1},...,T_{3,N}$¡tact_end¿. Describe each object and then select the
smoothest.
|  ASSISTANT: a) The surface feels yielding, exhibits significant roughness
and there are small bumps and ridges present. Overall, it presents
| a soft and rough surface with small bumps. b) The surface feels somewhat
firm and flexible, exhibits no roughness and there are big bumps on
| the surface. Overall, it presents a moderately hard and smooth surface with
big bumps. c) The surface feels soft and yielding, exhibits a slight
| roughness and there are tiny textural variations due to the fibrous
structure. Overall, it presents a soft and slightly rough surface with small
| bumps. Conclusion: b) is the smoothest.
POM |  USER: Given three tactile videos: a) ¡tact_start¿$T_{1,1},...,T_{1,N}$
|  ¡tact_end¿, b) ¡tact_start ¿$T_{2,1},...,T_{2,N}$¡tact_end¿, c)
¡tact_start¿$T_{3,1},...,T_{3,N}$¡tact_end¿. Describe the object in each
video, then
| match each video to one of the following objects in alphabetical order: 1) a
mandarin orange, 2) a cotton ball, 3) a roll of masking tape.
|  ASSISTANT: a) The surface feels firm, exhibits a slight roughness and there
are small textural irregularities and fine ridges. Overall, it
| presents a hard and slightly rough surface with small bumps. b) The surface
feels soft and yielding, exhibits no roughness and there are no
| prominent bumps or ridges. Overall, it presents a soft and smooth surface
with no bumps. c) The surface feels soft, exhibits a slight roughness
| and there are tiny bumps present. Overall, it presents a soft and slightly
rough surface with small bumps. Conclusion: a) is a roll of masking
| tape, b) is a cotton ball and c) is a mandarin orange.
PSR |  USER: Describe these two tactile videos: a) ¡tact_start¿$T_{1,1},...,T_{1,N}$¡tact_end¿, b) ¡tact_start ¿$T_{2,1},...,T_{2,N}$¡tact_end¿.
|  ASSISTANT: a) The surface feels hard and solid, exhibits a lack of
roughness and there are no textural variations or fine ridges. Overall, it
| presents a hard and smooth surface with no bumps. b) The surface feels
yielding, has a subtle roughness and there are small irregularities and
| bumps. Overall, it presents a soft and slightly rough surface with small
bumps.
|  USER: In an emergency, which of the objects above can be used to break
through thin ice covering a car outside? Select only one most
| appropriate object for this scenario based on physical property descriptions
of the two objects. Use the format ‘The most suitable object is x),
| because xxx’.
|  ASSISTANT: _The most suitable object is a) a hard and smooth surface with
no bumps, because it is more likely to penetrate the thin ice_
| _covering a car exterior and not get stuck._

PhysiCLeAR’s training and evaluation suite comprises five physical reasoning
tasks (Table III). All five tasks use tactile data and natural language
instructions as inputs (Table IV). Since the tactile data is in video form, we
follow prior LVLM work and represent it as a sequence of frames:
$X_{1},...,X_{N}$. We further detail each task’s motivation, setup, evaluation
details and whether they are used for training [T] and/or evaluation [E]
below:

Object Property Description (OPD) [T, E]. This task addresses property-based
description: generating both unstructured and structured descriptions of an
object’s hardness, roughness, and bumpiness from tactile videos. It parallels
existing image or video captioning tasks, which use natural visuals, and aids
the LLM in interpreting tactile signals. The unstructured description provides
more complex descriptions in cases where they are visible in the tactile video
(e.g. “fibrous structure” for toilet paper or “grains” for a scoop of rice).
The language instructions are variants of “Describe the physical properties of
¡tact_start¿$T_{1},...,T_{N}$¡tact_end¿.” The unstructured description is
generated using ChatGPT 3.5 and manually cleaned to produce a diverse
description of our objects based on our three physical properties. The
structured description of an object’s physical properties using our
annotations is formatted as: “Overall, it presents a hardness_label and
roughness_label surface with bumpiness_label.”

Property Comparison (PC) [T, E]. Given two tactile videos, each of a different
object, and a specified physical property, and its comparative adjective,
determine whether the comparative adjective accurately describes the two
videos. From a training perspective, this task helps a model distinguish
between the various descriptions of physical properties, thereby aligning its
comprehension of physical characteristics with our defined categories of
_hardness_ , _roughness_ , and _bumpiness_. This alignment ability may improve
a model’s ability to interpret and reason about the physical world in a manner
consistent with human understanding.

Property Superlative Selection (PSS) [T, E]. For three tactile videos, each of
a different object, and a specified physical property and its superlative
adjective (e.g. hardest for the hardness property), choose the video that the
superlative adjective best describes. This task is similar to the PC task and
helps the LLM align its physical understanding with that of our physical
property descriptions. Furthermore, since prior work has shown that LLMs might
perform differently when the polarity of the comparative adjective changes
[51], this task seeks to enhance the LLM’s resilience to various comparative
descriptions of physical properties.

Property-object Matching (POM) [T, E]. This task requires matching physical
properties to objects: given three tactile videos (each featuring a different
object) and three specified objects, the goal is to correctly associate each
video with an object. This helps to align a model’s existing knowledge of
object properties with our haptic perception, as our annotations are based on
human touch and serve as the reference for the physical properties and their
labels.

Property Scenario Reasoning (PSR) [E]. We provide two tactile videos, each
showcasing a different object, along with a real-world scenario that relies on
one or more of our defined physical properties. The task is to choose the
video that represents the object whose physical properties best meet the
scenario’s demands. This approach allows us to assess a model’s zero-shot
physical reasoning capabilities. Details of the scenarios are presented in
Table V.

TABLE V: Scenario Questions with Selected Object and/or Properties. We list the three scenarios we use to evaluate our model’s zero-shot physical reasoning capabilities using the three selected physical properties. For each scenario, we either specify the target object (along with the properties it has) or the target properties to select the target and non-target objects. Scenario Question | Target Object | Target Properties
---|---|---
Which object is most suitable for removing stains from a non-stick pan without scratching it? | Microfiber cloth | hardness, roughness
Which object would be most easily grippable when wet and slippery? | - | roughness, bumpiness
In an emergency, which of the objects above can be used to break through thin ice covering a car outside? | - | hardness

##  IV Octopi - Vision-language Property-guided Physical Reasoning

Figure 3: Octopi Framework. Our framework consists of CLIP’s visual encoder, a
projection module with two linear layers and Vicuna v1.5 as the LLM. Language
embeddings are derived through tokenization and then Vicuna’s word embedding
layer, with ¡tact_start¿ and ¡tact_end¿ being newly trained word embeddings
indicating the start and end of a tactile frame sequence from a single tactile
sensor. Tactile frames are fed into the visual encoder and then the projection
module to derive tactile embeddings with the same dimension as the word
embeddings.

The Octopi framework comprises three trained components: 1) tactile input
encoder, 2) projection module, and 3) LLM, similar to prior LVLM models [33,
35, 61]. A summary of our overall framework is shown in Fig. 3.

We leverage the capabilities of pre-trained vision models, notably the CLIP
[38] visual encoder ViT-L/14, as the foundation for our tactile encoder to
derive meaningful feature representations. The encoder’s output is then mapped
to the LLM’s word embedding space using a projection module, typically
consisting of one or two trainable layers. Our projection module, inspired by
LLaVA [33, 32], employs two linear layers with an intermediate GELU activation
[20]. Lastly, the LLM serves as the language understanding component in
Octopi. The performance of the LLM is largely influenced by its pre-training
datasets. We utilize the open-source LLaMA-based LLM, Vicuna [11], recognized
for its dialogue capabilities.

The inference process is illustrated in Fig. 3. Octopi receives an instruction
to evaluate the physical properties of uncooked rice. The text is tokenized
and fed into the LLM’s word embedding layer to produce word [W] embeddings. A
sequence of five tactile images is processed through the tactile encoder, with
the output embeddings sent to the projection module to obtain the final
tactile [T] embeddings. Newly trained word embeddings, represented by
¡tact_start¿ and ¡tact_end¿, mark the beginning and end of the tactile data,
respectively. These tactile embeddings are then merged with the word
embeddings at designated positions to form the final instruction embeddings
for the LLM.

We follow a three-step training methodology: (i) encoder fine-tuning, (ii)
tactile feature alignment, and (iii) end-to-end fine-tuning. In the following,
we describe each of these steps in greater detail.

###  IV-A Encoder Fine-tuning

Existing LVLM models take natural videos as input and can use CLIP’s visual
encoder without modification. However, our work involves vision-based tactile
inputs, which marks a significant distribution shift from natural images,
necessitating additional fine-tuning to derive useful representations from
these inputs.

We fine-tune our visual encoder to obtain useful representations from tactile
inputs using multitask physical property classification. We adopt the
architecture of ViFi-CLIP [39] so that our visual encoder can be trained on
video inputs. In ViFi-CLIP, frame-level embeddings from CLIP’s visual encoder
are average-pooled to obtain a video-level representation.

We then append learnable prompts to the pre-trained CLIP visual encoder
ViT-L/14 following Visual Prompt Tuning (VPT) [24] and initialize ViFi-CLIP’s
visual encoder with the new visual encoder. Specifically, we attach 8 task-
specific learnable prompts and a shared linear layer to the input sequence of
each Transformer [50] layer in the visual encoder and freeze the entire pre-
trained Transformer backbone.

Finally, we add three separate classification heads to ViFi-CLIP, each of
which predicts a label for one property (i.e. hardness, roughness or
bumpiness), and train all three classification heads simultaneously using the
cross-entropy loss. The model achieving the highest combined validation
accuracy — correctly predicting all three properties for an object — is
selected.

###  IV-B Tactile Feature Alignment

We discard the fine-tuned CLIP’s classification layers and use the outputs
from its visual encoder as output embeddings. To align the output embeddings
from the fine-tuned visual encoder with the LLM, the projection module is
trained on language annotations while the encoder and the LLM are frozen. We
also fine-tune the embedding layer due to the two new word tokens (i.e.
¡tact_start¿ and ¡tact_end¿).

###  IV-C End-to-end Fine-tuning

Finally, we used end-to-end fine-tuning to improve the coherence of the LLM’s
responses and increase the similarity between its responses and the language
annotations. In this stage, only the visual encoder is frozen while the word
embedding layer, projection module, and LLM are fine-tuned. We fine-tune the
LLM using low-rank adaptation (LoRA) [22] for parameter-efficient fine-tuning.

##  V Experimental Setup

In this section, we evaluate the physical property prediction and reasoning
capabilities of our proposed method. We design several experiments to answer
the following questions:

  1. 1.

Are our physical properties prediction useful for Octopi to reason about
everyday scenarios effectively?

  2. 2.

Can Octopi be used in real robots to help them accomplish tasks using physical
reasoning?

  3. 3.

Can Octopi’s understanding of the physical properties generalize to the unseen
daily life objects?

###  V-A Data Processing

The tactile videos were processed into frames. To focus on a few salient
frames for better efficiency, we selected frames that have the top 30% total
pixel intensity difference with their preceding frames. We randomly sampled 5
frames from these salient frames during training and selected 5 frames at
uniform intervals from the first salient frame during evaluation. Data
augmentation was performed during training in the form of random horizontal
and vertical flips with 50% chance for each flip.

###  V-B Training Hyperparameters

Encoder fine-tuning was performed for 30 epochs using the AdamW optimizer [34]
with no weight decay, a learning rate of $10^{-3}$, batch size of 32, and a
cosine annealing learning rate schedule. During tactile feature alignment, the
projection module is trained using 8k PhysiCLeAR samples using the AdamW
optimizer [34] with no weight decay, a learning rate of $2\times 10^{-5}$,
batch size of 16, and a cosine annealing learning rate schedule.

For end-to-end fine-tuning, both the projection module and the LLM’s LoRA
parameters are trained using 3k PhysiCLeAR samples using the AdamW optimizer
[34] with no weight decay, batch size of 16 and a cosine annealing learning
rate schedule. Learning rates of $2\times 10^{-5}$ and $2\times 10^{-4}$ were
used for the projection module and the LLM, respectively. We use a scaling
factor of 256, a rank of 128, and a dropout rate of 0.05 for LoRA.

###  V-C Training Requirements

Encoder fine-tuning took 6 hours and required less than 5GB of GPU VRAM.
Tactile feature alignment together with end-to-end fine-tuning took 5 hours
for Octopi-7b and 6.5 hours for Octopi-13b. We used 1 NVIDIA RTX A6000 for
Octopi-7b and 2 NVIDIA RTX A6000s for Octopi-13b.

##  VI Experimental Results

To address the above questions, we evaluated Octopi using (i) accuracy on the
physical understanding tasks in PhysiCLeAR’s test set, (ii) accuracy on
scenario reasoning tasks, (iii) task success rate on a real robot, and (iv)
property prediction accuracy on unseen objects. We tested two versions of
Octopi, Octopi-7b and Octopi-13b, which use Vicuna-7b v1.5 and Vicuna-13b v1.5
as their LLMs, respectively.

###  VI-A Tactile-grounded Physical Understanding with Object Property
Descriptions

TABLE VI: Results on PhysiCLeAR Physical Understanding Tasks. Octopi’s performance on physical understanding tasks improves with object property descriptions (OPD). Performance also increases with larger LLM size, with Octopi-13b outperforming Octopi-7b across all three tasks. | Random | Octopi-7b | Octopi-7b (no OPD)
---|---|---|---
PC | 33.33 | 48.10 | 46.51
PSS | 33.33 | 74.67 | 39.88
POM | 16.67 | 44.39 | 23.23
| Random | Octopi-13b | Octopi-13b (no OPD)
PC | 33.33 | 55.06 | 40.70
PSS | 33.33 | 84.00 | 39.88
POM | 16.67 | 60.43 | 18.71

During tactile feature alignment and end-to-end fine-tuning, we trained Octopi
with comparison tasks (i.e. PC, PSS and POM) to align its physical
understanding of our physical properties and objects with our labels. We
evaluated Octopi’s physical understanding with the same single-step prompts
used during training and on 500 question-answer pairs in total across the
three tasks. The results for physical understanding of unseen test objects are
shown in Table VI.

Our results show that both Octopi-7b and Octopi-13b perform well on all three
physical understanding tasks when they are trained to predict property
descriptions. Using physical property descriptions, Octopi-7b achieves
accuracies of 48.10% on PC, 74.67% on PSS and 44.39% on POM. Octopi-13b
outperforms Octopi-7b by 6.96% on PC, 9.33% on PSS and 16.04% on POM. This
suggests that Octopi’s physical understanding improves significantly with LLM
size.

Further, we explored the effect of using physical property descriptions by
fine-tuning both Octopi-7b and Octopi-13b on the physical understanding tasks
without intermediate physical property predictions. We found that predictions
based on object properties notably improve physical understanding in both
Octopi-7b and Octopi-13b.

###  VI-B Zero-shot Scenario Reasoning

TABLE VII: Results on PhysiCLeAR Scenario Reasoning Tasks. During scenario reasoning, we do not provide ground-truth property descriptions. Our experiments show that leveraging object properties significantly improves scenario reasoning for Octopi.  | Random | Octopi-7b | Octopi-13b
---|---|---|---
PSR | 50.00 | 69.57 | 67.39
PSR (w/o OPD) | 50.00 | 63.04 | 39.13

We assessed the usefulness of our physical property categories by testing how
Octopi can reason about everyday scenarios using the physical properties. For
reference, the different scenario questions are provided in Table V with the
prompts shown in Table IV.

Our results are summarized in Table VII. For both Octopi-7b and Octopi-13b,
including the object property significantly improves performance, which
supports our overall hypothesis that leveraging these properties is helpful
for these tasks. Interestingly, we observed that the 7b model marginally
outperformed the 13b model.

Figure 4: Rice (Cooked v.s. Uncooked) Reasoning. Octopi-13b is prompted to
reason about whether a scoop of rice is more likely to be cooked or uncooked
based on a tactile video of a scoop on uncooked rice. It reasons about the
rice state correctly without being trained to do so. Figure 5: Toothbrush Part
Reasoning. Given a tactile video of a toothbrush’s handle and the same
toothbrush’s bristles, Octopi-13b is prompted to reason which video belongs to
the handle and which video belongs to the bristles.

We further provide two qualitative examples to show Octopi-13b performs zero-
shot commonsense physical reasoning. In the first task, we provide a tactile
video of a scoop of uncooked rice and first instruct it to describe the
tactile video. We then follow up with an instruction to Octopi-13b to
determine if the rice is uncooked or cooked. Octopi-13b is able to reason that
the scoop of rice is uncooked due to its rough surface, as shown in Fig. 4.

Next, we gave Octopi-13b two tactile videos corresponding to two different
parts of the same toothbrush - the handle and the bristles. It is instructed
to describe both objects using the physical properties. We then instruct it to
determine which tactile video belongs to each object part using the physical
properties. Fig. 5 shows that Octopi-13b is able to reason about the property-
object match correctly.

###  VI-C Zero-shot Avocado Ripeness Classification

To evaluate Octopi’s usefulness as a tactile-grounded physical reasoning
system for real world tasks, we integrated two GelSight sensors on a 7-DoF
Franka Emika Panda robot and used it for avocado ripeness classification.
While ripe avocados generally appear in a shade of brown, their ripeness is
difficult to determine using vision alone. At the same time, ripe avacados are
softer then unripe ones and thus, tactile sensations can improve
classification.

We performed property prediction and ripeness classification evaluations using
a set of 10 avocados with 20 tactile samples collected from each avocado (i.e.
200 total samples). During ripeness classification, 100 pairs of avocado
samples were selected and Octopi was tasked to identify which avocado is
riper. Each pair contains avocados in different stages of ripeness. At test
time, the Franka robot grasped each avocado once to collect the tactile
readings, before passing it to the model. The model gave instructions on which
avocado to place in the ripe bin, and this was relayed through ROS for
execution.

TABLE VIII: Zero-Shot Avocado Property Prediction and Ripeness Classification Results. Octopi-13b predicts avocado properties reasonably well with only a pressing motion. For avocado ripeness classification, Octopi-13b is able to leverage its commonsense knowledge to use both hardness and bumpiness properties. | Random | Octopi-13b | PG-InstructBLIP
---|---|---|---
Property Prediction | 3.70 | 35.50 | 0.00
Hardness | 33.33 | 57.50 | 37.50
Roughness | 33.33 | 71.00 | 3.00
Bumpiness | 33.33 | 64.00 | 9.50
Ripeness Classification | 50.00 | 63.00 | -

Table VIII shows Octopi-13b’s performance on both avocado property prediction
and avocado ripeness classification. We split the evaluation into two parts
and evaluate the avocado property prediction and ripeness classification
separately.

For avocado property prediction, Octopi-13b achieves an accuracy of 35.50%,
which is significantly higher than that of the random baseline (3.70%). Unlike
the PhysiCLeAR dataset, these tactile videos are collected with only pressing
and without any rotation. This suggests that Octopi-13b’s physical property
prediction capability is robust to differences in tactile exploratory
procedures.

Octopi-13b obtains a zero-shot ripeness prediction of 63.00%. To set up this
task, we first elicit Octopi’s commonsense reasoning for avocado ripeness
using our properties by querying “You will be given tactile descriptions that
consist of three physical properties: hardness, roughness, bumpiness. Hardness
categories in increasing hardness: soft, moderately hard, hard. Roughness
categories in increasing roughness: smooth, slightly smooth, rough. Bumpiness
categories in increasing bumpiness: no bumps, small bumps, big bumps. Which of
these properties help to determine avocado ripeness? Rank them.”. Octopi-13b
responds with “The properties that help determine avocado ripeness are
hardness and bumpiness. Roughness is not a reliable indicator of avocado
ripeness. In terms of hardness, a ripe avocado will be moderately hard, while
an unripe avocado will be hard. In terms of bumpiness, a ripe avocado will
have small bumps, while an unripe avocado will have no bumps or small bumps.”
This exchange is provided as context to Octopi-13b during ripeness
classification.

For comparison, we evaluated a physically-grounded vision-language model, PG-
InstructBLIP [16], on property predictions of our avocados. PG-InstructBLIP
was trained to infer a predetermined set of physical properties from _visual_
images of real objects in the EgoObjects dataset [63]. Table VIII shows PG-
InstructBLIP’s performance on property prediction for our avocados was poor.
Possible reasons for this are that (i) the definitions of the physical
properties may not be well-aligned with PhysiCLeAR, and/or (ii) the physical
properties of avocados are not clearly apparent using only the visual
modality. We could not coax the PG-InstructBLIP model to directly classify
avocado ripeness despite trying various prompts; it would always pick the
first object.

###  VI-D Object Property Description Prediction

TABLE IX: Results on PhysiCLeAR Object Property Description Test Set. FT CLIP is the combination of the fine-tuned CLIP visual encoder and the three separate trained classification layers. Octopi-7b and Octopi-13b perform above the random baseline for object property predictions and have similar performance to the fine-tuned CLIP. Octopi-13b performs better than Octopi-7b on the prediction task. | Random | FT CLIP | Octopi-7b | Octopi-13b
---|---|---|---|---
Combined | 3.70 | 57.89 | 47.37 | 55.26
Hardness | 33.33 | 86.84 | 71.05 | 73.68
Roughness | 33.33 | 76.32 | 73.68 | 78.95
Bumpiness | 33.33 | 71.05 | 81.58 | 78.95

The physical understanding and scenario reasoning capabilities of Octopi
depends on its initial physical property predictions. We evaluated Octopi’s
physical property prediction on the PhysiCLeAR test set and show the results
in Table IX. Both Octopi-7b and Octopi-13b perform well above the random
baseline for combined and individual property prediction and have similar
performance to the fine-tuned CLIP model, indicating that Octopi can be used
for object property prediction. Octopi-13b has a higher combined accuracy
(i.e. all three physical properties are correctly predicted for a given
object) when compared to Octopi-7b, suggesting there are performance gains
with larger LLMs for tactile signal grounding (apart from the bumpiness
property).

##  VII Ablations

In this section, we describe ablation studies to examine (i) the impact of the
encoder’s learned representations on physical property prediction and (ii) the
influence of end-to-end fine-tuning data quantity on physical reasoning. For
the following sections, we report test accuracy on unseen objects.

###  VII-A Ablation: The Impact of Encoder Fine-tuning

TABLE X: CLIP Fine-tuning Ablation Results on Object Property Prediction. FT refers to fine-tuned. Using the CLIP fine-tuned on property prediction improves Octopi’s performance in property prediction. | Octopi-7b (FT CLIP) | Octopi-7b (base CLIP)
---|---|---
Combined | 47.37 | 39.47
Hardness | 71.05 | 81.58
Roughness | 73.68 | 52.63
Bumpiness | 81.58 | 55.26
| Octopi-13b (FT CLIP) | Octopi-13b (base CLIP)
Combined | 55.26 | 50.00
Hardness | 73.68 | 76.32
Roughness | 78.95 | 65.79
Bumpiness | 78.95 | 76.32

We used vision-based tactile inputs in this work and pre-trained vision
foundation models (i.e. CLIP) have shown impressive zero-shot performance on
vision tasks. To test whether additional fine-tuning improves the pre-trained
CLIP encoder’s representations for physical property prediction using tactile
images, we conducted ablation experiments. We compared the performance of two
Octopi versions — one trained with the off-the-shelf CLIP encoder and the
other trained with the fine-tuned CLIP encoder.

In Table X, our Object Property Description results show that Octopi-7b
trained with a fine-tuned CLIP encoder outperforms one trained with an
unmodified CLIP encoder by 7.90% on combined accuracy. Similarly, Octopi-13b
with the fine-tuned CLIP visual encoder performs better on the combined,
roughness, and bumpiness predictions, with the combined accuracy being 5.26%
higher. This suggests that a fine-tuned CLIP generally improves its learned
representations for physical property prediction in an end-to-end LVLM system.

TABLE XI: CLIP Fine-tuning Ablation Results on Physical Understanding Tasks. Using a fine-tuned CLIP improves Octopi’s performance in physical understanding tasks for both Octopi-7b and Octopi-13b. | Octopi-7b (fine-tuned CLIP) | Octopi-7b (base CLIP)
---|---|---
PC | 48.10 | 30.38
PSS | 74.67 | 42.67
POM | 44.39 | 36.36
PSR | 69.57 | 69.57
| Octopi-13b (fine-tuned CLIP) | Octopi-13b (base CLIP)
PC | 55.06 | 47.47
PSS | 84.00 | 75.33
POM | 60.43 | 57.22
PSR | 67.39 | 45.65

We further tested both Octopi versions on physical understanding tasks with
results in Table XI. For Octopi-7b, the version trained with a fine-tuned CLIP
encoder performs better across the three physical understanding tasks (by
17.72% on PC, 32.00% on PSS, 8.03% on POM). Similarly, Octopi-13b with the
fine-tuned CLIP encoder has a better performance for physical understanding
tasks, which suggests that fine-tuning generally helps physical understanding
and zero-shot physical reasoning performance.

###  VII-B Ablation: The Impact of End-to-end Fine-tuning

TABLE XII: End-to-end Fine-tuning Physical Property Prediction Result Comparisons. End-to-end fine-tuning with LoRA generally improves physical property prediction accuracies. | Octopi-7b (w/ LoRA) | Octopi-7b (w/o LoRA)
---|---|---
Combined | 47.37 | 39.47
Hardness | 71.05 | 65.79
Roughness | 73.68 | 76.32
Bumpiness | 81.58 | 71.05
| Octopi-13b (w/ LoRA) | Octopi-13b (w/o LoRA)
Combined | 55.26 | 23.68
Hardness | 73.68 | 36.84
Roughness | 78.95 | 73.68
Bumpiness | 78.95 | 71.05

Table XII shows Octopi’s performance on the property prediction task before
and after end-to-end fine-tuning with LoRA. For both Octopi-7b and Octopi-13b,
the fine-tuned variants generally performed better. We see sharp improvements
for Octopi-13b model with improvements across the properties. Our results
suggest that end-to-end fine-tuning improves physical property prediction
accuracy. Similar to the property prediction task, we observed that fine-
tuning with LoRA also improves Octopi’s performance on physical understanding
tasks (Table XIII).

TABLE XIII: End-to-end Fine-tuning Physical Understanding Result Comparisons. End-to-end fine-tuning for physical understanding tasks significantly improves physical understanding for both Octopi-7b and Octopi-13b. | Rand. | Octopi-7b (LoRA) | Octopi-7b (w/o LoRA)
---|---|---|---
PC | 33.33 | 48.10 | 40.51
PSS | 33.33 | 74.67 | 30.00
POM | 16.67 | 44.39 | 12.30
PSR | 50.00 | 69.57 | 63.04
| Rand. | Octopi-13b (LoRA) | Octopi-13b (w/o LoRA)
PC | 33.33 | 55.07 | 52.53
PSS | 33.33 | 84.00 | 77.33
POM | 16.67 | 60.43 | 34.76
PSR | 50.00 | 67.39 | 60.87

##  VIII Conclusion and Discussion

In this work, we extended large vision-language models (LVLMs) to process and
describe tactile inputs using physical properties. We proposed a tactile
dataset called PhysiCLeAR, comprising data from vision (Camera) and tactile
(GelSight) sensors collected from everyday objects, along with physical
property annotations. We also present Octopi, a large tactile-language model
trained using datasets like PhysiCLeAR to perform physical property reasoning
using tactile input.

Our experiments show that Octopi is able to describe tactile signals from
novel unseen objects and that inferred physical properties can be used for
zero-shot physical reasoning and robot task completion in scenarios with
visual ambiguity. We studied the impact of different components in Octopi, and
found that using a task-specific visual encoder that is fine-tuned on our
labels improves performance significantly across all tasks. This suggests that
improvements to the visual encoder will yield benefits. In addition,
parameter-efficient LLM fine-tuning consistently improved performance.

Our work opens up future work in tactile robotics. We are currently working on
visual encoder improvement and more diverse exploratory procedures to obtain
additional physical properties. It would also be interesting to combine
different datasets (e.g., those using other tactile sensors [43, 44]), along
with other modalities such as robot proprioception. We plan to also perform
physical understanding alignment with object images and LLM fine-tuning with
additional physical understanding data [51, 30].

## References

  * Alayrac et al. [2022] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al.  Flamingo: a visual language model for few-shot learning.  _Advances in Neural Information Processing Systems_ , 35:23716–23736, 2022.
  * Aroca-Ouellette et al. [2021] Stéphane Aroca-Ouellette, Cory Paik, Alessandro Roncone, and Katharina Kann.  Prost: Physical reasoning of objects through space and time.  _arXiv preprint arXiv:2106.03634_ , 2021.  URL https://arxiv.org/pdf/2106.03634.pdf.
  * Bai et al. [2023] Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou.  Qwen-vl: A frontier large vision-language model with versatile abilities.  _arXiv preprint arXiv:2308.12966_ , 2023.  URL https://arxiv.org/pdf/2308.12966.pdf.
  * Bakhtin et al. [2019] Anton Bakhtin, Laurens van der Maaten, Justin Johnson, Laura Gustafson, and Ross Girshick.  PHYRE: A New Benchmark for Physical Reasoning.  2019\.  URL https://arxiv.org/pdf/1908.05656.pdf.
  * Bergmann Tiest [2010] Wouter M. Bergmann Tiest.  Tactual perception of material properties.  _Vision Research_ , 50(24):2775–2782, 2010.  ISSN 0042-6989.  doi: https://doi.org/10.1016/j.visres.2010.10.005.  URL https://www.sciencedirect.com/science/article/pii/S0042698910004967.  Perception and Action: Part I.
  * Bisk et al. [2020] Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al.  Piqa: Reasoning about physical commonsense in natural language.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 34, pages 7432–7439, 2020.  URL https://arxiv.org/pdf/1911.11641.pdf.
  * Brohan et al. [2023] Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, et al.  Rt-2: Vision-language-action models transfer web knowledge to robotic control.  _arXiv preprint arXiv:2307.15818_ , 2023.  URL https://arxiv.org/pdf/2307.15818.pdf.
  * Cao et al. [2023] Guanqun Cao, Jiaqi Jiang, Danushka Bollegala, and Shan Luo.  Learn from Incomplete Tactile Data: Tactile Representation Learning with Masked Autoencoders.  In _2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , pages 10800–10805. IEEE, 2023.  URL https://arxiv.org/pdf/2307.07358.pdf.
  * Chen et al. [2023] Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, Zechun Liu, Pengchuan Zhang, Raghuraman Krishnamoorthi, Vikas Chandra, Yunyang Xiong, and Mohamed Elhoseiny.  Minigpt-v2: large language model as a unified interface for vision-language multi-task learning.  _arXiv preprint arXiv:2310.09478_ , 2023.  URL https://arxiv.org/pdf/2310.09478.pdf.
  * Chen et al. [2009] X. Chen, Fei Shao, Cathy Barnes, Tom Childs, and Brian Henson.  Exploring Relationships between Touch Perception and Surface Physical Properties.  _International Journal of Design_ , 3:67–76, 08 2009.  URL https://arxiv.org/pdf/1704.03822.pdf.
  * Chiang et al. [2023] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing.  Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023.  URL https://lmsys.org/blog/2023-03-30-vicuna/.
  * Duan et al. [2021] Jiafei Duan, Samson Yu, and Cheston Tan.  Space: A simulator for physical interactions and causal learning in 3d environments.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 2058–2063, 2021.
  * Duan et al. [2022a] Jiafei Duan, Samson Yu, Soujanya Poria, Bihan Wen, and Cheston Tan.  PIP: Physical Interaction Prediction via Mental Simulation with Span Selection.  In _European Conference on Computer Vision_ , pages 405–421. Springer, 2022a.  URL http://phys101.csail.mit.edu/papers/phys101_bmvc.pdf.
  * Duan et al. [2022b] Jiafei Duan, Samson Yu, Hui Li Tan, Hongyuan Zhu, and Cheston Tan.  A survey of embodied ai: From simulators to research tasks.  _IEEE Transactions on Emerging Topics in Computational Intelligence_ , 6(2):230–244, 2022b.
  * Duan et al. [2023] Jiafei Duan, Yi Ru Wang, Mohit Shridhar, Dieter Fox, and Ranjay Krishna.  Ar2-d2: Training a robot without a robot.  _arXiv preprint arXiv:2306.13818_ , 2023.
  * Gao et al. [2023a] Jensen Gao, Bidipta Sarkar, Fei Xia, Ted Xiao, Jiajun Wu, Brian Ichter, Anirudha Majumdar, and Dorsa Sadigh.  Physically grounded vision-language models for robotic manipulation.  _arXiv preprint arXiv:2309.02561_ , 2023a.  URL https://arxiv.org/pdf/2309.02561.pdf.
  * Gao et al. [2022] Ruohan Gao, Zilin Si, Yen-Yu Chang, Samuel Clarke, Jeannette Bohg, Li Fei-Fei, Wenzhen Yuan, and Jiajun Wu.  Objectfolder 2.0: A multisensory object dataset for sim2real transfer.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 10598–10608, 2022.  URL https://arxiv.org/pdf/2204.02389.pdf.
  * Gao et al. [2023b] Ruohan Gao, Yiming Dou, Hao Li, Tanmay Agarwal, Jeannette Bohg, Yunzhu Li, Li Fei-Fei, and Jiajun Wu.  The ObjectFolder Benchmark: Multisensory Learning With Neural and Real Objects.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 17276–17286, June 2023b.  URL https://arxiv.org/pdf/2306.00956.pdf.
  * Gao et al. [2016] Yang Gao, Lisa Anne Hendricks, Katherine J Kuchenbecker, and Trevor Darrell.  Deep learning for tactile understanding from visual and haptic data.  In _2016 IEEE international conference on robotics and automation (ICRA)_ , pages 536–543. IEEE, 2016.  URL https://arxiv.org/pdf/1511.06065.pdf.
  * Hendrycks and Gimpel [2016] Dan Hendrycks and Kevin Gimpel.  Gaussian error linear units (gelus).  _arXiv preprint arXiv:1606.08415_ , 2016.
  * Hong et al. [2024] Yining Hong, Zishuo Zheng, Peihao Chen, Yian Wang, Junyan Li, and Chuang Gan.  Multiply: A multisensory object-centric embodied large language model in 3d world.  _arXiv preprint arXiv:2401.08577_ , 2024.
  * Hu et al. [2021] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen.  Lora: Low-rank adaptation of large language models.  _arXiv preprint arXiv:2106.09685_ , 2021.  URL https://arxiv.org/pdf/2106.09685.pdf.
  * Huang et al. [2022] Hung-Jui Huang, Xiaofeng Guo, and Wenzhen Yuan.  Understanding dynamic tactile sensing for liquid property estimation.  _arXiv preprint arXiv:2205.08771_ , 2022.  URL https://arxiv.org/pdf/2205.08771.pdf.
  * Jia et al. [2022] Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim.  Visual prompt tuning.  In _European Conference on Computer Vision_ , pages 709–727. Springer, 2022.
  * Jiang and Luo [2022] Jiaqi Jiang and Shan Luo.  Robotic perception of object properties using tactile sensing.  In _Tactile Sensing, Skill Learning, and Robotic Dexterous Manipulation_ , pages 23–44. Elsevier, 2022.
  * Klatzky and Reed [2016] Roberta Klatzky and Catherine L Reed.  Haptic exploration.  _Scholarpedia of Touch_ , pages 177–183, 2016.
  * Lee [2000] Mark H. Lee.  Tactile sensing: New directions, new challenges.  _The International Journal of Robotics Research_ , 19(7):636–643, 2000.  doi: 10.1177/027836490001900702.  URL https://doi.org/10.1177/027836490001900702.
  * Li et al. [2023a] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi.  Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.  _arXiv preprint arXiv:2301.12597_ , 2023a.  URL https://dl.acm.org/doi/10.5555/3618408.3619222.
  * Li et al. [2023b] KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao.  Videochat: Chat-centric video understanding.  _arXiv preprint arXiv:2305.06355_ , 2023b.  URL https://arxiv.org/pdf/2305.06355.pdf.
  * Li et al. [2023c] Lei Li, Jingjing Xu, Qingxiu Dong, Ce Zheng, Xu Sun, Lingpeng Kong, and Qi Liu.  Can language models understand physical concepts?  In Houda Bouamor, Juan Pino, and Kalika Bali, editors, _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 11843–11861, Singapore, December 2023c. Association for Computational Linguistics.  doi: 10.18653/v1/2023.emnlp-main.726.  URL https://aclanthology.org/2023.emnlp-main.726.
  * Li and Adelson [2013] Rui Li and Edward H. Adelson.  Sensing and recognizing surface textures using a gelsight sensor.  In _2013 IEEE Conference on Computer Vision and Pattern Recognition_ , pages 1241–1247, 2013.  doi: 10.1109/CVPR.2013.164.
  * Liu et al. [2023a] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee.  Improved baselines with visual instruction tuning.  _arXiv preprint arXiv:2310.03744_ , 2023a.
  * Liu et al. [2023b] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee.  Visual instruction tuning.  _arXiv preprint arXiv:2304.08485_ , 2023b.  URL https://arxiv.org/pdf/2304.08485.pdf.
  * Loshchilov and Hutter [2017] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  _arXiv preprint arXiv:1711.05101_ , 2017.
  * Maaz et al. [2023] Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan.  Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models.  _arXiv preprint arXiv:2306.05424_ , 2023.  URL https://arxiv.org/pdf/2306.05424.pdf.
  * Melnik et al. [2023] Andrew Melnik, Robin Schiewer, Moritz Lange, Andrei Muresanu, Mozhgan Saeidi, Animesh Garg, and Helge Ritter.  Benchmarks for Physical Reasoning AI.  _arXiv preprint arXiv:2312.10728_ , 2023.  URL https://arxiv.org/pdf/2312.10728.pdf.
  * Purri and Dana [2020] Matthew Purri and Kristin Dana.  Teaching cameras to feel: Estimating tactile physical properties of surfaces from images.  In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXVII 16_ , pages 1–20. Springer, 2020.  URL https://arxiv.org/pdf/2004.14487.pdf.
  * Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In _International conference on machine learning_ , pages 8748–8763. PMLR, 2021.  URL https://arxiv.org/pdf/2103.00020.pdf.
  * Rasheed et al. [2023] Hanoona Rasheed, Muhammad Uzair Khattak, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan.  Fine-tuned clip models are efficient video learners.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 6545–6554, 2023.
  * Sakamoto and Watanabe [2017] Maki Sakamoto and Junji Watanabe.  Exploring Tactile Perceptual Dimensions Using Materials Associated with Sensory Vocabulary.  _Frontiers in Psychology_ , 8, 2017.  URL https://api.semanticscholar.org/CorpusID:14038261.
  * Saygin Seyfioglu et al. [2023] Mehmet Saygin Seyfioglu, Wisdom O Ikezogwo, Fatemeh Ghezloo, Ranjay Krishna, and Linda Shapiro.  Quilt-LLaVA: Visual Instruction Tuning by Extracting Localized Narratives from Open-Source Histopathology Videos.  _arXiv e-prints_ , pages arXiv–2312, 2023.  URL https://arxiv.org/pdf/2312.04746.pdf.
  * Takahashi and Tan [2019] Kuniyuki Takahashi and Jethro Tan.  Deep visuo-tactile learning: Estimation of tactile properties from images.  In _2019 International Conference on Robotics and Automation (ICRA)_ , pages 8951–8957. IEEE, 2019.  URL https://arxiv.org/pdf/1803.03435.pdf.
  * Taunyazov et al. [2020] Tasbolat Taunyazov, Weicong Sng, Hian Hian See, Brian Lim, Jethro Kuan, Abdul Fatir Ansari, Benjamin CK Tee, and Harold Soh.  Event-driven visual-tactile sensing and learning for robots.  _arXiv preprint arXiv:2009.07083_ , 2020.  URL https://arxiv.org/pdf/2009.07083.pdf.
  * Taunyazov et al. [2021] Tasbolat Taunyazov, Luar Shui Song, Eugene Lim, Hian Hian See, David Lee, Benjamin CK Tee, and Harold Soh.  Extended tactile perception: Vibration sensing through tools and grasped objects.  In _2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , pages 1755–1762. IEEE, 2021.
  * Team et al. [2023] Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al.  Gemini: a family of highly capable multimodal models.  _arXiv preprint arXiv:2312.11805_ , 2023.  URL https://arxiv.org/pdf/2312.11805.pdf.
  * Tegin and Wikander [2005] Johan Tegin and Jan Wikander.  Tactile sensing in intelligent robotic manipulation—a review.  _Industrial Robot: An International Journal_ , 32, 02 2005.  doi: 10.1108/01439910510573318.
  * Tian et al. [2019] Stephen Tian, Frederik Ebert, Dinesh Jayaraman, Mayur Mudigonda, Chelsea Finn, Roberto Calandra, and Sergey Levine.  Manipulation by feel: Touch-based control with deep predictive models.  In _2019 International Conference on Robotics and Automation (ICRA)_ , pages 818–824. IEEE, 2019.  URL https://arxiv.org/pdf/1903.04128.pdf.
  * Touvron et al. [2023a] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al.  Llama: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_ , 2023a.
  * Touvron et al. [2023b] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_ , 2023b.  URL https://arxiv.org/pdf/2307.09288.pdf.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  _Advances in neural information processing systems_ , 30, 2017.
  * [51] Yi Ru Wang, Jiafei Duan, Dieter Fox, and Siddhartha Srinivasa.  NEWTON: Are Large Language Models Capable of Physical Reasoning?
  * Wei et al. [2023] Haoran Wei, Lingyu Kong, Jinyue Chen, Liang Zhao, Zheng Ge, Jinrong Yang, Jianjian Sun, Chunrui Han, and Xiangyu Zhang.  Vary: Scaling up the Vision Vocabulary for Large Vision-Language Models.  _arXiv preprint arXiv:2312.06109_ , 2023.  URL https://arxiv.org/pdf/2312.06109.pdf.
  * Wu et al. [2016] Jiajun Wu, Joseph J Lim, Hongyi Zhang, Joshua B Tenenbaum, and William T Freeman.  Physics 101: Learning Physical Object Properties from Unlabeled Videos. In _BMVC_ , volume 2, page 7, 2016.  URL http://phys101.csail.mit.edu/papers/phys101_bmvc.pdf.
  * Wu et al. [2023] Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji, and Tat-Seng Chua.  Next-gpt: Any-to-any multimodal llm.  _arXiv preprint arXiv:2309.05519_ , 2023.  URL https://arxiv.org/pdf/2309.05519.pdf.
  * Yang et al. [2022] Fengyu Yang, Chenyang Ma, Jiacheng Zhang, Jing Zhu, Wenzhen Yuan, and Andrew Owens.  Touch and go: Learning from human-collected vision and touch.  _arXiv preprint arXiv:2211.12498_ , 2022.  URL https://arxiv.org/pdf/2211.12498.pdf.
  * Yi et al. [2019] Kexin Yi, Chuang Gan, Yunzhu Li, Pushmeet Kohli, Jiajun Wu, Antonio Torralba, and Joshua B Tenenbaum.  Clevrer: Collision events for video representation and reasoning.  _arXiv preprint arXiv:1910.01442_ , 2019.  URL https://arxiv.org/pdf/1910.01442.pdf.
  * Yuan et al. [2016] Wenzhen Yuan, Mandayam A. Srinivasan, and Edward H. Adelson.  Estimating object hardness with a gelsight touch sensor.  In _2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , pages 208–215, 2016.  doi: 10.1109/IROS.2016.7759057.
  * Yuan et al. [2017] Wenzhen Yuan, Siyuan Dong, and Edward H. Adelson.  Gelsight: High-resolution robot tactile sensors for estimating geometry and force.  _Sensors_ , 17(12), 2017.  ISSN 1424-8220.  doi: 10.3390/s17122762.  URL https://www.mdpi.com/1424-8220/17/12/2762.
  * Yuan et al. [2018] Wenzhen Yuan, Yuchen Mo, Shaoxiong Wang, and Edward H Adelson.  Active clothing material perception using tactile sensing and deep learning.  In _2018 IEEE International Conference on Robotics and Automation (ICRA)_ , pages 4842–4849. IEEE, 2018.  URL https://arxiv.org/pdf/1711.00574.pdf.
  * Zandonati et al. [2023] Ben Zandonati, Ruohan Wang, Ruihan Gao, and Yan Wu.  Investigating Vision Foundational Models for Tactile Representation Learning.  _arXiv preprint arXiv:2305.00596_ , 2023.  URL https://arxiv.org/pdf/2305.00596.pdf.
  * Zhang et al. [2023a] Hang Zhang, Xin Li, and Lidong Bing.  Video-llama: An instruction-tuned audio-visual language model for video understanding.  _arXiv preprint arXiv:2306.02858_ , 2023a.
  * Zhang et al. [2023b] Yiyuan Zhang, Kaixiong Gong, Kaipeng Zhang, Hongsheng Li, Yu Qiao, Wanli Ouyang, and Xiangyu Yue.  Meta-transformer: A unified framework for multimodal learning.  _arXiv preprint arXiv:2307.10802_ , 2023b.  URL https://arxiv.org/pdf/2307.10802.pdf.
  * Zhu et al. [2023] Chenchen Zhu, Fanyi Xiao, Andrés Alvarado, Yasmine Babaei, Jiabo Hu, Hichem El-Mohri, Sean Chang, Roshan Sumbaly, and Zhicheng Yan.  Egoobjects: A large-scale egocentric dataset for fine-grained object understanding.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2023.
  * Zhu et al. [2020] Yixin Zhu, Tao Gao, Lifeng Fan, Siyuan Huang, Mark Edmonds, Hangxin Liu, Feng Gao, Chi Zhang, Siyuan Qi, Ying Nian Wu, et al.  Dark, beyond deep: A paradigm shift to cognitive ai with humanlike common sense.  _Engineering_ , 6(3):310–345, 2020.

## Appendix

In the appendices below, we provide more details for our PhysiCLeAR dataset,
specifically for the annotations, objects, properties, sample videos and
prompts.

##  Appendix A Annotation Details

TABLE XIV: PhysiCLeAR Property Category Annotation Guidelines. For each property and its three categories, we detail the general guideline we used to choose the most appropriate category for a tactile sample. The annotator rating for each property and its categories are given as [X] where X is the rating. Property Category | Annotation Guideline
---|---
Soft [0] | compressible upon pressing with little force
Moderately hard [1] | compressible upon pressing with moderate force
Hard [2] | incompressible with high human pressing force
Smooth [0] | minimal feeling of friction upon finger sliding
Slightly rough [1] | slight feeling of friction upon finger sliding
Rough [2] | significant feeling of friction upon finger sliding
No bumps [0] | no bumps visible on object surface
Small bumps [1] | bumps are less than $1/4$ of the tactile image
Big bumps [2] | bumps are more than $1/4$ of the tactile image

While the annotations for PhysiCLeAR are done separately, we agree upon a
general set of guidelines to align the annotations for the properties and
reduce excessive variance. We show the guidelines in Table XIV. While the
hardness and roughness properties are generally annotated with actual haptic
feedback from human exploratory procedures, the bumpiness property is
generally annotated visually from the GelSight tactile images.

While the annotations for PhysiCLeAR are done separately, we agree upon a
general set of guidelines to align the annotations for the properties and
reduce excessive variance. We show the guidelines in Table XIV. While the
hardness and roughness properties are generally annotated with actual haptic
feedback from human exploratory procedures, the bumpiness property is
generally annotated visually from the GelSight tactile images.

TABLE XV: PhysiCLeAR Annotation Ratings. We show the annotation ratings for each property and its three categories for all objects. The ratings are given as [X, Y, Z] where X is the rating by annotator 1, Y by annotator 2 and Z by annotator 3. The mapping between the ratings and the property categories can be found in Table XIV. Sample Name | Hardness | Roughness | Bumpiness
---|---|---|---
baseball | [2, 2, 2] | [2, 2, 2] | [0, 1, 0]
basket | [2, 2, 2] | [2, 2, 2] | [2, 1, 2]
blanket | [1, 1, 1] | [0, 0, 0] | [0, 0, 0]
bread_knife_blade | [2, 2, 2] | [0, 0, 0] | [0, 0, 0]
bread_knife_handle | [2, 2, 2] | [1, 2, 2] | [0, 0, 0]
rubber_bands | [1, 0, 2] | [1, 0, 1] | [2, 2, 2]
wires | [2, 1, 2] | [0, 0, 0] | [2, 1, 2]
cotton_ball | [0, 0, 0] | [1, 0, 1] | [0, 0, 0]
---|---|---|---
rug | [1, 0, 1] | [2, 1, 2] | [2, 2, 2]
feather_duster | [0, 0, 1] | [0, 0, 0] | [1, 1, 1]
water_bottle | [1, 1, 1] | [0, 0, 0] | [2, 2, 2]
strainer_handle | [2, 2, 2] | [0, 0, 0] | [0, 0, 0]
strainer_base | [1, 2, 1] | [2, 2, 2] | [1, 2, 1]
controller_base | [2, 2, 2] | [0, 0, 0] | [0, 0, 0]
controller_buttons | [2, 2, 2] | [1, 1, 1] | [0, 1, 0]
controller_keypad | [2, 2, 2] | [1, 1, 1] | [2, 1, 2]
controller_shoulderpads | [2, 2, 2] | [0, 0, 1] | [1, 0, 1]
controller_stick | [2, 2, 2] | [1, 1, 1] | [1, 1, 1]
gauze_pad | [1, 2, 2] | [1, 1, 1] | [1, 2, 0]
lanyard_string | [1, 1, 2] | [0, 0, 1] | [1, 2, 0]
lanyard_card | [2, 2, 2] | [0, 0, 0] | [2, 0, 2]
leather_book | [2, 2, 2] | [0, 0, 0] | [0, 0, 0]
lemon | [1, 1, 2] | [1, 1, 1] | [1, 2, 1]
mandarin_orange | [1, 1, 1] | [1, 1, 1] | [1, 1, 1]
mop_head | [1, 1, 1] | [1, 1, 1] | [0, 1, 1]
rubber_slippers | [1, 1, 1] | [1, 1, 1] | [2, 1, 1]
scissor_blade | [2, 2, 2] | [0, 0, 0] | [0, 0, 0]
scissor_handle | [2, 2, 2] | [0, 0, 0] | [0, 0, 0]
paper_towel | [0, 0, 0] | [1, 1, 1] | [1, 1, 2]
avocado_kinda_ripe | [1, 2, 1] | [2, 2, 2] | [1, 1, 2]
kiwi_kinda_ripe | [1, 1, 1] | [2, 2, 2] | [1, 2, 0]
pen_base | [2, 2, 2] | [0, 0, 0] | [0, 0, 0]
pen_pad | [1, 0, 2] | [1, 0, 1] | [2, 0, 2]
pillow | [0, 0, 0] | [0, 0, 0] | [0, 0, 0]
potato | [2, 2, 2] | [1, 1, 1] | [1, 1, 1]
rice_spatula_handle | [2, 2, 2] | [0, 0, 0] | [0, 0, 0]
rice_spatula_base | [2, 2, 2] | [2, 2, 2] | [2, 2, 2]
masking_tape | [1, 2, 2] | [1, 1, 1] | [1, 1, 0]
toilet_paper | [1, 1, 1] | [1, 1, 1] | [1, 2, 1]
wood_ruler | [2, 2, 2] | [0, 0, 0] | [0, 0, 0]
rice | [1, 1, 2] | [2, 2, 2] | [2, 2, 2]
aluminum_foil | [1, 1, 2] | [1, 0, 1] | [1, 1, 2]
stress_ball | [0, 0, 1] | [0, 0, 0] | [0, 0, 0]
tennis_ball | [1, 2, 1] | [1, 1, 1] | [1, 1, 1]
toilet_brush_handle | [1, 0, 2] | [0, 0, 0] | [0, 0, 0]
tomato | [1, 1, 2] | [0, 0, 0] | [0, 0, 0]
toothbrush_handle | [2, 2, 2] | [0, 0, 0] | [2, 0, 2]
tv_remote_back | [2, 2, 2] | [0, 0, 0] | [0, 0, 0]
tv_remote_buttons | [1, 1, 2] | [2, 2, 2] | [1, 2, 0]
eraser | [1, 1, 2] | [0, 0, 0] | [0, 0, 0]
insulating_holder | [0, 0, 1] | [2, 2, 2] | [1, 2, 1]
orange | [1, 1, 1] | [1, 1, 1] | [1, 2, 1]
oven_mitt | [1, 1, 2] | [2, 2, 2] | [1, 2, 1]
egg | [2, 2, 2] | [0, 0, 0] | [0, 0, 0]
avocado_unripe | [2, 2, 2] | [2, 2, 2] | [1, 2, 0]
tissue_ball | [0, 0, 0] | [0, 0, 0] | [1, 2, 0]
hairbrush_bristles | [2, 2, 2] | [2, 2, 2] | [2, 2, 1]
sponge_rough | [1, 1, 2] | [2, 2, 2] | [1, 2, 1]
hairbrush_bristles_side | [1, 2, 1] | [2, 2, 2] | [2, 2, 2]
toothbrush_bristles | [0, 0, 1] | [2, 2, 2] | [1, 2, 1]
sponge_soft | [0, 0, 0] | [1, 1, 1] | [1, 2, 0]
steel_wool | [0, 0, 1] | [2, 2, 2] | [1, 1, 2]
feather_duster_handle | [2, 2, 1] | [0, 0, 0] | [0, 0, 1]
nylon_shirt | [1, 1, 1] | [1, 1, 1] | [0, 2, 0]
denim | [1, 1, 1] | [1, 1, 1] | [1, 2, 0]
millet | [1, 1, 2] | [2, 2, 2] | [1, 2, 1]
bubble_wrap | [1, 1, 1] | [2, 1, 2] | [2, 1, 2]
tsa_lock_numbers | [2, 2, 2] | [2, 2, 2] | [1, 1, 2]
bath_towel | [1, 1, 1] | [2, 2, 2] | [1, 2, 1]
ice_block | [2, 2, 2] | [0, 0, 0] | [0, 0, 0]
clothes_peg | [2, 2, 2] | [1, 0, 1] | [0, 1, 0]
hairbrush_handle | [2, 2, 2] | [0, 0, 0] | [0, 0, 0]
microfiber_cloth | [1, 1, 1] | [1, 1, 1] | [1, 1, 0]
toilet_brush_bristles | [0, 0, 1] | [2, 2, 2] | [1, 2, 1]
hairbrush_bristles | [2, 2, 2] | [2, 2, 2] | [2, 2, 1]

##  Appendix B Object Details

TABLE XVI: PhysiCLeAR Training Set Objects. We list all objects in our training set and their sample names. Object | Sample Name | Split
---|---|---
a baseball | baseball | train
a basket | basket | train
a blanket | blanket | train
a bread knife’s blade | bread_knife_blade | train
a bread knife’s handle | bread_knife_handle | train
a bunch of rubber bands | rubber_bands | train
a bunch of wires | wires | train
a cotton ball | cotton_ball | train
a dishwashing cloth | rug | train
a feather duster’s head | feather_duster | train
a filled disposable water bottle | water_bottle | train
a fine mesh strainer’s handle | strainer_handle | train
a fine mesh strainer’s scoop | strainer_base | train
a game controller’s body | controller_base | train
a game controller’s buttons | controller_buttons | train
a game controller’s keypad | controller_keypad | train
a game controller’s shoulder buttons | controller_shoulderpads | train
a game controller’s thumbstick | controller_stick | train
a gauze pad | gauze_pad | train
a lanyard | lanyard_string | train
a lanyard’s card holder | lanyard_card | train
a leather book cover | leather_book | train
a lemon | lemon | train
a mandarin orange | mandarin_orange | train
a mop’s head | mop_head | train
a pair of rubber slippers | rubber_slippers | train
a pair of scissors’ blade | scissor_blade | train
a pair of scissors’ handle | scissor_handle | train
a sheet of paper towel | paper_towel | train
a ripe avocado | avocado_kinda_ripe | train
a ripe kiwi | kiwi_kinda_ripe | train
a pen’s barrel | pen_base | train
a pen’s grip | pen_pad | train
a pillow | pillow | train
a potato | potato | train
a rice spatula’s handle | rice_spatula_handle | train
a rice spatula’s scoop | rice_spatula_base | train
a roll of masking tape | masking_tape | train
a roll of toilet paper | toilet_paper | train
a ruler | wood_ruler | train
a scoop of rice | rice | train
a sheet of aluminium foil | aluminium_foil | train
a stress ball | stress_ball | train
a tennis ball | tennis_ball | train
a toilet brush’s handle | toilet_brush_handle | train
a tomato | tomato | train
a toothbrush’s handle | toothbrush_handle | train
a TV remote’s back | tv_remote_back | train
a TV remote’s buttons | tv_remote_buttons | train
an eraser | eraser | train
an insulating holder | insulating_holder | train
an orange | orange | train
an oven mitt | oven_mitt | train
an uncracked egg | egg | train
an unripe avocado | avocado_unripe | train
crumpled tissue paper | tissue_ball | train
the ends of a hairbrush’s bristles | hairbrush_bristles | train
the rough side of a sponge | sponge_rough | train
the sides of a hairbrush’s bristles | hairbrush_bristles_side | train
the sides of a toothbrush’s bristles | toothbrush_bristles | train
the soft side of a sponge | sponge_soft | train
TABLE XVII: PhysiCLeAR Validation and Test Set Objects. We list all objects in our validation and test sets and their sample names. Object | Sample Name | Split
---|---|---
a ball of steel wool | steel_wool | val
a feather duster’s handle | feather_duster_handle | val
a nylon shirt | nylon_shirt | val
a pair of jeans | denim | val
a scoop of millet | millet | val
a sheet of bubble wrap | bubble_wrap | val
numbers on a TSA lock | tsa_lock_numbers | val
a bath towel | bath_towel | test
a block of ice | ice_block | test
a clothes peg | clothes_peg | test
a hairbrush’s handle | hairbrush_handle | test
a microfiber cloth | microfiber_cloth | test
a toilet brush’s bristles | toilet_brush_bristles | test
the ends of a hairbrush’s bristles | hairbrush_bristles | test

In Tables XVI and XVII, we provide the list of objects in PhysiCLeAR, their
associated video name and the set (i.e. train, val or test) they belong to. We
use the object names in the left column during the Property-object Matching
task for more semantic specificity.

TABLE XVIII: PhysiCLeAR Object Parts. We list the objects in PhysiCLeAR that have distinct semantic components, the names of their semantic components and the components’ associated sample names. Object | Part | Part Sample Name
---|---|---
Bread knife | Blade | bread_knife_blade
| Handle | bread_knife_handle
Feather duster | Handle | feather_duster_handle
| Head | feather_duster
Fine mesh strainer | Handle | strainer_handle
| Scoop | strainer_base
Game controller | Body | controller_base
| Buttons | controller_buttons
| Keypad | controller_keypad
| Shoulder buttons | controller_shoulderpads
| Thumbstick | controller_stick
Hairbrush | Ends of bristles | hairbrush_bristles
| Handle | hairbrush_handle
| Sides of bristles | hairbrush_bristles_side
Lanyard | Card holder | lanyard_card
| String | lanyard_string
Pair of scissors | Blade | scissor_blade
| Handle | scissor_handle
Pen | Barrel | pen_base
| Grip | pen_pad
Rice spatula | Handle | rice_spatula_handle
| Scoop | rice_spatula_base
Sponge | Rough side | sponge_rough
| Soft side | sponge_soft
Toilet brush | Bristles | toilet_brush_bristles
| Handle | toilet_brush_handle
Toothbrush | Handle | toothbrush_handle
| Sides of bristles | toothbrush_bristles
TV remote | Back | tv_remote_back
| Buttons | tv_remote_buttons

In Table XVIII, we list objects that have semantically meaningful parts for
part-based physical reasoning tasks. Specifically, we note that their semantic
parts can be used for more precise Property-object Matching.

TABLE XIX: PhysiCLeAR Object Surface Materials. We list all materials in PhysiCLeAR and their associated objects. Surface Material | Object
---|---
fabric | bath_towel
| blanket
| cotton_ball
| denim
| feather_duster
| gauze_pad
| hairbrush_bristles
| hairbrush_bristles_side
| lanyard_string
| microfiber_cloth
| mop_head
| nylon_shirt
| oven_mitt
| pillow
| rug
| tennis_ball
| toothbrush_bristles
food | avocado_kinda_ripe
| avocado_unripe
| egg
| ice_block
| kiwi_kinda_ripe
| lemon
| mandarin_orange
| millet
| orange
| potato
| rice
| tomato
leather | baseball
| leather_book
metal | aluminium_foil
| bread_knife_blade
| scissor_blade
| steel_wool
| strainer_handle
| strainer_base
| tsa_lock_numbers
paper | masking_tape
| paper_towel
| tissue_ball
| toilet_paper
plastic | bubble_wrap
| clothes_peg
| controller_base
| controller_buttons
| controller_keypad
| controller_shoulderpads
| controller_stick
| feather_duster_handle
| hairbrush_handle
| lanyard_card
| pen_base
| rice_spatula_base
| rice_spatula_handle
| scissor_handle
| sponge_rough
| sponge_soft
| toilet_brush_bristles
| toilet_brush_handle
| toothbrush_handle
| tv_remote_back
| water_bottle
| wires
rubber | eraser
| pen_pad
| rubber_bands
| rubber_slippers
| stress_ball
silicone | insulating_holder
| tv_remote_buttons
wood | basket
| bread_knife_handle
| wood_ruler

We list the surface materials of PhysiCLeAR objects in Table XIX. PhysiCLeAR
contains nine different common materials: fabric, food, leather, metal, paper,
plastic, rubber, silicone and wood. While we do not use these materials for
any task currently, we catalog them for future extensions (e.g. material
classification using properties).

##  Appendix C Property Statistics

TABLE XX: PhysiCLeAR Property Category Count. We list all property categories and the object count for each category. Property Category | Object Count | Percentage (%)
---|---|---
Soft | 26 | 35.14
Moderately hard | 14 | 18.92
Hard | 34 | 45.95
Smooth | 36 | 48.65
Slightly rough | 16 | 21.62
Rough | 22 | 29.73
No bumps | 26 | 35.14
Small bumps | 34 | 45.95
Big bumps | 14 | 18.92

We show the object count and percentage of each property category in Table XX.
There is at least 18.92% of each category for each property such that the
dataset is sufficiently balanced.

TABLE XXI: PhysiCLeAR Property Combination Count. We list all property combinations and the count for each combination. Property Combination | Count
---|---
Soft, smooth, no bumps | 5
Soft, slightly rough, no bumps | 0
Soft, Rough, no bumps | 0
Soft, smooth, small bumps | 2
Soft, slightly rough, small bumps | 7
Soft, Rough, small bumps | 8
Soft, smooth, big bumps | 1
Soft, slightly rough, big bumps | 0
Soft, Rough, big bumps | 3
Moderately hard, smooth, no bumps | 2
Moderately hard, slightly rough, no bumps | 0
Moderately hard, rough, no bumps | 0
Moderately hard, smooth, small bumps | 2
Moderately hard, slightly rough, small bumps | 4
Moderately hard, rough, small bumps | 3
Moderately hard, smooth, big bumps | 2
Moderately hard, slightly rough, big bumps | 0
Moderately hard, rough, big bumps | 1
Hard, smooth, no bumps | 17
Hard, slightly rough, no bumps | 1
Hard, rough, no bumps | 1
Hard, smooth, small bumps | 1
Hard, slightly rough, small bumps | 4
Hard, rough, small bumps | 3
Hard, smooth, big bumps | 4
Hard, slightly rough, big bumps | 0
Hard, rough, big bumps | 3

Table XXI shows the count for each combination of the three properties:
hardness, roughness and bumpiness. We note that some combinations are commonly
found in everyday objects (e.g. the combination of [Hard, smooth, no bumps]
for man-made objects) while some combinations are very rare. Furthermore, the
curse of dimensionality means that it is harder to represent all combinations
well as the number of properties increases. Our future work will include
representing the combinations in a more balanced way and incorporating better
representation learning methods to handle this imbalance.

##  Appendix D Sample Video Statistics

TABLE XXII: Sample Video Length Statistics. The average tactile sample video has 112.30 frames, a minimum of 50 frames and a maximum of 126 frames. Average | Min | Max
---|---|---
112.30 | 50 | 126

There are 5 to 7 tactile videos for each object in PhysiCLeAR. Table XXII
shows the average, minimum and maximum number of frames in our tactile videos
at 112.30, 50 and 126 respectively.

###  D-A Language Prompt Details

We list the questions for each task in this section.

⬇

object_property_description = [

"Describe the physical properties of <tact_start> <img_tokens <tact_end>.",

"How does this tactile video <tact_start> <img_tokens> <tact_end> feel?"

]

property_comparison = [

"Is the object in the tactile video <tact_start> <img_tokens> <tact_end>
<more_property> than the one in <tact_start> <img_tokens> <tact_end>? Describe
both objects before answering.",

"Is the object in <tact_start> <img_tokens> <tact_end> <more_property> than
the object in <tact_start> <img_tokens> <tact_end>? Describe both objects
before answering.",

"Is the object in the tactile video <tact_start> <img_tokens><tact_end>
<less_property> than the one in <tact_start> <img_tokens> <tact_end>? Describe
both objects before answering.",

"Is the object in <tact_start> <img_tokens> <tact_end> <less_property> than
the object in <tact_start> <img_tokens> <tact_end>? Describe both objects
before answering."

]

property_superlative_selection = [

"Given three tactile videos: a) <tact_start> <img_tokens> <tact_end>, b)
<tact_start> <img_tokens> <tact_end>, c) <tact_start> <img_tokens> <tact_end>.
Describe each object and then select the <most_property>.",

"You have tactile videos of one object each: a) <tact_start> <img_tokens>
<tact_end>, b) <tact_start> <img_tokens> <tact_end>, c) <tact_start>
<img_tokens> <tact_end>. Describe each object and then select the
<most_property> object.",

"Given these tactile videos: a) <tact_start> <img_tokens> <tact_end>, b)
<tact_start> <img_tokens> <tact_end>, c) <tact_start> <img_tokens> <tact_end>.
Describe each object and then select the <least_property>.",

"You have tactile videos of one object each: a) <tact_start> <img_tokens>
<tact_end>, b) <tact_start> <img_tokens> <tact_end>, c) <tact_start>
<img_tokens> <tact_end>. Describe each object and then select the
<least_property> object."

]

property_object_match = [

"Given three tactile videos: a) <tact_start> <img_tokens> <tact_end>, b)
<tact_start> <img_tokens> <tact_end>, c) <tact_start <img_tokens <tact_end>.
Describe the object in each video, then match each video to one of the
following objects in alphabetical order: 1) <object>, 2) <object>, 3)
<object>.",

"You have tactile videos of one object each: a) <tact_start> <img_tokens>
<tact_end>, b) <tact_start> <img_tokens> <tact_end>, c) <tact_start>
<img_tokens> <tact_end>. Describe the object in each video, then match each
video to one of the following objects in alphabetical order: 1) <object>, 2)
<object>, 3) <object>."

]

property_scenario_reasoning = [

"Given these descriptions of three objects: a) <description> b) <description>
c) <description>. <scenario_question>. Select the most appropriate object in
the format ’The most suitable object is x).’"

]

##  Appendix E PG-InstructBLIP Avocado Property Prediction Details

Figure 6: Cropped Avocado Image for Vision-only Property Prediction. The
cropping of the avocado images allows for an evaluation of PG-InstructBLIP
that is closer to the original dataset it was trained on, since the target
object takes up a bigger portion of an image.

For the avocado experiments with PG-InstructBLIP, We take 20 RGB images of
each avocado from a top-down view on a flat gray surface. We vary the position
and orientation of each avocado for each image and use cropped images (example
shown in Fig. 6); the cropping of the avocado images makes it closer to the
dataset PG-InstructBLIP was trained on.

We use the original prompt provided in PG-InstructBLIP’s example script and
prompt the model for each of our physical properties individually. The
specific prompts are:

  * •

Hardness - “Classify the object as hard, soft or medium? Respond unknown if
you are not sure. Short answer:”

  * •

Roughness - “Classify the object as rough, smooth or medium? Respond unknown
if you are not sure. Short answer:”

  * •

Bumpiness - “Classify the object as having big bumps, small bumps or no bumps?
Respond unknown if you are not sure. Short answer:”

PG-InstructBLIP is not trained on our three physical properties and we find
that it never chooses “moderately hard” and “slightly rough”. Hence, we change
those categories to “medium” for hardness and roughness.