# Physical Reasoning and Object Planning for Household Embodied Agents

Ayush Agrawal<EMAIL_ADDRESS>
National University of Singapore
Raghav Prabhakar<EMAIL_ADDRESS>
International Institute of Information Technology - Hyderabad
Anirudh Goyal<EMAIL_ADDRESS>
Deepmind, London
Dianbo Liu<EMAIL_ADDRESS>
National University of Singapore

###### Abstract

In this study, we explore the sophisticated domain of task planning for robust
household embodied agents, with a particular emphasis on the intricate task of
selecting substitute objects. We introduce the CommonSense Object Affordance
Task (COAT), a novel framework designed to analyze reasoning capabilities in
commonsense scenarios. This approach is centered on understanding how these
agents can effectively identify and utilize alternative objects when executing
household tasks, thereby offering insights into the complexities of practical
decision-making in real-world environments.Drawing inspiration from human
decision-making, we explore how large language models tackle this challenge
through three meticulously crafted commonsense question-and-answer datasets,
featuring refined rules and human annotations. Our evaluation of state-of-the-
art language models on these datasets sheds light on three pivotal
considerations: 1) aligning an object’s inherent utility with the task at
hand, 2) navigating contextual dependencies (societal norms, safety,
appropriateness, and efficiency), and 3) accounting for the current physical
state of the object. To maintain accessibility, we introduce five abstract
variables reflecting an object’s physical condition, modulated by human
insights to simulate diverse household scenarios. Our contributions include
insightful Object-Utility mappings addressing the first consideration and two
extensive QA datasets (15k and 130k questions) probing the intricacies of
contextual dependencies and object states. The datasets, along with our
findings, are accessible at: https://github.com/com-phy-affordance/COAT. This
research not only advances our understanding of physical commonsense reasoning
in language models but also paves the way for future improvements in household
agent intelligence.

##  1 Introduction

Humans, as beings innately attuned to their surroundings, traverse a world
where conversations, decisions, behaviors, and understanding are deeply
embedded in the underlying fabric of situation. Their engagement with the
world entails commonsense (background) knowledge about entities–properties,
spatial relations, events, causes and effects, and other social norms
((McCarthy, 1959); (Winograd, 1972); (Davis & Marcus, 2015)). The importance
of situational awareness is starkly evident in our daily tasks, where choosing
objects for specific activities showcases our adaptability to different
settings. Consider the straightforward task of cutting a cake—how do we
determine which object is suitable for this task? When a person needs to
select an object to accomplish this task, there can be an array of factors
that might affect our choice. For example: we must choose something that is
capable of cutting (Utility), suitable for cutting a cake (contextual
appropriateness), and likely in an appropriate physical condition to be used
(physical state). These considerations would be to ensure the appropriateness,
ease, and safety of those cutting the cake as well as who will eat the cake.
These considerations although might seem trivial and intuitive to us humans,
are still an important aspect to consider when developing embodied household
agents. Such reasoning capabilities can be potentially leveraged by embodied
agents to generate action plans for human requirements represented in natural
language. In this work, we propose a CommonSense Object Affordance Task: a
textual physical commonsense task to evaluate most appropriate object
selection capabilities in the presence of various alternative objects.

Figure 1: We divide the whole decision-making process into 3 phases. Pruning
out options firstly based on Utility then Contextual Appropriateness and
finally on Physical State. This shows human adeptness in comparing
appropriateness across an array of factors and coming up with a substitute
object even in the absence of the ideal object [Cake Knife]. Our work provides
QA datasets about this type of commonsense reasoning

Recent advancements in large language models (LLMs) (Zhu et al., 2023; Peng et
al., 2023; Zhang et al., 2023; Brown et al., 2020; Chowdhery et al., 2022;
Touvron et al., 2023; OpenAI, 2023) have significantly enhanced our ability to
extract rich commonsense knowledge from extensive web data. To analyze this
task and evaluate the current capabilities of Language Models across such
human commonsense-oriented reasoning; we develop this task as a decision-
making process spanning across 3 major aspects: Utility: The concept of
Utility, a focal point in previous research Speer et al. (2017), elucidates
our understanding of an object’s functionality in a variety of situations.
Although ConceptNet (Speer et al., 2017) has been a crucial tool for
identifying object-utility relationships, its nature as a human-compiled
knowledge graph has led to the pursuit of more dynamic sources. We curate
Object-Utility mappings pertaining to this aspect and use this to further
evaluate decision-making across both remaining aspects. Context: Our decision-
making extends beyond mere utility. To account for the various situational
factors such as safety, social norms adherence, effort optimization,
efficiency, and situational appropriateness; we introduce the second aspect
as: contextual appropriateness This adeptness in judgment arises from our
ingrained commonsense, sculpted by experience and intuitive physical
understanding. To evaluate the reasoning capabilities of various language
models across this aspect, we generate Object-Concept-Task mappings and curate
a QA dataset consisting of 15K MCQ question-answer pairs. Physical State:
Previous work Li et al. (2023) has shown how object choice depends on various
physical variables. However, we humans don’t think in terms of these
variables. In order to make this aspect more human commonsense-oriented, we
add a layer of abstraction and introduce 5 abstract variables to depict the
current physical state of the object of interest. To observe how the object
usability evolves with various abstract physical state variations, we generate
human preference mappings and curate 2 QA datasets consisting of 130K MCQ
question-answer pairs.

To illustrate an example of using these 3 aspects to select the best feasible
object, let’s continue with the task of cutting a cake where the following
objects are available to use: a Broken Knife, Clean Scissors, a Clean Pillow,
and a Clean Knife. As we begin, we subconsciously evaluate each object’s
utility for the task, and thus shift our focus primarily on Knife and
Scissors. Further analysis typically leads to the dismissal of Scissors as a
suitable tool for cake cutting. Finally, upon considering the physical state
of Knives, the Clean Knife emerges as the obvious choice. This scenario
highlights the three key factors humans prioritize when selecting an object
for a task: the utility of the object, its contextual appropriateness, and its
current physical state.

Such commonsense reasoning capabilities not only allow us to judge the
appropriateness of an object in the context of the given task but also help us
in successfully coming up with an appropriate substitute object in the absence
of the most ideal object(Here: Cake Knife). Such skills if equipped with
embodied agents, will enhance their reasoning capabilities and make them adept
in planning tasks in scenarios where the ideal object is not available. .

Main Contributions: In this study, we made the following contributions:

  * •

Creation and provision of human-preference mappings across all 3 aspects of
the CommonSense Object Affordance Task(COAT)

  * •

Introduction of three major novel CommonSense-based QA Datasets, facilitating
an in-depth analysis of how object usability evolves under different
contextual scenarios and physical states

  * •

Evaluation of Large Language Model baselines on these datasets, accompanied by
a detailed analysis of their performance in multi-step abstract reasoning
scenarios.

##  2 Dataset Creation

To systematically investigate the capacity of LLM to conduct human-style
physical commonsense reasoning and preferences across three crucial factors,
we have devised an experimental framework centered around 75 household tasks,
carefully curated to span 22 distinct utilities. The experiment involves a
diverse inventory of 100 objects sourced from the AI2Thor Simulator (Speer et
al., 2017), ensuring relevance and diversity within a household context.

  1. 1.

Tasks: are high-level household activities that could be accomplished by a
human or embodied agent. Example: Cutting a Cake. See Task List

  2. 2.

Utilities: are extremely low-level utilities that could be assigned to a High-
level task. Like, for the example of Cutting Cake, the utility that gets
activated is Cutting. See Table 2(a)

  3. 3.

Objects: are a subset of objects available in AI2Thor (Kolve et al., 2022)
Simulator. See Table 2(b)

Utilities
---
Carrying | Comfort | Heating(vessel)
Cleaning | Washing | Mixing(tool)
Disposing | Cutting | Mixing (vessel)
Storage | Entertainment | Heating(source)
Reading | Breaking | Increasing Height
Eating | Writing | Physical Activity
Decoration | Light Source | Surface Support
(a) A representational subset of utilized Utilities

Objects
---
Bowl | Bed | Pan
DishSponge | SinkBasin | Spoon
GarbageCan | Knife | Cup
Fridge | Laptop | Microwave
Newspaper | BaseballBat | Chair
Apple | Pen | Dumbell
Statue | Floor Lamp | CounterTop
(b) A representational subset of utilized Objects

The following section gives an overview of the Annotation Tasks and the
process of creation of CommonSense Reasoning Datasets.

###  2.1 Human Preference Collection

####  2.1.1 Utility

Incorporating GPT3.5-turbo (Brown et al., 2020) along with human commonsense
annotations, we meticulously established a mapping between utilities and
objects. These are called Utility Objects. Notably, each object may be
associated with multiple utilities, and conversely, a single utility can be
linked to various objects. Table LABEL:tab:concepts-objects provides an
overview of the utilities along with their associated objects utilized in our
experiments. More Information about the annotation process can be found in
Appendix LABEL:appendix:_annotation-process

####  2.1.2 Contextual Appropriateness

In evaluating object utility, it is crucial to recognize that suitability for
specific tasks can vary significantly. Take, for example, the multifaceted use
of a candle. While it possesses the inherent ability to generate heat,
employing a candle for the purpose of heating soup introduces a range of
practical limitations. This observation underscores the complexity of human
preference and decision-making in the context of object utility. Key factors
influencing these choices include efficiency (as illustrated by the
impracticality of using a candle for heating soup), safety considerations
(such as the risks associated with standing on an armchair), social norms and
constructs (exemplified by the unconventional choice of serving wine in a
bowl), and the overall appropriateness of an action (e.g., the disposal of
eggshells in a sink basin). To systematically explore these dynamics, we
engaged human annotators in a study designed to assess the selection of
appropriate objects for specified tasks and utilities

####  2.1.3 Physical State

The selection of objects for specific tasks is influenced not only by
intangible factors such as safety and social constructs but also by the
object’s current physical state. Prior research, including the works of Li et
al. (2023) and Gao et al. (2023), has employed various physical parameters to
examine Language Learning Models’ (LLMs) comprehension of an object’s physical
attributes. In our study, we shift the focus to task planning under non-ideal
conditions, necessitating reasoning about potential substitute objects. To
this end, we have developed five distinct variables, each represented by
abstract symbolic terms. These variables have been derived directly from the
AI2Thor Simulator, facilitating their broader applicability and potential
integration into the burgeoning field of Embodied AI. Table 1 delineates these
variables and their corresponding abstract values. Here, Already In Use
variable is used to represent the availability of an object for use. Some
examples of an object in reversible-using state are: the object getting
recharged, a wet object or an object in a reversible state (meaning it will
need time to get back to the ideal state or is temporarily being used by
someone else). Whereas in an irreversible-using state, the object could be
broken, depleted, out of stock, and thus is in an irreversible state of use.
Further details about the chosen physical variables are elaborated in Appendix
A.1

Variables | Abstract Values
---|---
material | Metal, Wood, Plastic, Glass, Ceramic, Stone, Wax, Fabric, Rubber, Food, Paper Sponge, Organic, Soap
mass | Light, Medium, Heavy, Super-Heavy
temperature | Cold, RoomTemp, Hot
already in use | Free, Reversible-Using , Irreversible-Using
condition | Dirty, Clean, Broken
Table 1: Abstract Values for Various Variables

Gathering Common Object Configuration In the context of this study, a
Configuration denotes the physical state of an object characterized by five
variables. While a wax chair might be conceivable in the realm of Madame
Tussauds, it remains highly improbable in everyday household scenarios. Thus
to ensure the relevance of configurations to common household scenes, human
annotators were tasked with selecting plausible and frequently occurring
variable values for each object. (See Appendix LABEL:appendix:_annotation-
process)

Ranking Object Configurations In our study, we not only provided
configurations that occur commonly but also tasked the annotators with
categorizing the configurations of an object into three distinct classes:
Ideal, Moderate, and Bad. This classification was predicated on their
assessment of the anticipated time required for an agent to commence the task
with a given object configuration. Utilizing these categorizations, we
constructed two comprehensive datasets comprising 130,000 questions
specifically designed to assess the physical commonsense reasoning
capabilities of Large Language Models. Further details on this process are
elaborated in Appendix LABEL:appendix:_annotation-process

###  2.2 CommonSense QnA Datasets

Based on Contextual Appropriateness and Physical State. We created 3
CommonSense QA datasets.

  1. 1.

Task-0111These are different from the tasks(activities) used to curate
datasets: This experiment was based on pruning Objects based on contextual
factors affecting the appropriateness of an object for a particular task. We
utilized Object Level Dataset for its evaluation.

  2. 2.

Task-1 & Task-2: These experiments were based on pruning out Object
Configurations (Physical State) represented by 5 symbolic variables. We
utilized Variable Level Datasets for its evaluation.

Figure 3: A mapping created between Tasks and Concepts was utilized to sample
out <Task,Utility> combination to frame a question for all 3 datasets (1
Object Level, 2 Variable Level)

####  2.2.1 Object Level Dataset

To evaluate the reasoning capabilities of LLM across choosing objects over
contextual factors we curate an Object Level QA dataset. Here, previously
recorded Context Mappings were kept as Ground Truth. (See Annotation Task
2.1.2)

##### Question

Every question can be assigned a <Task, Utility> combination and was framed in
the way shown below:

Question What object would you be choosing for $<$utility$>$ when you are
tasked to $<$task$>$?

##### Options

Based on the sampling strategy and the number of options provided in the
prompt, we created 4 variations of object level dataset. An example of such
variation is shown below.

  1. 1.

Variation-1 : For each question, we randomly sampled 1 context object and 1
utility object both belonging to the same utility.222Details about other
variations for Task 0,1,2 can be found here Appendix B

Example Task 0 Variation 1 Question ID: 1,
Utility: heating(source),
Question: Which of the following objects would be best suited for the purpose
of "heating(source)" when tasked to "reheating coffee"?
Options: (A) Toaster (B) StoveBurner
Correct Answer: B

####  2.2.2 Physical Configuration Level Dataset

Based on Common Configurations generated in the annotation task [1], we create
2 Variable Level QA datasets to analyze the reasoning capabilities of Language
Models on pruning out options based on their current physical state. The 2
datasets differ in the level of difficulty and the level of reasoning required
to answer the questions correctly. We describe the creation process in this
section. The question in both datasets remains the same as that of Object
Level Dataset However, unlike the first dataset where the options were
objects, this time we give various Configurations of Context Objects as
options.

Configuration object name: Microwave, mass: super-heavy, temperature:
RoomTemp, material: Metal already in use: free condition: clean

Here, a distinct approach is employed compared to the Object level Dataset,
where Context Objects were previously sampled randomly based on a combination
of the question’s Task and Utility. In this study, we have classified the
configurations of Context Objects into three broad categories: "Ideal,"
"Moderate," and "Bad." Each category is defined by specific annotation
variables that delineate their characteristics. The "Ideal" category
represents configurations in their optimal states, facilitating the specified
task without the need for additional adjustments. In contrast, the "Moderate"
category includes configurations that deviate from these ideal states,
resulting in both time and material costs for their utilization. The models
assess these options based on their estimated penalties. Lastly, the "Bad"
category comprises configurations that render the Context Objects ineffective,
even when considering potential penalties. Both "Moderate" and "Bad"
configurations are grouped under Sub-Optimal Configurations, offering a
nuanced understanding of the varying degrees of object usability.

By sampling options from these 3 sets of configurations [1], we divide our
effort into 2 datasets:

##### A. Ideal Configuration Dataset

In alignment with its name, the "Ideal Configuration" dataset involves
questions with the correct answer as Ideal Configuration of Context Object of
the question’s associated

<Task,Utility> combination. To systematically analyze the behavior of models,
we introduce 12 distinct variations of this dataset. The creation of these
variations is designed to progressively augment the complexity of the
datasets, facilitating a comprehensive analysis of model behaviors. Each of
the 12 variations comprises approximately 5,000 question-answer pairs, with
differing counts of options—ranging from 5 options to 2 options per question.
Along with the varying number of options, we also ablated on various sampling
techniques. This deliberate variation in the number of options aims to
evaluate the impact on success rates of Large Language Models (LLMs) as the
level of reasoning complexity increases. Whereas, the different sampling
techniques help us study their behavior concerning different object
distribution.

Process: To create these 12 variation datasets, we sampled a Task for $n$
number of times, where $n$ is proportional to the total count of all Commonly
Occurring Configurations of its Utility Objects. [Annotation Task 2.1.1] For a
given Question’s <Task, Utility> Combination, we randomly sample a Context
Object from the pool of Context objects. (obtained from 2.1.2). An example of
sampling the remaining options is explained below:

For 5 option datasets:

  1. 1.

Variation-1 : randomly selected Context Object’s Ideal Configuration + 4
randomly sampled sub-optimal configurations of the same Context Object

  2. 2.

Variation-2 : randomly selected Context Object’s Ideal Configuration + 2
randomly sampled sub-optimal configurations of the same Context Object + 2
randomly sampled sub-optimal configurations of different Context Object
belonging to the same <Task,Utility> combination 333The remaining variations
in sampling techniques and option count can be found in Appendix B

Example for Task 1 Variation 1 Question ID: 1,
Utility: heating(source),
Question: Which of the following objects would be best suited for the purpose
of "heating(source)" when tasked to "reheating coffee"?
Options: (A) object name: Microwave, mass: super-heavy, temperature: RoomTemp,
material: Metal, already in use: free, condition: clean (B) object name:
StoveBurner, mass: super-heavy, temperature: RoomTemp, material: Metal already
in use: irreversible-using, condition: dirty (C) object name: CoffeeMachine,
mass: heavy, temperature: RoomTemp, material: Metal already in use:
irreversible-using, condition: dirty (D) object name: Microwave, mass: super-
heavy, temperature: RoomTemp, material: Metal already in use: reversible-
using, condition: clean (E) object name: Microwave, mass: super-heavy,
temperature: RoomTemp, material: Metal, already in use: irreversible-using,
condition: broken Correct Answer: A

##### B. Sub-Optimal Configuration Dataset

The process of selecting an ideal configuration, while challenging for
language models, typically does not require intricate multi-step reasoning or
the consideration of a wide range of factors. To more rigorously evaluate
their reasoning abilities, particularly when faced with only sub-optimal
options, we have intentionally excluded all ideal configurations from our
sampling methodology. This deliberate exclusion necessitates that the models
engage in more sophisticated reasoning, considering various physical state
variables, thereby highlighting their capacity for abstract reasoning. By
focusing exclusively on sub-optimal configurations, this methodological shift
enables a more thorough investigation into the language models’ ability to
navigate and reason through complex scenarios in the absence of clear-cut
ideal solutions.

Process: To comprehensively assess language models’ abstract reasoning
capabilities when confronted with sub-optimal configurations, we create
another Variable Level QA dataset and introduce 14 variations of this dataset.
Like the previous dataset, each dataset is constructed using distinct sampling
strategies and by varying number of options. Across all 14 datasets, we
maintain a consistent structure of nearly 5,000 questions.

Each question in this dataset variation is associated with a Task and Utility
combination. While the set of questions remains consistent with previous
datasets, the sampling of each task is now proportional to the count of
"Moderate Configurations + Bad Configurations" (i.e the count of Sub-Optimal
Configurations for that question’s associated <Task, Utility> combination). An
example of 2 sampling techniques used for generating variation datasets is
explained below:

For 5 option dataset

  1. 1.

Variation-1: We sample all 5 options from Moderate Configurations of the
Context object of the question’s associated <Task,Utility> combination.

  2. 2.

Variation-2: We sample 4 options from Moderate Configurations and 1 option
from the Bad Configurations of the Context object of the question’s associated
<Task,Utility> combination. 444The remaining variations in sampling techniques
and option count can be found in Appendix B

Example Task 2 Variation 1 Question ID: 1,
Utility: heating(source),
Question: Which of the following objects would be best suited for the purpose
of "heating(source)" when tasked to "reheating coffee"?
Options: (A) object name: CoffeeMachine, mass: heavy, temperature: RoomTemp,
material: Metal, already in use: reversible-using, condition: dirty (B) object
name: StoveBurner, mass: heavy, temperature: RoomTemp, material: Metal,
already in use: reversible-using, condition: clean (C) object name:
StoveBurner, mass: heavy, temperature: RoomTemp, material: Metal, already in
use: free, condition: dirty (D) object name: CoffeeMachine, mass: heavy,
temperature: RoomTemp, material: Metal, already in use: reversible-using,
condition: clean (E) object name: Microwave, mass: heavy, temperature:
RoomTemp, material: Metal, already in use: reversible-using, condition: clean
Correct Answer: C

##  3 Experimental Setup & Results

Upon curation of Datasets, we evaluate various Large Language Models to
benchmark their performance on:

  1. 1.

R1: Object-level Commonsense Reasoning (performance on various contextual
factors including: social constructs, feasibility aspects, etc.)

  2. 2.

R2: Physical State level Commonsense Reasoning (performance on commonsense
understanding of various physical variables and how they affect decision
making)

We evaluate and compare the performances of various large Language Models on
the following metrics:

  1. 1.

Accuracy: The fraction of a number of questions answered correctly by the
Language Model.

  2. 2.

Bad Rate: The fraction of questions in which the chosen answer belonged to the
"Bad" configuration pool.

###  3.1 Dataset Summary

Task | #Variation | #Q | Av | Options | GT
---|---|---|---|---|---
0 | 4 | 15.5K | 3.8K | Utility Objects |  Context Objects [Õ]
1 | 12 | 58,7K | 4.9K |  Õ’s Configurations |  Õ’s Ideal Configurations
2 | 14 | 68.9K | 4.9K |  Õ’s SuboptConfigurations |  Õ’s BestSubopt Configurations
Table 2: Summary of Datasets Used for Experiments 44footnotetext: #Q=question
count; Av=average question count

###  3.2 Glossary

Term | Definition
---|---
objects | a set of 100 household objects
utilities | a set of 22 abstract utilities
tasks | a set of 75 household activities
question |  a <Task,Utility> combination indicating the specific task aspect to emphasize
variable | a symbolic variable used to explain an object’s physical state
configuration | complete description of an object using 5 symbolic variables
utility-mapping |  mapping between utility and object; facilitates Utility-Objects
context-mapping |  mapping between task and object; facilitates Context-Objects
ideal-configuration | A state where an object is ready to perform a task without requiring additional time or effort,
| marked by the simultaneous occurrence of all ideal variable values in its
description
moderate-configuration | A state requiring additional effort to reach an ideal condition for task performance,
| marked by the presence of moderate variable values.
bad-configuration | An inoperative state of an object with irreparable issues, marked by the presence of "Bad Variable"
| values.
sub-optimal configuration | group of "Moderate" and "Bad" configurations

###  3.3 Results

Task 0 Analysis: We observe from Table 3; that the performance of GPT3.5-Turbo
and PaLM outperform other models with a much smaller number of parameters.
This may be attributed to their size as well as the amount of internet data
they’ve been trained on. They both showcased similar performance suggesting
similar object-level reasoning capabilities suggesting an impressive object-
level commonsense Reasoning. Even though the performance of every model was
observed to be impressive; Mistral-7B outshone all other models of similar
size as well as both 13B models. Upon analyzing the trend of average accuracy
across various datasets for Task-0[Figure 4] we note an important trend
implying a drop in accuracy as we increase the number of options. This
suggests degradation in reasoning capabilities as the number of comparisons
increases. This trend was observed in Task 1 and Task 2 as well555See Figure
10(a), 10(b) for trends in Task:1 and Task:2 Average Accuracies for each
dataset type. Table 3 and Figure 4 provide a fair evaluation of the reasoning
capabilities of Language Models over Object Level Affordance Reasoning tasks.
[R1]

Model | Task-0 Variations
---|---
2-opt |  3-opt |  4-opt |  5-opt
|  v1 |  v2 |  v3 |  v4
PaLM |  90.0 |  74.2 |  68.3 |  65.0
GPT3.5-Turbo |  88.8 |  71.9 |  68.3 |  66.9
2-5 vicuna13B |  71.0 |  54.5 |  49.3 |  46.2
LLama2-13B |  76.5 |  58.2 |  50.9 |  46.9
2-5 Vicuna7b |  51.0 |  34.5 |  28.5 |  26.5
Mistral-7B |  76.2 |  57.9 |  50.2 |  47.2
2-5 ChatGLM-6B |  62.0 |  42.0 |  34.2 |  27.6
ChatGLM2-6B |  62.9 |  44.6 |  34.3 |  35.4
Table 3: Accuracies for Different Models when evaluated on Task-0 : Object
Level QA Dataset

Figure 4: Average Accuracy of Various models on Task 0 as we increase option
count

Task 1 Analysis: Table LABEL:tab:performance-t1 summarizes the performance
accuracy of different models on Task-1 Datasets where models were tasked to
reason based on Physical Configuration of Objects (using Ideal Configuration
Datasets). This task was aimed at judging if language models have an
understanding of the difference between Ideal Configuration and Sub-Optimal
Configurations. Here as well we witness the superior reasoning capabilities of
GPT3.5-Turbo and PaLM, with the latter outperforming the former on each
dataset by an average of 8.8%. Amongst the smaller models, we see Mistral7B
dominating all other 7B and 6B models with Vicuna7B and ChatGLM-6B performing
very close to random performances. For 13B models, LLama2-13B showcased its
superior reasoning capabilities and was on average 7.6% more accurate than
Vicuna13B. Here apart from the falling average accuracy with increasing
options, we also notice some interesting behaviors when we increased the
Object Diversity (i.e an increase in the number of sub-optimal configurations
of different context object (of same <Task, Utility> Combination) other than
the object who’s Ideal Configuration is already in the options as the correct
answer).

Our work opens up an avenue for improving the language model’s abstract multi-
step reasoning for estimating the physical affordance of everyday objects that
are used in household activities. Future efforts would be directed towards
integrating these datasets to train Embodied Language agents and proving their
competence of our 3-step architecture in successful task completion when
situations aren’t ideal. Judging the variable values in the real world could
be a tricky affair, thus although the current work focused on handcrafted
variables, calculating these variables and learning new latent variables from
multi-modal inputs for effective analysis and reasoning about an object’s
applicability seems a foreseeable domain to explore.

## Limitations

This work focuses on dealing with contextual connotations associated with an
object when deciding whether to use it as a substitute for task execution. We
further considered abstract physical variable level analysis to highlight the
evolution of usability with various physical abstractions. While determining
the values of these variables may appear straightforward in the Ai2Thor
Simulator, achieving the same in real-life scenarios requires a resilient
model. Even if we are able to calculate the variables, there is a limitation
to which an object’s state could be represented using abstract physical
variables. When comparing objects, sometimes we need to understand their exact
situation to make a decision about their usability. To develop robust embodied
agents capable of dealing with such explicit reasoning along with abstract
commonsense reasoning capabilities, further work needs to be directed along
with integrating multi-modal reasoning capabilities in addition to commonsense
reasoning. In addition to this, in this study, we assumed that all the objects
were allowed to be used by the agent. In some cases, it might be possible that
the human companion of the agent might have kept an object because of a
certain way and didn’t want it disturbed. Thus the agent might need to re-
calculate the object use preference in accordance with this newly imposed
human preference. Further works along this line would enable us to move an
inch closer toward Embodied agents capable of such constrained planning
capabilities in addition to multi-modal commonsense reasoning.

##  5 Related Works

Previous work has been done in the domains related to the scope of this paper.
In this section, we summarize some of them:

##### Probing Language Models

Understanding what LMs know after large-scale pre-training is an active
research area (Rogers et al., 2020). Various probing methods have been
developed (Tenney et al., 2019b); (Petroni et al., 2019), and investigations
show that LMs capture linguistic (Tenney et al., 2019a); (Liu et al., 2019),
factual (Petroni et al., 2019); (Roberts et al., 2020); (Dai et al., 2022),
commonsense knowledge (Wang et al., 2019); (Forbes et al., 2019), and even
acquire grounded concepts (Patel & Pavlick, 2021).

##### CommonSense QA Datasets

Evaluating to what level commonsense world understanding LMs possess has been
explored by many. (Gu et al., 2023) analyses mental models of LLMs and aligns
them with improved models about everyday things; (Bisk et al., 2019) consisted
of questions requiring physical commonsense reasoning. Recently, there has
been a lot of work in NLP to utilize commonsense for QA, NLI, etc. (Sap et
al., 2019); (Talmor et al., 2019). Many of these approaches seek to
effectively utilize ConceptNet by reducing the noise retrieved from it (Lin et
al., 2019) (Kapanipathi et al., 2020) There have been several other QA
Datasets to benchmark CommonSense Reasoning abilities in Language Models, Some
of them include: (Geva et al., 2021); (Yang et al., 2018); (Mihaylov et al.,
2018);

##### Reasoning in LLMs

Reasoning is a crucial aspect of intelligence, influencing decision-making,
problem-solving, and other cognitive abilities. (Huang & Chang, 2023) presents
the current state of research on LLMs’ reasoning abilities, exploring
approaches to improve and evaluate their reasoning skills. (Dziri et al.,
2023) investigates problems associated with multistep reasoning with LLMs Some
of the works dealing with tackling reasoning in small models are: (Magister et
al., 2023) (Fu et al., 2023) (Shridhar et al., 2023)

## References

  * Bisk et al. (2019) Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi.  Piqa: Reasoning about physical commonsense in natural language, 2019.
  * Brown et al. (2020) Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei.  Language models are few-shot learners, 2020.
  * Chiang et al. (2023) Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing.  Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023.  URL https://lmsys.org/blog/2023-03-30-vicuna/.
  * Chowdhery et al. (2022) Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel.  Palm: Scaling language modeling with pathways, 2022.
  * Dai et al. (2022) Yuqian Dai, Marc de Kamps, and Serge Sharoff.  BERTology for machine translation: What BERT knows about linguistic difficulties for translation.  In _Proceedings of the Thirteenth Language Resources and Evaluation Conference_ , pp. 6674–6690, Marseille, France, June 2022. European Language Resources Association.  URL https://aclanthology.org/2022.lrec-1.719.
  * Davis & Marcus (2015) Ernest Davis and Gary Marcus.  Commonsense reasoning and commonsense knowledge in artificial intelligence.  _Commun. ACM_ , 58(9):92–103, aug 2015.  ISSN 0001-0782.  doi: 10.1145/2701413.  URL https://doi.org/10.1145/2701413.
  * Du et al. (2022a) Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang.  Glm: General language model pretraining with autoregressive blank infilling.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pp. 320–335, 2022a.
  * Du et al. (2022b) Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang.  Glm: General language model pretraining with autoregressive blank infilling.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pp. 320–335, 2022b.
  * Dziri et al. (2023) Nouha Dziri, Ximing Lu, Melanie Sclar, Xiang Lorraine Li, Liwei Jiang, Bill Yuchen Lin, Peter West, Chandra Bhagavatula, Ronan Le Bras, Jena D. Hwang, Soumya Sanyal, Sean Welleck, Xiang Ren, Allyson Ettinger, Zaid Harchaoui, and Yejin Choi.  Faith and fate: Limits of transformers on compositionality, 2023.
  * Forbes et al. (2019) Maxwell Forbes, Ari Holtzman, and Yejin Choi.  Do neural language representations learn physical commonsense?, 2019.
  * Fu et al. (2023) Yao Fu, Hao Peng, Litu Ou, Ashish Sabharwal, and Tushar Khot.  Specializing smaller language models towards multi-step reasoning, 2023.
  * Gao et al. (2023) Jensen Gao, Bidipta Sarkar, Fei Xia, Ted Xiao, Jiajun Wu, Brian Ichter, Anirudha Majumdar, and Dorsa Sadigh.  Physically grounded vision-language models for robotic manipulation.  In _arXiv preprint arXiv:2309.02561_ , 2023.
  * Geva et al. (2021) Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant.  Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies, 2021.
  * Gu et al. (2023) Yuling Gu, Bhavana Dalvi Mishra, and Peter Clark.  Do language models have coherent mental models of everyday things?  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pp. 1892–1913, Toronto, Canada, July 2023. Association for Computational Linguistics.  doi: 10.18653/v1/2023.acl-long.106.  URL https://aclanthology.org/2023.acl-long.106.
  * Huang & Chang (2023) Jie Huang and Kevin Chen-Chuan Chang.  Towards reasoning in large language models: A survey, 2023.
  * Jiang et al. (2023) Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed.  Mistral 7b, 2023.
  * Kapanipathi et al. (2020) Pavan Kapanipathi, Veronika Thost, Siva Sankalp Patel, Spencer Whitehead, Ibrahim Abdelaziz, Avinash Balakrishnan, Maria Chang, Kshitij Fadnis, Chulaka Gunasekara, Bassem Makni, Nicholas Mattei, Kartik Talamadupula, and Achille Fokoue.  Infusing knowledge into the textual entailment task using graph convolutional networks.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 34(05):8074–8081, Apr. 2020.  doi: 10.1609/aaai.v34i05.6318.  URL https://ojs.aaai.org/index.php/AAAI/article/view/6318.
  * Kolve et al. (2022) Eric Kolve, Roozbeh Mottaghi, Winson Han, Eli VanderBilt, Luca Weihs, Alvaro Herrasti, Matt Deitke, Kiana Ehsani, Daniel Gordon, Yuke Zhu, Aniruddha Kembhavi, Abhinav Gupta, and Ali Farhadi.  Ai2-thor: An interactive 3d environment for visual ai, 2022.
  * Krippendorff (2011) Klaus Krippendorff.  Computing krippendorff’s alpha-reliability.  _Departmental Papers (ASC). University of Pennsylvania_ , 2011.  URL https://repository.upenn.edu/asc_papers/43.
  * Li et al. (2023) Lei Li, Jingjing Xu, Qingxiu Dong, Ce Zheng, Qi Liu, Lingpeng Kong, and Xu Sun.  Can language models understand physical concepts?, 2023.
  * Lin et al. (2019) Bill Yuchen Lin, Xinyue Chen, Jamin Chen, and Xiang Ren.  Kagnet: Knowledge-aware graph networks for commonsense reasoning, 2019.
  * Liu et al. (2019) Nelson F. Liu, Matt Gardner, Yonatan Belinkov, Matthew E. Peters, and Noah A. Smith.  Linguistic knowledge and transferability of contextual representations.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pp. 1073–1094, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.  doi: 10.18653/v1/N19-1112.  URL https://aclanthology.org/N19-1112.
  * Magister et al. (2023) Lucie Charlotte Magister, Jonathan Mallinson, Jakub Adamek, Eric Malmi, and Aliaksei Severyn.  Teaching small language models to reason, 2023.
  * McCarthy (1959) John McCarthy.  Programs with common sense.  In _Proceedings of the Teddington Conference on the Mechanization of Thought Processes_ , pp. 75–91, London, 1959. Her Majesty’s Stationary Office.  URL http://www-formal.stanford.edu/jmc/mcc59.html.
  * Mihaylov et al. (2018) Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal.  Can a suit of armor conduct electricity? a new dataset for open book question answering, 2018.
  * OpenAI (2023) OpenAI.  Gpt-4 technical report, 2023.
  * Patel & Pavlick (2021) Roma Patel and Ellie Pavlick.  Mapping language models to grounded conceptual spaces.  In _International Conference on Learning Representations_ , 2021.
  * Peng et al. (2023) Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao.  Instruction tuning with gpt-4, 2023.
  * Petroni et al. (2019) Fabio Petroni, Tim Rocktäschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander H. Miller, and Sebastian Riedel.  Language models as knowledge bases?, 2019.
  * Roberts et al. (2020) Adam Roberts, Colin Raffel, and Noam Shazeer.  How much knowledge can you pack into the parameters of a language model?  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pp. 5418–5426, Online, November 2020. Association for Computational Linguistics.  doi: 10.18653/v1/2020.emnlp-main.437.  URL https://aclanthology.org/2020.emnlp-main.437.
  * Rogers et al. (2020) Anna Rogers, Olga Kovaleva, and Anna Rumshisky.  A primer in BERTology: What we know about how BERT works.  _Transactions of the Association for Computational Linguistics_ , 8:842–866, 2020.  doi: 10.1162/tacl_a_00349.  URL https://aclanthology.org/2020.tacl-1.54.
  * Sap et al. (2019) Maarten Sap, Ronan Le Bras, Emily Allaway, Chandra Bhagavatula, Nicholas Lourie, Hannah Rashkin, Brendan Roof, Noah A. Smith, and Yejin Choi.  Atomic: An atlas of machine commonsense for if-then reasoning.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 33(01):3027–3035, Jul. 2019.  doi: 10.1609/aaai.v33i01.33013027.  URL https://ojs.aaai.org/index.php/AAAI/article/view/4160.
  * Shridhar et al. (2023) Kumar Shridhar, Alessandro Stolfo, and Mrinmaya Sachan.  Distilling reasoning capabilities into smaller language models, 2023.
  * Speer et al. (2017) Robyn Speer, Joshua Chin, and Catherine Havasi.  Conceptnet 5.5: An open multilingual graph of general knowledge.  In _Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence_ , AAAI’17, pp. 4444–4451. AAAI Press, 2017.
  * Talmor et al. (2019) Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant.  Commonsenseqa: A question answering challenge targeting commonsense knowledge, 2019.
  * Tenney et al. (2019a) Ian Tenney, Dipanjan Das, and Ellie Pavlick.  BERT rediscovers the classical NLP pipeline.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pp. 4593–4601, Florence, Italy, July 2019a. Association for Computational Linguistics.  doi: 10.18653/v1/P19-1452.  URL https://aclanthology.org/P19-1452.
  * Tenney et al. (2019b) Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R Thomas McCoy, Najoung Kim, Benjamin Van Durme, Samuel R. Bowman, Dipanjan Das, and Ellie Pavlick.  What do you learn from context? probing for sentence structure in contextualized word representations, 2019b.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom.  Llama 2: Open foundation and fine-tuned chat models, 2023.
  * Wang et al. (2019) Cunxiang Wang, Shuailong Liang, Yue Zhang, Xiaonan Li, and Tian Gao.  Does it make sense? and why? a pilot study for sense making and explanation.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pp. 4020–4026, Florence, Italy, July 2019. Association for Computational Linguistics.  doi: 10.18653/v1/P19-1393.  URL https://aclanthology.org/P19-1393.
  * Winograd (1972) Terry Winograd.  Understanding natural language.  _Cognitive Psychology_ , 3(1):1–191, 1972.  ISSN 0010-0285.  doi: https://doi.org/10.1016/0010-0285(72)90002-3.  URL https://www.sciencedirect.com/science/article/pii/0010028572900023.
  * Yang et al. (2018) Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning.  Hotpotqa: A dataset for diverse, explainable multi-hop question answering, 2018.
  * Zhang et al. (2023) Renrui Zhang, Jiaming Han, Chris Liu, Peng Gao, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, and Yu Qiao.  Llama-adapter: Efficient fine-tuning of language models with zero-init attention, 2023.
  * Zhu et al. (2023) Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny.  Minigpt-4: Enhancing vision-language understanding with advanced large language models, 2023.

##  Appendix A Appendix

###  A.1 Dataset Specifics

(a) Plot showing number of objects(x) for each utility(y), as obtained after
Utility based pruning 2.1.1

(b) Plot showing number of tasks(x) for each utility(y)

###  A.2 Variables

Here we describe the variables used to describe an object’s physical state. We
kept it at an abstract level to judge basic commonsense reasoning
capabilities.

  1. 1.

Mass: Based on an estimate of the weight of an object:  (i) light[0,1 Kg]
(ii) medium[1,5 Kg]  (iii) heavy[5,10 Kg]  (iv) super-heavy[> 10 Kg]

  2. 2.

Material: what material is used to make that object

  3. 3.

Temperature: the surface temperature of the object: Cold/Hot/RoomTemp

  4. 4.

already in use: tells us about the availability of the object: reversible-
using/irreversible-using/free

  5. 5.

condition: tells us about the condition of the object: broken/clean/dirty

###  A.3 Human Annotations: Object-Utility Mappings

Table 2.1.1 summarizes the collected and refined Object-Utility pairings.
Throughout this work, we have referred to these as Utility Mappings.

##  Appendix B Dataset Creation

In addition to the variations explained in 2.2.2, we further create 3 more
types of datasets for each of the 3 tasks. These would be each consisting of
4, 3, and 2 options. The method we choose for sampling these options enables
us to analyze the performance and reasoning capabilities of LLMs in zero-shot
commonsense reasoning. The datasets are:

###  B.1 Task 0

  1. 1.

Variation-2: For each question, we sampled 1 context object and 2 utility
objects belonging to the same utility.

  2. 2.

Variation-3: For each question, we sampled 1 context object and 3 utility
objects belonging to the same utility.

  3. 3.

Variation-4: For each question, we sampled 1 context object and 4 utility
objects belonging to the same utility.

###  B.2 Task 1

##### 5 option datasets

:

  1. 1.

Variation-3: Random context object’s Ideal Configuration + 4 randomly sampled
sub-optimal configurations of same Task and Utility’s different context object

##### 4 option datasets

:

  1. 1.

Variation-4 : Random context object’s Ideal Configuration + 3 randomly sampled
sub-optimal configurations of the same context object

  2. 2.

Variation-5: Random context Object’s Ideal Configuration + 2 randomly sampled
sub-optimal configurations of the same context object + 1 randomly sampled
sub-optimal configurations of different context object belonging to the same
<Task,Utility> combination

  3. 3.

Variation-6: Random context Object’s Ideal Configuration + 1 randomly sampled
sub-optimal configurations of the same context object + 2 randomly sampled
sub-optimal configurations of different context object belonging to the same
<Task,Utility> combination

  4. 4.

Variation-7: Random context object’s Ideal Configuration + 3 randomly sampled
sub-optimal configurations of the different context object belonging to the
same <Task,Utility> combination

##### 3 option datasets

:

  1. 1.

Variation-8 : Random context object’s Ideal Configuration + 2 randomly sampled
sub-optimal configurations of the same context object

  2. 2.

Variation-9: Random context object’s Ideal Configuration + 1 randomly sampled
sub-optimal configuration of the same context object + 1 randomly sampled sub-
optimal configuration of different context object belonging to the same
<Task,Utility> combination

  3. 3.

Variation-10: Random context object’s Ideal Configuration + 2 randomly sampled
sub-optimal configurations of the different context object belonging to the
same <Task,Utility> combination

##### 2 option dataset

  1. 1.

Variation-11: Random context object’s Ideal Configuration + 1 randomly sampled
sub-optimal configurations of the same context object.

  2. 2.

Variation-12: Random context object’s Ideal Configuration + 1 randomly sampled
sub-optimal configurations of the different context object belonging to the
same <Task,Utility> combination

###  B.3 Task 2

##### 5 option dataset

  1. 1.

Variation-3: We sample 3 options from the Moderate Configurations and 2
options from the Bad Configurations of the same <Task, Utility> combination’s
context objects

  2. 2.

Variation-4: We sample 2 options from the Moderate Configurations and 3
options from the Bad Configurations of the same <Task, Utility> combination’s
context objects

  3. 3.

Variation-5: We sample 1 option from the Moderate Configurations of the
context objects of that particular <Task, Utility> combination, we allow
sampling equivalent options as long as either of them is not the correct
answer. We also sample 4 options from the Bad Configurations of context
objects of that particular <Task, Utility> combination

##### 4 option dataset

  1. 1.

Variation-6: We sample 4 options from the Moderate Configurations of the
context objects of that particular <Task, Utility> combination, here we allow
sampling equivalent options as long as either of them is not the correct
answer.

  2. 2.

Variation-7: We sample 3 options from the Moderate Configurations of the
context object of that particular <Task,Utility> combination, we allow
sampling equivalent options as long as either of them is not the correct
answer. We also sample 1 option from the Bad Configurations of the random
context objects of that particular <Task, Utility> combination

  3. 3.

Variation-8: We sample 2 options from the Moderate Configurations of the
context object of that particular <Task,Utility> combination, we allow
sampling equivalent options as long as either of them is not the correct
answer. We also sample 2 options from the Bad Configurations of the random
context objects of that particular <Task, Utility> combination

  4. 4.

Variation-9: We sample 1 option from the Moderate Configurations of the
context objects of that particular <Task, Utility> combination, here we allow
sampling equivalent options as long as either of them is not the correct
answer. We also sample 3 options from the Bad Configurations of context
objects of that particular <Task, Utility> combination

##### 3 option dataset

  1. 1.

Variation-10: We sample 3 options from the Moderate Configurations of the
context objects of that particular <Task, Utility> combination, here we allow
sampling equivalent options as long as either of them is not the correct
answer.

  2. 2.

Variation-11: We sample 2 options from the Moderate Configurations of the
context objects of that particular <Task, Utility> combination, here we allow
sampling equivalent options as long as either of them is not the correct
answer. We also sample 1 option from the Bad Configurations of context objects
of that particular <Task,Utility> combination

  3. 3.

Variation-12: We sample 1 option from the Moderate Configurations of the
context object of that particular <Task,Utility> combination. We also sample 2
options from the Bad Configurations of the random context objects of that
particular <Task, Utility> combination

##### 2 option dataset

  1. 1.

Variation-13: We sample 2 options from the Moderate Configurations of the
context objects of that particular <Task, Utility> combination, here we allow
sampling equivalent options as long as either of them is not the correct
answer.

  2. 2.

Variation-14: We sample 1 option from the Moderate Configurations of the
context object of that particular <Task,Utility> combination. We also sample 1
option from the Bad Configurations of the random context objects of that
particular <Task, Utility> combination

##  Appendix C Results

(a) Average Accuracy of Various models on Task 1 as we increase option count

(b) Average Accuracy of Various models on Task 2 as we increase option count