# Why Lift so Heavy?
Slimming Large Language Models by Cutting Off the Layers

Shuzhou Yuan∗1, Ercong Nie∗2,3, Bolei Ma2,3, Michael Färber1
1Karlsruhe Institute of Technology, Germany
2LMU Munich, Germany
3Munich Center for Machine Learning (MCML), Germany
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Large Language Models (LLMs) possess outstanding capabilities in addressing
various natural language processing (NLP) tasks.††∗ Equal contribution.
However, the sheer size of these models poses challenges in terms of storage,
training and inference due to the inclusion of billions of parameters through
layer stacking. While traditional approaches such as model pruning or
distillation offer ways for reducing model size, they often come at the
expense of performance retention. In our investigation, we systematically
explore the approach of reducing the number of layers in LLMs. Surprisingly,
we observe that even with fewer layers, LLMs maintain similar or better
performance levels, particularly in prompt-based fine-tuning for text
classification tasks. Remarkably, in certain cases, models with a single layer
outperform their fully layered counterparts. These findings offer valuable
insights for future work aimed at mitigating the size constraints of LLMs
while preserving their performance, thereby opening avenues for significantly
more efficient use of LLMs.

Why Lift so Heavy?
Slimming Large Language Models by Cutting Off the Layers

Shuzhou Yuan∗1, Ercong Nie∗2,3, Bolei Ma2,3, Michael Färber1 1Karlsruhe
Institute of Technology, Germany 2LMU Munich, Germany 3Munich Center for
Machine Learning (MCML), Germany<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>

##  1 Introduction

Stacking decoder layers enables Large Language Models (LLMs) to generate deep
representations of input text, but also enlarges the model size of LLMs.
Beginning with the vanilla Transformers using 6 decoder layers (Vaswani et
al., 2017), language models have been rapidly growing to include more layers.
Examples are Llama2 7B with 32 layers (Touvron et al., 2023), OPT with 24
layers (Zhang et al., 2022), and GPT2-XL with even 48 decoder layers (Radford
et al., 2019). While LLMs with multiple layers have achieved state-of-the-art
performance in Natural Language Processing benchmarks (Wang et al., 2018),
adding just one layer could introduce millions of additional parameters to the
LLM.

Figure 1: Our research demonstrates that retaining only a single layer in
LLMs, by simply cutting off additional layers, sustains equivalent performance
compared to utilizing the full complement of layers.

Researchers aiming to reduce the training cost of LLMs have explored various
strategies, including freezing the parameters of LLMs and incorporating
additional trainable modules. These approaches encompass techniques such as
Adapter tuning (Houlsby et al., 2019), Prompt tuning (Lester et al., 2021),
and Prefix tuning (Li and Liang, 2021). Additionally, several studies have
investigated the disparities in representations across different layers,
highlighting the significance of updating the last few layers for language
models (Kovaleva et al., 2019; Liu et al., 2019; Lee et al., 2019). While
these methods demonstrate promising performance improvements and offer
substantial savings in training costs, they do not address the issue of the
cumbersome size of LLMs. Consequently, significant resources are still
required for the storage and inference of these models.

To address the challenges associated with the large storage demand of LLMs and
to explore the impact of the number of layers in LLMs, we propose a novel
approach of reducing the number of decoder layers in LLMs by removing specific
layers directly. In contrast to previous methods that involve freezing the
parameters of LLMs while retaining the complete architecture, our approach
involves simply trimming down the layers and retaining only the remaining
layers in the model. Through experimentation across various classification
tasks employing different styles, we surprisingly observe that a model with
just one layer maintains performance comparable to that of the full model. By
pruning the layers of the model, we effectively remove 93% of the parameters
in GPT2-XL (Radford et al., 2019) and 88% of the parameters in OPT (Zhang et
al., 2022).

Demonstrating a significant yet efficient finding, our work contributes to the
field of NLP in the following aspects:

  1. i)

We systematically investigate the influence of the number of layers in LLMs on
their performance and discover that reducing the number of layers does not
lead to a decrease in the model’s performance for classification tasks.

  2. ii)

Our findings provide valuable insights for future research aimed at reducing
the size of models, thereby offering a potential solution to the significant
challenges of huge computational budgets in LLM research.

##  2 Related Work

Pruning, a prevalent technique for model compression method, seeks to discard
non-essential parameters within language models. This approach ranges from
removing individual weights (unstructured pruning) to excising higher-
granularity structures (structured pruning), aiming to enhance efficiency
without compromising performance (LeCun et al., 1989; Han et al., 2016; Sanh
et al., 2020; Xu et al., 2021; Xia et al., 2022; Wang et al., 2023). Fan et
al. (2019) pioneered the concept of layer-wise pruning, a method that involves
randomly dropping out some layers of the model in the training to enhance the
model’s robustness towards layer reduction at inference time. Building on this
foundation, Peer et al. (2022) and Sajjad et al. (2023) aimed to identify and
remove an optimal subset of layers directly from the pretrained models for use
in downstream tasks. Recent studies have explored the application of pruning
techniques to LLMs (Xia et al., 2023; Jha et al., 2023; Frantar and Alistarh,
2023; Chen et al., 2023; Ashkboos et al., 2024). Unlike previous work, we
investigate the effect of layer-wise pruning on LLMs with prompt-based fine-
tuning quantitatively.

##  3 Experiment

###  3.1 Layer Pruning

We systematically adopt a top-layer removing strategy by only remaining the
first $n$ layers and drop all other layers. Specifically, for GPT2-XL, we keep
the models with 1, 2, 12, and 24 layers, while for OPT, we consider models
with 1, 2, 6, and 12 layers.111The complete GPT-2 XL model consists of 48
layers, and the OPT-1.3b model comprises 24 layers.

###  3.2 Problem Statement

In this work, we formulate text classification task as prompt-based language
modeling task. With only a few examples provided as input, our objective is to
determine the class of a given text. Let $M$ denote a language model, and let
$\mathcal{L}$ represent a set of label words. For instance, in a binary
classification task, we define a prompt as follows:

$X=t_{1},l_{1},t_{2},l_{2},t_{i}$ (1)

Here, $t_{1},l_{1}$ and $t_{2},l_{2}$ denote the examples and their label
words, $t_{i}$ represents the text to be classified. Utilizing the prompt $X$,
the language model $M$ aims to predict the next token $l_{i}$, which is then
assigned as the label word for $t_{i}$. $M$ is initialized with pretrained
parameters $\phi$, and updated by minimizing the cross-entropy loss.

###  3.3 Dataset

We conduct text classification tasks using four widely utilized datasets
spanning various domains: AGNews: AG’s news topic classification dataset,
utilized for topic classification with 4 labels (Zhang et al., 2015). EmoC:
EmoContext, facilitating 4-label emotion classification (Chatterjee et al.,
2019). SST-2: Stanford Sentiment Treebank Binary, employed for sentiment
analysis (Socher et al., 2013). TREC: Text REtrieval Conference Question
Classification (TREC), designed for question type classification, encompassing
6 types (Li and Roth, 2002; Hovy et al., 2001).

###  3.4 Setting

$n$ | Param | AGNews | EmoC | SST-2 | TREC | Average | $n$ | Param | AGNews | EmoC | SST-2 | TREC | Average
---|---|---|---|---|---|---|---|---|---|---|---|---|---
GPT2-XL | OPT
48 | 1.6B | 85.34 | 73.70 | 68.97 | 80.16 | 77.04 | 24 | 1.3B | 76.10 | 74.80 | 64.31 | 76.80 | 73.00
24 | 819M | 86.66 | 75.44 | 72.41 | 72.41 | 76.73 | 12 | 711M | 77.52 | 78.46 | 69.13 | 80.44 | 76.39
12 | 450M | 86.42 | 75.34 | 73.76 | 84.72 | 80.06 | 6 | 409M | 77.48 | 77.22 | 69.34 | 81.72 | 76.44
2 | 143M | 85.80 | 75.78 | 74.22 | 85.12 | 80.23 | 2 | 297M | 80.60 | 74.66 | 70.92 | 81.88 | 77.01
1 | 112M | 85.30 | 76.76 | 72.89 | 83.16 | 79.53 | 1 | 157M | 81.32 | 77.42 | 69.17 | 82.12 | 77.51

Table 1: Prompt-based fine-tuning with different numbers of layers using
language modeling head. $n$ denotes the number of layers.

$n$ | Param | AGNews | EmoC | SST-2 | TREC | Average | $n$ | Param | AGNews | EmoC | SST-2 | TREC | Average
---|---|---|---|---|---|---|---|---|---|---|---|---|---
GPT2-XL | OPT
48 | 1.6B | 85.48 | 73.34 | 67.13 | 81.04 | 76.75 | 24 | 1.3B | 73.96 | 76.72 | 64.08 | 75.92 | 72.67
24 | 819M | 85.94 | 71.94 | 71.40 | 82.68 | 77.99 | 12 | 711M | 76.02 | 75.86 | 68.60 | 80.72 | 75.30
12 | 450M | 86.06 | 73.82 | 72.18 | 83.28 | 78.83 | 6 | 409M | 78.56 | 77.56 | 71.15 | 80.52 | 76.95
2 | 143M | 85.28 | 75.52 | 73.19 | 84.12 | 79.53 | 2 | 297M | 77.92 | 72.76 | 69.38 | 80.80 | 75.22
1 | 112M | 85.50 | 75.28 | 73.12 | 83.40 | 79.33 | 1 | 157M | 79.88 | 72.38 | 71.10 | 82.56 | 76.48

Table 2: Prompt-based fine-tuning with different numbers of layers using
classification head. $n$ denotes the number of layers.

$n$ | Param | AGNews | EmoC | SST-2 | TREC | Average | $n$ | Param | AGNews | EmoC | SST-2 | TREC | Average
---|---|---|---|---|---|---|---|---|---|---|---|---|---
GPT2-XL | OPT
48 | 1.6B | 86.62 | 72.60 | 72.96 | 81.64 | 78.46 | 24 | 1.3B | 78.94 | 76.02 | 68.35 | 81.92 | 76.31
24 | 819M | 86.02 | 75.42 | 73.05 | 84.40 | 79.72 | 12 | 711M | 83.30 | 75.12 | 67.02 | 81.72 | 76.79
12 | 450M | 86.96 | 76.58 | 73.76 | 85.04 | 80.59 | 6 | 409M | 83.82 | 78.22 | 70.85 | 82.88 | 78.94
2 | 143M | 86.64 | 74.32 | 72.94 | 84.28 | 79.54 | 2 | 297M | 84.38 | 76.54 | 70.94 | 82.44 | 78.58
1 | 112M | 85.56 | 73.66 | 73.30 | 81.12 | 78.41 | 1 | 157M | 81.86 | 77.80 | 70.87 | 83.76 | 78.57

Table 3: Fine-tuning with different numbers of layers using classification
head. $n$ denotes the number of layers.

(a)

(b)

Figure 2: The average accuracy of different layers for three classification
methods. LM denotes language modeling head, P-CLS denotes prompt-based fine-
tuning with classification head, CLS denotes classification head without
prompt.

To create prompts, we select one demonstration per class and append the sample
to be predicted at the end of the prompt. These demonstrations are drawn from
the original training set of the datasets and are excluded from subsequent
training. For training, we randomly select 200 training samples from the
original training set to ensure adequate model training. Additionally, we
allocate 1000 samples from the original training set for validation purposes,
while 1000 samples from the original test set are reserved for evaluation.
Model selection is based on the accuracy achieved on the validation set, with
the best-performing model then evaluated on the test set.

We maintain consistent hyperparameters across models, regardless of the number
of layers. Specifically, we employ AdamW (Loshchilov and Hutter, 2019) as the
optimizer with a learning rate of 5e-5. All models are trained for 50 epochs,
with early stopping implemented after 15 epochs. Due to resource constraints,
we set the batch size to 1. Five different random seeds 222Random seeds: [0,
42, 421, 520, 1218] are utilized for the experiments, and the average accuracy
across these seeds is reported as the final results.

###  3.5 Main Results

The results are presented in Table 1. For the GPT2-XL model, intriguingly, the
model’s performance demonstrates minimal variance across varying layer
configurations. When fine-tuned with the complete set of layers, the average
accuracy reaches 77.04%. Even with a reduction to 24 layers, the accuracy only
experiences a slight decrease to 76.73%. Notably, a reduction to 2 layers
yields the highest accuracy at 80.23%, surpassing the full-layer model.
Remarkably, a one-layer model outperforms its full-layer counterpart,
indicating the potential for significant parameter reduction from 48 to 1,
saving nearly 1.5 billion parameters.

Similarly, OPT exhibits a consistent trend of performance improvement with
decreasing layer count. While the 24-layer configuration achieves an average
accuracy of 73%, performance consistently improves as layers are reduced. OPT
with 12 layers achieves an accuracy of 76.39%, with 6 layers, 76.44%, with 2
layers, 77.01%, and with a single layer, the highest accuracy of 77.51% is
attained. The reduction of model layers from 24 to 1 results in a saving of
1.14 billion parameters.

###  3.6 Replacement of Classification Head

Given that the classification tasks are framed as language modeling endeavors,
an inquiry arises regarding whether the consistent performance across varied
layer configurations stems from the semantic understanding facilitated by the
language modeling head. To explore this hypothesis, we substitute the language
modeling head with a classification head, maintaining the experimental setting
outlined in §3.4.

The outcomes of employing a classification head for prompt-based fine-tuning
are delineated in Table 2. In comparison to the language modeling head, the
performance of the classification head exhibits a slight decrease under
identical layer counts. Nonetheless, the average accuracy of the
classification head maintains consistent performance with marginal
fluctuations. For GPT2-XL, employing the full complement of layers yields an
average accuracy of 76.75%. Remarkably, the model achieves its highest
accuracy with only 2 layers, with even a single-layer configuration
outperforming the full-layer model. A similar trend is observed for the OPT
model, wherein a reduction in layers correlates with performance enhancement.
While the full-layer configuration achieves an average accuracy of 72.67%, the
single-layer model surpasses it with an accuracy of 76.48%.

###  3.7 Further Discussion: Classification

We delve deeper into the classification task employing the conventional
classification paradigm, wherein text categorization relies solely on the
input text without demonstrations. By removing demonstrations and providing
the model with input examples, we apply a classification head to categorize
the text, adhering to the experimental settings articulated in §3.4.

As depicted in Table 3, the average accuracy of the classification setting
conforms to the trend observed in prompt-based fine-tuning. The reduction in
layers does not exert a discernible impact on the model’s performance. For
GPT2-XL, the model consistently maintains an average accuracy of approximately
80%, irrespective of the number of layers. Optimal accuracy is attained by the
model with 12 layers. Analogously, OPT exhibits similar performance regardless
of layer count, with the single-layer configuration outperforming the full-
layer model, both for GPT2-XL and OPT.

Furthermore, we visually represent the results of the three classification
methods we have implemented in Figure 2. Notably, these methods yield similar
outcomes, with layer count exerting negligible influence on model performance.
Intriguingly, models with fewer layers often outperform those with full
layers. These findings suggest that neither the language modeling head nor the
classification head singularly dictates the consistent performance observed
with decreasing layer count. Consequently, this discovery holds promise for
optimizing resources in classification tasks by economizing training and
storage resources.

##  4 Conclusion

This study delves into the impact of layer count on the performance of LLMs.
Our investigation unveils that variations in the number of layers do not
result in performance degradation for classification tasks employing LLMs.
Remarkably, even with a reduction in the number of LLM layers from 48 to 1,
both GPT2-XL and OPT consistently maintain or even enhance performance. This
discovery offers valuable insights into reducing the training and storage
costs associated with LLMs, presenting promising avenues for optimizing
resource allocation in such models.

## Limitations

This study exclusively focuses on examining the impact of the number of layers
in LLMs on classification tasks, without extending the investigation to other
NLP tasks. Furthermore, we compare the performance of various classification
schemes but refrain from delving deeper into elucidating the underlying
reasons behind the observed phenomenon where the models exhibit consistent
performance regardless of the number of layers.

## References

  * Ashkboos et al. (2024) Saleh Ashkboos, Maximilian L Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, and James Hensman. 2024.  Slicegpt: Compress large language models by deleting rows and columns.  _arXiv preprint arXiv:2401.15024_.
  * Chatterjee et al. (2019) Ankush Chatterjee, Kedhar Nath Narahari, Meghana Joshi, and Puneet Agrawal. 2019.  SemEval-2019 task 3: EmoContext contextual emotion detection in text.  In _Proceedings of the 13th International Workshop on Semantic Evaluation_ , pages 39–48, Minneapolis, Minnesota, USA. Association for Computational Linguistics.
  * Chen et al. (2023) Tianyi Chen, Tianyu Ding, Badal Yadav, Ilya Zharkov, and Luming Liang. 2023.  Lorashear: Efficient large language model structured pruning and knowledge recovery.  _arXiv preprint arXiv:2310.18356_.
  * Fan et al. (2019) Angela Fan, Edouard Grave, and Armand Joulin. 2019.  Reducing transformer depth on demand with structured dropout.  In _International Conference on Learning Representations_.
  * Frantar and Alistarh (2023) Elias Frantar and Dan Alistarh. 2023.  Sparsegpt: Massive language models can be accurately pruned in one-shot.  In _International Conference on Machine Learning_ , pages 10323–10337. PMLR.
  * Han et al. (2016) Song Han, Huizi Mao, and William J. Dally. 2016.  Deep compression: Compressing deep neural network with pruning, trained quantization and huffman coding.  _4th International Conference on Learning Representations, ICLR_.
  * Houlsby et al. (2019) Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019.  Parameter-efficient transfer learning for NLP.  In _Proceedings of the 36th International Conference on Machine Learning_ , volume 97 of _Proceedings of Machine Learning Research_ , pages 2790–2799. PMLR.
  * Hovy et al. (2001) Eduard Hovy, Laurie Gerber, Ulf Hermjakob, Chin-Yew Lin, and Deepak Ravichandran. 2001.  Toward semantics-based answer pinpointing.  In _Proceedings of the First International Conference on Human Language Technology Research_.
  * Jha et al. (2023) Ananya Harsh Jha, Tom Sherborne, Evan Pete Walsh, Dirk Groeneveld, Emma Strubell, and Iz Beltagy. 2023.  How to train your (compressed) large language model.
  * Kovaleva et al. (2019) Olga Kovaleva, Alexey Romanov, Anna Rogers, and Anna Rumshisky. 2019.  Revealing the dark secrets of BERT.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 4365–4374, Hong Kong, China. Association for Computational Linguistics.
  * LeCun et al. (1989) Yann LeCun, John Denker, and Sara Solla. 1989.  Optimal brain damage.  _Advances in neural information processing systems_ , 2.
  * Lee et al. (2019) Jaejun Lee, Raphael Tang, and Jimmy Lin. 2019.  What would elsa do? freezing layers during transformer fine-tuning.  _arXiv preprint arXiv:1911.03090_.
  * Lester et al. (2021) Brian Lester, Rami Al-Rfou, and Noah Constant. 2021.  The power of scale for parameter-efficient prompt tuning.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 3045–3059, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Li and Liang (2021) Xiang Lisa Li and Percy Liang. 2021.  Prefix-tuning: Optimizing continuous prompts for generation.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 4582–4597, Online. Association for Computational Linguistics.
  * Li and Roth (2002) Xin Li and Dan Roth. 2002.  Learning question classifiers.  In _COLING 2002: The 19th International Conference on Computational Linguistics_.
  * Liu et al. (2019) Nelson F. Liu, Matt Gardner, Yonatan Belinkov, Matthew E. Peters, and Noah A. Smith. 2019.  Linguistic knowledge and transferability of contextual representations.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 1073–1094, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Loshchilov and Hutter (2019) Ilya Loshchilov and Frank Hutter. 2019.  Decoupled weight decay regularization.  In _International Conference on Learning Representations_.
  * Peer et al. (2022) David Peer, Sebastian Stabinger, Stefan Engl, and Antonio Rodríguez-Sánchez. 2022.  Greedy-layer pruning: Speeding up transformer models for natural language processing.  _Pattern Recognition Letters_ , 157:76–82.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9.
  * Sajjad et al. (2023) Hassan Sajjad, Fahim Dalvi, Nadir Durrani, and Preslav Nakov. 2023.  On the effect of dropping layers of pre-trained transformer models.  _Computer Speech and Language_ , 77:101429.
  * Sanh et al. (2020) Victor Sanh, Thomas Wolf, and Alexander Rush. 2020.  Movement pruning: Adaptive sparsity by fine-tuning.  _Advances in Neural Information Processing Systems_ , 33:20378–20389.
  * Socher et al. (2013) Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013.  Recursive deep models for semantic compositionality over a sentiment treebank.  In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing_ , pages 1631–1642, Seattle, Washington, USA. Association for Computational Linguistics.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Advances in Neural Information Processing Systems_ , pages 5998–6008.
  * Wang et al. (2018) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018.  GLUE: A multi-task benchmark and analysis platform for natural language understanding.  In _Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_ , pages 353–355, Brussels, Belgium. Association for Computational Linguistics.
  * Wang et al. (2023) Xinpeng Wang, Leonie Weissweiler, Hinrich Schütze, and Barbara Plank. 2023.  How to distill your BERT: An empirical study on the impact of weight initialisation and distillation objectives.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 1843–1852, Toronto, Canada. Association for Computational Linguistics.
  * Xia et al. (2023) Mengzhou Xia, Tianyu Gao, Zhiyuan Zeng, and Danqi Chen. 2023.  Sheared llama: Accelerating language model pre-training via structured pruning.  In _Workshop on Advancing Neural Network Training: Computational Efficiency, Scalability, and Resource Optimization (WANT@ NeurIPS 2023)_.
  * Xia et al. (2022) Mengzhou Xia, Zexuan Zhong, and Danqi Chen. 2022.  Structured pruning learns compact and accurate models.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1513–1528, Dublin, Ireland. Association for Computational Linguistics.
  * Xu et al. (2021) Dongkuan Xu, Ian En-Hsu Yen, Jinxi Zhao, and Zhibin Xiao. 2021.  Rethinking network pruning – under the pre-train and fine-tune paradigm.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 2376–2382, Online. Association for Computational Linguistics.
  * Zhang et al. (2022) Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022.  Opt: Open pre-trained transformer language models.  _arXiv preprint arXiv:2205.01068_.
  * Zhang et al. (2015) Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015.  Character-level convolutional networks for text classification.  In _Advances in Neural Information Processing Systems_ , volume 28. Curran Associates, Inc.


# From Cooking Recipes to Robot Task Trees – Improving Planning Correctness
and Task Efficiency by Leveraging LLMs with a Knowledge Network

Md. Sadman Sakib, and Yu Sun
Yu Sun and Md. Sadman Sakib are members of the Robot Perception and Action Lab
(RPAL), which is part of the Department of Computer Science & Engineering at
the University of South Florida, Tampa, FL, USA.
Email<EMAIL_ADDRESS>
###### Abstract

Task planning for robotic cooking involves generating a sequence of actions
for a robot to prepare a meal successfully. This paper introduces a novel task
tree generation pipeline producing correct planning and efficient execution
for cooking tasks. Our method first uses a large language model (LLM) to
retrieve recipe instructions and then utilizes a fine-tuned GPT-3 to convert
them into a task tree, capturing sequential and parallel dependencies among
subtasks. The pipeline then mitigates the uncertainty and unreliable features
of LLM outputs using task tree retrieval. We combine multiple LLM task tree
outputs into a graph and perform a task tree retrieval to avoid questionable
nodes and high-cost nodes to improve planning correctness and improve
execution efficiency. Our evaluation results show its superior performance
compared to previous works in task planning accuracy and efficiency.

##  I Introduction

Robotic cooking has emerged as a highly promising domain within robotics,
presenting notable advantages such as convenience and the potential for
enhanced efficiency and precision in meal preparation. To effectively automate
cooking tasks, the key component is efficient task planning. This entails
generating a series of actions guiding the robot in accomplishing a specific
goal. However, this is an intricate field of research due to the fact that
cooking tasks typically involve lengthy sequences of actions encompassing
various ingredients and tools. Moreover, they necessitate the attainment of
numerous crucial ingredient states throughout the process. Additionally, the
cooking conditions, processes, and requirements are exceptionally diverse.
Approaches like state-space planning, learning from demonstration, and even
knowledge network retrieval encounter difficulties when confronted with unseen
starting conditions and requests.

In cooking tasks, ingredients or objects can vary in form, shape, and size,
and there are multiple states to consider during recipe execution. The
manipulation of an ingredient depends on its specific state, and certain
ingredients may not be readily available in the desired state. Additionally,
robots have varying capabilities, making some actions easier for them to
perform than others. A task planning method should consider these factors and
propose a plan that is most suitable for the robot to execute efficiently.
Previous work has created a knowledge network consisting of 140 cooking
recipes called the Functional Object-Oriented Network (FOON) [1, 2]. However,
generating plans in novel scenarios where FOON lacks knowledge about the
recipe or an ingredient proved challenging. Furthermore, expanding the
knowledge base was difficult due to the reliance on manual annotation.

Recently, the emergence of Large Language Models (LLMs) [3, 4, 5] has
addressed the limitation of limited knowledge. These LLMs possess the ability
to generate “likely” viable solutions for different scenarios and requests.
While their results may not always be correct or optimal, their notable
capacity for generalization can help overcome the limitations of search-based
task tree retrieval methods. The search-based retrieval approach with a
comprehensive knowledge network on the other hand can detect, remove and
replace the wrong elements in the LLM outputs.

The primary focus of this research paper is to tackle the task planning
challenge in robotic cooking through the introduction of an innovative task
tree generation approach (Figure 1). We aim to generate a task plan that is
both error-free and cost-effective. To enhance the accuracy of the task plan,
we employ a method that involves detecting incorrect components within the
task trees generated by GPT-3 and search for alternative options either within
other task trees or within the FOON knowledge graph. This approach allows us
to improve the overall quality and reliability of the generated task plan. By
carefully selecting the most optimized plan from these alternatives, the
pipeline ensures effective resource utilization while achieving the desired
objectives. The effectiveness of the task tree generation pipeline is
evaluated through a comparative analysis with a previous approach. The results
demonstrate the superiority of the proposed method, showcasing enhanced task
planning accuracy and improved cost-efficiency.

Figure 1:  Overview of our approach. Given a meal preparation instruction, the
model generates a list of tasks specifically designed for the robot.

Our contributions in this paper are as follows: (i) We propose a novel task
tree generation approach that accepts any dish of the user’s choice and
produces a robot task tree with state-of-the-art accuracy and efficiency; (ii)
We fine-tune GPT-3 to convert natural language instructions into a task tree
structure; (iii) We improve the accuracy of the task plan by detecting
incorrect components in the GPT generated task trees and finding alternatives
either in other task trees or FOON; (iv) We optimize execution costs by
performing weighted retrieval in a mini-FOON combined from multiple GPT
outputs or FOON. We demonstrate the superiority of our model through a
comparison with a previous approach.

##  II Background

###  II-A Functional Object-Oriented Network

FOON and related knowledge graphs have been used in many tasks for robots,
such as robotic cooking [6] and furniture assembly [7, 8, 9]. The one used
here is a knowledge graph constructed through manual annotation of video
demonstrations. It consists of two types of nodes: object nodes and motion
nodes. These nodes are connected by directed edges, which depict the
preconditions and effects of actions. The functional unit is the fundamental
building block of FOON, representing a single action observed in the video
demonstration. It consists of one or more input nodes, one or more output
nodes, and a single motion node. The input nodes specify the required state of
objects before the action, while the output nodes describe the resulting state
after the action is executed. The motion node represents the action itself.
Functional units provide a detailed and vivid representation of the actions
observed in the video demonstrations. Figure 2 shows two functional units of
slicing an onion and placing onion to cooking pan. The current FOON dataset
(available in [10]) consists of 140 annotated recipes sourced from platforms
such as YouTube, Activity-Net [11], and EPIC-KITCHENS [12].

Figure 2:  Two functional units from FOON depicting slicing an onion and
placing it to the cooking pan. Objects and motions nodes are denoted by green
circle and red square respectively.

####  II-A1 Task Planning with FOON

The utilization of FOON as a knowledge base for task planning offers several
advantages, including the ability to provide recipe variations. Task planning
with FOON involves searching the network to find a goal node and retrieving a
path, referred as a task tree, that leads to achieving the desired objective.
A task tree, consists of a sequence of functional units that need to be
executed in order to prepare the dish. To illustrate, consider the task tree
associated with boiling water, which comprises actions such as placing a pot
on the stove, pouring water, turning on the stove, and turning off the stove.
Each of these procedural steps is represented by input object nodes,
signifying the prerequisites for executing the action; a motion node, denoting
the action itself; and output object nodes, denoting the effect of executing
the action. The task tree retrieval algorithm proposed in [1] focuses on
finding a path that utilizes only the ingredients available in the kitchen. On
the other hand, [13] retrieves a plan that can be executed with human-robot
collaboration. Nevertheless, these approaches have a limitation when it comes
to generating a plan for a recipe that is not explicitly available in FOON.
For example, if a user asks for a plan to prepare a mango milkshake, but there
is no dedicated recipe for it in FOON, the system may be unable to provide a
plan, even if there is a recipe for a banana milkshake. To address this
limitation, a novel task tree retrieval method [14] was introduced that can
learn from similar recipes in FOON and make necessary modifications to match
the user’s requirements. While this approach introduces some level of
generalization, the quality of the generated plan heavily relies on the
availability of closely matched recipes in FOON. In this work, we leverage
LLMs to overcome this dependency on closely matched recipes and generate high-
quality task trees for any recipe, thereby enhancing the flexibility and
effectiveness of the task planning process.

###  II-B Related Works

In the domain of robotic cooking and task planning, several strategies have
been proposed to tackle the challenges associated with generating effective
action sequences for executing user instructions. One prominent approach
involves the use of knowledge graphs to address this challenge. Notably, the
KNOWROB framework [15, 16] has made significant contributions in this area by
leveraging a knowledge base constructed from data collected in sensor-equipped
environments. [17] introduced a task generalization scheme that relaxes the
requirement of having multiple task demonstrations to perform tasks in unknown
environments. This scheme integrates the task plan with a knowledge graph
derived from observations in a virtual simulator. The impact of knowledge
graphs on a robot’s decision-making process was further investigated in [18].
However, these approaches heavily rely on the limited information contained in
their respective knowledge bases. In contrast, our approach harnesses the
power of Language Models (LLMs) to alleviate the burden of creating a
knowledge base, offering a more comprehensive and flexible solution.

Recently, task planning with LLMs has become a prominent area of research,
capitalizing on the impressive language understanding and generation
capabilities of LLMs. Various studies have explored the use of LLMs to
generate step-by-step plans for long-horizon tasks. For instance, Erra et al.
[19] proposed an approach that employs LLMs to generate plans for complex
tasks. [20, 21, 22] have also utilized LLMs for plan generation in different
domains. However, these works often do not explicitly consider the robot’s
capability to perform specific actions. One limitation of relying solely on
LLMs is the lack of interaction with the environment. To address this
limitation, SayCan [23] introduced a framework that combines the high-level
knowledge of LLMs with low-level skills, enabling the execution of plans in
the real world. By grounding LLM-generated plans with the robot’s capabilities
and environmental constraints, SayCan bridges the gap between language-based
planning and physical execution. In addition, recent research efforts such as
Text2Motion [24], ProgPrompt [25] have integrated LLMs with learned skill
policies. They exhibit trust in the LLM-generated plan and proceed to execute
it whereas we focus on enhancing LLM’s accuracy to generate an optimal task
plan.

##  III Proposed Method

Our objective is to develop a robust pipeline that generates highly accurate
and executable task trees for robotic operations. To achieve this, we employ a
multi-step approach that leverages the capabilities of LLMs and FOON.
Initially, we utilize ChatGPT [26] to generate a recipe based on the user’s
meal specifications. However, the output is in natural language, which may
pose challenges for direct robot execution. To address this, we employ a fine-
tuned GPT-3 model to convert the recipe instructions into a task tree format.
Due to the uncertainty of the generative model, the task plan may not be
always correct or most efficient. To enhance reliability and efficiency, we
look for alternative options in other task trees generated by GPT-3 or in
FOON. From these alternatives, the selected task tree is expected to be
accurate and easier for the robot to execute. A visual representation of our
pipeline and its key components are presented in Figure 3. In the following
subsections, we will provide detailed explanations of each component.

Figure 3: Overview of our task tree generation procedure. Starting with a meal
specification as the query, our pipeline generates a task plan represented as
task tree 7.

###  III-A Prompt engineering for recipe generation

Our system is designed to accommodate dish specifications provided by the
user. The user can specify a list of desired ingredients or exclude certain
ingredients. Additionally, specifications such as gluten-free, vegetable-
based, or non-dairy options are also accepted. Based on this information, we
engine a prompt and retrieve the recipe from ChatGPT. To facilitate easier
parsing, we have designed the prompt to include numbered instructions within
the response from ChatGPT.

###  III-B Converting instructions to a task tree

When a robot performs an action, several factors need to be considered, such
as preconditions, effects, and the objects involved. Additionally,
understanding the state of these objects is crucial for determining the
appropriate grasp or manipulation technique. However, extracting all this
information from a recipe written in natural language poses significant
challenges. To address this complexity, we propose translating the
instructions into structured functional units that encapsulate all the
necessary details. By organizing these functional units into a task tree, we
provide a step-by-step guide for the robot to execute the task effectively. To
accomplish this, we have created a dedicated dataset for fine-tuning a GPT-3
Davinci model. This model takes a recipe as input and translates it into a
task tree representation. The dataset comprises 180 recipe examples sourced
from FOON, each consisting of natural language instructions and a
corresponding FOON task tree. Due to the limitation of maximum token count,
some recipes had to be divided into multiple parts, resulting in multiple task
trees for a single recipe.

###  III-C Creating a mini-FOON

To address the potential presence of errors in the task plans generated by the
fine-tuned model, we adopt a strategy of generating multiple task trees for
the same recipe. Our aim is to search for a task tree that is error-free and
one that is efficient for the robot to execute from the combined graph mini-
FOON. FOON has revealed that merging recipes in a graph structure can lead to
the emergence of novel cooking methods. This merging process allows recipes to
share information and learn diverse approaches for accomplishing subtasks.
Inspired by this idea of exploring new paths, we employ a similar graph
structure to merge the five task trees generated by GPT. This merged structure
is referred to as a mini-FOON.

####  III-C1 Merging task trees

During the merging process, our objective is to eliminate any incorrect
functional units and remove duplicates. An incorrect functional unit can arise
in two ways: (i) syntax error and (ii) an erroneous object-action
relationship. Syntax verification involves checking whether the functional
unit includes the necessary components such as input and output objects, as
well as a motion node. Additionally, it verifies if each object has an
assigned state. On the other hand, validating the object-action relationship
poses the challenge of determining if the state transition for an action is
correct. To tackle this challenge, we compiled a comprehensive list of all
valid state transitions from FOON. Based on this list, we can assess the
correctness of a transition. For instance, if a transition such as “sliced
$\rightarrow$ whole” is not present in FOON, it would be identified as
incorrect. Functional units that successfully pass the verification criteria
are then added to the mini-FOON.

###  III-D Creating a super-FOON

We integrate the mini-FOON with the original FOON, forming a combined network
known as the super-FOON. During this merging process, our primary focus is on
node consolidation, as the mini-FOON and FOON may use different names for the
same object or motion node. To achieve consolidation, we follow a set of basic
rules. For instance, we convert all object names to their singular form. We
observed that GPT-3 often generates plural forms such as ”strawberries” and
”onions,” while FOON represents them as ”strawberry” and ”onion” respectively.
By applying these rules, we try to ensure consistency and compatibility
between the node names in the mini-FOON and FOON within the super-FOON
network.

###  III-E Task tree retrieval

Taking the desired dish as the goal node, we employ a search procedure similar
to [13] to retrieve all paths leading to the goal. We execute the same search
algorithm in both the mini-FOON and super-FOON. This approach often yields
multiple task plans, exceeding five in number, which may differ in the number
of cooking steps involved. For instance, when preparing a banana milkshake,
one plan may suggest adding the whole peeled banana to the blender, while
another plan may propose cutting the banana in half before blending. Once the
incorrect functional units have been filtered out, the task tree retrieval
procedure does not select them. Instead, it prioritizes the available correct
functional units to construct the task plan. For instance, if the functional
unit for “slicing an apple” is found to be incorrect in the first generated
tree but correct in the other four task trees, the search procedure will
choose the functional unit of slicing an apple from those four task trees.
From the generated plans, we must select the most feasible one for the robot
to execute. The feasibility of executing an action depends on the robot’s
configuration. For example, a robot with only one hand may find pouring easier
than chopping. Consequently, the success rate of executing a task tree varies
among different robots. Following the approach of [13], we assign a cost value
ranging from 0 to 1 to each action. These values are determined by three
factors: 1) the physical capabilities of the robot, 2) its past experiences
and ability to perform actions, and 3) the tools or objects that the robot
needs to manipulate. A higher cost value indicates a more challenging action
to execute. Based on these costs, we select task tree 6 from the mini-FOON and
task tree 7 from the super-FOON. Ideally, task tree 7 should never be worse
than task tree 6 since the super-FOON encompasses all the task trees from the
mini-FOON. Task tree 7 serves as the final output of this pipeline. Figure 4
illustrates an example of cost optimization using the super-FOON, where two
pouring actions are preferred over scooping due to the significantly lower
cost assigned to pouring (0.1) compared to scooping (0.4). We assigned a low
cost to pouring based on the successful pouring accuracy achieved by Huang et
al. [27] with a robot.

(a) cost of execution = 0.7

(b) cost of execution = 0.5

Figure 4: Example of cost optimization: Comparison between task trees
retrieved from the mini-FOON and super-FOON. The assigned costs for scooping,
pouring, and mixing are 0.4, 0.1, and 0.1 respectively. (a) The task tree from
the mini-FOON (b) The task tree from the super-FOON.

##  IV Experiments and Results

Our experiment aims to assess both the quality of the generated task trees and
the associated execution costs. Simultaneously, we seek to compare the
performance of our model in generating recipes across different dish
categories. To accomplish this, we curated a dataset consisting of 60 randomly
selected recipes from the Salad, Drink, and Muffin categories. These recipes
were extracted from Recipe1M+ [28], a comprehensive collection of over one
million recipes encompassing a wide range of dish types and ingredients.

###  IV-A Evaluation Metric

Validating the plan of a cooking task in an automated manner is challenging
due to the absence of a fixed method for preparing a dish. Two task plans for
the same dish can differ in their cooking approaches, yet both can be deemed
correct. As a result, manual verification becomes necessary. However, the
original format of a task tree can be difficult for humans to comprehend. To
address this, we convert the task trees into progress lines as used in [14] to
illustrate how the ingredients are manipulated and undergo changes throughout
the cooking process. This simplified visualization facilitates the detection
of errors in the task plan by humans. We consider a recipe correct if the
progress lines for all ingredients used in the recipe are accurate. An example
of progress lines for a Greek Salad recipe is provided in Figure 5.

Figure 5:  Progress lines for a Greek Salad recipe.

###  IV-B Task Planning Accuracy

We employed four different methods to generate task trees for the selected
recipes. The quality of the generated trees was assessed using the progress
line, and the corresponding accuracy results are shown in Figure 6. When
relying solely on FOON, the task trees obtained for Salad and Drink recipes
exhibited good quality. This was expected as FOON contained an ample number of
recipes (10 each) for these categories. However, for Muffin recipes, the
quality of the generated task trees suffered due to the scarcity of available
examples in FOON (only one recipe). The FOON-search based approach heavily
depends on finding a similar recipe in FOON as a reference for making
necessary modifications to the task plan. Consequently, a high number of
adjustments were required, leading to inaccuracies in the task plan. In the
case of the fine-tuned GPT-3 model, errors in functional units frequently
resulted in task plan failures. However, the introduction of the Mini-FOON
helped mitigate these errors by providing a wider range of alternatives to
achieve the desired objectives. Integrating FOON into our approach enabled us
to choose a path from a broader set of options, resulting in higher accuracy.
Compared to [14], our approach achieved a 4% higher accuracy for Salad, 6%
higher accuracy for Drink, and a significant 45% higher accuracy for Muffin
recipes. Notably, our fine-tuned model demonstrated good accuracy for Muffin
recipes, despite not being specifically trained on this particular dish. This
highlights the significant advantage of employing an LLM. Once the LLM is
fine-tuned to comprehend the structure of a task tree, it can effectively
generalize to various types of recipes.

Figure 6:  Comparison of different approaches’ accuracy on Salad, Drink, and
Muffin dishes.

###  IV-C Execution cost

The objective of this experiment is to evaluate the extent to which our
approach can optimize the execution cost of recipes. If a recipe cannot be
optimized, it implies that there are no superior alternatives in FOON compared
to the initial output generated by the fine-tuned model (task tree 1). In
Figure 7, we present the number of optimized recipes by generating different
numbers of task trees. When the number of task trees is 2, and we select the
plan with the lower cost, it yields a better solution in 5% of the cases.
Similarly, by gradually increasing the number of task trees up to 5 and
selecting the one with the minimum cost, we obtain a better solution in 15% of
the cases. More optimization occurs when we choose task tree 6 from the Mini-
FOON, as it combines subtasks from five different task trees, resulting in a
lower cost. Ultimately, task tree 7, the final output from our pipeline,
maximizes the advantages of FOON and minimizes the execution cost compared to
task tree 1 in 40% cases.

Figure 7:  Number of recipes that were optimized by generating varying numbers
of task trees in comparison to Task Tree 1 (generated by the fine-tuned
model).

##  V Discussion

###  V-A Finetuning a GPT-3

We examined how the model’s understanding of the task tree structure improves
with the addition of new training data (Figure 8). Initially, the training
began with a dataset consisting of only 30 examples. Consequently, the model
struggled to grasp the syntax of functional units, resulting in grammatical
errors in the generated functional units. For instance, it would include
multiple motion nodes within a single functional unit, whereas, according to
the definition, a functional unit should contain only one motion node. As we
increased the number of recipes, the model gradually reduced its syntactical
errors. However, it still exhibited logical mistakes, such as incorrect state
transitions or missing actions. Finally, after finetuning with 180 examples,
the model achieved an accuracy of 67%.

Figure 8:  Impact of training dataset size on model accuracy.

###  V-B Executing a task tree

A task tree provides a high-level plan that lacks interaction with the
environment. However, executing actions often requires additional information,
such as the geometric position of objects or the initial quantity of
ingredients in a container. For instance, a task plan might involve adding ice
to an empty glass, but the glass could be positioned upside down on a table.
Therefore, before pouring the ice, the glass would need to be rotated back to
its original position. This crucial step is missing in our high-level
planning. Hence, there is a need for hierarchical planning, where the task
tree can be converted into a low-level plan that can be executed in the real
world. Paulius et al. [29] proposed a method to convert a task tree into a
representation using Planning Domain Definition Language (PDDL) [30]. Each
functional unit is treated as a planning operator, and a plan is generated
based on the robot’s low-level motion primitives.

###  V-C Limitations of our approach

(i) The generation of a task tree involves making 5 API calls to the fine-
tuned model. Each API call takes approximately 5 seconds, resulting in a slow
pipeline. The focus of this research was not on time complexity. In the
future, if we aim to enhance the system’s speed, it may be necessary to
explore fine-tuning locally installed LLMs. (ii) The generated plan sometimes
introduces new names for ingredients, states or motions such as garnish. These
unknown labels in functional units pose a challenge when attempting to find
alternative options in FOON, as proper mapping to existing functional units
becomes difficult. Furthermore, the detection of incorrect transitions is also
hindered, as the possible transition list may not include these new labels.
(iii) A fine-tuned GPT-3 model has a limitation where the combined query and
completion cannot exceed 2048 tokens. Due to this constraint, generating a
task tree becomes challenging when dealing with complex recipes that require a
higher number of functional units.

##  VI Conclusion

In this study, our objective was to propose a novel pipeline for task tree
generation, leveraging the advantages offered by LLMs. We utilized ChatGPT to
respond to user queries, and then fine-tuned a GPT-3 model to convert the
response into a task tree representation. To enhance the accuracy and
execution cost of the task tree, we integrated the output of the fine-tuned
model with FOON, exploring multiple possibilities to achieve the desired
objectives. Through our experiments, we demonstrated its superior performance,
highlighting its remarkable generalization capabilities. In future, we intend
to focus on addressing the challenges of task tree correction and re-planning
in cases of planning or execution failures. It is worth noting that our
pipeline exhibits a high degree of flexibility, allowing for the seamless
substitution of GPT and FOON with more advanced Language Models or knowledge
networks. We aim to incorporate image inputs into our system by utilizing the
newly released GPT-4, which can handle both textual questions and accompanying
images. This would allow users to upload images of dishes and inquire about
their preparation methods.

## References

  * [1] D. Paulius, Y. Huang, R. Milton, W. D. Buchanan, J. Sam, and Y. Sun.  Functional Object-Oriented Network for Manipulation Learning.  In Intelligent Robots and Systems (IROS), 2016 IEEE/RSJ International Conference on, pages 2655–2662. IEEE, 2016.
  * [2] Md Sadman Sakib, Hailey Baez, David Paulius, and Yu Sun.  Evaluating Recipes Generated from Functional Object-Oriented Network.  arXiv preprint arXiv:2106.00728, 2021.  (Featured in 18th International Conference on Ubiquitous Robots (UR 2021)).
  * [3] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, T. J. Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeff Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei.  Language models are few-shot learners.  ArXiv, abs/2005.14165, 2020.
  * [4] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.  arXiv preprint arXiv:1810.04805, 2018.
  * [5] Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le.  Finetuned language models are zero-shot learners.  ArXiv, abs/2109.01652, 2021.
  * [6] Kota Takata, Takuya Kiyokawa, Ixchel G. Ramirez-Alpizar, Natsuki Yamanobe, Weiwei Wan, and Kensuke Harada.  Efficient task/motion planning for a dual-arm robot from language instructions and cooking images.  In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 12058–12065, 2022.
  * [7] Ross A. Knepper, Todd Layton, John Romanishin, and Daniela Rus.  IkeaBot: An autonomous multi-robot coordinated furniture assembly system.  In 2013 IEEE International Conference on Robotics and Automation, pages 855–862, 2013.
  * [8] Yizhak Ben-Shabat, Xin Yu, Fatemeh Sadat Saleh, Dylan Campbell, Cristian Rodriguez-Opazo, Hongdong Li, and Stephen Gould.  The IKEA ASM Dataset: Understanding People Assembling Furniture through Actions, Objects and Pose.  2021 IEEE Winter Conference on Applications of Computer Vision (WACV), pages 846–858, 2021.
  * [9] Alexandros Vassiliades, Nikos Zarkadas, Nick Bassiliades, and Theodore Patkos.  Onto-IKEA: A Knowledge Retrieval Framework Based on IKEA Ontology.  In JOWO, 2021.
  * [10] FOON Website: Graph Viewer and Videos.  http://www.foonets.com, 2021.  Accessed: 2023-06-06.
  * [11] Bernard Ghanem Fabian Caba Heilbron, Victor Escorcia and Juan Carlos Niebles.  Activitynet: A large-scale video benchmark for human activity understanding.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 961–970, 2015.
  * [12] Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and Michael Wray.  Scaling Egocentric Vision: The EPIC-KITCHENS Dataset.  In European Conference on Computer Vision (ECCV), 2018.
  * [13] D. Paulius, K. S. P. Dong, and Y. Sun.  Task Planning with a Weighted Functional Object-Oriented Network.  In 2021 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2021.
  * [14] Md. Sadman Sakib, David Paulius, and Yu Sun.  Approximate task tree retrieval in a knowledge network for robotic cooking.  IEEE Robotics and Automation Letters, 7:11492–11499, 2022.
  * [15] M. Beetz, D. Beßler, Andrei Haidu, M. Pomarlan, A. Bozcuoğlu, and Georg Bartels.  KnowRob 2.0 — A 2nd Generation Knowledge Processing Framework for Cognition-Enabled Robotic Agents.  2018 IEEE International Conference on Robotics and Automation (ICRA), pages 512–519, 2018.
  * [16] Michael Beetz, Ulrich Klank, Ingo Kresse, Alexis Maldonado, Lorenz Mösenlechner, Dejan Pangercic, Thomas Rühr, and Moritz Tenorth.  Robotic roommates making pancakes.  In 2011 11th IEEE-RAS International Conference on Humanoid Robots, pages 529–536, 2011.
  * [17] Angel Andres Daruna, Lakshmi Nair, Weiyu Liu, and S. Chernova.  Towards robust one-shot task execution using knowledge graph embeddings.  2021 IEEE International Conference on Robotics and Automation (ICRA), pages 11118–11124, 2021.
  * [18] Angel Andres Daruna, Devleena Das, and S. Chernova.  Explainable knowledge graph embedding: Inference reconciliation for knowledge inferences supporting robot actions.  2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1008–1015, 2022.
  * [19] Chao Zhao, Shuai Yuan, Chunli Jiang, Junhao Cai, Hongyu Yu, Michael Wang, and Qifeng Chen.  Erra: An embodied representation and reasoning architecture for long-horizon language-conditioned manipulation tasks.  IEEE Robotics and Automation Letters, PP:1–8, 06 2023.
  * [20] Wenlong Huang, P. Abbeel, Deepak Pathak, and Igor Mordatch.  Language models as zero-shot planners: Extracting actionable knowledge for embodied agents.  ArXiv, abs/2201.07207, 2022.
  * [21] Dhruv Shah, Blazej Osinski, Brian Ichter, and Sergey Levine.  Lm-nav: Robotic navigation with large pre-trained models of language, vision, and action.  In Conference on Robot Learning, 2022.
  * [22] Andy Zeng, Adrian S. Wong, Stefan Welker, Krzysztof Choromanski, Federico Tombari, Aveek Purohit, Michael S. Ryoo, Vikas Sindhwani, Johnny Lee, Vincent Vanhoucke, and Peter R. Florence.  Socratic models: Composing zero-shot multimodal reasoning with language.  ArXiv, abs/2204.00598, 2022.
  * [23] Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alexander Herzog, Daniel Ho, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Eric Jang, Rosario Jauregui Ruano, Kyle Jeffrey, Sally Jesmonth, Nikhil Jayant Joshi, Ryan C. Julian, Dmitry Kalashnikov, Yuheng Kuang, Kuang-Huei Lee, Sergey Levine, Yao Lu, Linda Luu, Carolina Parada, Peter Pastor, Jornell Quiambao, Kanishka Rao, Jarek Rettinghouse, Diego M Reyes, Pierre Sermanet, Nicolas Sievers, Clayton Tan, Alexander Toshev, Vincent Vanhoucke, F. Xia, Ted Xiao, Peng Xu, Sichun Xu, and Mengyuan Yan.  Do as i can, not as i say: Grounding language in robotic affordances.  In Conference on Robot Learning, 2022.
  * [24] Kevin Lin, Christopher Agia, Toki Migimatsu, Marco Pavone, and Jeannette Bohg.  Text2motion: From natural language instructions to feasible plans.  arXiv preprint arXiv:2303.12153, 2023.
  * [25] Ishika Singh, Valts Blukis, Arsalan Mousavian, Ankit Goyal, Danfei Xu, Jonathan Tremblay, Dieter Fox, Jesse Thomason, and Animesh Garg.  ProgPrompt: Generating situated robot task plans using large language models.  In International Conference on Robotics and Automation (ICRA), 2023.
  * [26] OpenAI.  Chatgpt.  https://openai.com/, 2021.  Accessed on 8th June 2023.
  * [27] Yongqiang Huang, Juan Wilches, and Yu Sun.  Robot gaining accurate pouring skills through self-supervised learning and generalization.  Robotics and Autonomous Systems, 136:103692, 2021.
  * [28] Javier Marin, Aritro Biswas, Ferda Ofli, Nicholas Hynes, Amaia Salvador, Yusuf Aytar, Ingmar Weber, and Antonio Torralba.  Recipe1M+: A Dataset for Learning Cross-Modal Embeddings for Cooking Recipes and Food Images.  IEEE Trans. Pattern Anal. Mach. Intell., 2019.
  * [29] David Paulius, Alejandro Agostini, and Dongheui Lee.  Long-horizon planning and execution with functional object-oriented networks.  2022\.
  * [30] Drew McDermott, Malik Ghallab, Adele E. Howe, Craig A. Knoblock, Ashwin Ram, Manuela M. Veloso, Daniel S. Weld, and David E. Wilkins.  Pddl-the planning domain definition language.  1998\.