# Instruction Following with Goal-Conditioned Reinforcement Learning in
Virtual Environments

Zoya Volovikova,1,2 Alexey Skrynnik,1,3 Petr Kuderov,1,2 Aleksandr I.
Panov1,2,3

###### Abstract

In this study, we address the issue of enabling an artificial intelligence
agent to execute complex language instructions within virtual environments. In
our framework, we assume that these instructions involve intricate linguistic
structures and multiple interdependent tasks that must be navigated
successfully to achieve the desired outcomes. To effectively manage these
complexities, we propose a hierarchical framework that combines the deep
language comprehension of large language models with the adaptive action-
execution capabilities of reinforcement learning agents. The language module
(based on LLM) translates the language instruction into a high-level action
plan, which is then executed by a pre-trained reinforcement learning agent. We
have demonstrated the effectiveness of our approach in two different
environments: in IGLU, where agents are instructed to build structures, and in
Crafter, where agents perform tasks and interact with objects in the
surrounding environment according to language commands.

## Introduction

The ability to solve complex tasks, formulated in natural language, that
require a long sequence of actions in the environment is a fundamental
property of human intelligence. The recent significant success of large
language models (LLMs) in instruction following and explanation generation
demonstrates their powerful capabilities in solving commonsense, general
knowledge, and code generation problems within the verbal domain. However,
success rate of multi-step task completion for autonomous agents driven by
general purpose LLMs is still low [Liu et al., 2024]. Moreover, LLMs are often
trained solely on textual data, which limits their ability to understand and
perform actions in real-world-like environments. Consequently, even ChatGPT
[Ouyang et al., 2022] exhibits poor spatial reasoning [Bang et al., 2023]. On
the other hand, reinforcement learning (RL) has proven effective in learning
sequences of fine-grained actions for specific tasks within an environment.
Thus, investigating the combination of LLMs for natural language understanding
and high-level planning, along with RL for learning environmental
manipulation, represents a promising research direction.

Figure 1: The task of collaborative interaction between the agent, the
environment, and the user involves the following: the user provides
instructions to the agent, and the agent executes actions within the
environment to accomplish the task based on these instructions.

LLMs can be regarded as universal knowledge bases that allow human users to
interact in natural language and solve complex tasks [Huang et al., 2022, Zeng
et al., 2022]. Recent studies have shown that pre-trained LLMs can construct
high-level action plans in both simulated and real environments [Bara et al.,
2021, Min et al., 2022, Murray and Cakmak, 2022]. However, these LLM-based
approaches necessitate manual prompt engineering, handcrafted translation of
language commands into embodied actions, and a strategy for goal-aware action
selection from the distribution of potential options generated by language. In
this context, several studies [Ahn et al., 2022, Ouyang et al., 2022] have
demonstrated that the rough action plans extracted from language models can be
refined using RL.

In our research, we present the hierarchical framework IGOR (Instruction
Following with Goal-Conditioned RL), which combines the capabilities of large
language models for understanding complex natural language constructions with
RL-based policies to develop effective behavior in the embodied environment.
The framework relies on two main components: a Language Module, which
translates instructions in natural language into a high-level action plan with
generated subgoals, and a Policy Module, tasked with executing this plan and
achieving subgoals.

Furthermore, we introduce independent learning strategies for each module. We
have developed efficient learning strategies for the LLM for limited datasets.
Some of these strategies are based on data augmentation using the ChatGPT
model, while others rely on subdividing subtasks by altering data formats and
decomposing these subtasks into primitives. We set a learning task for the RL
agents based on goals and curriculum. This approach promotes highly efficient
task execution in dynamic environments beyond the training sample.

The effectiveness of our approach was rigorously tested in two embodied
environments. The first is IGLU [Zholus et al., 2022], where the ’Builder’
agent constructs structures based on natural language instructions from the
’Architect’ agent (see an example in Fig. 1). The second is the Modified
Crafter environment, where the instruction-following agent needs to adapt to
dynamically changing environments. Our results demonstrate that the
application of our method not only outperforms the algorithms presented in the
NeurIPS IGLU competition111https://www.iglu-contest.net [Kiseleva et al.,
2022b, a, 2023], but also surpasses the baselines based on Dreamer-v3 in
Crafter environment [Hafner, 2022].

The main contributions222Our code is available at https://github.com/AIRI-
Institute/IGOR of our paper are:

  1. 1.

We proposed a novel task decomposition approach that facilitates the
incorporation of augmentations, curriculum learning, and potentially other
techniques to multi-modal setups involving LLM and RL learning for virtual
environments.

  2. 2.

In addition to the known IGLU environment, we presented a modified Crafter
environment by introducing a textual modality and prepared a corresponding
dataset to support this enhancement.

  3. 3.

We conducted extensive experiments to compare our approach with other methods
in both the Crafter and IGLU environments, demonstrating significant
improvements.

## Related work

Planing with LLM. Recent studies are actively exploring the generation of
action plans using language models. Some works focus on prompt engineering for
effective plan generation [Singh et al., 2022], while others address the
challenge of translating the language model’s output into executable actions
[Huang et al., 2022]. In [Ahn et al., 2022], models are trained with RL for
selecting feasible actions and executing elementary actions. Many of these
works [Shridhar et al., 2020, Lynch et al., 2023] aim to control robots
interactively in real time. Maintaining a dialogue with humans is an essential
area of research for robotics [Padmakumar et al., 2022, Gao et al., 2022].

Language Grounding Problem. The language grounding problem in intelligent
agents involves linking objects across modalities, such as matching textual
instructions with objects in virtual environments. Methods to address this
include using CLIP for visual-textual links [Radford et al., 2021], cross-
attention mechanisms, and compressing data into hidden subspaces, exemplified
by Dynalang [Lin et al., 2023]. Some strategies integrate language processing
with reinforcement learning, using text embeddings as observations [Hafner et
al., 2023, Lin et al., 2023]. Others connect textual descriptions to
environmental entities using transformer models like Emma and EmBERT [Wang and
Narasimhan, 2021, Suglia et al., 2021]. Additionally, some approaches use
multiple modules trained independently, with pre-trained language models
aiding in planning and adapting actions, addressing the lack of real-world
experience [Huang et al., 2022, Brohan et al., 2023, Li et al., 2022].
Innovatively, models like JARVIS-1 combine pre-trained memory blocks with
tools like CLIP, enhancing multimodal memory and scheduling [Wang et al.,
2023, Fan et al., 2022].

Embodied environments. In the field of embodied reinforcement learning,
several platforms have been developed to train agents based on text
instructions. Among these, AI2Thor [Kolve et al., 2017] and Habitat [Szot et
al., 2021] , offer tasks that are simple and adhere to strict rules, which
simplifies the process of linking actions to text using straightforward
syntactic structures (Messenger [Wang and Narasimhan, 2021], HomeGrid [Lin et
al., 2023], TWOSOME [Tan et al., 2024]).

Furthermore, advancements have been made to enhance the Crafter [Hafner, 2022]
environment, resulting in the creation of the Text-Crafter [Du et al., 2023]
version. Similarly, the MineDojo [Fan et al., 2022] platform, which is based
on Minecraft, has been introduced. These platforms are designed for more
intricate linguistic and planning tasks. Additionally, the IGLU [Kiseleva et
al., 2023] environment stands out for its complexity. In IGLU, agents must
follow detailed instructions to construct structures within a virtual world.
These environments are characterized by a vast state space and involve complex
tasks that are formulated by humans.

## IGOR: Follow Instruction with Goal-based RL

Figure 2: The IGOR framework has three modules: a Language module that solves
language understanding problems and provides a high-level plan of subtasks, a
Task Manager that encodes the subtasks for the Policy module, and a Policy
module that executes actions in the environment based on visual observations
and subtasks. Figure 3: The diagram displays the IGOR system, where the
"Language Module" transforms text instructions into subtasks. The "Task
Manager" coordinates the subtasks and monitors their execution. The "Policy
Module" operates in a virtual environment based on the subtasks. Dotted lines
indicate the training process of the modules, while solid lines show how the
modules interact during inference.

The IGOR framework is designed to solve the challenges of natural language
understanding and instruction execution within virtual environments, enabling
the processing of instructions that contain complex linguistic structures and
specific terminologies. The virtual environments in which the intelligent
agent operates are characterized by extensive observation areas and require
the execution of multiple interconnected tasks. The framework is composed of
three key modules.

The Language Module, implemented using Large Language Models (LLM), analyzes
incoming instructions and converts them into a high-level plan consisting of a
set of specific subtasks that need to be executed.

The Policy Module, implemented using reinforcement learning methods based on
the Proximal Policy Optimization (PPO) algorithm, is responsible for the
strategic interaction in the environment, including the execution of the
interconnected tasks.

The Task Manager acts as a wrapper over the virtual environment, transforming
the list of subtasks provided by the Language Module into a format
understandable to the Policy Module. This module also ensures that the tasks
specified in the instructions are completed and concludes the episode after
their execution.

Thus, the inference pipeline operates by initially receiving instructions
which are processed by the Language Module, where they are translated into a
series of subtasks — a high-level execution plan. Subsequently, the Policy
Module executes the necessary actions in the virtual environmrnt to achieve
the objectives outlined in the instructions. An example of using our pipeline
can be found in the Figure ( 2)

Training for each learnable module is conducted separately, which provides
flexibility in the integration of training methods and techniques. The
training of the Language Module involves various augmentations and
modifications to the dataset to prevent overfitting. The Policy Module is
trained using a goal-based approach, which allows for training on a broader
set of potential objectives than those available in the initial data. The
inclusion of a curriculum in the training process also significantly enhances
the quality of the final agent.

### Language Module Training Techniques

The core of our language module utilizes a large pre-trained language model,
which has been further finetuned on a dataset specific to the environment.
This dataset contains information on how instructions translate into specific
subtasks, enabling the model to understand and decompose complex commands
effectively. The training leverages a specific format that maps instructions
to their corresponding subtasks (e.g., Instruction -> Subtask 1, Subtask 2,
Subtask 3).

Due to the difficulty of obtaining comprehensive datasets for training models
to translate language instructions into commands, we face additional
challenges. Typically, datasets for training large language models (LLMs) on
such tasks are manually curated, which is a labor-intensive process. This
often limits both the size and the quality of the datasets available. In
response, we employ techniques to prevent overfitting, especially when working
with these limited datasets. Our experiments demonstrate that these methods
effectively enhance training quality by ensuring the model can generalize well
from smaller, varied linguistic datasets, leading to a more robust
understanding of instructions.

Augmentation with LLM. In this technique, we begin by understanding the
structure and specific terminology of the dataset. Our approach involves
iteratively modifying the list of subtasks necessary to execute a given
instruction for each dataset element. Subsequently, ChatGPT or another LLM is
tasked with rewriting the instruction to incorporate these modified subtasks.

The prompt request is structured as follows:

1) Description of the Environment: Provide the LLM with a detailed
understanding of the setting by explaining the overarching themes and key
specific concepts. Use succinct descriptions for familiar ideas and detailed
explanations for unique aspects.

2) Few-shot Example: Introduce the original instruction alongside its required
subtasks. Optionally, you can also provide an example of how the instruction
might change if the subtasks are altered.

3) Task Modification Request: Specify new subtasks for the target instruction
and request the LLM to revise the instruction accordingly.

It is important to emphasize that the LLM is not creating the instruction from
scratch. We aim to start with the existing instruction and suggest
modifications, ensuring that the style of the original instruction is
preserved as much as possible.

Subtasks decomposition. The second technique entails decomposing original
subtasks into "primitives", which involves modifying the structure and format
of subtasks in the dataset. Essentially, it suggests consolidating frequently
co-occurring subtasks into a single common subtask, thereby reducing the data
volume required for processing by the language model. These aggregated
subtasks are termed "primitives".

We explore two methods for creating primitives. One method utilizes unique
tokens, such as emojis, to encode each primitive. Emojis are chosen for their
diverse range, making them a convenient means of representing a broad array of
subtasks. The second method involves crafting primitives in a manner that
aligns logically with the content of the instructions. This approach aims to
enhance the coherence between the instructional context and the subtask
structure and is the approach employed in our experiments.

### Task Manager

The Task Manager serves as an intermediary to connect the Language Module and
the Policy Module during execution and to allocate subtasks from the dataset
to the Policy Module throughout its training phase. It retrieves a list of
subtasks and systematically supplies these to the RL agent as components of
its observational input. Based on observations collected during the agent’s
interaction with the environment, the Task Manager determines whether a
subtask has been successfully completed and decides whether to proceed to the
next task or to conclude the episode. Upon successful completion of a subtask,
the Task Manager assigns a positive reward ($r=+1$) to reinforce the agent’s
behavior.

### Training the Policy Module

Instead of training a RL agent to tackle all subtasks required by a complex
instruction simultaneously, we propose a goal-based learning approach. In this
approach, each episode presents the agent with an observation and only one of
the subtasks from the instruction. By doing this, we simplify the agent’s
task, reducing complexity of the task and facilitating more focused learning
on specific aspects of the overall problem.

We examine a visual-based reinforcement learning environments characterized as
a Partially Observable Markov Decision Process (POMDP). In this framework, the
observation function is augmented by subtask encodings provided by the Task
Manager. The primary objective of the agent is to develop a policy $\pi$ that
selects actions to maximize the expected cumulative reward. To achieve this,
we employ policy gradient methods, specifically the Proximal Policy
Optimization (PPO) algorithm. PPO has been demonstrated to offer substantial
robustness across various tasks, attributed to its effective balancing of
exploration and exploitation by optimizing a clipped surrogate objective
function
$\mathbb{E}\left[\min(\rho_{t}(\theta)\hat{A}_{t},\text{clip}(\rho_{t}(\theta),1-\epsilon,1+\epsilon)\hat{A}_{t})\right]$,
where
$\rho_{t}(\theta)=\frac{\pi_{\theta}(a_{t}|s_{t})}{\pi_{\text{old}}(a_{t}|s_{t})}$
and $\hat{A}_{t}$ denotes the advantage estimate at time $t$ [Schulman et al.,
2017].

During training we utilize curriculum task sampler that dynamically adjusts
the probability of selecting subtasks based on their performance, inspired by
curriculum learning approaches [Matiisen et al., 2019, Nesterova et al.,
2022]. Specifically, the selection probability for each task $i$ in
$\mathcal{T}$ is modified according to:

$q_{i}=\begin{cases}\frac{1}{d}&\text{if }r_{i}\geq\tau\\\ 1+(\delta_{i}\cdot
d)&\text{if }r_{i}<\tau\end{cases}$

where $r_{i}$ is the task’s average reward, $\delta_{i}$ measures the
variability in reward, $d$ is a scaling coefficient, and $\tau$ is a success
threshold. Probabilities are normalized using a softmax function to form a
distribution from which tasks are sampled. A detailed description of the
algorithm, its pseudocode and ablation study can be found in the Appendix.

## Experimental Setup

To investigate and test the capabilities of intelligent agents, we have chosen
environments with high combinatorial complexity. These environments allow us
to assess how agents cope with tasks requiring the execution of many
interrelated subtasks to achieve a target state.

The first environment is IGLU, where the agent’s task is to build three-
dimensional constructions based on textual descriptions. It is important to
note that the complexity of the environment largely lies in the fact that,
depending on the instructions, there can be a vast number of potential target
states. To be successful in such an environment, an agent must possess
advanced text interpretation skills, as well as the ability to think spatially
and model ordering to adequately recreate the required structures.

The second environment is Crafter, where the agent needs to follow textual
instructions to perform a variety of tasks, such as gathering resources and
crafting items. This environment tests the agent’s ability to understand
natural language and effectively plan sequences of actions in response to
changing conditions.

Below is the general pipeline for applying our approach to these virtual
environments:

  1. 1.

Fine-tune the LLM: Fine-tune the large language model (LLM) with an
environment-specific dataset to translate instructions into subtasks for the
reinforcement learning (RL) agent. Add environment-specific techniques if
needed: augmentation, data modification, and dataset expansion.

  2. 2.

Train the RL agent: Train the RL agent in a goal-based mode on environment-
relevant subtasks. If needed, add environment-specific techniques: curriculum
learning, hierarchical RL, reward shaping, and curiosity-driven exploration.

  3. 3.

Measure performance: Measure performance on a test dataset for each
environment. The LLM predicts subtasks from instructions, which the trained RL
agent then executes.

### IGLU Environment

#### Environment.

IGLU is an environment333https://github.com/iglu-contest/gridworld where an
embodied agent can build spatial structures (figures) of blocks of different
colors. The agent’s goal is to complete a task expressed as an instruction
written in natural language.

The observation space consists of point-of-view (POV) image $(64,64,3)$,
inventory item counts $(6)$, and the pitch and yaw angles $(5)$. The agent can
navigate over the building zone, place, and break blocks, and switch between
block types. Additionally, the environment provides a textual observation in
the form of a dialogue from the dataset, which defines a building task. The
examples of such target tasks is presented in Fig. 4.

Figure 4: IGLU is a 3D environment where agents are tasked with constructing
structures in a designated area, guided by descriptions provided in natural
language and the agent’s first person perspective.

Target utterances define the rest of the blocks needed to be added. The
environment provides two modes of action for choice: walking and flying. In
our experiments, we create agents for use in both flying and walking modes.
The action space combines discrete and continuous subspaces: a discrete space
of 13 actions (noop, four camera rotation actions, break block, place block,
choose block type 1-6) and a set of 6 continuous actions for movement in all
three directions.

#### Metrics.

We employed the F1 metric, as proposed in the IGLU competition, to evaluate
the quality of the approaches.

#### Baselines.

We have selected the three best solutions from the IGLU 2022 competition for
comparison. The second and third-place solutions, which utilized T5 [Raffel et
al., 2020] and Pegasus [Zhang et al., 2020] models respectively, are based on
the organizers’ proposed solution but differ in the NLP models employed. We
also include the solution of the first-place team, called
BrainAgent444https://github.com/kakaobrain/brain-agent, whose approach differs
significantly from the others. They used end-to-end learning, with the RL
agent taking the embedding of a frozen NLP model as input, along with
environment information and manually added features.

#### Dataset.

The training and testing datasets contain 109 and 41 English instructions,
respectively, with corresponding grids that need to be constructed. Each
instruction consists of individual building steps with a corresponding voxel
for each step.

For training the language model, we transform voxel shapes into text — a list
of block coordinates that need to be placed. Grids are converted into a
sequence of block coordinates and colors in the form of a string of tuples.
For this dataset, a single block (its coordinates and a color) represents a
single subtask in our terminology.

The order of the blocks in the sequence corresponds to the appearance of the
blocks in the grid and have the following ordering rule: first by $x$, then by
$z$, and then by $y$, where $x$ and $z$ are horizontal axes and $y$ is
vertical axis. In many instructions, the horizontal position of the first
block is not specified, such that it can appear anywhere along $x$ and $z$
randomly. This is detrimental to the model during training as it will lead to
high loss values for tokens that cannot be derived from the input data.
Therefore, we change the sequence so that the first block of the first step of
the instruction is placed at the center of the space, and the other blocks are
shifted accordingly.

Figures Classification

Based on the capabilities an AI agent needs to construct figures in a dataset,
we have developed our own rule-based classification of figures. This helps to
more precisely determine which types of figures pose challenges and which
descriptions of figure types might cause difficulties in construction by a
language model. For visual examples, see Appendix. The classes, defined by
their spatial characteristics, are presented below:

  * •

flat figures are characterized primarily by their length and width, with
minimal emphasis on height.

  * •

tall figures prioritize height over both length and width.

  * •

air figures are distinguished by their lack of contact with the ground,
signifying that they are airborne.

  * •

floor figures are defined by their complete contact with the ground, resting
entirely on the surface.

#### Training details.

The Language Module for the IGLU environment was trained to predict, based on
dialogue instructions, a list of coordinates of blocks that need to be placed
or removed, where each coordinate corresponds to a subtask.We created a second
dataset with a modified format of subtasks, grouping blocks that occur
together in instructions into parallelepipeds. An example of the subtasks
format for the IGLU dataset can be found in 1. Additionally, we augmented the
dataset using ChatGPT, where we tasked the model to rotate the object by 180
degrees. The modification we applied for ChatGPT augmentation involved
rotating the object by 180 degrees. We requested the LLM to adjust the
building instructions to match this modification, ensuring the textual
description aligned with the altered figure orientation.

As the base for the Language Module, the Flan-T5 base model was used. The LLM
models are trained in a seq2seq manner, converting the instruction into a
sequence of blocks as described above. Training hyperparameters and hardware
parameters are listed in the appendix F.

Table 1: Example of coords and prims subtask formats in the IGLU environment, corresponding to the instruction: "<Architect> place 5 red blocks in a row, one row north of center." Format |  Description |  Example
---|---|---
coords |  (x, y, z, colorID):  • x, y, z: coordinates • colorID: block color id |  (0, 5, 5, 3), (0, 6, 5, 3), (0, 7, 5, 3), (0, 8, 5, 3), (0, 9, 5, 3)
prims |  (start), (size), rotation, color:  • (start): initial block (x, y, z) • (size): dimensions (x, y, z) • rotation: alignment • color: block color name |  (0, 5, 5), (1, 1, 5), eastsky, red

For training Policy Module in the IGLU environment, our objective is to train
the agent to either place or remove a single block on a virtual field. At the
beginning of an episode, a list of subtasks is randomly generated or taken
from a training dataset. A specific block from this list is then selected as
the target. Depending on this selection, we generate the initial state of the
environment by positioning the blocks leading up to the target block.

The reward mechanism used during training is the same as proposed by the
competition’s authors.

To enhance the quality of training for the RL agent, we incorporate a
curriculum that controls the complexity of the list of subtasks. This
structured approach helps in progressively challenging the agent, thus
improving its learning efficiency and capability to handle complex tasks.

Solving visual goal-based tasks can be accomplished by a wide range of modern
RL approaches, both model-free [Berner et al., 2019, Espeholt et al., 2018,
Petrenko et al., 2020] and model-based [Hafner, 2022, Hafner et al., 2023]. In
this work, we used an open-source implementation of asynchronous
PPO555https://github.com/alex-petrenko/sample-factory [Petrenko et al., 2020],
which demonstrated better performance in terms of GPU hours to final score
based on our preliminary experiments.

### Crafter Environment

Figure 5: Crafter is a 2D environment reminiscent of Minecraft, where players
must gather food and water, acquire resources, fend off creatures, and
construct tools.

#### Environment.

In the original Crafter environment666https://github.com/danijar/crafter,
agents are primarily tasked with exploration, gathering resources, and
battling monsters. We have adapted this framework so that each episode now
provides a free-form textual instruction, directing the agent to perform the
number of specific tasks. In our modified version, each task given to the
agent corresponds exactly to one achievement in the original game. Thus,
within a single episode, following these instructions means completing
individual achievements directly tied to the game’s original goals. The
examples of the agent in Crafter environment is presented in Fig. 5.

#### Dataset.

To create a dataset of textual tasks for the agent in the Crafter environment,
we followed a specific pipeline: First, based on the known list of tasks and
achievements in the crafter environment, we randomly generated a list of
subtasks. Second, to obtain descriptive instructions for these tasks, we used
Mistral-7B-Instruct777https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.1.
The example of such prompt and response is presented in Fig.6. Third, to
obtain more varied descriptions of instructions, using the same Mistral model
to stylize the received instructions (for example of such reformulation see
Fig. 7).

Rewrite the list of subtasks into one instructive sentence for an agent in a
virtual environment, replacing words from the list with synonyms. Subtasks:
["Defeat Zombie", "Collect Iron with count 1", "Make Iron Sword"]

Vanquish the undead foe, gather a single unit of metallic mineral, and forge
an iron weapon.

Figure 6: The example depict prompt and response interactions used to generate
the dataset for the modified Crafter environment.

Rewrite the instruction in the style of Tolkien with the same SHORT sentence
size adding causality. Instruction: Vanquish the undead foe, gather a single
unit of metallic mineral, and forge an iron weapon.

Slay the wights, gather a nugget of mithril, and forge a blade of iron.

Figure 7: This figure shows an examples prompt and response for the
reformulation of an instruction generated by the Mistral-7B-Instruct.

#### Metrics.

To assess the quality of an agent’s performance in an environment where it
follows language instructions, we used a metric called "Success Rate." This
metric measures whether all the subtasks required by the given instructions to
the agent were completed. Accordingly, for one instruction, the metric can be
$1$ if the agent completed all the tasks, or $0$ if any task was not
completed.

#### Baselines.

We compared our IGOR Agent with Dynalang [Lin et al., 2023]. To run Dynalang
on a text-adapted Crafter, we employ the T5 model to encode the text,
mirroring the approach taken by the paper’s authors. In each episode, we
sequentially transmit all instructions and images to Dynalang, one token at a
time. This process allows us to evaluate Dynalang’s ability to identify the
necessary subtasks from the instructions, aligning with our methodology. We
adopt the same reward system used in our experiments.

#### Training details.

A Language module for the crafting environment was trained on the provided
data without any modifications. Following the instructions within this
environment, the module is tasked with determining which achievements the
agent should collect in the virtual setting. Additionally, the Flan-T5-base
model was utilized as a reference.

For Policy Module we use self-supervised goal sampling to train the agent on
these subtasks efficiently. In this approach, each training episode for the
agent targets a distinct goal selected randomly, aligning with the goals
possibly derived from the LLM. The reward function for the RL agent is
designed to provide positive feedback for the successful completion of
subtasks, motivating the agent to achieve its goals. This function uses both
the original Crafter environment reward and an extra reward for the completion
of a subtask by the agent. The original reward is scaled by a factor of $0.1$,
and the extra reward for subtasks accomplishing is set to $1$. Through this
process, the agent learns to solve individual subtasks, contributing to the
overall action plan devised by the LLM.

## Experimental Results

In this section, we compare our approach with other state-of-the-art methods.
The section is organized as follows: First, we present the results in the
modified Crafter environment. Second, we show the results for the IGLU
environment, alongside ablation experiments for different data processing
techniques.

### Crafter Environment

The results of IGOR and Dynalang in the Crafter environment are shown in the
bar chart presented in Fig. 8. The x-axis displays various categories, where
the Total category represents the overall success rate across all test
instructions. The other categories illustrate the agents’ performance on
specific subtasks within the instructions, highlighting how effectively each
agent solves individual subtasks when attempting to complete the entire
instruction. For instance, a low value for a particular subtask on the plot
indicates that the agent often fails to complete the instructions when that
subtask is involved.

Figure 8: A comparison of the performance of IGOR and Dynalang approaches, on
the Crafter environment with textual tasks. The success rate metric for each
subtask is averaged across instructions requiring that specific subtask. Total
bar represents the overall success rate for all instructions of test dataset.

Overall, the total score of IGOR is $0.6$, indicating that the agent
successfully completes all required subtasks in $60\%$ of cases, significantly
outperforming Dynalang, which only achieves $36.4\%$. Our approach
demonstrates superior performance in $19$ out of $22$ subtasks. Furthermore,
it achieves results that are at least twice as good in $8$ out of $22$ types
of instructions. Additionally, our approach, with the integration of
curriculum learning, shows strong capabilities in managing tasks that are
especially challenging, such as Collect Diamond, Collect Iron, Make Iron
Sword, and Make Iron Pickaxe.

### IGLU Environemnt

The comparative results in the IGLU environment are presented in Table 2. We
ran the IGOR approach in both Flying and Walking modes. Additionally, we
conducted experiments with different data variants for training the LLM agent,
utilizing both primitives and coordinates as subtasks. Across all
configurations, our method consistently outperforms the competitors, achieving
significantly better results in all figure categories. Notably, IGOR’s lowest
score is $0.45$, which surpasses the overall performance of the winning team,
BrainAgent, which scored $0.36$. For simpler figures such as Floor using
primitives, our agent scores $0.75$ compared to BrainAgent’s $0.53$. A clear
advantage is also observed for more complex tasks such as Air, where IGOR
scores $0.46$ versus $0.25$. Even when using coordinates as subtasks, our
agent’s advantage, although reduced, persists across all figure types.

Table 2: A comparative analysis of the F1 score (higher is better) for the IGOR approach versus the IGLU-2022 competition winners on the test dataset. Approach | Prim | Flying | Floor | Flat | Tall | Air | Total
---|---|---|---|---|---|---|---
IGOR | ✓ | ✓ | 0.72 | 0.51 | 0.46 | 0.40 | 0.52
IGOR | $\times$ | ✓ | 0.68 | 0.44 | 0.36 | 0.34 | 0.45
IGOR | ✓ | $\times$ | 0.75 | 0.46 | 0.32 | 0.33 | 0.46
IGOR | $\times$ | $\times$ | 0.68 | 0.42 | 0.29 | 0.31 | 0.45
BrainAgent | n\a | $\times$ | 0.53 | 0.36 | 0.23 | 0.25 | 0.36
MHB | n\a | $\times$ | 0.08 | 0.05 | 0.03 | 0.04 | 0.05
Pegasus | n\a | $\times$ | 0.08 | 0.06 | 0.02 | 0.04 | 0.06

#### Comparing Data Processing Technics.

We evaluated various data processing strategies for training NLP models, as
detailed in Table 3. These strategies included expanding the original dataset
(strategy "Coords"), employing the ChatGPT augmentation technique outlined
previously (strategy "ChatGPT"), and modifying the dataset’s format (strategy
"Prim"). We also enhanced the dataset with color augmentation by altering the
color of the block and its corresponding descriptor in the instructions
(strategy "Color").

Table 3:  A comparative analysis of various data processing strategies for LLM training. The table displays the F1 scores, which measure discrepancy between LLM-predicted figures and target figures. The highest scores in each category are highlighted in bold.  Prim |  Color |  ChatGPT | Floor | Flat | Tall | Air | Total
---|---|---|---|---|---|---|---
✓ |  ✓ |  ✓ | 0.75 | 0.51 | 0.44 | 0.36 | 0.50
✓ |  ✓ |  $\times$ | 0.73 | 0.52 | 0.44 | 0.35 | 0.50
✓ |  $\times$ |  $\times$ | 0.69 | 0.42 | 0.34 | 0.25 | 0.42
$\times$ |  ✓ |  ✓ | 0.53 | 0.40 | 0.33 | 0.29 | 0.39
$\times$ |  ✓ |  $\times$ | 0.39 | 0.36 | 0.33 | 0.29 | 0.35
$\times$ |  $\times$ |  $\times$ | 0.45 | 0.35 | 0.30 | 0.28 | 0.34

It can be observed that adaptation to primitives, even in the absence of
augmentations, improves the F1 metric by $0.08$ (from $0.34$ to $0.42$). When
combined with augmentations, the F1 score sees a $0.16$ boost ($0.50$) versus
$0.01$ increase via augmentations alone on coordinates ($0.35$).

While the ChatGPT augmentations don’t yield a significant increase on
primitives, an analysis of the F1 scores across different shape categories
reveals their effectiveness. On coordinates, the ChatGPT augmentation
outperforms color augmentations, showing a $0.4$ relative improvement ($0.35$
vs $0.39$).

This suggests that the augmentation strategies have different impact depending
on subtask format. Additionally, it is apparent that the primary challenge for
the language model arises from instructions requiring the prediction of blocks
suspended in the air—these tasks exhibit the highest error rates. This may be
due to the limited number and specific nature of such instructions in the
training dataset.

## Limitations

Exploring previous studies in this area reveals that creating a general
approach for multimodal environments presents a significant challenge [Zhong
et al., 2021]. While many approaches show potential, they often underperform
when applied outside their original domains. Aiming for a universally
applicable method is an admirable goal, yet it currently has limited practical
use.

Our work introduces a new framework that separates the training of RL and LLM
components. The proposed decomposition aims to achieve architectural
flexibility, facilitating the easy integration of various techniques such as
curriculum learning and reward shaping for RL, as well as data augmentation
and human feedback for LLM training. However, our approach requires the
identification of known subtasks, which may not be present in some domains,
thus necessitating additional data collection to identify them.

## Conclusion

In this paper, we introduce the Instruction Following with Goal-Conditioned
Reinforcement Learning in Virtual Environments (IGOR) method, a novel approach
that translates natural language instructions into a sequence of executable
subtasks. The IGOR method integrates two independently trainable modules along
with an intermediary Task Manager module. The Language Module is responsible
for converting textual instructions into a defined sequence of subtasks.
Meanwhile, the RL agent is structured in a goal-conditioned manner, making it
adept at handling a wide array of tasks across various environmental contexts.

The modular decomposition of the IGOR approach allows for the incorporation of
additional training techniques such as data augmentation, goal-based
strategies, and curriculum learning in RL. A detailed analysis of the results
shows that this approach not only allows for more flexible training
customization but also yields a significant improvement in performance.

We demonstrate these advantages through experiments in two environments. In
the IGLU environment, where the agent is required to construct a figure in a
virtual setting based on dialogue with a human, our method surpasses the
winners of the IGLU 2022 competition. Additionally, we have adapted the
Crafter environment to require the agent to achieve specific achievements
based on instructions, showing that our approach outperforms the Dynalang
method based on Dreamer V3.

## References

  * Ahn et al. [2022] Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, et al.  Do as i can, not as i say: Grounding language in robotic affordances.  _arXiv preprint arXiv:2204.01691_ , 2022.
  * Bang et al. [2023] Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, Quyet V. Do, Yan Xu, and Pascale Fung.  A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity, 2023.
  * Bara et al. [2021] Cristian-Paul Bara, CH-Wang Sky, and Joyce Chai.  Mindcraft: Theory of mind modeling for situated dialogue in collaborative tasks.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, 2021.
  * Berner et al. [2019] Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemysław Dębiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, et al.  Dota 2 with large scale deep reinforcement learning.  _arXiv preprint arXiv:1912.06680_ , 2019.
  * Brohan et al. [2023] Anthony Brohan, Yevgen Chebotar, Chelsea Finn, Karol Hausman, Alexander Herzog, Daniel Ho, Julian Ibarz, Alex Irpan, Eric Jang, Ryan Julian, et al.  Do as i can, not as i say: Grounding language in robotic affordances.  In _Conference on robot learning_ , pages 287–318. PMLR, 2023.
  * Du et al. [2023] Yuqing Du, Olivia Watkins, Zihan Wang, Cédric Colas, Trevor Darrell, Pieter Abbeel, Abhishek Gupta, and Jacob Andreas.  Guiding pretraining in reinforcement learning with large language models.  In _International Conference on Machine Learning_ , pages 8657–8677. PMLR, 2023.
  * Espeholt et al. [2018] Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, Iain Dunning, et al.  Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures.  In _International conference on machine learning_ , pages 1407–1416. PMLR, 2018.
  * Fan et al. [2022] Linxi Fan, Guanzhi Wang, Yunfan Jiang, Ajay Mandlekar, Yuncong Yang, Haoyi Zhu, Andrew Tang, De-An Huang, Yuke Zhu, and Anima Anandkumar.  Minedojo: Building open-ended embodied agents with internet-scale knowledge.  _Advances in Neural Information Processing Systems_ , 35:18343–18362, 2022.
  * Gao et al. [2022] Xiaofeng Gao, Qiaozi Gao, Ran Gong, Kaixiang Lin, Govind Thattai, and Gaurav S Sukhatme.  Dialfred: Dialogue-enabled agents for embodied instruction following.  _IEEE Robotics and Automation Letters_ , 7(4):10049–10056, 2022.
  * Hafner [2022] Danijar Hafner.  Benchmarking the spectrum of agent capabilities.  In _International Conference on Learning Representations_ , 2022.  URL https://openreview.net/forum?id=1W0z96MFEoH.
  * Hafner et al. [2023] Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap.  Mastering diverse domains through world models.  _arXiv preprint arXiv:2301.04104_ , 2023.
  * Huang et al. [2022] Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch.  Language models as zero-shot planners: Extracting actionable knowledge for embodied agents.  In _International conference on machine learning_ , pages 9118–9147. PMLR, 2022.
  * Kiseleva et al. [2022a] Julia Kiseleva, Ziming Li, Mohammad Aliannejadi, Shrestha Mohanty, Maartje ter Hoeve, Mikhail Burtsev, Alexey Skrynnik, Artem Zholus, Aleksandr Panov, Kavya Srinet, et al.  Interactive grounded language understanding in a collaborative environment: Iglu 2021.  In _NeurIPS 2021 Competitions and Demonstrations Track_ , pages 146–161. PMLR, 2022a.
  * Kiseleva et al. [2022b] Julia Kiseleva, Alexey Skrynnik, Artem Zholus, Shrestha Mohanty, Negar Arabzadeh, Marc-Alexandre Côté, Mohammad Aliannejadi, Milagro Teruel, Ziming Li, Mikhail Burtsev, et al.  Iglu 2022: Interactive grounded language understanding in a collaborative environment at neurips 2022.  _arXiv preprint arXiv:2205.13771_ , 2022b.
  * Kiseleva et al. [2023] Julia Kiseleva, Alexey Skrynnik, Artem Zholus, Shrestha Mohanty, Negar Arabzadeh, Marc-Alexandre Côté, Mohammad Aliannejadi, Milagro Teruel, Ziming Li, Mikhail Burtsev, et al.  Interactive grounded language understanding in a collaborative environment: Retrospective on iglu 2022 competition.  In _NeurIPS 2022 Competition Track_ , pages 204–216. PMLR, 2023.
  * Kolve et al. [2017] Eric Kolve, Roozbeh Mottaghi, Winson Han, Eli VanderBilt, Luca Weihs, Alvaro Herrasti, Matt Deitke, Kiana Ehsani, Daniel Gordon, Yuke Zhu, et al.  Ai2-thor: An interactive 3d environment for visual ai.  _arXiv preprint arXiv:1712.05474_ , 2017.
  * Li et al. [2022] Shuang Li, Xavier Puig, Chris Paxton, Yilun Du, Clinton Wang, Linxi Fan, Tao Chen, De-An Huang, Ekin Akyürek, Anima Anandkumar, et al.  Pre-trained language models for interactive decision-making.  _Advances in Neural Information Processing Systems_ , 35:31199–31212, 2022.
  * Lin et al. [2023] Jessy Lin, Yuqing Du, Olivia Watkins, Danijar Hafner, P. Abbeel, Dan Klein, and Anca D. Dragan.  Learning to model the world with language.  _ArXiv_ , abs/2308.01399, 2023.  URL https://api.semanticscholar.org/CorpusID:260438420.
  * Liu et al. [2024] Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang.  Agentbench: Evaluating LLMs as agents.  In _The Twelfth International Conference on Learning Representations_ , 2024.  URL https://openreview.net/forum?id=zAdUB0aCTQ.
  * Lynch et al. [2023] Corey Lynch, Ayzaan Wahid, Jonathan Tompson, Tianli Ding, James Betker, Robert Baruch, Travis Armstrong, and Pete Florence.  Interactive language: Talking to robots in real time.  _IEEE Robotics and Automation Letters_ , 2023.
  * Matiisen et al. [2019] Tambet Matiisen, Avital Oliver, Taco Cohen, and John Schulman.  Teacher–student curriculum learning.  _IEEE transactions on neural networks and learning systems_ , 31(9):3732–3740, 2019.
  * Min et al. [2022] So Yeon Min, Devendra Singh Chaplot, Pradeep Kumar Ravikumar, Yonatan Bisk, and Ruslan Salakhutdinov.  FILM: Following instructions in language with modular methods.  In _International Conference on Learning Representations_ , 2022.  URL https://openreview.net/forum?id=qI4542Y2s1D.
  * Murray and Cakmak [2022] Michael Murray and Maya Cakmak.  Following natural language instructions for household tasks with landmark guided search and reinforced pose adjustment.  _IEEE Robotics and Automation Letters_ , 7(3):6870–6877, 2022.
  * Nesterova et al. [2022] Maria Nesterova, Alexey Skrynnik, and Aleksandr Panov.  Reinforcement learning with success induced task prioritization.  In _Mexican International Conference on Artificial Intelligence_ , pages 97–107. Springer, 2022.
  * Ouyang et al. [2022] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al.  Training language models to follow instructions with human feedback.  _Advances in Neural Information Processing Systems_ , 35:27730–27744, 2022.
  * Padmakumar et al. [2022] Aishwarya Padmakumar, Jesse Thomason, Ayush Shrivastava, Patrick Lange, Anjali Narayan-Chen, Spandana Gella, Robinson Piramuthu, Gokhan Tur, and Dilek Hakkani-Tur.  Teach: Task-driven embodied agents that chat.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , pages 2017–2025, 2022.
  * Petrenko et al. [2020] Aleksei Petrenko, Zhehui Huang, Tushar Kumar, Gaurav Sukhatme, and Vladlen Koltun.  Sample factory: Egocentric 3d control from pixels at 100000 fps with asynchronous reinforcement learning.  In _International Conference on Machine Learning_ , pages 7652–7662. PMLR, 2020.
  * Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever.  Learning transferable visual models from natural language supervision.  In _International Conference on Machine Learning_ , 2021.  URL https://api.semanticscholar.org/CorpusID:231591445.
  * Raffel et al. [2020] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _J. Mach. Learn. Res._ , 21(140):1–67, 2020.
  * Schulman et al. [2017] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov.  Proximal policy optimization algorithms.  _arXiv preprint arXiv:1707.06347_ , 2017.
  * Shridhar et al. [2020] Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, and Dieter Fox.  Alfred: A benchmark for interpreting grounded instructions for everyday tasks.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 10740–10749, 2020.
  * Singh et al. [2022] Ishika Singh, Valts Blukis, Arsalan Mousavian, Ankit Goyal, Danfei Xu, Jonathan Tremblay, Dieter Fox, Jesse Thomason, and Animesh Garg.  Progprompt: Generating situated robot task plans using large language models.  _arXiv preprint arXiv:2209.11302_ , 2022.
  * Suglia et al. [2021] Alessandro Suglia, Qiaozi Gao, Jesse Thomason, Govind Thattai, and Gaurav S. Sukhatme.  Embodied bert: A transformer model for embodied, language-guided visual task completion.  _ArXiv_ , abs/2108.04927, 2021.  URL https://api.semanticscholar.org/CorpusID:236975859.
  * Szot et al. [2021] Andrew Szot, Alex Clegg, Eric Undersander, Erik Wijmans, Yili Zhao, John Turner, Noah Maestre, Mustafa Mukadam, Devendra Chaplot, Oleksandr Maksymets, Aaron Gokaslan, Vladimir Vondrus, Sameer Dharur, Franziska Meier, Wojciech Galuba, Angel Chang, Zsolt Kira, Vladlen Koltun, Jitendra Malik, Manolis Savva, and Dhruv Batra.  Habitat 2.0: Training home assistants to rearrange their habitat.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2021.
  * Tan et al. [2024] Weihao Tan, Wentao Zhang, Shanqi Liu, Longtao Zheng, Xinrun Wang, and Bo An.  True knowledge comes from practice: Aligning llms with embodied environments via reinforcement learning.  _ArXiv_ , abs/2401.14151, 2024.  URL https://api.semanticscholar.org/CorpusID:267212097.
  * Wang and Narasimhan [2021] H. J. Austin Wang and Karthik Narasimhan.  Grounding language to entities and dynamics for generalization in reinforcement learning.  _ArXiv_ , abs/2101.07393, 2021.  URL https://api.semanticscholar.org/CorpusID:231639188.
  * Wang et al. [2023] Zihao Wang, Shaofei Cai, Anji Liu, Yonggang Jin, Jinbing Hou, Bowei Zhang, Haowei Lin, Zhaofeng He, Zilong Zheng, Yaodong Yang, Xiaojian Ma, and Yitao Liang.  Jarvis-1: Open-world multi-task agents with memory-augmented multimodal language models.  _arXiv preprint arXiv: 2311.05997_ , 2023.
  * Zeng et al. [2022] Andy Zeng, Adrian Wong, Stefan Welker, Krzysztof Choromanski, Federico Tombari, Aveek Purohit, Michael Ryoo, Vikas Sindhwani, Johnny Lee, Vincent Vanhoucke, et al.  Socratic models: Composing zero-shot multimodal reasoning with language.  _arXiv preprint arXiv:2204.00598_ , 2022.
  * Zhang et al. [2020] Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter Liu.  Pegasus: Pre-training with extracted gap-sentences for abstractive summarization.  In _International Conference on Machine Learning_ , pages 11328–11339. PMLR, 2020.
  * Zholus et al. [2022] Artem Zholus, Alexey Skrynnik, Shrestha Mohanty, Zoya Volovikova, Julia Kiseleva, Artur Szlam, Marc-Alexandre Coté, and Aleksandr I. Panov.  IGLU Gridworld: Simple and Fast Environment for Embodied Dialog Agents.  In _CVPR 2022 Workshop on Embodied AI_ , 2022.  URL http://arxiv.org/abs/2206.00142https://embodied-ai.org/papers/2022/12.pdf.
  * Zhong et al. [2021] Victor Zhong, Austin W Hanjie, Sida I Wang, Karthik Narasimhan, and Luke Zettlemoyer.  Silg: The multi-environment symbolic interactive language grounding benchmark.  _arXiv preprint arXiv:2110.10661_ , 2021.

##  Appendix A Curriculum Learning

We use curriculum learning technique, inspired by principles outlined in
[Matiisen et al., 2019, Nesterova et al., 2022]. The core objective of this
method is to dynamically adjust the selection probabilities of training tasks
to optimize learning progression based on empirical performance measures. The
pseudocode is outlined in Algorithm 1.

Our algorithm commences with an equal likelihood of selecting any given task
from a list of tasks $\mathcal{T}$, each characterized by two parameters: an
average reward $r_{i}$, computed using an exponential moving average, and a
reward variability measure $\delta_{i}=|r_{i}^{t}-r_{i}^{t-1}|$, which
captures the absolute change in rewards between consecutive trials. This
dynamic adjustment is governed by two key parameters: a scaling coefficient
$d$ and a success rate threshold $\tau$.

The task selection probability is modified according to the task’s performance
relative to $\tau$. For tasks where the reward $r_{i}$ meets or exceeds
$\tau$, the algorithm assigns a lower selection probability of
$q_{i}=\frac{1}{d}$, effectively deprioritizing tasks that have already
surpassed a certain success benchmark. Conversely, tasks that fall below this
threshold are given a higher probability of selection, formulated as
$q_{i}=1+(\delta_{i}\cdot d)$. This scheme favors tasks with greater
variability in performance, hypothesizing that such tasks may yield more
informative learning experiences due to their higher potential for
improvement.

To ensure a probabilistic selection, the probabilities are normalized using a
softmax function, yielding a distribution $\mathbf{p}$. A task is then
selected randomly according to this distribution, with the likelihood of
selecting task $i$ given by $p_{i}$ from $\mathbf{p}$, effectively integrating
an element of stochasticity into task selection which is critical for
exploring a variety of learning experiences. This methodology allows for a
balanced exploration-exploitation trade-off, adapting task difficulty based on
learner performance to potentially accelerate skill acquisition in complex
learning environments.

This adaptive approach to task selection represents a significant enhancement
to the efficiency of curriculum learning strategies, ensuring that learners
are consistently challenged with tasks appropriate to their evolving
capabilities. This strategy not only maintains engagement but also maximizes
learning outcomes by strategically modulating the difficulty of tasks
presented over the course of training.

1

Input: List of tasks $\mathcal{T}$, each task $i$ characterized by parameters
$(r_{i},\delta_{i})$ where $r_{i}$ is the average reward for task $i$ using
exponential moving average, and $\delta_{i}=|r_{i}^{t}-r_{i}^{t-1}|$ is the
average difference between subsequent rewards for task $i$, also using
exponential moving average. Constants $d$ (a coefficient that influences
sampling probability) and $\tau$ (a success rate threshold).

Output: Selected task

2

3

4

5Function  _AdaptiveSelect(_$\mathcal{T},d,\tau$_)_:

6 Initialize a vector $\mathbf{q}\leftarrow[]$;

7 for  _each task $i\in\mathcal{T}$_ do

8 if  _$r_{i}\geq\tau$_ then

// Assign a lower probability for tasks that exceed the success threshold
$\tau$

9 $q_{i}\leftarrow\frac{1}{d}$;

10

11 else

// Assign higher probability to tasks with more variability and below the
threshold

12 $q_{i}\leftarrow 1+(\delta_{i}\cdot d)$;

13

14 end if

15 Append $q_{i}$ to $\mathbf{q}$;

16

17 end for

// Normalize probabilities

18 $\mathbf{p}\leftarrow\text{Softmax}(\mathbf{q})$;

19

// Sample task index $j$ according to the probability distribution
$\mathbf{p}$

20 $j\sim\text{Categorical}(\mathbf{p})$;

21

22 return $\mathcal{T}[j]$;

23

24 end

Algorithm 1 Task Selection Based on Success Rate Figure 9: Comparison of
probabilities of sampling a subtask during training between a uniform sampling
policy and a curriculum learning-based policy. Figure 10: Success rates for
each subtask during training, comparing a uniform sampling policy to a
curriculum learning-based policy.

To highlight the importance of curriculum learning techniques, we conducted an
additional experiment in the Crafter environment. Here, we compare the
learning processes of a PPO agent with and without curriculum learning. In
Figure 9, we show the probability of sampling each subtask during training.
For the uniform policy, this is represented by a line parallel to the x-axis.
In Figure 10, we provide the success rates for both approaches. It is clear
that the version with curriculum learning shows better results for tasks such
as collect_iron, collect_diamond, make_iron_pickaxe, and make_iron_sword. This
improvement can be attributed to the dynamics shown in the first plot. For
example, for the collect_iron task, as the agent begins to master this
problem, the sampling probability increases. Once performance plateaus, the
probability decreases. This pattern also applies to other tasks; if the agent
has already mastered a subtask, e.g., place_table or make_wood_pickaxe, the
probability of sampling these subtasks decreases below that of uniform
sampling, thereby freeing up resources for learning other subtasks.

##  Appendix B Coordinates Generation Prompt

First, we include a description of the IGLU environment in the prompt. This
way, we provide information on how the coordinates change in relation to
cardinal directions (north, west, sky). We also provide information about
certain patterns present in the dataset (rows, squares, columns, towers) and
what they signify in the dataset.

Environment have size (11 11 9). Coordinates are (x y z). x coordinate
increases towards south. y coordinate increases towards east. z coordinate
increases towards the sky. So the highest south-west coordinate would be ( 11
11 9). And the lowest nort-west coordinate is (0 0 0). Middle of
enviroment/grid is (5 5 0). If I will start from the middle of
enviroment/grid, but on the floor, face north and put one block, it coords
will be (4 50). I can place blocks in this environment. For task: "Facing
north, build three stacks of two orange blocks, then destroy the bottom orange
block on all three stacks"; I will get a cords for built figure: ((2 4 0), (2
5 0), (2 6 0)). If I want a column/tower of 6 yellow blocks in the middle of
the grid, the coords will be (5 5 0), (5 5 1), (5 5 2), (5 5 3), ( 5 5 4), (5
5 5). If I want a row with 3 blocks towards south: (5 5 0), (5 6 0), (5 7 0).
What coords will be for tower of 6 yellow blocks in the middle of the grid? As
aswer, send me only list of coords.

Next, we ask to add the color of the shape to the coordinate description,
which should be chosen for a specific block.

As aswer, send me only list of coords. Add color digit to tuples array with
the next color mapping: BLUE - 1 GREEN - 2 RED - 3 ORANGE - 4 PURPLE - 5
YELLOW - 6 and use 0 - for blocks needs to be removed

Now, in the chatbot’s single-session mode, we sequentially present the
instructions from the task without making any alterations to them.

##  Appendix C Instruction Augmentation Prompt

In this prompt, we also start by providing information about the environment
like in B and the dialogues. Then, we ask to rewrite the dialogue as if the
shapes were required to be built rotated by 180 degrees.

I have a dialog with instructions describing how to build a 3D figure on a
plane. The instructions have information on where to build the shapes and how
to build them. I will consistently give you dialogues, and you have to rewrite
the instructions so that they describe the construction of the same figure,
only the location is rotated 180 degrees clockwise. Size of eviroment is (9,
11, 11) Remember: column and tower are vertical structures, line and row are
horizontal. Do not change the instructions associated with the modification of
vertical structures (keywords top and bottom). The output is only the final
dialogue (be sure to save the ++ signs, I will then use them to separate the
dialogue into instructions).

##  Appendix D IGLU Structure Classes

Examples of structure classes in the IGLU environment are presented in Fig.
11. Each structure represents a typical example used in the experimental
results section. Please note that the same figure can be attributed to several
classes; for example, structure (a) is both tall and flying simultaneously.
The structure on the floor (c) is also a flat structure, etc.

The complexity of each class is highlighted by the experimental results. tall
and flying structures are more difficult than floor and flat structures. floor
is the easiest one, since it doesn’t require the ability to place or remove
proper supporting blocks, which are much more challenging tasks.

Figure 11: Example of IGLU figures classes.

##  Appendix E IGLU Reward Function

In the IGLU environment, the reward function used changes throughout the
training curriculum to balance guided exploration and precise task execution.
Initially, the primary reward is based on the distance from the modification
to the goal, calculated as $\frac{+1}{dist+1}$. As training progresses, this
reward structure transitions to a more rigorous system where the agent
receives $+1$ for correct modifications (indicating successful subtask
completion) and a constant penalty of $-0.5$ for incorrect modifications,
regardless of the distance. Additionally, in the early stages of the
curriculum, there is an auxiliary reward for reducing the distance between the
agent and the goal, provided only when moving positively towards the goal.
This proximity reward diminishes as training advances, encouraging the agent
to autonomously navigate and execute tasks by the end of the curriculum.

##  Appendix F Hyperparameters and Hardware Resources

Table 4 lists the hyperparameters used for training the RL module of the IGOR
approach. Table 5 provides the hyperparameters for training the Flan T5
language model, specifying only those parameters that were adjusted from the
default settings in the HuggingFace.

Table 4: The parameters utilized for training the PPO actor-critic model.  Hyperparameter | Value
---|---
Adam learning rate | $0.0001$
$\gamma$ (discount factor) | $0.99$
Rollout | $32$
Clip ratio | $0.1$
Batch size | $1024$
Optimization epochs | $1$
Entropy coefficient | $0.003$
Value loss coefficient | $0.5$
GAEλ | $0.95$
ResNet residual blocks | 3
ResNet number of filters | $64$
LSTM hidden size | $512$
Activation function | ReLU
Network Initialization | orthogonal
Rollout workers | $16$
Environments per worker | $32$
Table 5: Parameters used to finetune Flan-T5, which have been changed from the default parameters in the Transformers library.  Hyperparameter | Value
---|---
learning rate | $0.0001$
batch size | $16$
gradient accumulation steps | $16$

Table 6 shows the training time and hardware resources. IGOR uses 139 GPU
hours in Crafter and 32 in IGLU, while Dynalang requires 166 GPU hours in
Crafter. IGOR efficiently combines RL and LLM modules on various devices.

Table 6: Training time and hardware resources for different modules Training Module | GPU hours | Device
---|---|---
Dynalang (crafter) | 166 | TITAN RTX
IGOR (RL crafter) | 136 | TITAN RTX
IGOR (LLM crafter) | 3 | NVIDIA A100
IGOR (RL iglu) | 20 | TITAN RTX
IGOR (LLM iglu) | 12 | NVIDIA A100


#  How studying black hole theory
may help us to quantise gravity111presented at the Conference on “Eternity
between and Space and Time”, Padova, May 19-21, 2022.

Gerard ’t Hooft

(Faculty of Science, Department of Physics
Institute for Theoretical Physics
Princetonplein 5, 3584 CC Utrecht
The Netherlands http://www.staff.science.uu.nl/~hooft101)

> Abstract
>  Black holes are more than just odd-looking curiosities in gravity theory.
> They uniquely intertwine the basic principles of General Relativity with
> those of Quantum Theory. Just by demanding that they nevertheless obey
> acceptable laws of dynamics, just like stars and planets, we hit upon
> strange structures that must play key roles in the quantum effects that we
> expect in the gravitational force at ultrashort distance scales.
> It is explained why, in our approach to address the problem of information
> conservation, the usual expression for the temperature of Hawking’s
> radiation is off by a factor 2.

##  1 Introduction

Gravitation is arguably the most elementary force in physics. It appears to be
directly linked to a fundamental principle:

> Invariance of physical laws under general coordinate transformations in
> space and time.

As is well-known, other forces among the elementary particles of matter are
based on very similar symmetry principles, notably those of local gauge
theories, but the symmetry principle that governs the gravitational force
seems to be at the basis of all symmetries in nature, that of general
transformations among all coordinates for space and time. And more remarkably,
Combining this symmetry principle with quantum mechanics seems to lead to
novel and unique clashes with what we thought we knew about forces and
dynamics.

It was thought that history of science had given us a significant clue: _“We
will get this thing done within half a century or so !_

Unfortunately, in spite of a tremendous amount of work and numerous essential,
novel pieces of insight, we still are confronted with mysteries at this point.
What are we doing wrong?

Most conventional theories, focussing either on physics at the Planck scale
(some $10^{-33}$ cm or $10^{-44}$ seconds, or on the scale of cosmological
theories (ranging up to the size of the universe), assume that the quantum
formalism is mandatory:

_Start with the existence of a Hilbert space_ , formulate a law for computing
quantum amplitudes, and assume that all physical phenomena covered by the
theory can be described in terms of these amplitudes, _even if_ there is no
need to agree on what it is that these amplitudes actually describe. According
to many researchers, discussions on the foundations of quantum mechanics
itself have come to a dead end. Here however, we emphasise that quantum
amplitudes can actually be reduced to being a _vector representation_ of
phenomena based on extremely mundane forms of logic. [GtHCA] One may ignore
the usual conundrums of the ‘collapse of the wave function’, the role played
by ‘pilot waves’, and even the existence of uncountably many distinct
universes (the ‘many world’ hypothesis).

And indeed we must worry about the nature of general coordinate
transformations that we thought to have under control.

A deeper study of black holes [2], and their relation with the laws of quantum
mechanics [3], may teach us new and very important things. Black holes seem to
be just the most basic solutions of Einstein’s field equations, but it is not
automatically guaranteed that their connection with quantum mechanics will be
anything ordinary. In our telescopes it seems that black holes may be just a
special kind of burnt-out stars. But if conventional theories for the quantum
treatment are not completely wrong, one does not get ‘ordinary behaviour’ from
our equations. Something is wrong. So, we advertise a closer look at those
laws. Just by postulating that everything hangs together in a very orderly
manner, may force us to rephrase those equations, in a way that might become
very revealing.

##  2 Schwarzschild and the tortoise metric

Much of the material described in this section and the next one has appeared
in several previous reports by this author.[11, 12, 13] Readers familiar with
is can just briefly scan this part of the paper, but we shall refer to it when
we continue.

Shortly after Einstein published his theory of General Relativity, the
astronomer Karl Schwarzschild [2] realised that assuming spherical symmetry
enables us to write down an exact solution of these equations. The metric one
arrives at, in modern notation, reads

$\displaystyle{\rm d}s^{2}=-{\rm
d}t^{2}\Big{(}1-\frac{2GM}{r}\Big{)}+\frac{{\rm
d}r^{2}}{1-\frac{2GM}{r}}+r^{2}{\rm d}\Omega^{2}\ ,$ (2.1)

where $r$ represents distance from the origin, $t$ is a time coordinate, and
$\Omega$ stands for the solid angle coordinates $(\theta,\varphi)$. The
variables ${\rm d}r$ and ${\rm d}t$ stand for infinitesimal differences for
the coordinates of two adjacent points in space-time, and ${\rm d}\Omega$
stands short for their infinitesimal angular separation, ${\rm
d}\Omega^{2}={\rm d}\theta^{2}+\sin^{2}\theta\,{\rm d}\varphi^{2}$. $M$ is a
parameter that stands for the total mass as would be perceived by an observer
at infinity. See Fig. 1a.

time $\uparrow$

a) b)

Figure 1: a: The Schwarzschild metric. Vertical dashed lines: the event
horizon. Curved lines: light like radial geodesics: $y\,=\,$const (smooth),
and $x\,=\,$const (dashed). Cones: the orientation of the local light cones.
Angular coordinates $(\theta,\,\varphi)$ are not shown. b: The Tortoise
coordinates $(x,\,y)$, showing regions $I-IV$ and the orientations of the
local light cones. Curves show $r=$ const. lines.

This metric is independent of time $t$, and we see that the contributions from
${\rm d}r$ and ${\rm d}t$, as given in the first two terms in (2.1) both
switch signs when $r$ crosses the point $r=2GM$. Here, G is Newton’s
gravitational constant, and the freely adjustable parameter $M$ represents the
mass of this object. Much later, it would become known as a ‘black hole’.

The apparent singularity at $r=2GM$ is not real, it is a coordinate artefact.
One can replace the longitudinal coordinates $r$ and $t$ by a new set, $x$ and
$y$, see Fig. 1b, as follows:

$\displaystyle x\,y$ $\displaystyle=$
$\displaystyle\Big{(}\frac{r}{2GM}-1\Big{)}\,e^{\textstyle r/2GM}\ ;$ (2.2)
$\displaystyle y/x$ $\displaystyle=$ $\displaystyle e^{\textstyle t/2GM}\ .$
(2.3)

These coordinates are called after Kruskal [4] and Szekeres [5], and shall be
referred to here as ‘tortoise coordinates’ for short, just because they
replace the infinitely slow geodesics to and from the horizon, by ordinary
geodesics crossing a light front.

In these coordinates, the singularities at $r\rightarrow 2GM$ disappear:

$\displaystyle{\rm d}s^{2}\ =\ \frac{32(GM)^{3}}{r}\,e^{\textstyle-r/2GM}{\rm
d}x\,{\rm d}y\ +\ r^{2}{\rm d}\Omega^{2}\ .$ (2.4)

However, here we discover that there are two horizons, not one: the _future
event horizon_ , at $x=0$, to which all absorbed matter particles move, and
the _past event horizon_ , at $y=0$, which may emit particles. The tortoise
coordinates show that the Schwarzschild space-time has a natural extension,
from region $I$, where $x>0$ and $y>0$, to regions $II,\ III,$ and $IV$.
Region $I$ is the _physical region_. Here, we can draw the geodesics for all
in-going particles, and all out-going particles, see Fig. 2.

Figure 2: Region $I$, the physical region (see Fig. 1b), showing early and
late particles going in and out. Everywhere in this region the light cones are
oriented as the one shown.

Believing that black holes should _not_ be approached as magic wands, but
should be understood as normal physical objects that can absorb things and
emit things, leads one to believe that in-particles (short for particles going
into the black hole) transmit all information they contain towards the out-
particles. At first sight, this seems to violate causality, since the points
where in-particles cross the future event horizon lie in the far future while
the out-particles emerge in the far past (see Fig. 2).

However, they meet each other half way, and if this point lies much further
away than a Planck length from the horizon crossing point (point 0 in the
figure), then the contradiction becomes insignificant. This separation
decreases exponentially with the time span lying between the in-orbit and the
out-orbit, so the Planck domain is reached quite quickly. But more to the
point, we can compute what happens here. It is the _gravitational force
between in-particles and out-particles that does the job of information
transfer_ (as soon as their longitudinal coordinates meet), and, it acts
locally as a sonic boom; there is no contradiction with causality at all.

##  3 The Shapiro effect

How, by gravity, an in-particle 1 affects an out-particle 2 – and vice versa –
is an elementary exercise in General Relativity. Consider first a flat
background. One can take particle 1 to be at rest, and consider its metric as
a Schwarzschild metric. Then let this particle move with a velocity close to
$c$. Suppose now another light particle, 2, moves in the opposite direction.
It experiences the gravitational field of particle 1 as being highly
compressed by the Lorentz transformation. Immediately before, and immediately
after the encounter, space-time is experienced as being flat (in the home
frame of particle 1, the particles are then far apart). So the gravitational
field of particle 1 is experienced as a sonic boom, comparable to Cherenkov
radiation.[6]

During this encounter, what happens is closely related to the Shapiro effect
[7] when a light ray grazes past the Sun or another heavy body. The geodesic
of the light ray is delayed. An elementary calculation [9] shows that this
amounts to a shift $\delta u^{\mu}$ of particle 2 in the light cone direction
of the motion of particle 1. If particle 1 has a light cone momentum $p^{\mu}$
then one finds

$\displaystyle\delta u^{\mu}=-4G\,p^{\mu}\,\log|\tilde{x}-\tilde{x}^{\prime}|\
.$ (3.1)

Here, $\tilde{x}-\tilde{x}^{\prime}$ is the transverse separation at the
closest encounter (impact parameter).

This was the effect in a flat background space-time. Now consider particles
moving in and out of a black hole [10], entirely in the longitudinal direction
(close to the horizons, the transverse motion, and shift, are neglected for
the time being). For a particle 1 moving in the longitudinal $p^{-}$
direction, this shift can be written as [11, 12]

$\displaystyle\delta u^{-}(\Omega)=8\pi G\,f(\Omega,\Omega^{\prime})\,,$ (3.2)

where $\delta u^{-}$ is the shift in the orbit of particle 2. The function $f$
replaces the logarithm in Eq. (3.1). It is a Green function obeying

$\displaystyle(1-\Delta_{\Omega})f(\Omega,\Omega^{\prime})=\delta^{2}(\Omega,\,\Omega^{\prime})\
.$ (3.3)

An in-particle with momentum $p^{-}$ at solid angle $\Omega^{\prime}$ causes a
shift $\delta u^{-}$ at solid angle $\Omega$ . This Shapiro effect has a very
important property: _it is linear_ : the effects of all in-particles is just
the sum of the contributions of every single one. At the same time, it
displaces all out-particles by the same position-dependent amount $\delta
u^{-}(\Omega)$.

This enables us to write the combined effect as a property of functions on the
angles $\Omega,\ \Omega^{\prime}$. Write a single function $p^{-}(\Omega)$ as

$\displaystyle
p^{-}(\Omega)\equiv\sum_{i}p^{-}_{i}\delta^{2}(\Omega,\Omega_{i})\ ,$ (3.4)

where $\Omega_{i}$ are the spots where the in-particles enter the future event
horizon; for all out-particles we write:

$\displaystyle u^{-}(\Omega_{i})=u^{-}_{i}\ .$ (3.5)

The Shapiro effect can now be written as follows: if we add an in-particle
with momentum $\delta p_{i}^{-}$ at solid angle $\Omega_{i}$ then it modifies
the position $u^{-}(\Omega)$ of all out-particles, in accordance with the
equations

$\displaystyle u^{-}(\Omega)=8\pi G\int{\rm
d}^{2}\Omega^{\prime})\,f(\Omega,\Omega^{\prime})\,p^{-}(\Omega^{\prime})\ ,$
(3.6)

where the Green function $f(\Omega,\Omega^{\prime})$ obeys (3.3). We can write

$\displaystyle(1-\Delta_{\Omega})u^{-}(\Omega)=8\pi G\,p^{-}(\Omega)\ .$ (3.7)

Notice that we replaced the shift equation (3.2) for one particle going in and
one particle going out, by a single functional equation (3.6) for all
particles in and all particles out. The only difference this makes is that we
could omit the symbol $\delta$ that indicates _change_. Before and after any
change, out-positions relate to in-momenta in the same way. The equation
suggests that the origin, where all positions are zero and no particles are
going in at all, should be chosen as the origin of this space.

##  4 Quantum mechanics

In the previous section, we encountered fundamental new equations relating
positions and momenta of in and out going particles. This begs for an
investigation as to what happens when we introduce quantum mechanics. Close to
the origin222The process of information transfer, in which we are now
interested, takes place at Planckian distance scales. As long as the
Schwarzschild metric parameters, typically in the order of $GM$, are large
compared to this, our analysis applies., all particles obey

$\displaystyle{}[u^{\pm}_{i},\,p^{\mp}_{j}]=i\delta_{ij}\
,\qquad[u^{\pm},\,p^{\pm}]=0\ ,$ (4.1)

so that, for the functions $u^{\pm}(\Omega)$ and $p^{\pm}(\Omega)$, we can
write

$\displaystyle{}[u^{\pm}(\Omega),\,p^{\mp}(\Omega^{\prime})]$ $\displaystyle=$
$\displaystyle i\delta^{2}(\Omega,\,\Omega^{\prime})\ ,$ (4.2)
$\displaystyle{}[u^{\pm}(\Omega),\,p^{\pm}(\Omega^{\prime})]$ $\displaystyle=$
$\displaystyle 0\ .$ (4.3)

In combination with Eqs. (3.6) and (3.7), these equations become very
powerful. The equations are local now, and, most important, they are linear.
Linearity and locality might get broken when we refine these equations for
large transverse momenta, but we believe that the section for sufficiently low
transverse momenta can be kept by itself as the dominant contribution to the
information processing mechanism. Due to linearity, it is meaningful to expand
the functions $u^{\pm}$ and $p^{\pm}$ in spherical harmonics:

$\displaystyle u^{\pm}(\Omega)\equiv\sum_{\ell,m}u^{\pm}_{\ell m}Y_{\ell
m}(\Omega)\ ,\qquad p^{\pm}(\Omega)\equiv\sum_{\ell,m}p^{\pm}_{\ell m}Y_{\ell
m}(\Omega)\,,$ (4.4)

where the sum goes over all nonnegative integer values of $\ell$, and
$-\ell\leq m\leq\ell\ .$

For the spherical harmonics, the operator $\Delta$ diagonalises into
$-\ell(\ell+1)$, so that we can write

$\displaystyle u^{-}_{{\mathrm{out}},\,\ell m}=\frac{8\pi
G}{\ell^{2}+\ell+1}\,p^{-}_{{\mathrm{in}},\ell m}\,,\qquad
u^{+}_{{\mathrm{in}},\,\ell m}=-\frac{8\pi
G}{\ell^{2}+\ell+1}\,p^{+}_{{\mathrm{out}},\ell m}\,,$ (4.5)

where we wrote the subscripts ‘in’ and ‘out’, to indicate that these operator
refer to in- or out-particles. The minus sign in the last equation is
understood as a consequence of the antisymmetry of the commutator while no
minus signs had been inserted in Eqs (4.1):

$\displaystyle[u^{+}_{\ell m},\,u^{-}_{\ell^{\prime}\,m^{\prime}}]=\frac{8\pi
iG}{\ell^{2}+\ell+1}\,\delta_{\ell\ell^{\prime}}\delta_{mm^{\prime}}\ .$ (4.6)

Important remarks:

  * $\bullet$

There is only one real independent dynamical variable $u^{+}$, and a variable
$p^{-}$ canonically associated to it, at every value of the pair of integers
$(\ell,\,m)$. So the variables in Eq. (4.1) are replaced by a one-dimensional
quantum mechanical pair at each $\ell$ and $m$.

  * $\bullet$

Different $(\ell,\,m)$ values all commute. Therefore, in the approximations
used, the complete set of all quantum states will be just the product of all
quantum states $|u^{+}\rangle_{\ell m}$ at all $(\ell,\,m)$. By taking their
one-dimensional Fourier transforms, one gets the states $|p^{-}\rangle_{\ell
m}$ instead.

Thus, the out-particles emerge as being the Fourier transforms of the in-
particles, a very simple algebra at this stage.

Note that, replacing one single species of matter by a set of multiple matter
species, would lead to contradictions (there exists only one type of
gravitational force). There exists only one form of matter which we subject to
the rules (4.1), and therefore we cannot apply second quantisation in the
longitudinal direction; only the $\Omega$ dependence can be viewed as a
reduced, two-dimensional second quantisation process. Having single particle
states only for the $u^{\pm}$ dependence, but second quantisation only in the
$\theta,\varphi$ direction could be called
‘1.5${}^{\mathrm{th}}\,$quantisation’.

Figure 3: Penrose diagram generated from the tortoise coordinates $x$ and $y$,
by mapping these both on a compact segment $[-1,1]$. See text.

Everything seems to come together, but there are still some hurdles to take.
By mapping the Kruskal-Szekeres coordinates onto compact segments, one gets
the Penrose diagram for an eternal black hole, see Fig. 3. It was shown above
that the out-particles are obtained from the in-particles by Fourier
transforming the wave functions, call these $\langle
u^{+}|\psi\rangle_{\ell,m}$ . The Fourier transformation is a unitary process,
so one might think that we obtained the complete black hole evolution operator
$U(t)$. The fact that, so-far, we concentrated only on _eternal_ black holes,
is not a problem. At time scales longer than ${\mathcal{O}}(M\log M)$ in
Planck units, the wave functions get completely scrambled by the Shapiro
process, and the component of Hilbert space that would represent either
collapse, or a final evaporation event, is expected to be small compared to
the one spanned by the states that we do take into account.

A problem that we still have, at this stage, is that there are two asymptotic
regions, $I$ and $II$. Physically, we need unitarity only for one single
domain, region $I$ itself. But by generating the shifts $\delta u$, the
Shapiro process will transport states right across the horizon, and when we
attempt to calculate the combined effects, we find that unitarity is violated
by that proces: particle wave functions in region $I$ will generate Fourier
transforms that cover both regions $I$ and $II$.

It is true that, most of this process takes place in a Planckian domain close
to the origin as indicated in Figure 3, but, as time goes on, local Lorentz
boosts will quickly send these out-states to infinity, spreading themselves
both over regions $I$ and $II$.

In the past, we proposed to cure this problem by assuming region $II$ to
correspond to the _antipodes_ (in terms of the angles $\theta$ and $\varphi$)
of region $I$. But this did not work as assumed. In terms of the harmonic
functions $Y_{\ell m}(\Omega)$, going to the antipodes means multiplying the
angular wave functions by $(-1)^{\ell}$. For odd $\ell$, this actually cancels
the minus sign transporting us from region $I$ to region $II$ or back, causing
our calculations to fail.

We now believe that region $II$ is an _exact quantum copy_ [13] of all states
in region $I$. This means that we would limit ourselves to wave functions that
are _even_ under the interchange $I\leftrightarrow II$ This could also mean
that all information in region $I$ is accurately copied into region $II$, and
this remains to be the case for the Fourier transforms (the Fourier transform
of an even function is even as well.

This brings us very close to an ancient proposal by the author [14], to
identify the states in region $II$ as the set of bra states associated to the
kets in region $I$. At first sight this still may generate problems, since
energy reversal from $I$ to $II$ requires us to take real functions to be even
but imaginary functions to be odd. This property however, would not be passed
on correctly by the Fourier transforms. We suspect that this problem can be
solved by restricting ourselves to _real_ wave functions only.

Limiting ourselves to real wave functions only, now leads to serious questions
concerning the very foundations of quantum mechanics. The picture we obtained
so-far is that there are three representations for the wave functions of the
in- and out-particles: first, we have the original Schwarzschild metric, and
all in- and out-going particles can be represented as wave functions depending
on $r,\ t,\ \theta,$ and $\varphi$. We subsequently write the same metric in
terms of the tortoise coordinates $x,\ y,\ \theta,$ and $\varphi$. Using the
same wave functions, this gives us the quantum states in region $I$ when $x$
and $y$ are positive. Finally, we have the negative values of $x$ and $y$.
Again, the same wave functions now represent what goes on in region $II$.

The equations for these wave functions in $(x,\,y)$ space will be as usual.
The problems just mentioned above concern the time reversal in region $II$. It
is more than just an administrative difficulty. It seems that the concepts
_lowest energy state_ and _highest energy state_ are interchanged there. If
this is allowable then we can treat he entire system in terms of a single set
of quantum field theoretical equations, where the technical difficulties in
gluing the positive and negative values of $x$ and $y$ together must be more
carefully addressed than we were able to do here.

##  5 On the Hawking temperature

An issue not yet addressed in detail is the value of Hawking’s temperature,

$\displaystyle kT_{\mathrm{Hawking}}=\frac{1}{8\pi GM_{\mathrm{BH}}}\,.$ (5.1)

It is connected to the calculation of the Hawking entropy,

$\displaystyle\beta=\frac{1}{kT}=\frac{\partial S}{\partial E}=8\pi GM\ .$
(5.2)

Now these values represent the contributions of the quantum states both in
regions $I$ and $II$. If however, if region $II$ is nothing but a quantum
clone of region $I$ then it should not contribute to the entropy at all, so
that only half of the entropy is left:

$\displaystyle S={\textstyle\frac{1}{2}}\cdot 4\pi G\,M^{2}\
;\qquad\beta={\textstyle\frac{1}{2}}\,\beta_{\mathrm{Hawking}}\ ,\qquad
kT=2kT_{\mathrm{Hawking}}\ .$ (5.3)

This happens to be the temperature we anticipated in our treatment of Ref.
[14] using different arguments: the picture of regions $I$ and $II$ together
as representing bras and kets, turns our expressions into a density matrix. to
arrive at expectation values one should not take the absolute square of a
density matrix element but instead, trace it with the relevant operators.
Replacing the quadratic expressions by linear ones reduces the entropy by a
factor 2.

Now both arguments mentioned above, to indicate that the value of the
temperature is affected by a factor 2, seem to be mostly hand-waving. Author
warns Reader against believing that $N$ hand-waving arguments may add up to
something more believable than a single one. But we have a more powerful
argument, using the periodicity of the evolution operator in the Euclidean
time direction. This was the calculation first employed by Hawking [3] and
others.

Consider an outside observer $A$, and an observer $B$ in the effectively flat
spacetime at the origin of the Penrose diagram, Figs. 1b and 3. If the outside
observer $A$ performs a time translation by a constant amount,

$\displaystyle U(\delta t)=e^{\textstyle-i\delta t\,H}.$ (5.4)

The tortoise variable $x$ in Eqs. (2.3) is multiplied by
$e^{\textstyle\,\delta t\,/\,4GM}$ and $y$ is divided by the same amount. For
observer $B$ that is a Lorentz transformation at the origin. Now take the same
transformation but replace $\delta t$ by an imaginary number $-i\beta$. This
is a rotation:

$\displaystyle x\rightarrow x\,e^{\textstyle-i\beta/4GM}\ ,\qquad y\rightarrow
y\,e^{\textstyle i\beta/4GM}\ .$ (5.5)

Writing

$\displaystyle x=\varrho\,e^{\textstyle i\varphi}\ ,\qquad
y=\varrho\,e^{\textstyle-i\varphi}\ ,$ (5.6)

one gets a real, Euclidean spacetime in polar coordinates (by using the
coordinates $x+y$ and $-i(x-y)$). The standard calculation, computing the free
energy $F$ from ${\mbox{Tr}}\,(e^{\textstyle-\beta H})$, now assumes that only
full rotations, $\varphi\rightarrow\varphi+2\pi n$ lead spacetime back to its
original orientation. The Green functions to be considered are therefore
periodic with period $2\pi$ in $\varphi$. This is how $\beta$ takes the value
$8\pi GM$, The identification of the Euclidean periodicity $\beta$ with $1/kT$
is well-known in condensed matter physics and in lattice theories for
elementary particles, Thus one gets Eq. (5.2) and (5.1) for the Hawking
temperature.

Now in our theory, we have that region $II$ has $x$ and $y$ replaced by $-x$
and $-y$. There, it is a quantum clone of region $I$. All our green functions
must be the same in region $II$ and in region $I$ Thus, our Green functions
must return to their original values when $\varphi$ is replaced by
$\varphi+\pi$. This gives us the temperature (5.3).

##  6 Conclusion

In some sense, our theory for black holes is extremely conservative. In our
treatment there is no place for ‘entanglement’ issues, and not even for
attaching significant roles for the ‘Page time’ [15]. Page asks for explicit
attention to very large time scales, where a significant fraction of the
initial entropy is carried away by the radiated particles. But as soon as the
radiated particles have moved further than a few Planck lengths away from the
horizon crossing point, conventional laws of physics take over, entanglement
or not. Any respectable theory for the production of these particles must
include a chapter on how these particles are expected to make this transition
to free or almost free particle states in the outside world.

In fact, this seems to be easy enough. The total mass $M$ of a black hole
together with the surrounding particles is strictly constant. But only the
particles that are still – or already – close to their respective horizons,
contribute to the mass in practice. So, as soon as we see particles far enough
away from the black hole, we should modify our description in terms of a black
hole with mass $M^{\prime}$, slightly different from $M$, by subtracting the
contribution of the particles that are sufficiently far away, at any
particular moment of time.

Note that, in our approximations, the masses of the radiated particles are
small compared to that of the black hole itself, so that the time-dependent
corrections, as described above, are infinitesimal in any case. I think we
performed important ground work to formulate a precise framework for handling
the physics of black holes. In ancient times when solitons, magnetic
monopoles, field theoretical strings and membranes were regarded as novel
predictions of gauge theories and related concepts were discovered so that
their properties were calculated, no new physical principles had to be called
upon. For black holes, including of course the rotating and charged versions,
the situation is not at all that simple, and it could even be that the
physical laws at small scales will have to be thoroughly revised in order to
obtain a harmonic picture of what goes on. This would actually be welcome,
since out present views of Planckian scale physics appear to lead to
paradoxes: how should one handle energies beyond that scale, and how can we
reconcile that with quantum mechanics?

The author’s personal view is that both quantum mechanics and general
coordinate invariance are aspects of matter and geometry that are calling for
a more precise formulation of their foundations. It does not help to be
‘agnostic’, as some physicists declared, or to think that ‘chaos’ takes care
of all contradictions.

Particularly, our results on Hawking’s temperature are novel (they were
already mentioned in 1984 [14] but hardly noticed by string theorists). Some
further thinking strongly suggests that, actually, any attempt to identify the
information residing in region $II$ with what is already present in region
$I$, would reduce the Hawking entropy by a factor 2, and raise the temperature
of the radiation accordingly.

Since conservation of information (in the sense of entropy in thermodynamics)
is being considered more seriously these days, and in view of our considerably
revised outcome (5.3) for temperature and entropy in black holes, we conclude
that thorough revisions for quantum gravity itself may be opportune. Our
proposals for real wave functions rather than complex ones, also suggests
revisions for the foundation of quantum mechanics [16].

We thank N. Sanchez, N. Gaddam, F. Feleppa, N. Groenenboom, and S. Kumar for
discussions.

## References

  * [1]
  * [2] K. Schwarzschild, (1916). Über das Gravitationsfeld eines Massenpunktes nach der Einsteinschen Theorie. Sitzungsberichte der Königlich Preussischen Akademie der Wissenschaften. 7: 189-196. Bibcode:1916AbhKP1916..189S.
Translation: S. Antoci and A. Loinger, (1999). _On the gravitational field of
a mass point according to Einstein’s theory_. arXiv:physics/9905030.

  * [3] S.W. Hawking,  Particle creation by black holes, Commun. Math. Phys., 43(3) (1975) 199.
  * [4] M.D. Kruskal, _Maximal extension of Schwarzschild metric_ , Phys. Rev. 119 (1960) 1743 .
  * [5] G. Szekeres, _On the singularities of a Riemannian manifold,_ Publ. Math. Debrecen 7, 285 (1960) (now available at General Relativity and Gravitation 34 (2002) 2001).
  * [6] P.A. Cerenkov, _Visible Radiation Produced by Electrons Moving in a Medium with Velocities Exceeding that of Light_ , Phys. Rev. 52, 378 (1937).
  * [7] Irwin I. Shapiro (1964). _Fourth Test of General Relativity_. Phys. Rev. Letters. 13 (26): 789-791. Bibcode:1964PhRvL..13..789S. doi:10.1103/PhysRevLett.13.789.
  * [8] W.B. Bonnor, _The gravitational field of light_ , Commun. Math. Phys. 13 (1969) 163.
  * [9] P.C. Aichelburg and R.U. Sexl, _On the Gravitational field of a massless particle_ , Gen.Rel. and Gravitation 2 303-312 (1971);
  * [10] T. Dray and G. ’t Hooft, _The gravitational shock wave of a massless particle_ , Nucl. Phys. B253 (1985) 173.
  * [11] G. ’t Hooft, _On the quantum structure of a black hole_ , Nuclear Physics B256 (1985) 727-745.
  * [12] G. ’t Hooft, _Strings from gravity_ , Physica Scripta Vol T15 143-150, 1987.
  * [13] _Quantum clones inside black holes_ , universe-1927974; doi: 10.3390/universe8100537, http://arxiv.org/abs/2206.04608.
  * [14] G. ’t Hooft, _An ambiguity of the equivalence principle and Hawking’s temperature._ J. of Geometry and Physics 1 (1984) 45-52.
  * [15] D.N. Page, _Information in black hole radiation_ , Phys.Rev.Lett. 71 (1993) 3743-3746 arXiv:hep-th/9306083 (hep-th)
  * [16] G. ’t Hooft, _Explicit Construction of Local Hidden Variables for Any Quantum Theory Up To Any Desired Accuracy_ , in “Quantum Mechanics and Fundamentality”, V. Allori (ed.), Synthese Library 460, https://doi.org/10.1007/978-3-030-99642-0-13, arxiv:2103.04335[quant-ph].