*These authors contributed equally to this work

§ INTRODUCTION

Developing Artificial Intelligence (AI) agents that can attain human-level performance has been a long-standing goal for AI research <cit.>. Large Language Models (LLMs) <cit.> have emerged as promising tools in this endeavor, owing to their strong reasoning and generalization abilities. LLM-powered AI agents have exhibited significant potential across diverse domains, including code generation <cit.>, content creation <cit.>, tool utilization <cit.>, and robotics <cit.>.
Meanwhile, LLM-powered agents have exhibited the ability to mimic human-like behaviors when interacting with other players <cit.>, leading to a revolution in human-AI coordination.

AI agents powered by LLMs commonly rely on LLM APIs and hand-crafted complex prompts. A notable challenge within this paradigm is the high latency associated with the inference process due to API calls, ranging from seconds to minutes <cit.>. The inference time may not be regarded as bottleneck in scenarios with low-frequency interactions, such as code generation. However, the limitation of high inference latency becomes apparent in human-AI coordination applications, which requires real-time responses and high-frequency interactions, such as video games <cit.>.

In this work, we consider Overcooked as our real-time human-AI coordination testbed. Overcooked is a cooperative cooking game where players collaborate at a rate of approximately 3Hz to serve orders within a time budget. We further develop language-based communication in Overcooked to allow human-like cooperation. An ideal language agent is expected to exhibit real-time responsiveness, strong reasoning capabilities, and effective language-based communication with human players for the best performance in such a fast-paced game. Fig. <ref> shows a concrete example. When paired with an AI player, a human player might instruct the AI player by saying “Chop 3 tomatoes.” The AI player needs to accurately interpret and follow the command by swiftly picking up and chopping the specified number of tomatoes. Upon completion, the AI player needs to inform the human player of the ongoing progress,
e.g., by saying “I've chopped 3”, for future cooperation.
Afterward, the human player might directly say “one more”, which itself looks semantically ambiguous. The AI player must correctly infer the true command from the history commands and promptly chop one more tomato.

A concrete example of cooperation and communication between a human player and an AI player in Overcooked.

Traditional gaming AI usually employs smaller models or script policies, emphasizing fast inference for real-time responses to the game dynamics. Yet, this efficiency comes at the cost of limiting task completion and intra-player interaction abilities <cit.>.

We propose a  (HLA) for real-time human-AI coordination in Overcooked. Inspired by System 1 and System 2 thinking <cit.>, HLA combines both robust reasoning and interaction capabilities from a large model and real-time inference from a smaller model and a reactive policy.
In particular, HLA employs a hierarchical framework that consists of three modules: a proficient LLM for intention reasoning and language-based communication, a lightweight LLM for interpreting commands and high-level planning, and a script policy for executing low-level actions swiftly. We denote the proficient LLM as . The lightweight LLM, referred to as , generates macro actions, and the reactive script policy is referred to as , which transforms macro actions into atomic actions. Human commands are processed simultaneously through both the  and  to enhance real-time performance. The reactive policy further ensures the feasibility of actions and high-frequency interactions.

We consider three baseline agents, each lacking a specific HLA component. Our evaluation involves experiments on action latency, reasoning with simple and complex commands, and human studies. HLA demonstrates a remarkable advantage, being an order of magnitude faster than the best competitor in real-time action responsiveness. Furthermore, HLA outperforms the baseline agents significantly in command reasoning ability. Human studies confirm these findings, showing that HLA achieves approximately 50% higher game scores and receives the highest human preference.
More demonstrations can be seen on our website <https://sites.google.com/view/overcooked-hla/>.

§ RELATED WORKS

Language Agents. Prior works train instruction-following agents with paired datasets of text and trajectories in games <cit.>, visual navigation <cit.> and robotics <cit.>. However, these works are limited to simple domains. Recently, with the advances of Large Language Models, a class of works start to utilize the strong reasoning power of LLMs to interact with complex domains including web scenarios <cit.>, simulated environments <cit.>, real-world  environments <cit.>. These recent efforts use prompt engineering to elicit the power of LLMs <cit.>. Some recent works try to combine LLMs that play different functionalities in a cooperative manner <cit.>. In our work, we study the availability of language agents in response speed sensitive environments.

Human-AI Cooperation. Building AI agents that can cooperate with humans is a longstanding challenge. Prior works study human-AI cooperation without communication in games such as Hanabi <cit.> and Overcooked <cit.>, and in robotics  <cit.>. Language commands from humans are used to guide intelligent agents in visual navigation <cit.> and robotics <cit.>. Cicero <cit.> trains a language agent that can speak and make decisions like a human in the game of Diplomacy from a large volume of human play data. However, Diplomacy is a turn-based game and therefore has low real-time requirements. Recently, there are attempts in using LLM for human-AI interaction in domains including web scenarios <cit.>, health <cit.>, games <cit.> and other applications <cit.>. Similar to us, there are some concurrent works that use LLM for decision-making in the game of Overcooked <cit.>. We focus on improving real-time user experience in settings that demand rapid responses during human-AI interaction.

LLM Inference Speedup. The long inference latency of existing LLM-based agents is not desirable in domains that demand a fast response speed. “Skeleton-Of-Thought” <cit.> generates responses with a skeleton structure and uses parallel inference to reduce latency. Model compression methods achieve faster inference by distilling the knowledge of LLMs into smaller language models <cit.> and performing quantization <cit.>. In our work, we adopt a hierarchical design that leverages LLMs for reasoning and language interaction and small models for fast reaction in real-time gaming.

§ TESTBED: THE OVERCOOKED GAME

Cooking process.

Designed maps. From left to right are , , and

The cooking process and the maps in the Overcooked testbed.

§.§ Environment Details

Overcooked is a cooperative cooking game where participants must work together to prepare, cook, and promptly serve a variety of dishes.
The Overcooked environment <cit.> simulates and simplifies the original Overcooked game and provides an RL training interface as a common testbed for real-time human-AI coordination.
Throughout the game, orders continuously appear, each accompanied by a strict deadline. Players must prepare dishes in accordance with these orders and ensure they are delivered on time for rewards. Failed deliveries incur penalties.
To finish an order, players must follow a precise sequence of steps: retrieving the necessary ingredients, chopping them, mixing them as per the recipe, cooking them to make a dish, plating the dish before it overcooks, and finally, serving it at the counter. Fig. <ref> depicts the cooking process of the game. Each player can perform one of the 4 actions at each time step, i.e., up, down, left, and right, to control the character's position and to interact with other objects.

We implemented a collection of enhancements to the environment with full details described in Appendix <ref>.
In particular, we further extend the original environment by developing a chat interface to allow natural language communication between human and AI players.
Human players can either pause the game and send text messages to the AI through an additional chat window or directly speak to the AI player during gameplay. Similarly, the AI player can respond to human players through either text or speech.

We also design 4 distinct maps as shown in Fig. <ref>.
In each game, a human player (the pink beard character) collaborates with an AI player (the blue character) to complete the orders. The first two maps,  and , assess general human-AI cooperation capabilities. The third map, , completely separates the two players, therefore they must cooperate to finish any order.
In the fourth map, , we raise the number of concurrent orders and accelerate the action frequency to intensify the gameplay.

§.§ Challenges of Overcooked

Real-time Cooperation.
The Ovecooked game is particularly time-sensitive. It requires all players to adjust their game strategy in time and take swift actions, so that the orders do not timeout, and the dishes are not overcooked.
The real-time requirement of the game indicates that, if an AI agent wants to leverage LLM's strong capabilities, it cannot adopt the conventional approach of directly prompting the LLM to generate its next action. This approach leads to substantial latency, making it impractical for this game.

Command Reasoning.
When playing the original Overcooked game, human players often give commands that are semantically ambiguous or overly complex. We collect some common human commands and present the typical scenarios as follows.

* Quantity specification. The player asks others to “chop 3 tomatoes.”

* Semantic analysis. Instead of giving direct commands such as “cook the soup,” the player gives hints, “the soup order is about to timeout.”

* Ambiguous reference. The player first asks others to “chop some tomatoes,” then says “one more.” Here the “one” implicitly refers to a chopped tomato.

In such scenarios, the AI agent must consider various factors, such as human commands, environment composition, and historical actions, to determine the true intentions of the human player and respond accordingly. A basic reactive agent lacking reasoning capabilities will struggle to comprehend human commands, let alone collaborate effectively with humans to complete the game.

§ METHOD

§.§ Overview

Framework of , including a  for intention reasoning and language interaction, a  for macro actions generation, and an  to execute atomic actions.

As described in Sec. <ref>, the AI agent needs real-time responsiveness, command reasoning ability, and bilateral communication capability. A key observation in our testbed is that the AI agent can perform reasoning about human commands and communicate with humans at a regular frequency while atomic actions should be generated at a high frequency. Meanwhile, LLM-based agents are better at generating high-level moves than producing atomic actions. Therefore we propose  (HLA) as depicted in Fig. <ref>. HLA consists of three components: a proficient LLM, i.e., , that interprets human commands from the full game history and generates chat feedback; a lightweight LLM, i.e., , that delivers high-level moves, which we call “macro actions”, at a medium frequency; and a reactive policy, i.e., , that is implemented as pre-defined scripts and transforms macro actions into atomic actions to interact with the environment at a high frequency.
We will describe , , and  in the following sections respectively.

Workflow of .  employs a two-stage design. It reasons human intention according to human commands in the first stage, then generates chat message and performs completion assessment periodically in the second stage.

is empowered by a proficient LLM. The main functionality of  is to interpret vague human commands into concrete intentions, keep track of command completion progress, and provide responses containing key information and useful suggestions to the human partner. Also,  sends the inferred human intention to  and indicates to  whether the command is successfully completed.

To this end, we devise  to have two stages. The first stage, , infers human intention given the command and command history when the human issues a new command. The second stage, , periodically checks command completion and generates reply messages to the human partner based on the inferred intention. We use GPT-3.5 <cit.> here as GPT-3.5 features strong reasoning and communication power.

To interpret human commands, the LLM is provided with the command history and environmental information. The LLM interprets the vague command into concrete intentions that best reflect needs of the human partner. For instance, when the current soup orders are “Alice Soup” and “David Soup”, the command “Cook the first soup on the orders” should be interpreted as “Cook Alice Soup”. We note that this intention reasoning stage is carried out only once when a command is issued. The inferred intention is then stored by  and sent to . The prompt used in  is shown in Fig. <ref>.

[width=80mm, colback=gray!5!white, colframe=gray!150!black, left=1mm, top=0.5mm, right=1mm, bottom=0.5mm]

You are in a virtual environment where ...

Current orders: Alice Soup with plenty of time, ...

Human's message: “Cook the first soup”

Human's last intention: ...

Human's intention now: Cook Alice Soup

figure prompt in .

In , the LLM generates chat messages to the human partner and performs completion assessment. The chat messages cover different aspects of information including consent to the commands, planning of the AI agent, and information about the environment. The completion assessment is used as a tool to keep track of command completion progress. Once the human command is judged as completed, the inferred intention will be cleared and  will be reminded of completion of the command.

The LLM takes as input the inferred intention, current orders, environment state, and action history. We note that we use a compact representation for the action history by expressing consecutive repeated actions in the form of “action $\times$ repetition times” to reduce the length of the action history.

Different from  that is only executed once when a new command arrives,  runs periodically to actively monitor the command completion progress and provide positive messages to the human partner. Notably,  runs in different modes according to whether there exists an unaccomplished human command. When a human command is not completed yet,  runs at full functionality. When there are no ongoing human commands,  switches to a more casual mode that only generates chat messages and does not perform completion assessment.

Outline of the prompt used in  is shown in Fig. <ref>. We note that, when there are no ongoing human commands, completion assessment is disabled and the human's intention is ommited from the input prompt.

[width=80mm, colback=gray!5!white, colframe=gray!150!black, left=1mm, top=0.5mm, right=1mm, bottom=0.5mm]

You are in a virtual environment where ...

Current orders: Alice Soup (plenty of time), ...

Environment state: ...

Human's intention is: Cook Alice Soup once

Action you've done: Chop Lettuce twice, ...

Reasoning: I haven't cook Alice Soup yet.

My chat message is: Sure, I will cook it for you.

Intention satisfied? No

figure prompt in .

Workflow of .  is empowered by a lightweight LLM. It works with  cooperatively with a conditional prompt mechanism and avoids sub-optimal moves with an action-filtering mechanism.

As argued in Sec. <ref>,  produces high-level moves at a medium frequency while following human commands. We call these high-level moves macro actions. The current macro action set in our testbed includes chopping vegetables, serving soup, putting out the fire, and so on. A detailed description of macro actions can be found in Sec. <ref>.

Despite having superior decision-making capabilities, a powerful LLM as used in  is not applicable in  due to huge inference latency. Therefore a lightweight LLM is more suitable. On the other hand, while a lightweight LLM such as Llama2-13B-chat can successfully follow a human command when the command is concise and clear, it often generates sub-optimal moves that lead to a lower score when there are no ongoing commands and violates the command when the human partner gives a vague command.

Taking these factors into account, we design  that generates macro actions to interact with the environment, as depicted in Fig. <ref>.  is empowered by a lightweight LLM (quantized version <cit.> of Llama2-13B-chat <cit.>). To better ground human commands into moves,  works with  cooperatively with a conditional prompt mechanism. Lastly,  avoids sub-optimal moves with an action-filtering mechanism.

[width=80mm, colback=gray!5!white, colframe=gray!150!black, left=1mm, top=0.5mm, right=1mm, bottom=0.5mm]

You are in a virtual environment where ...

[If intention is unclear] Human sends a message: ...

[If intention is reasoned] Human's intention is: ...

[If intention is satisfied] Your chat message is: ...

My actions are: Chop Lettuce, <next action>

figure prompt.

§.§.§ Intention Selection

To better align macro action generation with vague human commands,  also uses the inferred intention from  as input. To prevent macro action generation from being blocked by the intention reasoning stage of ,  uses the raw human command as input before  successfully infers the intention. This asynchronous execution nature results in the conditional prompt mechanism used by  as depicted in Fig. <ref>.
When  is analyzing the command,  uses as input the raw command, action history, and availability of actions. As soon as  completes , the inferred intention is sent to , and  switches to use the inferred intention instead of the raw command to generate macro actions. Later when the command is evaluated as completed by , the conditional input for  is switched to the chat messages produced by , which helps the agent to better align its action with conversational response.

§.§.§ Macro Action Generation

We obtain action probability by the output probability of each macro action conditioned on the prompt prefix. Fig. <ref> illustrates the structure of the employed prompt. Each potential macro action candidate fills in the $<$next action$>$ variable to evaluate its output probability. The action history contained in the output prompt is to remind the agent of its past actions.

Lightweight LLM used in  may result in sub-optimal actions when managing complicated tasks. We therefore employ an action filter to filter out sub-optimal macro actions. The selection probabilities are calculated based on the probabilities output by the language model and the task-relevant value of each available macro action according to Eq. (<ref>).

\begin{equation}
\log U(a|s) \propto \log P_{LLM}(a|s) + \alpha V(a| s)
\label{eq:fastmind}
\end{equation}

Here, $U(a|s)$ represents the selection probability of macro action $a$ under current state $s$, $P_{LLM}(a|s)$ represents the probability of the language model outputting macro action $a$, $V(a| s)$ represents the value of macro action $a$ contributing to the task reward. The values for $V(a| s)$ are hardcoded, and their detailed information can be found in the Appendix <ref>. $\alpha$ is a dynamic adjustment term and is smaller when the human command is not assessed to be met and is larger when it is. Finally, we adopt a greedy selection scheme, i.e. macro action with the greatest $U(a|s)$ is selected.

At the lowest level of HLA,  employs a script policy to convert macro actions generated by  into atomic actions to interact with the environment.
The established macro action set includes the following types.

* Chop: Transform an ingredient into its chopped form.

* Mix: Combine chopped ingredients for cooking.

* Cook: Utilize mixed ingredients to cook a soup.

* Plate: Transfer a ready soup to a plate.

* Serve: Deliver a plated soup to the delivery point.

* Putout: Extinguish a fire on a pot using an extinguisher.

* Drop: Plate charred soup and discard it into a bin.

Most macro actions are equipped with a specified target, such as “Chop Lettuce” and “Cook Bob Soup”, consequently resulting in a total set of $21$ macro actions. Implementation details of the reactive policy can be found in Appendix <ref>. Given a macro action as a high-level goal,  chooses the most appropriate move and performs path planning to the target. We empirically found this to be particularly beneficial as shown in Sec. <ref>.

Note that the reactive policy of  shares similarities with goal-conditioned reinforcement learning <cit.>, which can, in turn, be trained by typical reinforcement learning algorithms <cit.>. We remark this as a topic for future studies.

§ EXPERIMENT

In this section, we consider three baseline agents to verify the hierarchical structure of HLA. We first test the real-time responsiveness of HLA and baselines by measuring action response latency. Moreover, we use a simple command set to evaluate the cooperative ability of each agent and a more complex command set to test command reasoning ability. Finally, we conduct human studies to collect the game scores and the human preference of all agents.

§.§ Baseline

To validate the effectiveness of the proposed hierarchical framework, we introduce three baseline agents, each lacking a certain component of the original HLA.

*  (SMOA). We remove the  and let the  produce macro actions. Current available macro actions are added to the input of the . The action filter is disregarded as GPT-3.5 is incapable of evaluating the probability for each macro action.

*  (FMOA). We remove the , including both  and . Due to the absence of , the dynamic adjustment term $\alpha$ in held static in Eq. (<ref>). We set a maximum length for action history, beyond which the human intention is assumed as fulfilled. History of human commands, current orders, and environment states are incorporated additionally into the input of .

*  (NEA). We remove the  and let the  choose atomic actions to control the agent directly. In addition to the original input, the  of the NEA incorporates environmental state information, such as the positions of all items on the map.

Details of the baseline agents can be found in Appendix <ref>.

§.§ Latency

We use macro action latency and atomic action latency to measure the real-time responsiveness of HLA and baseline agents. The macro action latency is defined as the time interval between receiving a human command and subsequently generating a macro action. And the atomic action latency is, in turn, the latency of an atomic action. In order to simulate the natural human-AI coordination, we build a command set and issue each command to the AI agent to evaluate its response latency. Details about latency measurement method and the command set can be found in Appendix <ref>.

The results of macro action latency and atomic action latency are depicted in Tab. <ref>, where lower number implies a faster action response time. The macro action latency of NEA is marked as “/” since it generates atomic actions directly. In HLA, when a human command is received, it is concurrently dispatched to both the  and the  therefore we report the macro action generation time of the .

Regarding the macro action latency, HLA produces $74.3\%$ lower latency than SMOA and $53.5\%$ lower latency than FMOA, suggesting that the hierarchical design significantly contributes to reducing response latency. FMOA exhibits a marginally shorter response time than SMOA, which can be attributed to the shorter inference time of the lightweight LLM, as well as the elimination of the intention completion assessment module in . Regarding the atomic action latency, HLA exhibits an order of magnitude advantage over the best competitor (0.28 vs. 0.08), demonstrating the real-time responsiveness of HLA. Among all agents, NEA performs the highest atomic action latency, indicating the importance of  that interacts with the environment with high frequency.

§.§ Performances with Simple Commands

An ideal AI agent should cooperate seamlessly with human players to achieve high game score, either in the absence of explicit human commands or after human commands is fulfilled.
Thus, we design two test cases where the human player either remains silent or speaks very little.
In both cases, the human player only chops ingredients and does not perform other tasks.

* No Command: The human player does not issue any command throughout the game.

* One Command: The human player asks the AI agent to prepare a specific soup at the start of the game, and the order for this soup only appears once at the start of the game.

We employ game score as a metric to assess the cooperative ability of HLA and baseline agents. The game score ascends every time an order is finished in time and descends whenever an order expires. We test both No Command and One Command on Map  and replicate the experiment five times with the same order sequence and keep the same experiment conditions. The average game scores under different test cases are depicted in Fig. <ref>, where the black line denotes standard deviation. Visualization of the gaming process can be found in Appendix <ref>.

Average game scores of HLA and baseline agents. Black line denotes standard deviation.

NEA fails to complete basic tasks, such as picking up things or approaching the target, and frequently gets stuck, therefore misses all soup orders and achieves minimum score of $-20$. This underscores the vital role played by the , which translates macro actions into atomic actions.
SMOA adequately follows the human command in the One Command case, but often overcooks dishes due to its high response latency, thus also performs poorly.
In the No Command case, FMOA serves the desired soups successfully but underperforms compared to HLA, primarily due to a high latency. However, in the One Command case, FMOA keeps making the requested soup even if it does not appear on the order list. In other words, FMOA fails to confirm the completion of a human command leads to suboptimal actions that do not fulfill the orders.

§.§ Interpreting Complex Commands

To see how well the agents comprehend and respond to commands of varying complexity, we design a complex command set comprising the following three challenges outlined in Sec. <ref>.

* Quantity specification (Quantity). We consider commands with specific numbers, e.g. “Chop 3 Lettuce.” or “Cook Bob Soup once.”

* Semantic analysis (Semantics). For example, the human gives a hint, “Alice Soup is about to timeout!” instead of a direct command like “Cook Alice Soup.”

* Ambiguous reference (Ambiguity). For example, after asking the agent to chop two onion, the human player asks, “Chop 1 more.” The AI agents need to infer the true intention based on environmental context and history commands.

We generate 10 different commands for each challenge. The details of the complex command set can be found in Appendix <ref>.
Each individual command is tested for 5 times. A command is considered successful if it succeeds at least 3 out of 5 attempts within 60s. In such cases, the command is labeled as passed, and its completion time is calculated as the average time taken for successful attempts. Otherwise, it is marked as failed and its completion time is marked as the maximum time of 60s. We report the average success rate and average completion time of commands in each challenge subset.

Tab. <ref> shows the results of HLA and baseline agents. NEA fails to execute any complex command and thus produces the worst performance.
Except NEA, FMOA exhibits the lowest success rate and the highest completion time when handling ambiguous commands, highlighting the significance of intention reasoning through the . SMOA performs poorly on quantity and semantics commands, displaying the longest completion time, indicating that although the  can interpret commands, it suffers from prolonged latency. HLA surpasses baseline agents with a notable advantage in both success rate and completion time, which demonstrates the effectiveness of the hierarchical design.

Ablation study on the two-stage design of .
The AI agent needs to infer human intentions and evaluate whether human commands are completed. In HLA, these tasks are performed separately by the two stages of .
In this section, we consider two variants of  to validate our two-stage design on interpreting complex commands.

* HLA without Intention Reasoning (no IR). We remove the  and use human commands directly as intentions for .

* HLA with one-stage  (one-stage). We combine the  and the  within , which now infers human intention, chats, and assesses intention completion simultaneously.

The results are shown in Tab. <ref>. HLA with one-stage  exhibits the longest completion time and the lowest success rate. We hypothesize that this is due to  having to handle multiple tasks simultaneously, resulting in poorer performance and longer inference time. HLA without Intention Reasoning performs worse than HLA(full) across all challenge subsets, particularly on ambiguous commands, indicating the significance of incorporating the .

§.§ Human Studies

§.§.§ Experiment Setting

We invite 60 volunteers for the human-AI experiment and divide them into 4 groups, each of which consists of 15 volunteers playing on a specific map. All volunteers are provided with a detailed introduction to the basic gameplay and the experiment process. They are fully aware of all their rights and experiments are approved with the permission of the department.

Considering the poor performance of NEA observed in Sec. <ref>, we only evaluate SMOA, FMOA, and HLA in this section. Each volunteer plays with the three AI players on the same map for two gaming phases, the preparation phase and the competition phase.
During the preparation phase, volunteers are required to collaborate with each of the three AI players for at least one round. In this process, human players familiarize themselves with the environment and engage in casual interactions with the AI players to explore their behaviors.
In the competition phase, volunteers can only play one round of the game with each of the three AI players to achieve the highest possible score.
We record the game scores and ask the volunteers to rank the three AI players based on their gaming experience.
We report the game scores and human preferences of SMOA, FMOA, and HLA in the following subsections.

§.§.§ Game Score

The average game scores on various maps from the competition phase are presented in Tab. <ref>. A high variance in scores can be attributed to the discrete nature of task rewards, where finishing an order yields $15$ to $20$ points.
SMOA exhibits the lowest scores on all maps due to its high response latency. Compared to SMOA, FMOA performs slightly better, averaging a 10-point advantage in scoring on each map. HLA outperforms both SMOA and FMOA across all maps with $\sim 50\%$ higher game scores, reflecting a significant advantage. In particular, HLA achieves a game score increase of over 40 points, i.e. serving at least two additional orders, on  and  when compared to baseline agents.
This highlights HLA's effective collaboration ability and real-time responsiveness.

Behavior Analysis.

We additionally calculate the ratio of valuable macro actions performed by each AI player, as well as the frequency of fire accidents on all maps during the competition phase.
A macro action is considered valuable if it produces a positive utility value upon execution. Conversely, useless macro actions may either be inherently invalid or become unexecutable due to high inference latency. We analyze three common macro actions, including Chop ingredients,Cook soups and Serve orders. The results are documented in Tab. <ref>. HLA stands out by executing the highest quantity of valuable macro actions and minimizing fire accidents, significantly outperforming SMOA and FMOA. Notably, the ratio of Serve macro action executed by HLA is 100%, indicating not only its capability to serve the correct order but also to do so promptly. This further helps to explain the excellent performance of HLA with regard to reasoning and real-time responsiveness.

§.§.§ Human Preference
In this section, we report the human preference on different AI players. Fig. <ref> and Fig. <ref> show the language communication preference and the overall preference of human participants respectively. Numbers indicate the difference of players who prefer row AI player over column AI player.

Communication Preference.
Fig. <ref> illustrates human feedback on communication accuracy, as well as consistency between chat messages and actions. The data is derived from both the preparation and competition phases. More than 20% of human players prefer HLA over SMOA and FMOA in terms of language communication accuracy as well as the consistency between chat messages and actions. We can also observe that FMOA outperforms SMOA with an advantage of $\sim$20% human preference, primarily due to SMOA's high latency according to the collected feedback. This underscores the pivotal role of real-time responsiveness of effective language communication.

Overall Preference.
Fig. <ref> reports the overall human preference of the preparation phase and the competition phase. Human participants expressed a strong preference for HLA over SMOA and FMOA, with a preference rate exceeding 30%. This preference was particularly evident during the competition phase, where HLA was favored over SMOA by as much as 50%. Human preference for FMOA remains higher than that for SMOA, at around 30%, aligning with the conclusion of communication preferences. Overall preference is a comprehensive metric of how human players evaluate an AI player's actions and communication. The strong preference on HLA indicates that it exhibits superior cooperative skills, faster responsiveness, and more consistent language communication.

Comm. Accuracy.

Comm. Consistency.

Human preference on communication accuracy, and consistency between chat message and actions.

Preparation Phase

Competition Phase

Overall human preference on different AI players.

§ CONCLUSION

We propose , an AI agent that can cooperate with humans using natural language in environments that require real-time execution.
Throughout the comprehensive experiments in an extended Overcooked, our method consistently excels in terms of game score, response latency, and human preference, showcasing reliable real-time human-AI cooperation.
Our method could be improved in a few key areas: substituting GPT-3.5 with GPT-4 in the  for enhanced semantic analysis, and replacing the scripted executor with an automatic one developed through goal-conditioned reinforcement learning to streamline scripting and boost low-level execution performance.

This research was supported by National Natural Science Foundation of China (No.62325405, U19B2019, M-0248), Tsinghua University Initiative Scientific Research Program, Tsinghua-Meituan Joint Institute for Digital Life, Beijing National Research Center for Information Science, Technology (BNRist) and Beijing Innovation Center for Future Chips.

We would suggest to visit our website <https://sites.google.com/view/overcooked-hla/> for more information.

§ ENVIRONMENT DETAILS

The original Overcooked benchmark <cit.> features a relatively straightforward menu and cooking process. In each gameplay session, a human player collaborates with an AI player to fulfill the orders. There is only one type of order, which is an onion soup made from three onions. Completing an order involves merely four steps, i.e., retrieving the necessary ingredients, mixing them in a pot to create a soup, plating it, and serving at the counter. Afterwards, the players receive a reward of 20.
Subsequent research expands upon this environment. For example, in <cit.>, players must use a fire extinguisher to put out the randomly appearing fire.

To make the benchmarks more challenging, we incorporate mechanisms from the original Overcooked game and make the following extensions:

* Diverse Ingredients. We offer three ingredients, i.e., onion, tomato, and lettuce.

* Chopping Mechanism. All ingredients must be chopped on the chop board before they are mixed and put in the pot. Player needs to interacts with the chop board 8 times to chop the ingredient.

* New Dishes.
We design four dishes: Alice Soup, made of one onion and one lettuce; Bob Soup, made of one tomato and one lettuce; Cathy Soup, made of one tomato and one onion; and David Soup, which includes all three ingredients. The cooking time for all soups is 15 seconds.

* Fire Mechanism. Leaving the cooked soup in pot for 25 seconds would overcook the soup and set the pot on fire. Players must use a fire extinguisher to put out the fire. The putout process takes 5 seconds. After the fire is put out, players can plate the overcooked food and discard it.

* Trash Can. Players can dispose of any unwanted ingredients or overcooked dishes in a trash can to free up space.

* Order Timeout. During the game, soup orders appear randomly. Orders for Alice, Bob, and Cathy Soup are valid for 60 seconds, while orders for David Soup last 70 seconds.  If finished in time, Alice, Bob, and Cathy soups give a reward of 15, and David soups give a reward of 20. Otherwise, the order disappears and gives a negative reward of -5.

* Human-AI Chat Interface: To simulate natural human interactions in the original game, we add a chat interface for communication between the human player and the AI player. The human player can either pause the game and communicate with the AI using a chat window or speak directly during gameplay. The AI can respond via text or speech.

The action space of the extended Overcooked testbed is thus composed of two parts: the atomic actions and the texts for communication. Specifically, the atomic actions include only four elements, i.e., up, down, left, and right, which control the character's position and its interaction with other objects.

We design 4 distinct maps for the extended Overcooked testbed. In each game, a human player (the pink beard character) collaborates with an AI player (the blue character) to complete the orders.
Whenever an order is fulfilled or timeouts, a new order appears, ensuring that there are 3 valid orders in the game at any given time.
Each game lasts for 100 seconds, and the default action frequency of the AI player is 2.5 Hz, resembling the natural action frequency of human players.
The first two maps assess the general cooperation.  employs a ring-like layout, while  creates a bottleneck that encourages the human player and the AI player to stay within a certain area.
The third map, , separates the two players, necessitating task division and cooperative coordination.
In the fourth map, , we raise the number of concurrent orders to 4 and the speed of the player to 3.5 Hz to intensify the gameplay.

To finish an order, players must follow a precise sequence of steps: retrieving the necessary ingredients, chopping them, assembling them as per the recipe, cooking them to make a dish, plating the dish before it is charred, and finally, serving it at the counter. If a soup cooks for a long time, it gets overcooked and the pot catches fire. Players must putout the fire with a fire extinguisher, plate the charred soup, and drop it into the trash bin.

§ METHOD DETAILS

§.§ Prompt Details

§.§.§ Prompt of  in

As discussed in Sec. <ref>, in , HLA takes in the human's chat message and reasons the human intention. Here, we divide the chat message into 6 categories, including useless message, short-term request, complicated instruction, long-term request, questions, and others. If the human player asks a question,  passes the question directly to , which then generates a chat response.

The full prompt is shown in Fig. <ref>. Mutable prompt components are marked in $<\,>$, such as $<Soup Orders>$.

§.§.§ Prompt of  in

As discussed in Sec. <ref>,  generates chat messages, whereas  assesses intention completion.
Depending on whether the human intention is satisfied,  works in different mode.
If the human intention is not satisfied, we ask the  to output its inner reasoning, intention completion assessment, and chat message. The full prompt of this mode is shown in Fig. <ref>. If the human intention is satisfied, we prompt the  to output chat message response only, as shown in Fig. <ref>.

As discussed in Sec. <ref>, the  uses a conditional prompt depending on whether the human intention is inferred and whether it is satisfied.

The full prompt is shown in Fig. <ref>, where the control conditions are marked in orange color.  takes human chat message, human intention reasoned by , or chat message generated by  as input, depending on the condition.

§.§ Script Policy

As detailed in Sec. <ref>, we have designed seven types of macro actions and implemented a hard-coded script policy to perform them in the . Each macro action type serves a specific function. Certain types of macro actions necessitate a specific target.

Within a specific environmental context, the value of each macro action is evaluated, as denoted by $V(a|s)$ in Eq. (<ref>). This evaluation draws upon human expert insights and considers various factors such as the items located on the map, current soup orders, and the readiness of each macro action. For example, the macro action “Plate Alice Soup” is given an initial utility value of $0.5$ if there's an active order for Alice Soup. In the absence of such an order, its utility value is set to $0$. However, if Alice Soup lingers in the pot for too long and is at risk of overcooking, the utility value for this action escalates to $1.0$. This increased value acts as a prompt for the agent to give higher priority to serving the soup quickly.

The seven types of macro actions are as follows.

* Chop: Transform an ingredient into its chopped form. The target can be Onion, Lettuce or Tomato. The value can be $0$ or $0.5$, depending on whether the ingredient needs to chop in current situation.

* Mix: Combine chopped ingredients for cooking. The target can be Alice Ingredients, Bob Ingredients, Cathy Ingredients, or David Ingredients. The value can be $0$ or $0.52$, depending on whether there is an unfinished order that needs the specified mixed ingredients.

* Cook: Utilize the mixed ingredients to cook a soup. The target can be Alice Soup, Bob Soup, Cathy Soup, or David Soup. The value can be $0$ or $0.54$, depending whether there is an unfinished order that needs cooking the specified soup.

* Plate: Plate a soup. The target can be Alice Soup, Bob Soup, Cathy Soup or David Soup. The value can be $0$ or $0.56-1$, depending on whether the soup needs serving in hurry, and how long it overcooks.

* Serve: Deliver a plated soup to the delivery point. The target can be Alice Soup, Bob Soup, Cathy Soup, or David Soup. The value can be $0$ or $0.58-1$, depending on the remaining time of soup order.

* Putout: Extinguish a fire on a pot using an extinguisher. No target needs to be specified. The value is always $0.6$.

* Drop: Plate overcooked soup and discard it into a bin. No target needs to be specified. The value is always $0.6$.

Most macro actions require at least one empty grid for execution. We have integrated an additional function to all of the macro actions to clean the grids if necessary.

Each macro action is flagged as available when it can be executed immediately. For instance, “Plate Alice Soup” becomes available when there is a reachable pot with cooked Alice Soup and a reachable plate. There might be instances when a macro action cannot proceed halfway, for example, the soup gets overcooked when the agent is plating it. Under these circumstances, the macro action will return a “Failed” message, prompting the agent to generate a new macro action. The implementation details of the script policy can be found in the open source code.

A macro action like “Chop Tomato” involves a sequence of atomic actions including directional movements such as up, down, left, or right. The script policy in this context employs Breadth-First Search (BFS) for efficient pathfinding, while also continuously tracking the environment's state during interaction. For example, in executing the macro action “Chop Tomato,” the agent follows a structured procedure: it navigates to the tile where tomatoes are located, picks up a tomato, places it on an unoccupied cutting board, and then chops it. After chopping, the agent moves the chopped tomato to a vacant counter area for future use. Within this workflow, specific actions like “move to tomato tile,” “place tomato on cutting board,” and “transfer chopped tomato to counter” are managed by the BFS algorithm. These actions are further broken down into a series of atomic actions, each involving movement in designated directions. A key aspect of this script policy is its design to minimize interference with the human player's actions.

§ IMPLEMENTATION OF BASELINE AGENTS

In  (SMOA),  is discarded. Thus,  in  takes in available macro actions as additional input and generates macro actions directly. If the generated macro action is not legal, the LLM regenerates a new one. The Action Filter in  of HLA needs to combine the value of each macro action and the probability of LLM choosing the specific action. Since GPT-3.5 does not provide a valid API to evaluate such probability, we discard Action Filter in SMOA.

The workflow of  is shown in Fig. <ref>, where  is discarded. The full prompt of  in  is shown in Fig. <ref>. In the prompt, we append an additional chat round to let the LLM generate a macro action.

Workflow of . The  is discarded, and  generates macro actions directly.

In  (FMOA),  is discarded, including both  and . The  in FMOA takes in current soup orders, current items, and the history of human commands as additional input. The LLM also generates a chat message aside from a macro action in . Due to the absence of , FMOA cannot tell whether the human intention is satisfied, and therefore cannot calculate the dynamic adjustment term $\alpha$ in Eq. (<ref>). We set $\alpha$ to be the same as in HLA when human intention is not satisfied. We set the maximum length for macro action history to be $9$, which is roughly the length of finishing $2$ soup orders from scratch (Cook 1 Alice Soup needs 2 Chop, 1 Mix and 1 Cook, while David Soup needs 3 Chop, 1 Mix and 1 Cook). The action history is cleared once the human issues a new command. We assume the human intention is satisfied once the action history reaches the maximum length.

The workflow of  is shown in Fig. <ref>, where  is discarded. The full prompt is shown in Fig. <ref>. The prompt conditions on whether the human intention is satisfied.

Workflow of . The  is discarded, and  generates chat message directly.

In  (NEA), the  is discarded, and the  has to generate atomic actions instead of macro actions. To provide spatial information for the , we append the positions of items and players to the input of .

The workflow of  is shown in Fig. <ref>, where  is discarded. The full prompt is shown in Fig. <ref>. In the prompt, we keep the conditional prompt module and add additional position information.

Workflow of . The  is discarded, and  generates atomic actions directly.

§ IMPLEMENTATION DETAILS OF COMMAND SET

§.§ Command Set for Latency Test

We construct a command set to simulate the real human commands. During the latency test, we do not concern about the actual actions of agents but the latency of their response, so we issue the commands sequentially with a 20-second interval in one round of game play. The response time of each agent is recorded. The test cases are outlined below.

* You are free to do anything.

* Try your best to earn more points.

* Focus on the orders.

* Chop 3 Lettuce.

* Chop 1 Onion.

* Chop 2 more.

* Cook Bob Soup.

* Cook it again.

* Alice soup is about to timeout!

* Watch out for the Cathy Soup order.

* Help me with the third soup on the orders please.

* Chop more vegetables.

* Aba Aba. Chop 1 potato.

* What are the orders?

* What is Alice Soup?

We use macro action latency and atomic action latency to measure the real-time responsiveness of HLA and baseline agents. The macro action latency is defined as the time interval between receiving a human command and subsequently generating a macro action.

The atomic action latency is the latency of an atomic action. For SMOA, FMOA and HLA, the AI agent only suffers from the latency of generating an macro action. In other words, these agents has the latency of generating the first atomic action after receiving a human command, but can output the following atomic actions swiftly once the macro action is generated. This is because  can translate macro action in to atomic action with minimal latency.
For NEA, the atomic action is directly generated by , which means NEA suffers from a latency every time it produces an atomic action.

To fairly compare the atomic action latency, we calculate the mean atomic action generation latency for SMOA, FMOA and HLA, which is more consistent with the subjective feelings of human players.
For SMOA, FMOA and HLA, the atomic action latency is defined as $T_a = \frac{T_m}{N_a}$, where $T_m$ is the generation latency of the first macro action after receiving a human command, and $N_a$ is the number of atomic actions in this macro action. For NEA, the atomic action latency is defined as the generation latency of the first atomic action after receiving a human command.

§.§ Complex Command Set

As discussed in Sec. <ref>, we construct a command set consist of a total of 30 complex commands for 3 challenges mentioned in Sec. <ref>. Experiment are conducted using the Map . The commands for quantity specification challenge are as follows.

* Chop 1 Onion.

* Chop two onions.

* Please chop 3 onions.

* Cut one Tomato.

* Help me cut 2 tomatoes.

* 3 chopped tomatoes please.

* Chop 1 Lettuce.

* 2 lettuces chop.

* help me to chop 3 lettuces.

* Cook Alice Soup once.

The commands for semantic analysis challenge are as follows.

* I need more onions

* Chop but except tomato and lettuce.

* Why are we always short of tomatoes?

* Just pass me toma and don't ask why.

* Can't you see the lettuce, uh?

* The green cabbage looks perfect!

* Bob Soup is about to timeout!

* Oh god, I forget the alice soup order.

* Come on! There is a cathy order!

* D soup!

The commands for ambiguous reference challenge are as follows.

* Chop 2 Onions -> Chop it again.

* Chop 3 Tomatoes -> One more please.

* Cut one lettuce -> Cut more!

* Cook Bob Soup. -> Cook it again!

* Cook 2 cathy soup, -> Can you do it again?

* Cook david soup once -> Help me with that again.

* Cook the first soup in the orders

* Cook the second order now!

* The third soup order should be cooked

* Please help me cook the last soup order

During the test of each command, the soup orders are carefully designed and fixed. For chopping commands, the target ingredient doesn't appear in any soup orders. For cooking commands, we ensure that the required soup is not part of any soup orders, except for the final 4 commands in ambiguous reference challenge, where the commands directly correspond with the orders.

§ ADDITIONAL RESULTS

All experiments were conducted on a computer equipped with A100-80G GPU.

§.§ Latency of Each Module

In addition to the end-to-end latency discussed in Sec. <ref>, we also measure the latency of each module of HLA and baseline agents, as shown in Tab. <ref>, where IR denotes the , CA denotes the , and MA denotes Macro Action Generation in the . Both SMOA and FMOA suffers from high latency, which originates from generating macro actions and giving chat message at the same time. This further validates the hierarchical design of HLA which can decouple these components.

§.§ Visualization of Simple Commands

As discussed in Sec. <ref>, we use two simple commands, No Command and One Command, to test the cooperative ability of HLA and baseline agents. The visualization results of  (NEA),  (SMOA),  (FMOA), and HLA are shown in Fig. <ref>, Fig. <ref>, Fig. <ref>, and Fig. <ref>, respectively. The game screenshots are captured at $1/3$ of gameplay, $2/3$ of gameplay, and upon completion of the game.
The game score can be found in the bottom-left corner of the screenshot. The blue text within the screenshot shows the human player chat message and AI agent chat response.

NEA is stuck at the top left corner of the map. This is because NEA continuously gives the atomic action of moving left and is stuck next to the tomato tile. SMOA successfully mixes the ingredients and cooks soups. But upon completion of the game, the AI player fails to plate the soup in time, and thus overcooks the soup and sets the pot on fire. FMOA gives hallucinating chat response. In No Command, at $2/3$ of gameplay, FMOA talks about “the burning pot”, but no pot is on fire at that time. In One Command, at $2/3$ of gameplay, FMOA still talks about “Bob Soup is almost ready” and tries to cook Bob Soup, but the Bob Soup is already served prior and there is no Bob Soup order at that moment.

HLA manages to use all 3 pots simultaneously upon completion of the game in One Command. In One Command, at $2/3$ of gameplay, HLA talks about “Cathy Soup is almost charred” and plates Cathy Soup afterwards, showing a consistency of its action and chat message. This chat message comes out before the screenshot is taken.

We suggest visiting our website for more video demonstrations.

No Command

One Command

Visualization result of NEA for latency test on simple command set. Screenshots are captured at $1/3$ of gameplay, $2/3$ of gameplay, and upon completion of the game.

No Command

One Command

Visualization result of SMOA for latency test on simple command set. Screenshots are captured at $1/3$ of gameplay, $2/3$ of gameplay, and upon completion of the game.

No Command

One Command

Visualization result of FMOA for latency test on simple command set. Screenshots are captured at $1/3$ of gameplay, $2/3$ of gameplay, and upon completion of the game.

No Command

One Command

Visualization result of HLA for latency test on simple command set. Screenshots are captured at $1/3$ of gameplay, $2/3$ of gameplay, and upon completion of the game.

§.§ Introducing Action Filter into SMOA

To assess the potential of adding an action filter to the Slow-Mind-Only Agent (SMOA), we set up GPT-3.5 to create both the macro action and its estimated probability independently, despite GPT-3.5 lacking a designated API for such probability evaluation. The prompt given to GPT-3.5 is to “output the action and the probability of each action.” In response, GPT-3.5 produces outputs like “{Chop Tomato: 0.33, Chop Lettuce: 0.33, Chop Onion: 0.33}.” An action filter is then applied to process both the action and its probability. This method is referred to as “SMOA+action filter”.

The results of our experiments are presented in Tab. <ref>, Tab. <ref>, and Tab. <ref>. In these tests, we found that adding an action filter to SMOA resulted in about a 50% increase in macro action latency compared to the standard SMOA. Furthermore, the performance for both simple and complex commands decreased notably. This decrease in performance is attributed to the extended output of GPT-3.5 when it is required to generate actions along with their probabilities, which in turn leads to longer response times. Another factor contributing to this decline is the method of generating action probabilities. Relying on the LLM to estimate the probability of each action candidate proved less accurate than utilizing the “logprob” feature from the LLM's output head. However, as this feature is currently not available in GPT-3.5, it impacted the overall performance negatively. Based on these findings, we decided to omit the action filter from SMOA in our comparisons to ensure fairness.

§.§ Comparison of Different LLMs

In this part, we analyze the performance differences when using various large language models (LLMs) within the  and the . Specifically, “HLA-Fast7B” indicates the use of Llama2-7B-chat as the  instead of Llama2-13B-chat, while “HLA-Slow70B” refers to the use of Llama2-70B-chat in the , replacing GPT-3.5. Both of the models use the same quantization technique as in HLA. The results regarding macro action latency and atomic action latency are detailed in Tab. <ref>. Tab. <ref> presents the average game scores for scenarios under No Command and One Command test cases on Map . Additionally, Tab. <ref> outlines the average success rate and completion time when processing complex commands.

Tab. <ref>, Tab. <ref> and Tab. <ref>, illustrate the performance of HLA-Fast7B, revealing a reduction of 29% in macro action latency and 25% in atomic action latency compared to HLA. This enhanced performance is primarily attributed to the lower computational requirements of Llama2-7B-chat. However, a notable limitation of HLA-Fast7B is its diminished efficacy in handling “One Command” test cases and in interpreting semantically complex commands. As a result, for the role of the , we have opted for Llama2-13B-chat, which offers a well-rounded balance between the reasoning capability and the inference speed.
On the other hand, HLA-Slow70B exhibits similar macro and atomic action latencies but struggles significantly with both simple and complex commands. This underscores the critical role of the Slow Mind's strong reasoning capabilities. Consequently, in our study, we have chosen GPT-3.5 as the  due to its superior performance in these areas.

§.§ Interpreting Complex Commands

Tab. <ref> in the main paper illustrates the success rate of HLA in interpreting different subsets of complex commands. It was observed that $10\%$ of semantically complex commands and $30\%$ of ambiguous commands were not successfully interpreted. Additionally, Tab. <ref> provides a detailed analysis of the failure rates attributable to the  and the  for these two subsets of commands. HLA(slowmind) denotes the failure rate linked to the , focusing on incorrect reasoning of intentions or assessment of task completion. HLA(fastmind) indicates the failure rate associated with the , where incorrect reasoning and assessment are identified and replaced with correct ones, followed by a recalculation of the failure rate.

As indicated in Tab. <ref>, the  is identified as the cause of all observed failures. The  incorrectly interprets human intentions as “None” or wrong intentions, which leads to the failures. For example, the semantic command (3), “Why are we always short of tomatoes?”. The  interprets the intention behind this command as “None”. Similarly, with the ambiguous command (6), “Cook David soup once -> Help me with that again”, the intention deduced by the  is incorrectly identified as “Cook Bob Soup once and Cook Alice Soup once.”

Further examination reveals that when we provide the correct human intentions and task completions to the , it still exhibits a 33% failure rate. For example, consider the ambiguous command (5), “Cook 2 Cathy soup, -> Can you do it again?”. Both the  and  fail in this instance. The  misinterprets the human intention as “None.” However, even when supplied with the correct intention, the  still executes some unnecessary macro actions before cooking the second Cathy Soup. This inefficiency results in time wastage and failure to complete the command within the designated time limit.

§.§ Human Studies

We additionally report the latency of each module, end-to-end latency, hit rate of macro actions generated by , detailed human preference, and visualized replay in the following subsections.

§.§.§ Details on Behavior Analysis

As mentioned in Sec. <ref>, we calculate the rate of valuable macro actions, i.e.,
$p_{valuable} = \frac{N_{valuable}}{N_{all}}$, where $N_{valuable}$ is the number of the specified macro actions that are valuable, and $N_{all}$ is the total number of the specified macro actions.
The detailed definition of useful macro actions are as follows.

* Chop. The moment the ingredient is put onto the cut board, a relevant unfinished order that hasn't been cooked exists, and no chopped form of this ingredient is on the map for this order.

* Cook. The moment the mixed ingredients are put into an empty pot, an unfinished order that hasn't been cooked exists.

* Serve. The moment the soup is served to the delivery point, an unfinished order of the specified soup exists.

Besides, we also examine the rate of fire accidents, i.e., $p_{fire} = \frac{R_{fire}}{R_{all}}$, where $R_{fire}$ is the number of game rounds that at least one pot catches fire, and $R_{all}$ is the number of total game rounds.

Additional results of each map can be found in Tab. <ref>, Tab. <ref>, Tab. <ref> and Tab. <ref>. In , the AI player cannot get access to the delivery point and doesn't serve any soup. The result of “Serve” is set to “/” in the table.

§.§.§ Latency of Each Module

Tab. <ref> shows the latency of each module of different AI players in the preparation phase of human studies, and Tab. <ref> shows the results in the competition phase.
IR denotes , CA denotes , and MA denotes Macro Action Generation in . The results are consistent with the previous results in Sec. <ref>. Simultaneous generation of macro actions and chat messages introduces significant latency into SMOA and FMOA.

§.§.§ End-to-end Latency

We report the end-to-end latency during the preparation phase and the competition phase, as shown in Tab. <ref>.
Due to the varying network conditions among participants, the latency of chat generation for both the HLA and SMOA increases.

§.§.§ Hit Rate of Immediate Macro Action in the

When human's chat message arrives,  generates a macro action directly based on it, which is referred to as an immediate macro action.
Additionally, we measure the hit rate of immediate macro actions in HLA, which denotes the proportion of immediate macro actions that are consistent with final macro actions generated according to the human intention. The results are shown in Tab. <ref>.
Despite notably faster, immediate macro actions generated by  satisfy human intentions most of the time. This suggests that HLA is capable of adhering to human commands with a rapid action response in most scenarios.

§.§.§ Human Preference

We ask the volunteers to rank and comment on HLA, SMOA, and FMOA both after the preparation phase and the competition phase. We provide 6 ranking metrics.

* AI Effectiveness: Whether the AI player can complete the dishes and obtain high scores.

* AI Assistance: Whether the AI player can help human complete orders.

* AI Responsiveness: Whether the AI player can respond in action quickly after human issues a command.

* AI Text Communication Accuracy: Whether the information conveyed by the AI player is correct.

* AI Text Communication and Action Consistency: Whether the answers given by the AI player is consistent with the actions it takes.

* Overall Performance: Evaluate the overall performance of the AI player based on gameplay experiences.

Detailed results of human preference of both phases are shown in Fig. <ref> and Fig. <ref>. HLA remains the most preferred in all aspects. FMOA is more preferred than SMOA in all aspects.

Response Time

Comm. Accuracy

Comm. Consistency

Human preference in the preparation stage of human studies. Numbers indicates difference of players who prefer row AI player over column AI player.

Response Time

Comm. Accuracy

Comm. Consistency

Human preference in the competition stage of human studies. Numbers indicates difference of players who prefer row AI player over column AI player.

§.§.§ Replay

We suggest visiting our website for more video demonstrations.

We present visualizations depicting four typical human players on different maps during the competition phase. These visualizations are shown Fig. <ref>, Fig. <ref>, Fig. <ref>, and Fig. <ref> respectively.
The player-128 remains silent through the whole game play. The player-221 and player-421 instruct AI players by texting messages while the player-322 instructs the AI player via speaking. The game screenshots are captured at $1/3$ of gameplay, $2/3$ of gameplay, and upon completion of the game.
The game score can be found in the bottom-left corner of the screenshot. The blue text within the screenshot shows the human player chat message and AI agent chat response.

Cooperating with silent players, AI agent should mainly help the human player get the highest score possible. Moreover, the pace of the game is faster when use speaking to communicate, and texting can the give human player more time to think.
As illustrated in the replay, collaborating with HLA results in the highest game scores.
The lower latency of HLA enables more ingredients to be chopped and placed on the counters compared to other AI players in both  and . Furthermore, HLA utilizes more pots simultaneously in  and .

Visualization result of player-128 in  during the competition phase. Screenshots are captured at $1/3$ of gameplay, $2/3$ of gameplay, and upon completion of the game.

Visualization result of player-322 in  during the competition phase. Screenshots are captured at $1/3$ of gameplay, $2/3$ of gameplay, and upon completion of the game.

Visualization result of player-221 in  during the competition phase. Screenshots are captured at $1/3$ of gameplay, $2/3$ of gameplay, and upon completion of the game.

Visualization result of player-421 in  during the competition phase. Screenshots are captured at $1/3$ of gameplay, $2/3$ of gameplay, and upon completion of the game.