#  Interaction is all You Need?
A Study of Robots Ability to Understand and Execute

Kushal Koshti  Nidhir Bhavsar
Universität Potsdam
{kushal.koshti<EMAIL_ADDRESS>
###### Abstract

This paper aims to address a critical challenge in robotics, which is enabling
them to operate seamlessly in human environments through natural language
interactions. Our primary focus is to equip robots with the ability to
understand and execute complex instructions in coherent dialogs to facilitate
intricate task-solving scenarios. To explore this, we build upon the Execution
from Dialog History (EDH) task from the TEACh benchmark. We employ a multi-
transformer model with BART LM. We observe that our best configuration
outperforms the baseline with a success rate score of $8.85$ and a goal-
conditioned success rate score of $14.02$. In addition, we suggest an
alternative methodology for completing this task. Moreover, we introduce a new
task by expanding the EDH task and making predictions about game plans instead
of individual actions. We have evaluated multiple BART models and an LLaMA2
LLM, which has achieved a ROGUE-L score of $46.77$ for this task. We make our
findings accessible over here111https://github.com/Nid989/TEACh_EDH.

Interaction is all You Need?
A Study of Robots Ability to Understand and Execute

Kushal Koshti and Nidhir Bhavsar Universität Potsdam {kushal.koshti,
<EMAIL_ADDRESS>

##  1 Introduction

To enable robots to operate efficiently in human environments, they need the
capability to engage in natural language interactions. Achieving this has long
been a goal in modeling AI agents, focusing on seamless interaction with
humans to assist in task-solving. This involves understanding and executing
instructions and facilitating dialog to clarify doubts and correct mistakes
Skrynnik et al. (2022).

In general, robots operating in the field should acquire the ability to
synchronize natural language with their operational environment, which
inherently ties into the concept of symbol grounding, concentrating on
establishing connections between language and static images Harnad (1990).
Over the years, various proposals have emerged to address the challenge of
human-AI collaboration, primarily focusing on humans instructing agents to
achieve specific goals (Shridhar et al., 2019; Gluck and Laird, 2019).
Typically, these tasks are executed in a simulated environment such as the
Minecraft virtual environment Narayan-Chen et al. (2019), AI2-THOR Kolve et
al. (2017), AI Habitat Savva et al. (2019), or Matterport 3D Anderson et al.
(2017). These environments confine the agent within limited boundaries, thus
accelerating its learning capabilities.

The introduction of TEACh (Task-driven Embodied Agents that Chat) by
Padmakumar et al. (2021) has brought forth benchmarks that effectively address
the aforementioned challenges, paving the way for the development of
conversational robotics in household tasks.

In this paper, we aim to address the strategy of using trained models to
attain elements of embodied intelligence. Similar to the TEACh benchmark
dataset, we tackle the task of Execution from Dialog history (EDH). This
involves utilizing the concept of instruction-oriented multi-party dialog
conversation and employing the ego-centric camera observations perceived by
the robot agent to carry out the desired task. Furthermore, we analyze the
dataset to uncover nuances and elements that hinder the performance of the
modeled system.

Overall, the problem aims at agents needing to understand the task
instructions provided at varying levels of abstraction across dialogs. We
conduct experiments aimed at achieving the task of EDH through the utilization
of language generation techniques. Consequently, we provide an explanation
regarding the potential unsuitability of this approach for the given dataset.
Finally, We try to gain valuable insights by leveraging the natural language
component of the dataset to assist in creating human-understandable plans for
executing a given task.

To summarize, our main contributions are:

  * •

C1: Modifying the baseline EDH (Execution from Dialog History) implementation
with GPLM-based language encodings and adjusted multimodal fusion.

  * •

C2: Introducing an alternative approach for the task and proposing a modeling
technique for it.

  * •

C3: Implementing a language-orientated task to generate game plans
constituting action-object pairs based on dialog instructions.

##  2 Related Work

###  2.1 Vision & Dialog Navigation and Task Completion

Certain embodied agent tasks demand the incorporation of visual observations
and language instructions to model behavior, particularly in executing
navigational actions such as turning or moving forward. The development of
models for such behavior within controlled environments has progressed
significantly, transitioning from using symbolic environments (MacMahon et
al., 2006; Chen and Mooney, 2011) to adopting photorealistic indoor Anderson
et al. (2017) and outdoor Narayan-Chen et al. (2019) settings.

Accordingly, some tasks entail combining such navigational actions to
accomplish a specific goal. These models encompass a combination of rules and
learned components that enable them to understand language Kollar et al.
(2017). Others operate within a fully observable blocks world (Bisk et al.,
2017; Misra et al., 2018). Each of these models depends on either human
demonstrations (Misra et al., 2018; Shah et al., 2021) or predefined annotated
tasks Shridhar et al. (2019) for replicability.

Moreover, different approaches have been adopted on a conglomerated scale to
explore how agents collaborate in achieving predefined objectives. These have
evolved into utilizing both the task description and the dialog interaction
between the user and the agent as the language component. For example, in a
Minecraft simulator, a collaborative building task involves an architect and a
builder Narayan-Chen et al. (2019). Similarly, the IGLU (Improving Grounding
and Language Understanding) task proposed by Kiseleva et al. (2022) models the
behavior of both the architect and the builder separately. While the architect
focuses on generating optimized and meaningful instructions, the builder aims
to understand the instructions and operate within a simulator to complete the
given task. Building upon the IGLU task, certain systems have been developed
to establish collaborative two-way communication, enabling the builder to seek
clarification from the architect Mehta et al. (2023).

###  2.2 Multimodal transformers and Language Models

Over the years, transformer-based language models have made significant
strides in various tasks Vaswani et al. (2017). Multimodal transformer
learning, focusing on integrating diverse modalities, has gained momentum over
the years, allowing for an unprecedented synchronization range. Certain works
use the transformer architecture to encode specific modalities Dosovitskiy et
al. (2020). In the field of vision, numerous works have actively utilized
language models and fusion techniques, closely linking them to contextually
encoded pairs of multimodal data. For example, ViLBERT encodes visual and
textual data, employing cross-attention at specific architecture layers Lu et
al. (2019). Similarly, LXMERT processes visual features and language data
using cross-modal attention Tan and Bansal (2019). In the context of embodied
agents, a multi-transformer setting is used to sequentially process visual
data guided by textual instructions Pashevich et al. (2021).

###  2.3 Language aspects of Embodied Agent Planning

Improving the natural language communication abilities of embodied agents is
crucial. Ideally, users should converse with the robot agent as they would
with a person, with the robot efficiently extracting information and
completing tasks. Additionally, integrating dialog acts that convey the
communication intent of the conversation is beneficial Gervits et al. (2021).
This inclusion of dialog acts and their corresponding utterances aids in
defining the overall plan required for specific task completion Gella et al.
(2022).

###  2.4 LLMs as Embodied Agents Planner

The current advancements in Large Language Models (LLMs) have led to various
contributions in addressing embodied agents’ challenges. These models leverage
their extensive generalized knowledge to assist in this task. In their study,
Song et al. (2023) explore the utilization of LLMs as planners for embodied
agents operating in visually perceived environments. They harness the
capabilities of large language models to perform few-shot planning for these
agents. Similarly, in their work, Zhang et al. (2023) demonstrate the
remarkable planning skills of LLMs in a single-agent embodied task. They also
successfully illustrate how these acquired concepts can be applied in multi-
agent scenarios where agents coordinate and communicate effectively in
confined environments. Using LLMs can help expand the constraint environment,
enabling the agent to become familiar with various scenarios Wang et al.
(2023).

##  3 Dataset Description

We utilize the TEACh dataset Padmakumar et al. (2022) in this task. The
dataset is structured into various formulations, each aligning with a
particular task. Essentially, it consists of dyadic conversations where human
annotators assume the roles of users (Commander) and robots (Follower),
working together on completing certain household tasks. In every dialog
session, the Follower has a specific high-level task, such as Make Plate of
Toast or Prepare Sandwich. The Commander possesses the details of this task,
while the Follower does not. The task involves multiple sub-goals that the
Follower must accomplish. To achieve this, the Commander conveys instructions
to the Follower through dialogs. The Follower can backchannel or communicate
with the Commander, seeking specific details and clarifications to enhance
task efficiency. These clarifications often relate to inquiries regarding
location, specific objects, or the steps to complete a task. The Follower now
incorporates this collected data into a simulator to achieve its presumed
goal. The data comprises around 3K recorded human-human gameplay sessions
assimilated in an AI2-THOR virtual environment Kolve et al. (2017).

In the TEACh dataset, every session includes an initial state denoted as
$S_{i}$ and a final state denoted as $S_{f}$. In addition, for a specific
task, we have access to the aggregated subgoals, a sequence of actions
performed by the Follower, represented as $A=(a_{1},a_{2},\ldots)$, and the
dialogs between the Commander and Follower. Furthermore, the dataset offers
synchronized visual frames extracted from the simulated environment in
conjunction with the action sequence. For future reference, we denote the
sequence of dialog actions as $A^{D}$ and all navigation and interaction
actions as $A^{I}$.

##  4 Task Formalisation

###  4.1 Execution from Dialog History (EDH)

In this task, we must automatically model how the Follower agent predicts
actions to achieve a specific task or subgoal.

An EDH instance is formed by taking a sub-sequence of a given gameplay
session. We define an EDH instance as
$E_{i}=(S^{E},A_{H},A^{I}_{R},I_{H},F^{E})$. Here, $S^{E}$ is the initial
environment state of the instance at time-step $t=0$, and $F^{E}$ is the
expected final environment state of the instance at $t=T_{f}$. The agent is
given the action history $A_{H}$ along with Follower ego-centric image history
$I_{H}$ till $t=T_{i}(0\leq T_{i}\leq T_{f})$. $E_{i}$ also comprises
reference (ground-truth) interaction actions ($A^{I}_{R}$) for supervision.

Let $\mathcal{A}$ be the set of all actions, and $\mathcal{I}$ be the set of
Follower ego-centric images. The task here is to learn a function
$f:\mathcal{A+I}\rightarrow\mathcal{A}$, that predicts the future sequence of
actions $A_{F}$, which lead to the final environment state of the instance
$F^{E}$. $A_{F}$ can be derived as:

$A_{F}=\\{a_{T_{i}},...,a_{T_{f}}\\}$ (1)
$a_{t}=f(\\{A_{H}+a_{t-1}\\},\\{I_{H}+i_{t-1}\\})$ (2)
$i_{t}=simulator(a_{t})$ (3)
$a_{t}\in\mathcal{A},i_{t}\in\mathcal{I},(T_{i}\leq t\leq T_{f})$

Equation 3 represents the Follower ego-centric image in the simulator after
processing action $a_{t}$.

###  4.2 EDH Plan Prediction

Gella et al. (2022) defines task planning as generating a sequence of symbolic
actions that guide a robot’s high-level behavior to complete a task. A plan
comprises a series of object manipulations that must be executed to achieve
success. For instance, the robot may need to navigate to specific objects, and
additional steps may be necessary depending on the environment’s state.
Despite this, generating a plan based solely on the dialog should be feasible.
We extend the EDH task by predicting the future plans $P_{F}$ instead of
future actions $A_{F}$.

A reference plan $P_{R}$ is created by making
$\\{\text{action},\text{object}\\}$ pair from the TEACh dataset 222We keep the
same splits as the original dataset for every interaction action taken by the
Follower and the object it interacts with, for a given EDH instance.

EDH plan prediction task can be modeled as a sequence-to-sequence task, where
given the dialog $A^{D}$ between the Follower and the Commander from an EDH
instance as input, the agent is tasked to predict a plan $P_{F}$, which
consists of all object interaction actions that happen during the EDH
instance. The predicted plan $P_{F}$ consists of alternating object
interaction actions $a_{i}$, $a_{i}\in A^{O}$ and object type $o_{i}$,
$o_{i}\in O_{R}$ ($O_{R}$ is the set of all objects that the Follower can
interact with in the simulator).

Let $\mathcal{P}$ be the set of all possible game plans. The task here is to
fine-tune a model $g:\mathcal{A}\rightarrow\mathcal{P}$ that predicts a
gameplan $P_{F}$ given the entire dialog $A^{D}$ for a given EDH instance.

$P_{F}=g(A^{D})$ (4)

##  5 Experiments

In this section, we present our methodological decisions to obtain conclusive
results for the task described in Section 4. Since this task involves multiple
modes, we evaluate the results and compare various training methods
simultaneously to gain insights into the significance of each modality. Due to
limited computational resources, we had to simplify our model configuration
choices, which prevented us from experimenting with larger models and
increased parameter sizes. The following sections elaborate on our adopted or
tested approaches to better understand the selected data.

###  5.1 Methodology

This section describes the architecture used to achieve optimal configuration
for multi-party embodied task completion. We have implemented a multi-
transformer-based architecture similar to the one used by Padmakumar et al.
(2021) to address the issue of EDH. This allows us to track the path taken by
the Follower using navigational and interactive actions. We investigate the
generative strategy to predict game plans to validate the effectiveness of the
language-only modality provided in the original dataset and demonstrate its
versatility. This approach enables us to identify a combination of interactive
actions and their corresponding object utilities that aim to achieve the goal
of task completion.

####  5.1.1 Episodic Transformer

We utilize a multi-transformer architecture for the task of EDH (Execution
form Dialog History), which involves predicting action sequence provided the
dialog history ($A^{D}$) between the Commander and Follower, the visual camera
observation taken from the AI2-THOR simulated household environment and the
previous actions ($A^{I}_{H}$) taken by the Follower considering all the
navigation and sub-task details. Our proposed architecture is primarily built
upon the E.T. (Episodic Transformer) model Pashevich et al. (2021).

Our architecture consists of four distinct modules: the BART AutoEncoder model
Lewis et al. (2019), the Faster R-CNN Ren et al. (2015) with a ResNet backbone
He et al. (2015), the Action Encoder, and a multimodal Cross-Attention module
combined with a transformer Encoder block. These modules work together to
produce likelihood scores for potential future action sequences. We opt for
the BART language model to encode the dialog conversations. We choose this
because BART excels in generating high-quality textual summaries and
seamlessly integrates with dialog utterances. We encode the dialog history
involving the Commander and the Follower and leverage the final hidden-state
representation to represent the language modality. Next, we employ a ResNet-50
backbone to embed individual visual observations. Specifically, we utilize a
Faster R-CNN trained ResNet-50 model for quicker computation of visual
representations, primarily focusing on highlighting object regions within the
images. Subsequently, we process these encoded visual representations through
a Convolution block and apply linear transformations to ensure they are mapped
to the same dimensional space as the language encoder. It should be noted that
these representations are encoded independently of each other. We use the
action encoder, a trainable embedding matrix, to map distinct action indices
into a higher-dimensional space.

In contrast to the original E.T. architecture, we employ a cross-attention
modeling technique that enhances hidden representations from various
modalities by integrating contextual information from each other Rajan et al.
(2022). To establish this cross-attention, we utilize the MultiHead Attention
(MHA) module Vaswani et al. (2017). Unlike the conventional self-attention
approach, which operates on similar representations for Query (Q), Key (K),
and Value (V), and calculates attention weights accordingly, in the cross-
attention model, we assign a source modality to serve as both K and V, while a
target modality functions as Q. The underlying idea behind this approach is to
explore cross-modal interactions by adapting the source modality to the target
modality. To ensure that this attention is not calculated over the future
action indices during training, we construct an attention mask that enables
the model to apply softmax over a specific indices position in the input.

Furthermore, it is important to note that we do not utilize the cross-modal
MHA within a self-attentive framework. To accomplish this, we employ a
multimodal encoder-block comprising multiple transformer-encoder layers
Vaswani et al. (2017). After obtaining embeddings from modality-specific
encoders and subsequent cross-modal attention ($h^{t}_{1:M}$, $h^{v}_{1:T}$,
$h^{a}_{1:T}$), the multimodal encoder generates output embeddings
($z^{t}_{1:M}$, $z^{v}_{1:T}$, $z^{a}_{1:T}$). Causal attention is applied in
the multimodal encoder to prevent visual and action embeddings from attending
to future time steps.

We finally utilize the contextually aware hidden state output from the
multimodal encoder, which reflects the visual observation indices. We then
apply a linear transformation to independently classify actions and
interactive object indices (See figure 1 for further reference).

###  5.2 Additional Experiments

####  5.2.1 Dialog-Plan Grounding

Additionally, we fine-tune models to predict future game plans within the EDH
plan prediction task context. For this, we fine-tune LLaMA2 LLM Touvron et al.
(2023) and variants of BART on the processed version of the EDH dataset. Our
selection of language model variants was driven by the need to explore a range
of models with varied sizes and capabilities.

We work on refining multiple BART LM versions, including the base and large
variants. Our primary focus revolves around understanding the performance
variations among the models based on parameter size for the dialog-to-plan
generation task. Furthermore, we train the BART-large model using synthetic
language data Shridhar et al. (2019) accessible along with the TEACh benchmark
dataset. To assess its adaptability and the potential benefits of synthetic
data in improving results, we refine it again on the current dataset.

Finally, we conduct experiments using LLaMA2 LLM because of its specialization
in dialog understanding and generation. Moreover, we aim to emphasize the
performance superiority inducted by utilizing LLM and its overall efficiency
in leveraging the given task.

####  5.2.2 MAF-BART

We conduct experiments to predict future action sequences in the EDH
(Execution from Dialog History) task. The prediction relies on input data,
including dialog history ($A^{D}$), prior actions ($A^{I}_{H}$), and
synchronized visual camera observations. Overall, the technique under
consideration resembles language generation, where the output at each time
step ($t$) depends on the encoded input information and the previously
generated output. To achieve this, we develop a modeling architecture based on
a BART LM and integrate the Multimodal Aware Fusion (MAF) Kumar et al. (2022)
module to seamlessly incorporate contextual knowledge from external
modalities. MAF employs a Multimodal Context-Aware (MCA2) attention Yang et
al. (2019), to efficiently combine textual and multimodal information,
resulting in enhanced predictive capabilities for future actions. Finally, we
utilize a decoder block constituting transformer-decoder layers Vaswani et al.
(2017) to predict future actions accordingly. For further detail relative to
the architecture, refer to section A.1.

####  5.2.3 Training & Inference

We conduct training for the tasks and configuration mentioned above using a
system equipped with Nvidia A100 GPU 80GB. However, we encountered challenges
related to certain driver issues and the AI2-THOR simulator’s incompatibility
with the remote cluster hosting the GPU. Consequently, we were unable to carry
out inference. As a result, we had to limit access to most of our model
configurations, making them compatible with a smaller machine equipped with an
Nvidia 1050Ti GPU 4GB. This led to the removal of several high-end model
configurations, as they were hindering the compliance of the vision modality
to the local machine.

In this section, we will outline the various hyperparameter settings that we
experimented with and provide supplementary meta-data information regarding
the proposed modeling techniques and attaining improved results for the
selected TEACh dataset.

In the context of Execution from Dialog History (EDH), we explore various
modeling approaches within the E.T. model architecture and analyze their
specific purposes through ablation experiments. However, on a broader scale,
we opt for a Faster-R-CNN pretrained ResNet-50 model to represent visual
observations, with each frame $\text{f}\in R^{(512\times 7\times 7)}$. We
utilize a GPLM, namely the BART-base333https://huggingface.co/facebook/bart-
base model to encode language instruction ($A^{D}$). BART contextually encodes
each token in the language input to $R^{768}$ dimensional vector space.
Furthermore, we conduct experiments using a fine-tuned version of the BART
model on the Synthetic dialog-action dataset provided along with the original
TEACh dataset. To conglomerate multiple modalities, we transform them into a
common subspace of $R^{768}$. Next, to facilitate the contextual self-
attentive fusion of multimodal features from each modality, we select an
encoder block with 2 transformer-encoder
layers444https://pytorch.org/docs/stable/generated/torch.nn.TransformerEncoder.html.
In the context of the EDH task, we receive a clear separation between history
(previously executed) and future actions. However, during training, we combine
them into sequences of visual observations and actions, with the option to
calculate loss that includes history time steps.

We finally execute dual linear classification on every visual index
representation obtained from the multimodal encoder to categorize actions and
their subsequent objects. We apply CrossEntropy loss to each classification
layer and then aggregate and backpropagate them as a sum. Table 3 presents the
selected hyperparameter attributes for this task.

Furthermore, to incorporate the impact of synthetic language simplification,
we refine the BART-base model using synthetic language
data555https://huggingface.co/Koshti10/BART-base-ET-synthetic. This data
aligns task descriptions within the ALFRED dataset Shridhar et al. (2019) and
associates them with action-oriented simplification similar to machine-
generated high-level code. We conduct the model fine-tuning in a conditional
generation setup over 20 epochs. Overall, approximately it takes around 20 hrs
to extract features, train and infer a model variant.

Within the context of the EDH plan prediction task, we used Colab
666https://colab.research.google.com/ with Nvidia A100 GPU (80GB) for fine-
tuning each of the 4 modeling strategies. Key hyperparameters for BART and
LLaMA2 LLM are provided in table 4. To fine-tune a very large model like
LLaMA2 LLM within the given runtime constraints, we used QLoRA Dettmers et al.
(2023), which backpropagate gradients through a frozen, 4-bit quantized pre-
trained language model into Low-Rank Adapters.

##  6 Result & Analysis

Model | Valid Seen
---|---
SR [TLW] |  GC [TLW]
TEACh Baseline
Lang |  0.72 [0.11] |  0.89 [0.15]
ET |  3.66 [0.72] |  6.82 [1.47]
\+ H |  5.78 [0.97] |  8.60 [1.99]
\+ S |  7.22 [1.78] |  10.93 [2.96]
\+ H + S |  7.45 [1.34] |  12.59 [3.80]
Ours
Lang |  1.99 [0.45] |  3.14 [1.12]
ET + BART |  5.08 [0.63] |  7.34 [1.28]
\+ H |  6.85 [1.21] |  9.67 [2.43]
\+ S |  7.12 [1.46] |  10.33 [3.39]
\+ H + S + CA |  8.85 [1.09] |  14.02 [3.72]

Table 1: The table presents results for different modeling strategies on
evaluation metrics, including success rate (SR), goal-conditioned success rate
(GC), and trajectory length weighted metrics (in brackets). We perform
ablations on history loss (H), language encoder initialization with ALFRED
synthetic language (S), and the use of cross-modal attention before fusion
(C).

Model | Valid-Seen
---|---
R-1 |  R-2 |  R-L
BART |  0.59 |  0.00 |  0.58
BART-base |  38.93 |  28.83 |  35.73
BART-large |  37.99 |  27.64 |  34.81
BART-synthetic |  38.22 |  27.74 |  35.43
LLaMA2 LLM |  54.14 |  41.22 |  46.77

Table 2: The table compares different fine-tuned models on ROGUE-1 (R-1),
ROGUE-2 (R-2) and ROGUE-L (R-L) metrics. The Vanilla BART-base model (without
any fine-tuning) is selected as the baseline

###  6.1 Evaluation Metrics

This section lists various evaluation metrics that assess the modeling
strategies discussed in Section 5, matching different task definitions and
functionalities.

EDH task is evaluated on 2 metrics for each instance, along with a trajectory
length weighted score for each metric:

  * •

Success: Success is 1 if _all_ expected actions $A^{I}_{R}$ are present in
$A_{F}$, else 0. Success Rate is calculated by averaging over all
instances777Authors use macro-average instead of micro-average.

  * •

Goal-Condition Success: Fraction of expected actions $A^{I}_{R}$ present in
$A_{F}$. Goal-Condition Success Rate is calculated by averaging over all
instances.

  * •

Trajectory Weighted Metrics: This is calculated for each of the above two
metrics for a metric value $m$ as follows :

$\mathrm{TLW}_{m}=\frac{m\times|A^{I}_{R}|}{\mathrm{max}(|A^{I}_{R}|,|A_{F}|)}$

To assess the effectiveness of the predicted game plan, we employ ROUGE scores
in a sequence-to-sequence task. ROUGE evaluates the quality of the generated
text by analyzing word and phrase overlaps with reference dialog
conversations. This approach ensures that the output is both linguistically
accurate and contextually appropriate. We utilize the ROUGE-1, ROUGE-2, and
ROUGE-L to evaluate our models, thus considering the unigram, bigram, and
longest-subsequence performance into account.

###  6.2 Quantitative Analysis

####  6.2.1 EDH (Execution from Dialog History)

Through table 1, we present a comprehensive evaluation of our proposed
approach using three distinct metrics: success rate (SR), goal-conditioned
success rate (GC), and trajectory length-weighted metrics (TLW). Our modeling
strategies consistently outperform the baseline models. Our approach leverages
a BART language model to encode dialog history ($A^{D}$), setting it apart
from the transformer-based encoder block used in the baseline models.
Additionally, it is evident from the table that the performance of the
employed architecture is subpar and that the model cannot be relied upon for
real-life applications as an embodied agent system. However, the implemented
changes still yield reasonable results for certain tasks, demonstrating their
relative reliability in specific subsamples of TEACh dataset.

The performance of the ablated baseline E.T. model in the original study by
Padmakumar et al. (2021) may differ from the ones mentioned in the table. This
is because the original study utilized a bigger variant of the training
dataset, which has now been reduced to only 5K instances. As a result, the
number of task-specific instances has also decreased. To compare our proposed
approach, we recreated the baseline models with a similar setup described in
the original study.

Firstly, the table clearly shows that the model’s performance is below average
when it receives only language-based information, i.e., the Commander and
Follower dialog interactions. This is because the model faces difficulty in
dealing with lengthy sequences of navigational actions and infrequent
interactive behaviors, as these largely depend on the camera observations from
preceding actions. Consequently, the model that relies solely on language
cannot provide enough contextual information to influence its predictions in
the long run. Moreover, since we use the BART language model, which excels at
summarizing dialog conversations, we achieve better-aligned performance
compared to a regular transformer-based language encoder with a vocabulary
solely focused on dataset-oriented words/tokens.

Incorporating visual observations into the data modeling is crucial, as
indicated in the table where there is a significant improvement in the
outcomes. Specifically, when evaluating the success rate based on goal
conditions, which assesses the fractional alignment of predicted output
actions with the ground-truth samples, we observe a substantial increase in GC
values for both the baseline E.T. model and our model. Nevertheless, owing to
the significance introduced by the BART encodings, we achieve a marginally
increased score on the validation examples for this metric. Furthermore,
considering the trajectory-length weighted values for both metrics in the
present scenario, we can see that our model prioritizes predicting shorter
trajectory lengths compared to the expected path. Thus indicating a higher
classification precision per instance across the validation split of TEACh
dataset.

Our analysis of model architectures that consider the complete history of
action subsequences for calculating cross-entropy loss during training yielded
noteworthy improvements. Considering the definitive classification of whole
subsequences of actions, we see a substantial 2% increase in the success rate
(SR) and improved trajectory length weighted (TLW) values. Moreover, this
trend also applies to the goal-conditions success rate (GC), where we could
classify almost one-tenth of the goal conditions for each instance in the
validation set.

We observe a slight decline in the performance of our E.T. model when compared
to the baseline. Our E.T. model uses the fine-tuned BART model on a synthetic
language dataset, achieving a predicted ROUGE-LSum score of 75%. Consequently,
the reason for this dip in performance can be attributed to the usage of
synthetic language encoder weights from the original work, as instructed by
Pashevich et al. (2021) in training the baseline E.T. model (replication). It
should be noted that the original work train their model on both the ALFRED
dataset and the synthetic dataset simultaneously, owing to a one-to-one
mapping of instances. However, we only train our model on the TEACh dataset,
without external influence from ALFRED data, thus showing a correlation with
the depletion of scores.

Finally, to demonstrate the efficacy of our approach, we take into account all
modeling variations, including the implementation of synthetic training to
standardize task descriptions, the utilization of interactions between the
three modalities ${\text{language},\text{frames},\text{actions}}$ via cross-
modal attention, and the consideration of history loss. Under this setting, we
attain a higher success rate (SR) and goal-conditioned success rate (GC). To
elaborate on the GC, we observe a notable 4% improvement in effectively
classifying action sequences in discrete goal-condition cases. Nevertheless,
it is essential to highlight that the TLW, specifically concerning SR, has
significantly diminished. This implies that accurately forecasting positive SR
invariably involves an extended trajectory length in action sequences.

####  6.2.2 EDH Plan Prediction

In table 2, we assess the performance of the fine-tuned model configurations
on the validation set prepared for the task. In this context, we utilize ROGUE
scores, which are discussed in subsection 6.1. Our baseline choice involves
opting for a zero-shot BART-based model without external training. This
approach enables us to assess how fine-tuning and model improvements affect
performance.

The vanilla BART model lacks prior knowledge about the game plans, resulting
in a generative response that does not include general information about any
game plans or actions. Nevertheless, when we fine-tune the BART-base model, we
observe performance improvement, as indicated in the table. Moreover, The
model learns and recognizes task-related cues and context within dialogs,
allowing it to generate plans closely aligned with the intended tasks.
Furthermore, it consistently maps similar dialogs to the same or similar
plans, improving task consistency and resulting in more reliable task
predictions.

A similar behavior is observed when considering the fine-tuned BART-large
model on the game-plan prediction dataset. However, this occurs because the
original EDH dataset contains many recurring instances of similar gameplay
across multiple instances. Consequently, the resulting dataset lacks variety
in distinct plan dialogs. The reason for this and certain model initialization
factors is that we observe a marginal degradation in performance.

Integrating the LLaMA2 LLM model into the task, as formalized previously,
presents an intriguing prospect for improving the performance and addressing
the challenges associated with this task compared to using BART models. LLaMA
models are designed to understand and generate human language with fluency and
context awareness. This enhanced language understanding capability is
particularly valuable for comprehending nuanced and context-rich dialogs in
the task. The longer context length of LLaMA allows it to capture more
extensive conversational history, thus better capturing relevant information.

###  6.3 EDH with Generative Action Decoding

Our attempts to transform the EDH (Execution from Dialog History) task into a
generative problem, wherein the actions for each subsequent time step are
decoded using the presented multimodal input data, yielded inconclusive
results. Due to the computational constraint at inference, we restrict our
model from camera observation aligning with future actions. Moreover, we
completely rely on the forecasting ability of the model to generate a
response.

We assess the proposed strategy using the F1 metric, which necessitates
predicting the output sequence alongside the ground-truth values. Our F1-score
is relatively low at $0.3562$, primarily due to the significant
overrepresentation of navigational action indices in the entire EDH dataset.
The substandard performance can be attributed mainly to the utilized
framework. Despite the potential benefits of the MAF strategy in
contextualizing modalities, the visual data fails to synchronize with the
discourse, contrary to SED dataset performance from the original work by Kumar
et al. (2022). As a result, even with recurrent training, the model can not
extract meaningful insights from the processed data.

###  6.4 Qualitative Analysis

Table 5 offers a general statistical summary of the task-specific information
in the TEACh dataset. In general, the dataset presents gameplay sessions with
varying task complexities, each demanding different cognitive skills for
execution by the follower. Certain tasks require the Commander and Follower to
engage in fewer dialog interactions and involve fewer variable parameters in
the form of objects for completion. Nonetheless, we can also affirm that the
provided dataset exhibits a greater prevalence of navigation action sequences.

Regarding the EDH dataset, we can demonstrate an excessive recurrence of
navigational actions within the ${A_{H},A^{I}_{R}}$ subsequences. To be
precise, the navigational actions, Forward, Pan Right, Pan Left, Turn Left,
Turn Right make up approximately 80% of the total actions executed by the
Follower in the task dataset. The table clearly illustrates that, for certain
specific tasks, object-specific interactive actions make up only one-tenth of
the total actions needed to complete the task. This is illustrated through
figure 2 sampled from the train-split of the EDH dataset. Here, the task is to
execute the following: i) N Slices of X in Y, and ii) Put All X in One Y.
However, this requires 138 actions to be executed, most of them being
consecutive navigational actions.

Generally, the model predicts and executes particular actions in real time. We
supply it with language instructions, action history, and visual observations,
enabling the model to establish a connection between the given instructions
and the subtasks necessary for task completion. Nevertheless, this acquired
characteristic of the model obstructs the overall task execution.
Additionally, the model tries to quickly identify the crucial step needed to
advance a specific subtask but, in doing so, neglects the intricate smaller
steps required for successful completion, which accumulates over time and
impacts the later stages. The problem is illustrated through figure 3. In
this, the task is to Make Sandwich. It constitutes of the following subtasks,
i.e., {Pickup Knife, Slice Bread, N Slice of Bread in Plate, Slice Tomato, N
Slice of Tomato in Plate}, however, the model fails to pick up either item and
place them into to the plate. Consequently, when the final state of the
simulator is observed, it could not identify sliced bread or sliced tomato,
thus resulting in task failure, with 2 GC satisfied.

According to our analysis, we realize that the authors of the original TEACh
benchmark dataset aim to replicate human behavior in AI agents. To achieve
this, they use human-human game roleplay recordings where participants can
perform tasks in distinctive ways, explicitly following specific predefined
instructions. The fundamental requirement for task completion involves
measuring the initial state $S_{I}$ and final state $S_{F}$ of the simulated
environment using metrics like SR and GC. However, this substantially impacts
the overall performance of the acquired embodied agent. The differences in
execution strategies employed during model learning for each instance become
apparent, highlighting variations in individuals’ approaches toward tasks.
Furthermore, the limited number of unique instances per task poses a challenge
for effective generalization, resulting in the model making errors in its
decisions.

The model excels in predicting the short sequence of actions for a specific
task and is invariant with additional navigational details. Particularly when
the reference frame remains nearly consistent and the model needs to carry on
with the task. Next, we demonstrate the inefficiency exhibited by the TEACh
benchmark dataset and outline flaws in the current evaluation system. In the
context of the AI2-THOR simulated environment, glitches have been observed to
occur in various scenarios in a stochastic manner. For example, in Figure 5,
we see the task of Clean Coffee Mug. The model accurately forecasts the
subsequent actions of Pour in Sink. Nevertheless, because of specific internal
technical problems, it cannot complete the task, resulting in incorrect
outcomes later.

In Figure 4, we observe an EDH instance that illustrates a 100% success rate
(SR) and a goal-conditioned success rate (GC). However, upon closer
examination, we discern that the modeling agent skips the step of Clean Pot
despite being assigned the task of Boil Potato. Furthermore, the evaluation
process relies heavily on locational information and attributes objects based
on their final state in the environment. Through their work, the authors
failed to include the essential practice of assessing the state of objects
under task consideration during the evaluation.

##  7 Conclusion & Future Work

In this work, we present a multi-transformer-based system that leverages a
BART GPLM and employs a cross-modal attention strategy to learn the task-
solving capabilities of an embodied agent. Despite the apparent inadequacy of
the task performance and its unsuitability for practical application, we
propose a deduction to enhance the data quality to achieve greater
generalizability. We discuss the the importance of the language modality and
the aspects of utilizing it to model a game plan, which serves as the high-
level perspective of a task. We demonstrate the ability of LLMs as planners to
generalize and induce human-oriented knowledge to execute tasks based on
dialog interactions. However, because of specific computational limitations,
we cannot achieve the optimal modeling configurations that are feasible with
the dataset under consideration.

We propose improvements in modeling strategies and data formulation to enhance
current performance. First, we recommend experimenting with a model like the
Vision Transformer to effectively represent camera observations because the
current model sometimes struggles with the abstract quality of the image.
Furthermore, we propose aligning image representation with dialog interactions
based on the insights learned from the unsuccessful experiments with the MAF-
BART model. Posing the EDH task as a topological classification strategy could
also negate the influence of navigational movements (imbalance). The use of
multimodal LLMs could be exploited to tackle the issue of embodied agents
under a few-shot setting. To enhance the performance of the mentioned models,
we can implement soft rules to limit the recording of multi-party gameplay and
carefully curate a larger dataset.

## References

  * Anderson et al. (2017) Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian D. Reid, Stephen Gould, and Anton van den Hengel. 2017.  Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments.  _CoRR_ , abs/1711.07280.
  * Bisk et al. (2017) Yonatan Bisk, Kevin J. Shih, Yejin Choi, and Daniel Marcu. 2017.  Learning interpretable spatial operations in a rich 3d blocks world.  _CoRR_ , abs/1712.03463.
  * Chen and Mooney (2011) David L. Chen and Raymond J. Mooney. 2011.  Learning to interpret natural language navigation instructions from observations.  In _Proceedings of the Twenty-Fifth AAAI Conference on Artificial Intelligence_ , AAAI’11, page 859–865. AAAI Press.
  * Dettmers et al. (2023) Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023.  Qlora: Efficient finetuning of quantized llms.  _arXiv preprint arXiv:2305.14314_.
  * Dosovitskiy et al. (2020) Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2020.  An image is worth 16x16 words: Transformers for image recognition at scale.  _CoRR_ , abs/2010.11929.
  * Gella et al. (2022) Spandana Gella, Aishwarya Padmakumar, Patrick Lange, and Dilek Hakkani-Tur. 2022.  Dialog Acts for Task-Driven Embodied Agents.  In _Proceedings of the 23nd Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDial)_ , pages 111–123.
  * Gervits et al. (2021) Felix Gervits, Antonio Roque, Gordon Briggs, Matthias Scheutz, and Matthew Marge. 2021.  How should agents ask questions for situated learning? an annotated dialogue corpus.  In _Proceedings of the 22nd Annual Meeting of the Special Interest Group on Discourse and Dialogue_ , pages 353–359, Singapore and Online. Association for Computational Linguistics.
  * Gluck and Laird (2019) Kevin A. Gluck and John E. Laird. 2019.  _Interactive Task Learning: Humans, Robots, and Agents Acquiring New Tasks through Natural Interactions_.  The MIT Press.
  * Harnad (1990) Stevan Harnad. 1990.  The symbol grounding problem.  _Physica D: Nonlinear Phenomena_ , 42(1):335–346.
  * He et al. (2015) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2015.  Deep residual learning for image recognition.  _CoRR_ , abs/1512.03385.
  * Kiseleva et al. (2022) Julia Kiseleva, Ziming Li, Mohammad Aliannejadi, Shrestha Mohanty, Maartje ter Hoeve, Mikhail Burtsev, Alexey Skrynnik, Artem Zholus, Aleksandr Panov, Kavya Srinet, Arthur Szlam, Yuxuan Sun, Marc-Alexandre Côté, Katja Hofmann, Ahmed Awadallah, Linar Abdrazakov, Igor Churin, Putra Manggala, Kata Naszadi, Michiel van der Meer, and Taewoon Kim. 2022.  Interactive grounded language understanding in a collaborative environment: Iglu 2021.
  * Kollar et al. (2017) Thomas Kollar, Stefanie Tellex, Matthew R. Walter, Albert Huang, Abraham Bachrach, Sachithra Hemachandra, Emma Brunskill, Ashis Gopal Banerjee, Deb Roy, Seth J. Teller, and Nicholas Roy. 2017.  Generalized grounding graphs: A probabilistic framework for understanding grounded commands.  _CoRR_ , abs/1712.01097.
  * Kolve et al. (2017) Eric Kolve, Roozbeh Mottaghi, Daniel Gordon, Yuke Zhu, Abhinav Gupta, and Ali Farhadi. 2017.  AI2-THOR: an interactive 3d environment for visual AI.  _CoRR_ , abs/1712.05474.
  * Kumar et al. (2022) Shivani Kumar, Atharva Kulkarni, Md Shad Akhtar, and Tanmoy Chakraborty. 2022.  When did you become so smart, oh wise one?! sarcasm explanation in multi-modal multi-party dialogues.
  * Lewis et al. (2019) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2019.  BART: denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  _CoRR_ , abs/1910.13461.
  * Lu et al. (2019) Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019.  Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks.  _CoRR_ , abs/1908.02265.
  * MacMahon et al. (2006) Matt MacMahon, Brian Stankiewicz, and Benjamin Kuipers. 2006.  Walk the talk: Connecting language, knowledge, and action in route instructions.
  * Mehta et al. (2023) Nikhil Mehta, Milagro Teruel, Patricio Figueroa Sanz, Xin Deng, Ahmed Hassan Awadallah, and Julia Kiseleva. 2023.  Improving grounded language understanding in a collaborative environment by interacting with agents through help feedback.
  * Misra et al. (2018) Dipendra Misra, Andrew Bennett, Valts Blukis, Eyvind Niklasson, Max Shatkhin, and Yoav Artzi. 2018.  Mapping instructions to actions in 3D environments with visual goal prediction.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 2667–2678, Brussels, Belgium. Association for Computational Linguistics.
  * Narayan-Chen et al. (2019) Anjali Narayan-Chen, Prashant Jayannavar, and Julia Hockenmaier. 2019.  Collaborative dialogue in Minecraft.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 5405–5415, Florence, Italy. Association for Computational Linguistics.
  * Padmakumar et al. (2021) Aishwarya Padmakumar, Jesse Thomason, Ayush Shrivastava, Patrick Lange, Anjali Narayan-Chen, Spandana Gella, Robinson Piramuthu, Gökhan Tür, and Dilek Hakkani-Tür. 2021.  Teach: Task-driven embodied agents that chat.  _CoRR_ , abs/2110.00534.
  * Padmakumar et al. (2022) Vishakh Padmakumar, Leonard Lausen, Miguel Ballesteros, Sheng Zha, He He, and George Karypis. 2022.  Exploring the role of task transferability in large-scale multi-task learning.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 2542–2550, Seattle, United States. Association for Computational Linguistics.
  * Pashevich et al. (2021) Alexander Pashevich, Cordelia Schmid, and Chen Sun. 2021.  Episodic transformer for vision-and-language navigation.  _CoRR_ , abs/2105.06453.
  * Rajan et al. (2022) Vandana Rajan, Alessio Brutti, and Andrea Cavallaro. 2022.  Is cross-attention preferable to self-attention for multi-modal emotion recognition?
  * Ren et al. (2015) Shaoqing Ren, Kaiming He, Ross B. Girshick, and Jian Sun. 2015.  Faster R-CNN: towards real-time object detection with region proposal networks.  _CoRR_ , abs/1506.01497.
  * Savva et al. (2019) Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, Devi Parikh, and Dhruv Batra. 2019.  Habitat: A platform for embodied AI research.  _CoRR_ , abs/1904.01201.
  * Shah et al. (2021) Rohin Shah, Cody Wild, Steven H. Wang, Neel Alex, Brandon Houghton, William H. Guss, Sharada P. Mohanty, Anssi Kanervisto, Stephanie Milani, Nicholay Topin, Pieter Abbeel, Stuart Russell, and Anca D. Dragan. 2021.  The minerl BASALT competition on learning from human feedback.  _CoRR_ , abs/2107.01969.
  * Shridhar et al. (2019) Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, and Dieter Fox. 2019.  ALFRED: A benchmark for interpreting grounded instructions for everyday tasks.  _CoRR_ , abs/1912.01734.
  * Skrynnik et al. (2022) Alexey Skrynnik, Zoya Volovikova, Marc-Alexandre Côté, Anton Voronov, Artem Zholus, Negar Arabzadeh, Shrestha Mohanty, Milagro Teruel, Ahmed Awadallah, Aleksandr Panov, Mikhail Burtsev, and Julia Kiseleva. 2022.  Learning to solve voxel building embodied tasks from pixels and natural language instructions.
  * Song et al. (2023) Chan Hee Song, Jiaman Wu, Clayton Washington, Brian M. Sadler, Wei-Lun Chao, and Yu Su. 2023.  Llm-planner: Few-shot grounded planning for embodied agents with large language models.
  * Tan and Bansal (2019) Hao Tan and Mohit Bansal. 2019.  LXMERT: learning cross-modality encoder representations from transformers.  _CoRR_ , abs/1908.07490.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurélien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023.  Llama 2: Open foundation and fine-tuned chat models.  _CoRR_ , abs/2307.09288.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  _CoRR_ , abs/1706.03762.
  * Wang et al. (2023) Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. 2023.  Voyager: An open-ended embodied agent with large language models.
  * Yang et al. (2019) Baosong Yang, Jian Li, Derek F. Wong, Lidia S. Chao, Xing Wang, and Zhaopeng Tu. 2019.  Context-aware self-attention networks.  _CoRR_ , abs/1902.05766.
  * Zhang et al. (2023) Hongxin Zhang, Weihua Du, Jiaming Shan, Qinhong Zhou, Yilun Du, Joshua B. Tenenbaum, Tianmin Shu, and Chuang Gan. 2023.  Building cooperative embodied agents modularly with large language models.

##  Appendix A Appendix

Hyperparameter |  Value
---|---
epochs |  20
batch size |  2
loss |  Cross-Entropy
optimizer |  AdamW
learning rate (lr) |  1e-5
lr scheduling type |  Linear
lr decay rate |  0.1
weight decay |  0.33
$\mathrm{action\\_loss}$ weight |  1
$\mathrm{object\\_loss}$ weight |  1
dropout rate |  0.2

Table 3: Displays the selected hyperparameter settings for training the E.T.
model in the context of the EDH task. These attributes remain fixed for all
modeling configurations related to this task, ensuring that the results
obtained are invariant with these values.

BART
---
Hyperparameter |  Value
epochs |  20
batch size |  8
learning rate |  5e-5
warmup steps |  500
weight decay |  0.01
LLaMA2 LLM
Hyperparameter |  Value
epochs |  1
batch size |  4
learning rate |  2e-4
weight decay |  0.001
LORA alpha |  16
LORA dropout |  0.1
LORA r |  64

Table 4: Displays the hyperparameter settings for finetuning BART models and
LLaMA2 LLM model Figure 1: The figure depicts the Episodic-Transformer
architecture, which incorporates the BART language model and a multimodal
cross-attention module. This module enables the contextual transfer of
information across different modalities. In the illustration, we observe an
inference scenario involving the dialog history $e_{1:M}$ (Lang), visual
observations from the start of the episode, denoted as $v_{1:t}$ (Frames), and
previously executed actions $a_{1:t-1}$ (Actions). The model operates by
processing $e_{1:M}$ using a BART AutoEncoder, transforming $v_{1:t}$ using a
ResNet backbone, and embedding $a_{1:t}$ via an embedding matrix. It then
generates actions and, if applicable, interactive objects at the current time
step, $t$.

|  Total Sessions |  Average Actions per Session |  Average Interactive Actions per Session
---|---|---|---
Water Plant |  91 |  $68.27\pm 44.29$ |  $7.05\pm 6.69$
Plate Of Toast |  80 |  $139.80\pm 81.81$ |  $19.80\pm 10.46$
Clean All X |  169 |  $98.30\pm 82.14$ |  $17.27\pm 14.82$
Sandwich |  89 |  $255.19\pm 116.28$ |  $41.76\pm 13.06$
N Slices Of X In Y |  144 |  $145.64\pm 121.01$ |  $19.44\pm 14.66$
Coffee |  145 |  $69.94\pm 46.44$ |  $10.55\pm 6.71$
Put All X In One Y |  163 |  $158.19\pm 115.81$ |  $9.57\pm 7.86$
Boil X |  81 |  $126.74\pm 96.09$ |  $18.70\pm 16.12$
Salad |  139 |  $258.13\pm 107.24$ |  $40.37\pm 14.14$
Breakfast |  113 |  $340.59\pm 151.04$ |  $56.30\pm 18.81$
Put All X On Y |  182 |  $101.73\pm 83.48$ |  $8.25\pm 6.40$
N Cooked Slices Of X In Y |  89 |  $185.70\pm 96.56$ |  $31.69\pm 13.52$

Table 5: showcases the meta-data information for the total gameplay session
available across the training split of the TEACh dataset. (Left) Displays all
the unique tasks found in the dataset. For each task, we show the average
total actions per session and the average total interactive actions per
session, along with their respective deviations in values.

Task Description |  Synthetic Language Simplification
---|---
Turn right toward the towel ring on the wall. Pick up the towel from the towel ring. Turn around and head to the toilet. Put the towel on the back of the toilet. Turn to the right, go toward the tub, and turn right. Pick up the towel from the ring on the wall. Turn around and go back to the toilet. Put the towel on the back of the toilet to the left of the bottle. |  Put two towels on the back of the toilet
Turn around and go to the sink. Grab the white lotion bottle with a brown pump. Turn around and back up towards the bathtub, then turn around to the sink again. Open the right cabinet door under the sink and place the lotion inside. |  Place the lotion on the sink underneath the cabinet.
Turn and go to the counter by the sink. Pick up the knife on the counter. Cut the tomato on the counter. Turn and go to the microwave. Put the knife in the microwave. Turn and go to the counter by the sink. Pick up a slice of tomato on the counter. Turn and go to the refrigerator. Chill the slice of tomato in the refrigerator. Turn and go to the microwave. Put the slice of tomato in the microwave. |  Put a chilled slice of tomato in the microwave.

Table 6: Showcase, instance sampled from the synthetic dataset provided
through the ALFRED dataset Shridhar et al. (2019). Figure 2: Shows the
Follower ego-centric camera observation from all of the available interactive
action timesteps of an EDH task instance. The task involves slicing a potato
and placing the slices into an empty pot. The suggested total number of
executable actions in the subsample is 138. Figure 3: Illustrates the ego-
centric camera observations of the Make Sandwich task, highlighting
interpretative actions carried out at each time step that are essential for
completion. Nevertheless, starting from t=38, the model struggles to correlate
visual perception with the task instructions, resulting in a continuous loop.
Figure 4: Showcases an EDH instance and compares the generated response
against the ground-truth action sequences. It highlights the shortcomings of
the employed evaluation strategy. Figure 5: Shocases a positive EDH instance
of executing the task of Clean Coffee Mug.

###  A.1 MAF-BART (extended)

We conduct experiments involving the prediction of future action sequences
within the context of the EDH (Execution from Dialog History) task. This
prediction relies on input data consisting of dialog history ($A^{D}$), prior
actions ($A^{I}_{H}$), and synchronized visual camera observations, all of
which influence the outcome. This task is similar to language generation,
where the output at time-step $t$ depends on both the encoded input
information and the previously generated output in the sequence (See equation
5 for further reference).

$P(X)=\prod_{t=1}^{M}P\left(a^{f}_{t}\mid a^{f}_{1:t-1},X_{in}\right)$ (5)

$X_{in}=\mathcal{F}\left(A^{D},A^{I}_{H},V_{H}\right)$ (6)

Here, $X_{in}$ represents the encoded information obtained from modeling the
input data. This information is subsequently used to condition future action
predictions.

We construct our model architecture based on a BART language model, which
incorporates encoder-decoder models to enable conditional generation
adaptability. To seamlessly integrate multimodal knowledge into the BART
architecture, we employ a Multimodal Aware Fusion (MAF) module, similar to the
one described by Kumar et al. (2022). The MAF module is adapter-based and
consists of Multimodal Context-Aware (MCA2) Yang et al. (2019) Attention and
Global Information Fusion (GIF) mechanisms. We use this module to incorporate
contextual knowledge from external modalities into the source-text
representations when given textual input in the form of pre-tokenized dialog
history $A^{D}$ and a sequence of action-frame cues.

Multimodal Aware Fusion, the standard cross-modal attention in the form of
dot-product relies on the direct interaction between textual representations
and other modalities. However, because modality representations may not have
the same subspace dimensionality as textual representations, this can result
in information loss. In the context of MAF, we utilize MCA2, which calculates
context-aware multimodal key and value representations followed by traditional
dot-product attention.

Given an intermediate representation H generated by a pre-trained language
model, query, key, and value vectors $Q$, $K$, and $V\in R^{n\times d}$ are
calculated. Next, given a multmodal representation $C\in R^{c\times d_{c}}$,
multimodal information informed key & value vectors $\hat{K}$ & $\hat{V}$ are
obtained. Next, via a learnable matrix $\lambda\in R^{n\times 1}$ and project
matrix $U_{k},U_{v}$, we derive attentive key & value representations with the
amalgamation of both the modalities under consideration (See equation 7).

$\begin{bmatrix}\hat{K}\\\
\hat{V}\end{bmatrix}=\left(1-\begin{bmatrix}\lambda_{k}\\\
\lambda_{v}\end{bmatrix}\right)\begin{bmatrix}K\\\
V\end{bmatrix}+\begin{bmatrix}\lambda_{k}\\\
\lambda_{v}\end{bmatrix}(C\begin{bmatrix}U_{k}\\\ U_{v}\end{bmatrix})$ (7)

Finally, the multimodal information-infused vectors $\hat{K}$ and $\hat{V}$
are used to compute the traditional scaled dot-product attention. Then, we
define modality-specific gates, assigning weights to each modality-aware
hidden representation concerning $H$, thus formulating the final information-
fused representation $\hat{H}$ (refer to equation 8).

$\hat{H}=H+g_{m_{1}}\odot H_{m_{1}}+\ldots$ (8)

Furthermore, as we use a decoder model to predict the next action sub-
sequence, we propose employing a fusion technique that combines previously
predicted actions with inferred camera observations through concatenation and
linear transformation.


# The Chermak–Delgado Measure as a Map on Posets

William Cocke  Language Technologies Institute, Carnegie Mellon University,
Pittsburgh, PA 15213<EMAIL_ADDRESS>and  Ryan McCulloch  Department of
Mathematics, University of Bridgeport, Bridgeport, CT 06604
<EMAIL_ADDRESS>
###### Abstract.

The Chermak–Delgado measure of a finite group is a function which assigns to
each subgroup a positive integer. In this paper, we give necessary and
sufficient conditions for when the Chermak–Delgado measure of a group is
actually a map of posets, i.e., a monotone function from the subgroup lattice
to the positive integers. We also investigate when the Chermak–Delgado
measure, restricted to the centralizers, is increasing.

###### Key words and phrases:

finite group theory, group theory, Chermak–Delgado measure, Chermak–Delgado
lattice

###### 2020 Mathematics Subject Classification:

Primary 20D30, 20E15

##  1\. Introduction.

The Chermak–Delgado measure is a function from the subgroup lattice of a
finite group $G$ to the positive integers that maps a subgroup $H$ to
$|H|\cdot|\mathbf{C}_{G}(H)|$. We write the Chermak–Delgado measure for a
finite group $G$ as $m_{G}$. Chermak and Delgado [7] introduced the function
$m_{G}$ in a slightly more general form. The function $m_{G}$ has the peculiar
property that the subgroups $H\leq G$ such that $m_{G}(H)$ has maximal value
form a sublattice, the so-called Chermak–Delgado lattice of $G$. We write
$\mathcal{CD}(G)$ for the Chermak–Delgado lattice of $G$. A proof that the
fiber of the maximum value of $m_{G}$ forms a lattice and other basic
properties of $\mathcal{CD}(G)$ can be found in Isaacs [10, Section 1.E]. More
modern research on the Chermak–Delgado often focuses on finding groups such
that the Chermak–Delgado lattice has a specified shape [2], or on specific
groups [12], or other structural properties of the lattice itself [4, 5, 11].
We direct students interested in the Chermak–Delgado lattice to the article by
Wilcox [17], in which she gives a non-exhaustive list of potential research
directions suitable for students.

We write $\mathcal{S}(G)$ for the the subgroup lattice of $G$. Then $m_{G}$ is
a map from $\mathcal{S}(G)$ to the positive integers $\mathbb{Z}^{+}$. In
Figure 1 we represent the map $m_{G}$ where $G=S_{3}$, the symmetric group on
3 symbols.

${{\mathcal{S}(G)}}$${{\mathbb{Z}^{+}}}$${{S_{3}}}$${9}$${{C_{3}}}$${6}$${{C_{2}}}$${{C_{2}}}$${{C_{2}}}$${4}$${1}$$\scriptstyle{m_{G}}$
Figure 1. A diagram for $G=S_{3}$ showing the Chermak–Delgado measure
$m_{G}:\mathcal{S}(G)\rightarrow\mathbb{Z}^{+}$. All of the subgroups
isomorphic to $C_{2}$ have measure $4$.

As suggested by the diagram in Figure 1, both the domain and the codomain of
the map $m_{G}$ are posets; where we take $x\preceq y$ in $\mathbb{Z}^{+}$ to
mean that $x\leq y$. However, the function $m_{G}$ does not in general
preserve this poset structure. To use the common vernacular of category
theory, we could say that $m_{G}$ forgets the poset structure of its domain
and codomain. In this paper we investigate when the map $m_{G}$ is actually a
map of posets, i.e., when $m_{G}:\mathcal{S}(G)\rightarrow\mathbb{Z}^{+}$ is
monotone increasing or decreasing. The question of when $m_{G}$ is a morphism
of posets was investigated by Fasolă [9] and we explain this connection in
Section 2.1.

We now present two examples of finite groups together with their
Chermak–Delgado measures, one such that the measure is monotone, and one where
the measure is not monotone. In Figure 2 we present the subgroup lattice and
Chermak–Delgado measure of the dihedral group of order 8 (written as $D_{8}$).
The subgroup lattice can be partitioned into two parts, each containing $5$
subgroups. Each of the nontrivial normal subgroups has a Chermak–Delgado
measure of $16$, while each of the other five subgroups has a Chermak–Delgado
measure of 8. Hence the Chermak–Delgado measure from $D_{8}$ to $\\{8,16\\}$
forms a poset. In contrast, the Chermak–Delgado measure from $A_{4}$, the
alternating group on 4 symbols, is not monotone. In Figure 3 we present a
diagram showing the map $m_{G}$ for $G=A_{4}$.

${{\mathcal{S}(G)}}$${{\mathbb{Z}^{+}}}$${{D_{8}}}$${16}$${{K_{4}}}$${{C_{4}}}$${{K_{4}}}$${{C_{2}}}$${{C_{2}}}$${{C_{2}}}$${{C_{2}}}$${{C_{2}}}$${1}$${8}$$\scriptstyle{m_{G}}$
Figure 2. A diagram for $G=D_{8}$ showing the Chermak–Delgado measure
$m_{G}:\mathcal{S}(G)\rightarrow\mathbb{Z}^{+}$. The two regions of the
diagram are the fibers of $m_{G}$. The map $m_{G}$ is a monotone map of posets
as well.
${{\mathcal{S}(G)}}$${{\mathbb{Z}^{+}}}$${16}$${{A_{4}}}$${12}$${{K_{4}}}$${9}$${{C_{3}}}$${{C_{3}}}$${{C_{3}}}$${{C_{3}}}$${8}$${{C_{2}}}$${{C_{2}}}$${{C_{2}}}$${1}$$\scriptstyle{m_{G}}$
Figure 3. A diagram for $G=A_{4}$ showing the Chermak–Delgado measure
$m_{G}:\mathcal{S}(G)\rightarrow\mathbb{Z}^{+}$. We note that each copy of
$C_{3}$ has measure $9$ and all of the copies of $C_{2}$ have measure 8. Since
$1$ has measure 12, the map $m_{G}$ is not a monotone map of posets.

Recall that there are two forms of monotone maps, increasing and decreasing.
We show in Proposition 7 that if $m_{G}$ is a decreasing function from
$\mathcal{S}(G)$ to $G$, then $G=1$. Consequently, we restrict our attention
to the case in which the Chermak–Delgado measure is increasing for the rest of
the paper.

Moreover, if $m_{G}$ is increasing, then the Chermak–Delgado lattice is
exactly the interval of subgroups between $\mathbf{Z}(G)$ and $G$, i.e.,
$H\in\mathcal{CD}(G)$ if and only if $\mathbf{Z}(G)\leq H$ (and of course
$H\leq G)$. We use the notation $[G/\mathbf{Z}(G)]$ for the interval of
$\mathcal{S}(G)$ between $G$ and $\mathbf{Z}(G)$. The following theorem shows
that the necessary conditions discovered by Fasolă [9] are in fact sufficient.

######  Theorem A.

Let $G$ be a finite group. Then the following are equivalent.

  1. (1)

The function $m_{G}$ is increasing on $\mathcal{S}(G)$.

  2. (2)

$m_{G}(H)=m_{G}(H\cap\mathbf{Z}(G))$ for all subgroups $H$ of $G$.

  3. (3)

$\mathcal{CD}(G)=[G/\mathbf{Z}(G)]$.

Of note, since the Chermak–Delgado lattice is self-dual under taking
centralizers, if $\mathcal{CD}(G)=[G/\mathbf{Z}(G)]$, then all of the
subgroups containing $\mathbf{Z}(G)$ are centralizers. We write
$\mathcal{C}(G)$ for the set of subgroups of $G$ that are centralizers in $G$
(which is a lattice). Theorem A implies that if $m_{G}$ is increasing then
$\mathcal{CD}(G)=\mathcal{C}(G)$.

Moreover, if we relax this condition to considering $m_{G}$ as a function not
from $\mathcal{S}(G)$ but from $\mathcal{C}(G)$, we obtain a similar result.
As an example of a sort of linguistic duality, we cannot help but point out
that _relaxing_ the condition on $m_{G}$ to only be increasing on
$\mathcal{C}(G)$ is equivalent to _restricting_ the domain of $m_{G}$.

######  Theorem B.

Suppose $G$ is a finite group. Then $m_{G}$ is increasing on $\mathcal{C}(G)$
if and only if $\mathcal{C}(G)=\mathcal{CD}(G)$, and in such case the group
$G$ is nilpotent.

We note that since all centralizers of $G$ contain $\mathbf{Z}(G)$, the
identification of $\mathcal{CD}(G)$ in Theorems A and B are both specific
instances of the following.

######  Theorem C.

Let $G$ be a finite group. If the Chermak–Delgado measure $m_{G}$ is
increasing on a poset $\mathcal{P}$ of subgroups of $G$ such that

  1. (1)

$\mathbf{Z}(G)\in\mathcal{P}$;

  2. (2)

$G\in\mathcal{P}$;

  3. (3)

$\mathcal{P}\cap\mathcal{CD}(G)\neq\varnothing$;

then $\mathcal{CD}(G)\cap\mathcal{P}=\\{H\in\mathcal{P}:\mathbf{Z}(G)\leq
H.\\}$

The paper proceeds as follows. In Section 2 we present background material,
including a discussion of Fasolă’s prior results, and general results we will
use. Then in Section 3 we prove Theorem A and present a classification of
finite groups with $m_{G}$ monotone on the full subgroup lattice. In Section 4
we prove Theorem B and investigate when $m_{G}$ is monotone on the centralizer
lattice.

##  2\. Background.

In this section we gather some relevant facts about centralizers, the
Chermak–Delgado lattice, and subnormal subgroups. We also discuss prior work
on when the Chermak–Delgado measure is increasing (see Section 2.1).

######  Proposition 1.

Recall the following facts about centralizers. Let $C\in\mathcal{C}(G)$, i.e.,
there exists a subgroup $H\leq G$ such that $C=\mathbf{C}_{G}(H)$.

  1. (1)

The subgroup $C$ is the centralizer of its centralizer, i.e.,
$C=\mathbf{C}_{G}(\mathbf{C}_{G}(C))$.

  2. (2)

The subgroup $C$ and its centralizer have the same Chermak–Delgado measure,
i.e.,
$m_{G}(C)=|C|\cdot|\mathbf{C}_{G}(C)|=|\mathbf{C}_{G}(C)|\cdot|\mathbf{C}_{G}(\mathbf{C}_{G}(C))|=m_{G}(\mathbf{C}_{G}(C))$.

Note that for any subset $S$ of a group $G$,
$\mathbf{C}_{G}(S)=\mathbf{C}_{G}(\langle S\rangle)\in\mathcal{C}(G)$, and so,
in particular, for any element $x\in G$, the subgroup
$\mathbf{C}_{G}(x)\in\mathcal{C}(G)$.

######  Proposition 2 ([10, Theorem 1.44]).

Let $G$ be a finite group. Then $\mathcal{CD}(G)\subseteq\mathcal{C}(G)$.

The next proposition will be used throughout the paper.

######  Proposition 3.

Suppose $m_{G}$ is increasing on a subset $\mathcal{P}$ of $\mathcal{S}(G)$.
If an abelian centralizer $C\in\mathcal{P}$ and
$\mathbf{C}_{G}(C)\in\mathcal{P}$, then $m_{G}$ is constant on
$\\{X\in\mathcal{P}\,\,:\,\,C\leq
X\leq\mathbf{C}_{G}(C)\\}=[\mathbf{C}_{G}(C)/C]_{\mathcal{P}}$.

###### Proof.

Let $X\in\mathcal{P}$ such that $C\leq X\leq\mathbf{C}_{G}(C)$. Since $m_{G}$
is increasing, we have $m_{G}(C)\leq m_{G}(X)\leq m_{G}(\mathbf{C}_{G}(C))$
and thus $m_{G}(X)=m_{G}(C)$. ∎

######  Corollary 4.

Suppose $m_{G}$ is increasing on a subset $\mathcal{P}$ of $\mathcal{S}(G)$.
If an abelian centralizer $C\in\mathcal{P}$ and
$\mathbf{C}_{G}(C)\in\mathcal{P}$, and if
$[\mathbf{C}_{G}(C)/C]_{\mathcal{P}}\cap\mathcal{CD}(G)\neq\emptyset$, then
$[\mathbf{C}_{G}(C)/C]_{\mathcal{P}}\subseteq\mathcal{CD}(G)$.

Let $\mathcal{S}\textit{n}(G)$ denote the collection of all subnormal
subgroups of a group $G$. Wielandt showed that for a finite group $G$,
$\mathcal{S}\textit{n}(G)$ is a sublattice of $\mathcal{S}(G)$.

######  Proposition 5 (Wielandt [10, Lemma 2.5]).

Let $G$ be a finite group and suppose that $S,T$ are subnormal subgroups of
$G$. Then $\langle S,T\rangle$ is subnormal in $G$.

Brewster & Wilcox in [3] showed that subgroups in $\mathcal{CD}(G)$ are
subnormal, see also [8].

######  Proposition 6.

Let $G$ be a finite group. Then
$\mathcal{CD}(G)\subseteq\mathcal{S}\textit{n}(G)$.

###  2.1. Previous work on when $m_{G}$ is increasing

This work was directly inspired by the 2024 paper of Georgiana Fasolă [9].
Fasolă examined the question of when subgroups with minimal Chermak–Delgado
measure form a sublattice. Her partial answers to this question relied on two
additional hypotheses:

  1. (1)

A bound on the number of values taken by the Chermak–Delgado measure $m_{G}$.

  2. (2)

Assuming that the Chermak–Delgado measure $m_{G}$ is increasing, i.e., $H\leq
K\Rightarrow m_{G}(H)\leq m_{G}(K)$.

Of note, Fasolă [9] showed that $m_{G}$ is increasing implies that $G$ is
nilpotent. She also proved that $(1)$ implies $(2)$ and $(3)$ in Theorem A.

##  3\. Monotone Chermak–Delgado measure on all subgroups.

In this section, we prove Theorem A, which states that the function $m_{G}$ is
increasing on the subgroup lattice if and only if
$\mathcal{CD}(G)=[G/\mathbf{Z}(G)]$. After proving Theorem A, we then collect
results showing when this condition is satisfied.

We start by showing that the Chermak–Delgado measure is decreasing only on the
trivial group.

######  Proposition 7.

Let $G$ be a finite group. If the Chermak–Delgado measure $m_{G}$ on $G$
satisfies $K\leq H\Rightarrow m_{G}(H)\leq m_{G}(K)$, then $G=1$.

###### Proof.

We note that $m_{G}(G)\leq m_{G}(1)$ implies that $\mathbf{Z}(G)=1$ and
$m_{G}(G)=m_{G}(1)$. Now for all $H\leq G$ we have $m_{G}(G)\leq m_{G}(H)\leq
m_{G}(1)$, and thus all elements of $\mathcal{S}(G)$ have the same
Chermak–Delgado measure. The only group $G$ such that the image of $m_{G}$ has
size $1$ is $1$. To see this, if $S$ is a nontrivial Sylow $p$-subgroup of
$G$, we have that $m_{G}(S)\neq m_{G}(1)$, since $|\mathbf{C}_{G}(S)|$ is
divisible by $p$. Therefore, the Sylow subgroups of $G$ are trivial and we
conclude that $G=1$. ∎

We now prove Theorem A. Since the theorem involved a few hypothesis, we
restate it before giving the proof.

######  Theorem A.

Let $G$ be a finite group. Then the following are equivalent.

  1. (1)

The function $m_{G}$ is increasing on $\mathcal{S}(G)$.

  2. (2)

$m_{G}(H)=m_{G}(H\cap\mathbf{Z}(G))$ for all subgroups $H$ of $G$.

  3. (3)

$\mathcal{CD}(G)=[G/\mathbf{Z}(G)]$.

###### Proof.

$(1)\rightarrow(2)$ is Lemma 2.3 in [9]. Note that if
$m_{G}(H)=m_{G}(H\cap\mathbf{Z}(G))$ for all subgroups $H$ of $G$, then
$m_{G}(H)=m_{G}(\mathbf{Z}(G))$ for all subgroups $H$ that contain
$\mathbf{Z}(G)$. This implies that $m_{G}$ is constant on $[G/\mathbf{Z}(G)]$,
and hence $\mathcal{CD}(G)=[G/\mathbf{Z}(G)]$. Thus, $(2)\rightarrow(3)$.

Now we show that $(3)\rightarrow(1)$. Suppose that
$\mathcal{C}\mathcal{D}(G)=[G/\mathbf{Z}(G)]$. Let $H\leq K$ be two subgroups
of $G$. We write $Z$ for $\mathbf{Z}(G)$. Then
$\mathbf{C}_{G}(HZ)=\mathbf{C}_{G}(H)$ and
$\mathbf{C}_{G}(KZ)=\mathbf{C}_{G}(K)$. Since both $HZ$ and $KZ$ are in
$\mathcal{C}\mathcal{D}(G)$, we know that
$m_{G}(HZ)=|HZ||\mathbf{C}_{G}(H)|=|KZ||\mathbf{C}_{G}(K)|=m_{G}(KZ)$. And so

$\frac{|H||Z|}{|H\cap Z|}|\mathbf{C}_{G}(H)|=\frac{|K||Z|}{|K\cap
Z|}|\mathbf{C}_{G}(K)|.$

Reordering terms, this is

(i) $m_{G}(H)\cdot\frac{|Z|}{|H\cap Z|}=m_{G}(K)\cdot\frac{|Z|}{|K\cap Z|}.$

Since $H\leq K$, we have $H\cap Z\leq K\cap Z$. Thus

(ii) $\frac{|Z|}{|H\cap Z|}\geq\frac{|Z|}{|K\cap Z|}.$

Combining equations i and ii shows that $m_{G}(H)\leq m_{G}(K)$. ∎

######  Proposition 8.

Let $G$ be a finite group, let $k$ be the number of divisors of
$|\mathbf{Z}(G)|$, and consider $m_{G}$ as a map from $\mathcal{S}(G)$ to
$\mathbb{Z}^{+}$. Then $|Im(m_{G})|\geq k$.

###### Proof.

Since $\mathbf{Z}(G)$ is abelian, $\mathbf{Z}(G)$ possesses a subgroup $H$ of
order $n$ for each $n$ dividing $|\mathbf{Z}(G)|$, and $m_{G}(H)=n\cdot|G|$.
Thus $\\{n\cdot|G|\,\,:\,\,n\,\,\mid\,\,|\mathbf{Z}(G)|\\}\subseteq
Im(m_{G})$, and the result follows. ∎

The class of groups $G$ in Theorem A achieve this lower bound on the size of
the image of $m_{G}$.

######  Corollary 9.

Let $G$ be a finite group, and let $k$ be the number of divisors of
$|\mathbf{Z}(G)|$. If $m_{G}$ is increasing on $\mathcal{S}(G)$, then
$|Im(m_{G})|=k$.

###### Proof.

If $m_{G}$ is increasing on $\mathcal{S}(G)$, then
$m_{G}(H)=m_{G}(H\cap\mathbf{Z}(G))$ for all subgroups $H$ of $G$
($(1)\rightarrow(2)$ in Theorem A), and it follows that
$\\{n\cdot|G|\,\,:\,\,n\,\,\mid\,\,|\mathbf{Z}(G)|\\}=Im(m_{G})$. ∎

An open question is “What can be said about groups $G$ with $|Im(m_{G})|=k$,
where $k$ be the number of divisors of $|\mathbf{Z}(G)|$?”. More questions
along these lines can be found in [16].

A classification of groups $G$ having $\mathcal{CD}(G)=[G/\mathbf{Z}(G)]$
appears in [15], and is based on the classification of groups $G$ having
$\mathcal{C}(G)=[G/\mathbf{Z}(G)]$ that appears in [14]. We collect two
results of Cheng, both in [6], that provide the classification for the
$p$-group case. Corollary 15 in the next section yields a classification of
the general case that is equivalent to the one in [15].

######  Proposition 10.

Let $G$ be a finite $p$-group. Then $\mathcal{C}(G)=[G/\mathbf{Z}(G)]$ if and
only if $G^{\prime}=\langle a\rangle$ is cyclic and $[a,G]\leq\langle
a^{4}\rangle$. If $p>2$, then, clearly $[a,G]\leq\langle a^{4}\rangle$, and so
in that case $\mathcal{C}(G)=[G/\mathbf{Z}(G)]$ if and only if $G^{\prime}$ is
cyclic.

Note that the case $G$ abelian is included in Proposition 10.

######  Proposition 11.

If $G$ is as in Proposition 10, $Z=\mathbf{Z}(G)$, and $Z\leq H\leq G$, then

$|G/Z|=|H/Z|\cdot|\mathbf{C}_{G}(H)/Z|.$

It is clear that the above two propositions together imply that a $p$-group
$G$ has $\mathcal{CD}(G)=\mathcal{C}(G)=[G/\mathbf{Z}(G)]$ if and only if $G$
is as in Proposition 10. This was also noted in [15].

##  4\. Monotone Chermak–Delgado Measure on Centralizers.

In this section, we prove Theorem B which classifies when the Chermak–Delgado
measure is increasing on centralizer subgroups of $G$.

######  Proposition 12.

Let $G$ be a finite group. Then the following are equivalent.

  1. (1)

$G$ is nilpotent.

  2. (2)

$\mathcal{S}(G)=\mathcal{S}\textit{n}(G)$.

  3. (3)

$\mathcal{C}(G)\subseteq\mathcal{S}\textit{n}(G)$.

###### Proof.

$(1)\leftrightarrow(2)$ is Lemma 2.1 in [10].

$(2)\rightarrow(3)$ is a tautological weakening of hypotheses.

To see that $(3)\rightarrow(2)$, let $H$ be an arbitrary subgroup of $G$. Now
for any $x\in G$ we have $\langle x\rangle\unlhd\mathbf{C}_{G}(x)$ which is
subnormal in $G$. Hence $H=\langle\langle x\rangle\,\,|\,\,x\in H\rangle$ is
subnormal by Proposition 5. ∎

We now prove Theorem B stating that the Chermak–Delgado measure is increasing
on centralizers if and only if the Chermak–Delgado lattice is exactly the
centralizer lattice. Moreover if the Chermak–Delgado measure is increasing on
centralizers, then the group is nilpotent.

###### Proof of Theorem B.

We can apply Corollary 4 taking $\mathcal{P}$ to be $\mathcal{C}(G)$. This
gives us that $\mathcal{C}(G)\subseteq\mathcal{CD}(G)$. Since
$\mathcal{CD}(G)$ consists of centralizers we conclude that
$\mathcal{CD}(G)=\mathcal{C}(G)$.

By Proposition 6, $\mathcal{CD}(G)\subseteq\mathcal{S}\textit{n}(G)$, and so
the nilpotency of $G$ follows from Proposition 12. ∎

Recall that finite nilpotent groups are direct products of their Sylow
subgroups. Moreover, both the Chermak–Delgado lattice and the centralizer
lattice respect direct products. For the Chermak–Delgado lattice, the result
appears in [3], and for the centralizer lattice it is an exercise in [14]. We
collect this together in the proposition below.

######  Proposition 13.

Let $G_{1},\dots,G_{n}$ be finite groups. Then

  * •

$\mathcal{C}(G_{1}\times\cdots\times
G_{n})=\mathcal{C}(G_{1})\times\cdots\times\mathcal{C}(G_{n})$.

  * •

$\mathcal{CD}(G_{1}\times\cdots\times
G_{n})=\mathcal{CD}(G_{1})\times\cdots\times\mathcal{CD}(G_{n})$.

We immediately obtain the following.

######  Corollary 14.

Suppose $G$ is a finite group. Then $m_{G}$ is increasing on $\mathcal{C}(G)$
if and only if $G=P_{1}\times\cdots\times P_{n}$, with $p_{1},\dots,p_{n}$
distinct primes and each $P_{i}$ a $p_{i}$-group with
$\mathcal{C}(P_{i})=\mathcal{CD}(P_{i})$.

Restricting to the case of Theorem A we obtain a classification.

######  Corollary 15.

Suppose $G$ is a finite group. Then $m_{G}$ is increasing on $\mathcal{S}(G)$
if and only if $G=P_{1}\times\cdots\times P_{n}$, with $p_{1},\dots,p_{n}$
distinct primes and each $P_{i}$ a $p_{i}$-group as in Proposition 10.

###### Proof.

First suppose that $G=P_{1}\times\cdots\times P_{n}$, with $p_{1},\dots,p_{n}$
distinct primes and each $P_{i}$ a $p_{i}$-group as in Proposition 10. Since
each $P_{i}$ has that
$\mathcal{CD}(P_{i})=\mathcal{C}(P_{i})=[P_{i}/\mathbf{Z}(P_{i})]$, and since
$(|P_{i}|,|P_{j}|)=1$ for $i\neq j$, we have that
$\mathcal{CD}(G)=\mathcal{C}(G)=[G/\mathbf{Z}(G)]$, and so $m_{G}$ is
increasing on $\mathcal{S}(G)$ by Theorem A.

Conversely, suppose that $m_{G}$ is increasing on $\mathcal{S}(G)$. Then
$m_{G}$ is increasing on $\mathcal{C}(G)$, and so by Corollary 14, we have
that $G=P_{1}\times\cdots\times P_{n}$, with $p_{1},\dots,p_{n}$ distinct
primes and each $P_{i}$ a $p_{i}$-group with
$\mathcal{C}(P_{i})=\mathcal{CD}(P_{i})$. Since $m_{G}$ is increasing on
$\mathcal{S}(G)$, by Theorem A,
$\mathcal{CD}(G)=\mathcal{C}(G)=[G/\mathbf{Z}(G)]$, and so it follows that
$\mathcal{CD}(P_{i})=\mathcal{C}(P_{i})=[P_{i}/\mathbf{Z}(P_{i})]$ for each
$i$, and so each $P_{i}$ is as in Proposition 10. ∎

It seems difficult to classify the $p$-groups $G$ with
$\mathcal{C}(G)=\mathcal{CD}(G)$. However, we can still observe many
interesting properties of these groups.

We will say that an abelian subgroup $A$ of a finite group $G$ is self-
centralizing, or maximal abelian, if $A=\mathbf{C}_{G}(A)$. Note that self-
centralizing abelian subgroups always exist, for if $A$ is an abelian subgroup
and $x\in\mathbf{C}_{G}(A)$, then $\langle A,x\rangle$ is abelian.

######  Proposition 16.

Suppose $G$ is a $p$-group with $\mathcal{C}(G)=\mathcal{CD}(G)$, and suppose
$|G|=p^{a}$ and $|\mathbf{Z}(G)|=p^{b}$. Then $a+b$ is even and every self-
centralizing abelian subgroup $A$ has order $p^{\frac{a+b}{2}}$.

###### Proof.

$G$, $\mathbf{Z}(G)$, and any self-centralizing abelian subgroup $A$ are all
in $\mathcal{C}(G)$. And because $\mathcal{C}(G)=\mathcal{CD}(G)$, such
subgroups all have the same Chermak–Delgado measure, and the result follows. ∎

The rest of the section investigates when $\mathcal{C}(G)$ forms a quasi-
antichain, i.e., the lattice contains a top and a bottom, and the join of any
two distinct elements is the top, and the meet of any two distinct elements is
the bottom. A group $G$ with $\mathcal{C}(G)$ a quasi-antichain is referred to
as an $\mathcal{M}$-group in [13].

We say that a centralizer $C$ of $G$ is minimal (maximal) if $C$ is an atom
(coatom) in $\mathcal{C}(G)$.

######  Proposition 17.

Let $G$ be a nonabelian group. If $M$ is a maximal centralizer, then
$M=\mathbf{C}_{G}(x)$ for some $x\in G-\mathbf{Z}(G)$.

###### Proof.

So $M=\mathbf{C}_{G}(H)$, and since $M<G$, $H\nsubseteq\mathbf{Z}(G)$. Let
$x\in H-\mathbf{Z}(G)$. Then $M=\mathbf{C}_{G}(H)\leq\mathbf{C}_{G}(x)<G$. And
since $M$ is a maximal centralizer, we have $M=\mathbf{C}_{G}(x)$. ∎

Note that the converse in not true, and so the centralizer of a non-central
element does not need to be a maximal centralizer. Take $G=D_{8}\times D_{8}$.
Then $\mathbf{C}_{G}((s,s))=\langle s,r^{2}\rangle\times\langle
s,r^{2}\rangle<\langle s,r^{2}\rangle\times D_{8}=\mathbf{C}_{G}((s,1))$.

The next result appears in [13], and we suggest the reader does not look it
up, but proves it as an exercise.

######  Proposition 18.

A group $G$ has that $\mathcal{C}(G)$ is a quasi-antichain if and only if
every maximal centralizer of $G$ is abelian.

######  Proposition 19.

A nonabelian group $G$ is equal to the union of its maximal centralizers.

###### Proof.

A centralizer must contain $\mathbf{Z}(G)$. If $x\in G-\mathbf{Z}(G)$, then
$x\in\mathbf{C}_{G}(x)\leq M$ for some maximal centralizer $M$. Thus $G$ is
equal to the union of its maximal centralizers. ∎

It is a well-known result that a group cannot equal the union of two of its
proper subgroups. This yields the following.

######  Corollary 20.

In a nonabelian group $G$, the number of maximal (minimal) centralizers is
greater than $2$. In particular, if $\mathcal{C}(G)$ is a quasi-antichain,
then the width of $\mathcal{C}(G)$ is greater than $2$.

###### Proof.

That the number of maximal centralizers is greater than $2$ is a consequence
of Proposition 19 and the fact that a group cannot equal the union of two of
its proper subgroups.

Now the centralizer map induces a duality on $\mathcal{C}(G)$, and so the
number of coatoms is equal to the number of atoms. Therefore the number of
minimal centralizers is greater than $2$. ∎

Quasi-antichain Chermak–Delgado lattices were studied in [4] and [1]. We
present one of the main results of [4].

######  Proposition 21.

If $G$ is a group with $\mathcal{CD}(G)$ a quasi-antichain of width $w\geq 3$
and $G\in\mathcal{CD}(G)$, then $G$ is nilpotent of class $2$; in fact, there
exists a prime $p$, a nonabelian Sylow $p$-subgroup $P$ with nilpotence class
$2$, and an abelian Hall $p^{\prime}$-subgroup $Q$ such that $G=P\times Q$,
$P\in\mathcal{CD}(P)$, and $\mathcal{CD}(G)\cong\mathcal{CD}(P)$ as lattices.
Moreover, there exist positive integers $a$, $b$ with $b\leq a$ such that
$|G/\mathbf{Z}(G)|=|P/\mathbf{Z}(P)|=p^{2a}$ and $w=p^{b}+1$.

Using the above results, we obtain the following.

######  Theorem 22.

Suppose that $G$ is a nonabelian $p$-group. Then
$\mathcal{C}(G)=\mathcal{CD}(G)$ is a quasi-antichain if and only if the
following all hold:

  * •

$G$ is nilpotent of class 2;

  * •

$\mathcal{C}(G)=\\{G,A_{1},\dots,A_{w},\mathbf{Z}(G)\\}$, where each $A_{i}$
is a maximal abelian subgroup of $G$ of order
$|A_{i}|=\sqrt{|G||\mathbf{Z}(G)|}$;

  * •

there exist positive integers $a$, $b$ with $b\leq a$ such that
$|G/\mathbf{Z}(G)|=p^{2a}$ and $w=p^{b}+1$.

###### Proof.

Suppose first that $\mathcal{C}(G)=\mathcal{CD}(G)$ is a quasi-antichain. So
$G,\mathbf{Z}(G)\in\mathcal{C}(G)$, and
$\mathcal{C}(G)=\mathcal{CD}(G)=\\{G,A_{1},\dots,A_{w},\mathbf{Z}(G)\\}$. The
fact that $A_{1},\dots,A_{w}$ are maximal abelian is Proposition 18. The fact
that $w\geq 3$ is Corollary 20. The rest follows directly from Proposition 16
and Proposition 21.

Conversely, if $\mathcal{C}(G)$ has the prescribed properties, then since
$\mathcal{CD}(G)\subseteq\mathcal{C}(G)$ and since $m_{G}$ is constant on
$\mathcal{C}(G)$, we have $\mathcal{C}(G)=\mathcal{CD}(G)$. ∎

Let $p$ be a prime and $n$ a positive integer. Let $G$ be the group of all
$3\times 3$ lower triangular matrices over $GF(p^{n})$ with $1$’s along the
diagonal. This example was given in [4] to show that quasi-antichain
Chermak–Delgado lattices with all atoms abelian exist for all possible widths
$p^{n}+1$. It is not hard to see that for each $x\in G-\mathbf{Z}(G)$,
$\mathbf{C}_{G}(x)$ is abelian. So it follows from Proposition 17 that every
maximal centralizer is abelian. By Proposition 18 we have that
$\mathcal{C}(G)$ is a quasi-antichain. Now there are exactly $p^{n}+1$ of
these maximal abelian subgroups, each of them having order $p^{2n}$, and so
width of $\mathcal{C}(G)$ is $p^{n}+1$, and since $m_{G}$ is constant on
$\mathcal{C}(G)$, we have that $\mathcal{CD}(G)=\mathcal{C}(G)$.

## References

  * An [2019] Lijian An.  Groups whose Chermak–Delgado lattice is a quasi-antichain.  _Journal of Group Theory_ , 22(3):529–524, 2019.
  * An [2022] Lijian An.  Twisted centrally large subgroups of finite groups.  _Journal of Algebra_ , 604:87–106, 2022.
  * Brewster and Wilcox [2012] Ben Brewster and Elizabeth Wilcox.  Some groups with computable Chermak–Delgado lattices.  _Bulletin of the Australian Mathematical Society_ , 86(1):29–40, 2012.
  * Brewster et al. [2014] Ben Brewster, Peter Hauck, and Elizabeth Wilcox.  Quasi-antichain Chermak–Delgado lattices of finite groups.  _Archiv der Mathematik_ , 103(4):301–311, 2014.
  * Burrell et al. [2023] David Burrell, William Cocke, and Ryan McCulloch.  On groups with few subgroups not in the Chermak–Delgado lattice.  _Arnold Mathematical Journal_ , 2023.  doi: https://doi.org/10.1007/s40598-023-00237-2.
  * Cheng [1982] Ying Cheng.  On finite $p$-groups with cyclic commutator subgroup.  _Archiv der Mathematik_ , 39:295–298, 1982.
  * Chermak and Delgado [1989] Andrew Chermak and Alberto Delgado.  A measuring argument for finite groups.  _Proceedings of the American Mathematical Society_ , 107(4):907–914, 1989.
  * Cocke [2020] William Cocke.  Subnormality and the Chermak–Delgado lattice.  _Journal of Algebra and its Applications_ , 19(8):2050141, 2020.
  * Fasolă [2024] Georgiana Fasolă.  On the Chermak–Delgado measure of a finite group.  _Results in Mathematics_ , 79(44), 2024.
  * Isaacs [2008] I. Martin Isaacs.  _Finite Group Theory_ , volume 92.  American Mathematical Soc., 2008.
  * McCulloch [2018] Ryan McCulloch.  Finite groups with a trivial Chermak–Delgado subgroup.  _Journal of Group Theory_ , 21(3):449–461, 2018.
  * Morresi Zuccari et al. [2018] Alessandro Morresi Zuccari, Valentina Russo, and Carlo Maria Scoppola.  The Chermak–Delgado measure in finite $p$-groups.  _Journal of Algebra_ , 502:262–276, 2018.
  * Schmidt [1970] Roland Schmidt.  Zentralisatorverbände endlicher gruppen.  _Rendiconti del Seminario Matematico della Università di Padova_ , 44:97–131, 1970.
  * Schmidt [1994] Roland Schmidt.  _Subgroup Lattices of Groups (De Gruyter Expositions in Mathematics Volume 14)_.  Walter de Gruyter, 1994.
  * Tărnăuceanu [2018] Marius Tărnăuceanu.  A note on the Chermak–Delgado lattice of a finite group.  _Communications in Algebra_ , 46(1):201–204, 2018.
  * Tărnăuceanu [2020] Marius Tărnăuceanu.  Finite groups with a certain number of values of the Chermak–Delgado measure.  _Journal of Algebra and its Applications_ , 19(5):2050088, 2020.
  * Wilcox [2016] Elizabeth Wilcox.  Exploring the Chermak–Delgado lattice.  _Mathematics Magazine_ , 89(1):38–44, 2016.