# LEGO: Learning EGOcentric Action Frame Generation
via Visual Instruction Tuning

Bolin Lai${}^{1,2\,\dagger}$ Xiaoliang Dai1 Lawrence Chen1 Guan Pang1 James M.
Rehg3 Miao Liu1
1GenAI, Meta 2Georgia Institute of Technology 3University of Illinois Urbana-
Champaign
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Generating instructional images of human daily actions from an egocentric
viewpoint serves a key step towards efficient skill transfer. In this paper,
we introduce a novel problem – egocentric action frame generation. The goal is
to synthesize the action frame conditioning on the user prompt question and an
input egocentric image that captures user’s environment. Notably, existing
egocentric datasets lack the detailed annotations that describe the execution
of actions. Additionally, the diffusion-based image manipulation models fail
to control the state change of an action within the corresponding egocentric
image pixel space. To this end, we finetune a visual large language model
(VLLM) via visual instruction tuning for curating the enriched action
descriptions to address our proposed problem. Moreover, we propose to Learn
EGOcentric (LEGO) action frame generation using image and text embeddings from
VLLM as additional conditioning. We validate our proposed model on two
egocentric datasets – Ego4D and Epic-Kitchens. Our experiments show prominent
improvement over prior image manipulation models in both quantitative and
qualitative evaluation. We also conduct detailed ablation studies and analysis
to provide insights on our method.

Figure 1: When the user queries for instructions on acquiring specific skills,
large language models (LLMs) tend to provide general instructions that may not
be readily applicable to the user’s current situation. While visual large
language models (VLLMs) can generate more specific guidance based on visual
prompt capturing the user’s context, the generated textual instructions still
lack clarity and ease for the user to interpret. In contrast, we present a
novel model — LEGO, which takes the user’s query and an egocentric image
captured from user’s perspective as inputs. It then generates an egocentric
action frame that vividly illustrates the execution of the query action.

††footnotetext: †This work was done during Bolin’s internship at GenAI, Meta.

##  1 Introduction

The emergence of Large Language Models (LLMs) [10, 6, 66, 99], such as
ChatGPT, has revolutionized the transfer of knowledge. However, an LLM alone
is not a sufficient tool for human skill transfer. Consider a question
answering example in Fig. 1(a). The AI agent can summarize general world
knowledge, yet it’s not directly applicable to the user’s current
circumstance. On the other hand, the egocentric visual perception provides a
novel means to capture the actions and intents as well as the surrounding
contexts of the camera wearer. As shown in Fig. 1(b), recent Visual Large
Language Models (VLLMs) [1, 47, 12, 53, 103] show the capability of generating
instructional responses based on the egocentric visual prompt. However, the
generated textual instructions are not the ideal vehicle for transferring
human skills, as neuroscience studies have revealed that human brain processes
images much faster than texts [4]. Moreover, it has also been evidenced that
humans can interpret an action from a static image [23]. Motivated by these
discoveries, we seek to design an egocentric AI system capable of synthesizing
an image that not only vividly depicts how an action should be conducted, but
also seamlessly aligns with the user’s visual perspective.

Formally, we introduce the novel problem of egocentric action frame generation
as shown in Fig. 1(c). Given a user query of how to perform a specific action
and an egocentric image capturing the moment before the action begins, the
goal is to synthesize an egocentric image illustrating the execution of the
action in the same egocentric context. In this paper, we address this problem
by harnessing diffusion models [69, 28], which have shown strong capability in
image manipulation tasks [101, 5, 26, 45, 58]. There are two major challenges
in using diffusion models to generate action frames from an egocentric
perspective. First, the action annotations of the existing egocentric datasets
[22, 13] are simply composed of a verb and nouns (see Fig. 1), and thus lack
the necessary details for diffusion models to learn the action state change
and to associate the action with correct objects and body movements. Second,
even when provided with detailed prompts about egocentric actions,
conditioning solely from a text encoder may not be able to effectively control
the action state change for action frame generation.

To address these challenges, we propose to Learn the EGOcentric (LEGO) action
frame generation model with visual instruction tuning. First, we design a data
curation pipeline that generates enriched action descriptions at scale, by
finetuning a VLLM with visual instruction tuning. Second, we leverage the
image and text embeddings from finetuned VLLM as a complement to the text
encoder from the diffusion model to improve the controllability for action
frame generation. Our experimental results suggest that the enriched action
descriptions and the utilization of VLLM embeddings both improve the image
generation performance. Overall, our contributions can be summarized as
follows:

  * •

We introduce the novel problem of egocentric action frame generation to
facilitate the process of skill transfer.

  * •

We leverage visual instruction tuning to enrich egocentric action
descriptions, and demonstrate how the enriched descriptions can help the
diffusion model understand the action state change from the egocentric
perspective.

  * •

We propose a novel model – LEGO that leverages the text and visual embeddings
from a visual large language model to improve the performance of the latent
diffusion model for egocentric action frame generation.

  * •

We conduct thorough experiments on Ego4D and Epic-Kitchens datasets to
validate the superiority of our model over prior approaches. We also showcase
the contribution of each component of our model design through ablation
studies. We further provide analysis on how the visual instruction tuned
embeddings benefit model performance.

##  2 Related Work

Text-Guided Image Manipulation. The recent emergence of diffusion models
enables text-guided image manipulation including image restoration [35], style
transfer [76], personalized image synthesis [70, 72, 88], pose generation [38,
30] and generic image editing [102, 83, 95, 36, 80, 40, 100, 85, 59, 84, 62,
45, 83, 19, 16]. SDEdit [58] converts the image to the latent space by adding
noise through a stochastic differential equation and then denoises the
representation for image editing. Rombach _et al_. [69] further expand SDEdit
from the original stroke-based editing to text-based editing. Null-text
inversion (NTI) [60] inverts a real image by DDIM [37] to yield the diffusion
process and then reconstructs the image. The image can then be edited
following the same strategies as Prompt-to-Prompt [26]. NTI relies on accurate
inversion process which can be improved by using coupled transformations [81]
or proximal guidance [25] and accelerated by a blended guidance strategy [63].
To associate the nouns with correct objects, DIFFEDIT [11] generates a mask to
localize the manipulated regions. However, most inversion-based methods
require accurate image captions, which are largely unavailable in the existing
egocentric dataset. Recently, InstructPix2Pix [5] demonstrates the potential
to edit a real image without the inversion process and original captions.
However, how to leverage the diffusion model to control the state change of an
action within the egocentric image plane remains unexplored.

Large Language Model for Image Generation. LLMs [10, 6, 66, 99, 79, 77] and
VLLMs [1, 47, 12, 53, 103, 96, 74, 24] have shown their strong capability of
understanding complex human instructions. Recently, LLMs and VLLMs are used to
guide image generation [94, 2, 9, 52, 7, 103, 8]. Wen _et al_. [89] use a
pretrained VLLM to pinpoint the misalignment of the text prompt and the
synthetic image and then correct it using the diffusion model. Lian _et al_.
[50] propose to generate a layout map using an LLM to improve the
understanding of prompts with spatial and numeracy reasoning. InstructEdit
[84] uses BLIP-2 [47] to infer the objects that will be edited and then
generates a mask with SAM [41] for object grounding. A pre-trained LLM can
also be used as a controller to connect with various foundation models [90,
71, 91]. GILL [42] learns text-relevant image embeddings from VLLM in a few
additional tokens for image generation. Importantly, all previous efforts
leverage the off-the-shelf image or text embeddings from the foundational
models. In contrast, our method uses visual instruction tuning to improve the
image and text embeddings from the VLLM and thereby facilitates the action
frame generation from the egocentric point of view.

Egocentric Vision. Recent efforts seek to understand human’s actions and
attentions [31, 32, 49, 17, 87, 75, 39, 43, 44], modeling hand-object
interactions [54, 67, 55, 21, 92], and estimating human body poses [78, 57,
82, 48] from the egocentric perspective. Here, we mainly discuss the most
relevant works on egocentric visual-language models and egocentric visual
content generation. Lin _et al_. [51] propose the first egocentric video-
language pre-training model – EgoVLP. Pramanick _et al_. [64] further improve
it by incorporating multi-modal fusion directly into the video and language
backbone. Ashutosh _et al_. [3] propose to learn a hierarchical video-language
embedding for long egocentric videos. Ramakrishnan _et al_. [68] introduce
NaQ, which is a data augmentation strategy to train models for long egocentric
video search with natural language queries. In terms of egocentric visual
generation, Jia _et al_. [34] leverage GANs [20] to generate future head
motion in hand forecasting task. Zhang _et al_. [97] leverage GANs to
facilitate future gaze anticipation. Ye _et al_. [93] propose the affordance
diffusion model that takes in the image of an object and generates possible
hand-object interactions in the egocentric view. In this paper, we present the
first work that investigates how to leverage VLLMs and diffusion models to
generate action state change on the egocentric image plane.

##  3 Method

Figure 2: Examples for data curation using GPT-3.5. We provide detailed action
descriptions of several example images as well as their action labels and
bounding boxes for in-context learning. In addition, we input the action label
and bounding boxes of another action as a query. GPT-3.5 is able to generate
descriptions with enriched information (highlighted by underlines) in the
response.

The problem setting of egocentric action frame generation is illustrated in
Fig. 1(c). Given an egocentric image frame $\mathcal{X}$ that captures the
user’s current visual context as well as a user query prompt $\mathcal{P}$
regarding how to perform an action, our goal is to synthesize the action frame
$\mathcal{Y}$ that visually depicts how the action should be conducted.

Figure 3: Overview of our proposed LEGO model. We first finetune a visual
large language model (VLLM) to generate the enriched action description with
visual instruction tuning. We then project image and text embeddings from the
finetuned VLLM to the feature space of the latent diffusion model (LDM).
Finally, we train the LDM to synthesize the egocentric action frame
conditioning on the input frame, enriched action description, as well as the
VLLM image and text embeddings.

The key insight of our proposed method is leveraging the strong capability of
a VLLM to enhance the diffusion model for egocentric action frame generation.
The annotations of existing egocentric datasets do not describe the details of
how actions are conducted. As a remedy, we leverage visual instruction tuning
to finetune a VLLM that enriches action descriptions based on the egocentric
visual prompt. In addition, the off-the-shelf CLIP text encoder of a
diffusion-based image manipulation model may have a domain gap between the
CLIP pre-training data and egocentric action datasets, resulting in limited
understanding of the action-related prompts used in our problem setting. To
bridge this gap, we propose LEGO – a novel approach that leverages VLLM
embeddings to control the state change of actions and to generate action
frames accordingly. We detail the VLLM-based data enrichment pipeline and our
model design in the following sections.

###  3.1 Egocentric Visual Instruction Tuning

Training Data for Visual Instruction Tuning. As shown in Fig. 2, we use
GPT-3.5 to generate detailed action descriptions based on an input query
composed of a short action label and object bounding boxes. First, we provide
several examples of possible inputs along with their expected output
descriptions for GPT-3.5 to perform _in-context learning_. These examples
cover a diverse set of scenes in the egocentric action dataset. Each example
is composed of an action label, a manually annotated action description, and
relative spatial information of hands and objects-of-interests represented by
bounding box coordinates. GPT-3.5 can learn from the given examples and
generate detailed action descriptions based on a new input query. The
resulting GPT-3.5 curated data is then further used for visual instruction
tuning. More details of the prompt structure are provided in the
supplementary.

Visual Instruction Tuning. We follow the finetuning strategy in prior work
[53], as shown in Fig. 3(a). Specifically, we use the pre-trained CLIP visual
encoder [65] $\phi$ to encode the visual representation and then apply a
linear projection layer $\tau$ to map the CLIP visual features into the
semantic space of the LLM, _i.e_. $\mathcal{H}_{i}=\tau(\phi(\mathcal{X}))$.
To construct the user prompt $\mathcal{P}$, we insert the action label
annotation into a prompt template to create a coherent query that is aligned
with the instruction-following nature of an LLM. We then tokenize
$\mathcal{P}$, and feed both the prompt text tokens and image tokens
$\mathcal{H}_{i}$ as inputs into the LLM. Finally, the LLM is trained to
generate enriched action description (denoted as $\mathcal{R}$) based on the
user prompt and image input.

User Prompt Enrichment at Scale. Note that the visual-instruction tuned VLLM
doesn’t rely on any object bounding boxes as input. Therefore, we can generate
enriched action descriptions for all egocentric action data at scale.

###  3.2 Egocentric Action Frame Generation

We leverage a latent diffusion model (LDM) [69] to synthesize the action frame
conditioning on the input frame and the detailed action description
$\mathcal{R}$ generated by our finetuned VLLM (see Fig. 3(b)). The input image
is first encoded into a latent space using a pre-trained auto-encoder
$\mathcal{E}$. The input to the denoising UNet is a concatentation of the
latent input representation $\mathcal{E}(\mathcal{X})$ and a Gaussian noise
$\mathcal{G}$.

Our key innovation is to design the U-Net conditioning component so that the
diffusion model can interpret the enriched action description. To start, we
follow [5] and adopt the conventional pre-trained CLIP text encoder $\psi$ to
extract a text representation of $\mathcal{R}$, _i.e_.
$\psi(\mathcal{R})\in\mathbb{R}^{N\times D}$ where $N$ is the maximum number
of text tokens and $D$ is the number of feature channels. We further leverage
the image and text embeddings from the visual instruction tuned VLLM as
additional LDM conditioning to alleviate the domain gap issue of the text
encoder.

Specifically, we feed the VLLM image embedding $\mathcal{H}_{i}$ into an extra
linear layer $\sigma$ to map it to LDM feature space, _i.e_.
$\sigma(\mathcal{H}_{i})\in\mathbb{R}^{M\times D}$, where $M$ is the number of
image tokens. Note that $\mathcal{H}_{i}$ is already projected to the semantic
space during visual instruction tuning, and therefore differs from the image
embedding $\mathcal{E}(\mathcal{X})$ from the auto-encoder. Moreover, we also
extract the text embedding $\mathcal{H}_{t}$ before the last linear layer of
LLM. We adopt a fixed token number $N$ and enforce padding or truncation
behavior, as in the CLIP text encoder. The text embedding is then fed to a
projection layer $\mu$. In LLM decoder, the response is generated iteratively
and each word embedding only conditions on the context ahead of it. To extract
the holistic semantic meaning of $\mathcal{H}_{t}$ in LDM feature space, we
further add self-attention layers $\pi$ after the projection, _i.e_.
$\pi(\mu(\mathcal{H}_{t}))\in\mathbb{R}^{N\times D}$. Thus, the U-Net
conditioning can be formulated as:

$\mathcal{C}=\Big{[}\psi(\mathcal{R}),\sigma(\mathcal{H}_{i}),\pi(\mu(\mathcal{H}_{t}))\Big{]}\in\mathbb{R}^{(2N+M)\times
D}.$ (1)

The conditioning $\mathcal{C}$ is fed into the denoising UNet at multiple
layers via the cross-attention mechanism [69]. We assume the intermediate
feature of a specific UNet layer is $\mathcal{U}$, which is learned from the
UNet input (_i.e_. the concatenation of input frame representation
$\mathcal{E}(\mathcal{X})$ and Gaussian noise $\mathcal{G}$). The cross-
attention at this UNet layer can be formulated as:

$CrossAtt(Q,K,V)=softmax\left(\frac{QK^{T}}{\sqrt{D}}\right)\cdot V,$ (2)

where $Q=W_{Q}\cdot\mathcal{U}$, $K=W_{K}\cdot\mathcal{C}$ and
$V=W_{V}\cdot\mathcal{C}$. Note that $W_{Q}$, $W_{K}$ and $W_{V}$ are
learnable matrices. We also adopt the classifier-free guidance following [5]
(see supplementary for details). Finally, the UNet output is converted to the
image domain by a pre-trained decoder.

###  3.3 Implementation Details

All parameters of the VLLM are initialized from the pre-trained LLaVA [53]
weights. During training, we freeze the CLIP image encoder and finetune the
projection layer and LLM with cross-entropy loss for 3 epochs. To improve the
diversity of the text prompts, we randomly select the question template from
10 candidates in each iteration. For LDM training, the text encoder, UNet and
auto-encoder are initialized with pre-trained weights [69]. The projection
layers $\sigma$ and $\mu$ and the self-attention layers $\pi$ are initialized
using the Xavier algorithm [18]. The text encoder is frozen and the remaining
weights are finetuned with L2 regression loss between the predicted noise and
real noise for 20,000 iterations. All input and target images are resized to a
resolution of 256$\times$256\. Please refer to the supplementary for more
details of training and inference.

##  4 Experiments

We first introduce the datasets, data preprocessing, and evaluation metrics
used in our experiments. Then we demonstrate the improvement of our model over
prior image manipulation approaches in both quantitative evaluation and
qualitative visualization. In addition, we ablate our model to show the
contribution of each component. Finally, we investigate the importance of VLLM
finetuning to the performance of the diffusion model.

Methods | Ego4D | Epic-Kitchens
---|---|---
CLIP | EgoVLP | EgoVLP+ |  FID $\downarrow$ | PSNR |  LPIPS $\downarrow$ | CLIP | EgoVLP | EgoVLP+ |  FID $\downarrow$ | PSNR |  LPIPS $\downarrow$
ProxNPI [25] | 0.682 | 0.445 | 0.727 | 33.011 | 11.876 | 0.409 | 0.658 | 0.323 | 0.528 | 51.350 | 11.064 | 0.464
SDEdit [58] | 0.734 | 0.501 | 0.729 | 33.347 | 11.813 | 0.416 | 0.748 | 0.338 | 0.568 | 27.413 | 11.296 | 0.433
IP2P [5] | 0.788 | 0.622 | 0.788 | 24.731 | 12.155 | 0.372 | 0.770 | 0.430 | 0.611 | 20.647 | 11.229 | 0.408
LEGO | 0.806 | 0.657 | 0.804 | 23.830 | 12.294 | 0.364 | 0.786 | 0.459 | 0.627 | 21.566 | 11.334 | 0.404
Table 1: Comparison with prior image manipulation approaches. $\downarrow$
means a lower score in this metric suggests a better performance. The best
results are highlighted with boldface. The  green row refers to our LEGO model
performance.

###  4.1 Data and Metrics

Datasets We conduct our experiments on two well-established egocentric action
datasets – Ego4D [22] and Epic-Kitchens [13]. Both datasets were densely
annotated with action starting time $t$ and ending time $\hat{t}$. In our
problem setting, we select an egocentric image frame $\delta_{i}$ seconds
before the action begins as the input $\mathcal{X}$, and an image $\delta_{o}$
seconds after the action begins as the target frame $\mathcal{Y}$. On the
Ego4D dataset, based on the annotations of Pre-Condition-15 time (PRE-15)
$t_{pre}$ , and Point-of-No-Return time (PNR) $t_{pnr}$, we set
$\delta_{i}=t-t_{pre}$ and $\delta_{o}=t_{pnr}-t$. For Epic-Kitchens, PNR and
PRE-15 annotations are not available. Instead, we empirically select
$\delta_{i}=0.25$ seconds and $\delta_{o}=t+\lambda*(\hat{t}-t)$, where
$\lambda=0.6$, for our experiments.

Data Preparation for Visual Instruction Tuning. For our dataset curation, we
randomly select 20,891 actions with bounding box annotations from the Ego4D
training set and 17,922 actions with VISOR [14] mask annotations from the
Epic-Kitchens training set. We then insert the original action label into a
prompt template to construct the full user prompt. In order to diversify the
prompt structure, we prepare 10 prompt templates (see supplementary) and
randomly select one for each action at training time.

Data Improvement for Action Frame Generation. Due to the possible drastic
camera motion, the egocentric image frames at $t-\delta_{i}$ and
$t+\delta_{o}$ may be blurry. As a mitigation, we first calculate aesthetic
scores [29] of the frames at $t-\delta_{i}$ and $t+\delta_{o}$ as well as $3$
frames before and after them. The corresponding frames with the highest
aesthetic score are used as the input and ground truth of our model. In
addition, the egocentric body motion may have huge variance depending on the
action type, meaning that the input frame and target frame may look almost
identical in more stationary actions (_e.g_. camera wearer is reading book),
or significantly different in more active actions (_e.g_. outdoor activities
with frequent head motion). Such large variances may incur additional barriers
for the diffusion model training. Therefore, we calculate the CLIP similarity
of the input frame and target frame, and we filter out the instances where the
CLIP similarity is lower than 0.81 or higher than 0.97. With these steps, we
ultimately curate 85521/9931 data samples for the train/test sets from Ego4D
and 61841/8893 data samples for the train/test sets from Epic-Kitchens.

Metrics. We adopt image-to-image similarity, image-to-text similarity, and
user study as metrics in our experiments.

  * •

_Image-to-Image Metrics._ We consider three types of metrics to evaluate
image-to-image similarity. (1) Distribution Divergence based metric: Fréchet
Inception Distance (FID) [27]. (2) Perception based metrics: Peak Signal-to-
Noise Ratio (PSNR) and Learned Perceptual Image Patch Similarity (LPIPS) [98]
with SqueezeNet [33] as the encoder. (3) Contrastive pre-training based
metrics: image-to-image CLIP score [65], EgoVLP score [51] and EgoVLP+ score
[51]. EgoVLP is a contrastive video-language pre-training model trained with
egocentric videos. Since EgoVLP takes multiple frames as inputs, we consider
two types of inputs: duplicating the output frame as a static input (_i.e_.
EgoVLP score) and combining the input frame with the output frame (_i.e_.
EgoVLP+ score). As as result, EgoVLP+ can effetively measure whether the
generated frame can depict the state change of an action. Note that instead of
measuring similarity of input and output frames as in prior works [12, 81], in
our problem setting, we measure the similarity between the generated action
frame and ground truth, which better reflects whether the generation results
can illustrate the execution of an action.

  * •

_Image-to-Text Metrics._ We find the widely-used image-to-text CLIP score can
not align actions with egocentric images due to the domain gap. Similar
misalignment problem is also observed in [61, 86, 15, 73]. In our experiments,
we utilize BLIP [46] to generate image captions of the output images and then
calculate text-to-text similarity using CLIP text encoder (following [36]). We
implement this metric with two BLIP structures: BLIP-B and BLIP-L. Though this
solution may still suffer from the same domain gap issue, it is a more
appropriate evaluation metric in our problem setting. See more evidence and
discussions in the supplementary.

  * •

_User Study._ We also conduct a user study on a subset of test data to further
validate the advantage of our model based on human preference. We sample 60
examples from each dataset and present the generated frames from our model as
well as the baseline models to raters on Amazon Mechanical Turk (AMT). We also
provide the input frames and the corresponding action labels during
evaluation. For each instance, we ask the raters to select the image that best
aligned with the provided action label while preserving the most contextual
information from the input frame. To minimize potential bias, we hire 5 AMT
raters to annotate each example thus resulting in 300 samples for each
dataset. The template of our user study is presented in the supplementary.

###  4.2 Comparison with Prior Approaches

We compare our proposed model with previous state-of-the-art methods for text-
guided image manipulation, including ProxNPI [25], SDEdit [58] and
InstructPix2Pix (IP2P) [5]. For a fair comparison, we finetune these baseline
methods with the same data used in our experiments. Specifically, we train
IP2P in an end-to-end way on the two datasets with existing egocentric action
labels as text conditioning. ProxNPI and SDEdit rely on the off-the-shelf
latent diffusion model to synthesize edited images and thus can not be trained
end to end. Therefore, we first finetune the latent diffusion model with
egocentric images and action labels and then use the finetuned latent
diffusion model parameters to implement ProxNPI and SDEdit approaches.

####  4.2.1 Quantitative Comparison

Methods | Ego4D | Epic-Kitchens
---|---|---
BLIP-B | BLIP-L | BLIP-B | BLIP-L
ProxNPI [25] | 0.177 | 0.174 | 0.237 | 0.234
SDEdit [58] | 0.198 | 0.197 | 0.215 | 0.213
IP2P [5] | 0.200 | 0.205 | 0.254 | 0.264
LEGO | 0.204 | 0.207 | 0.270 | 0.274
Table 2: Image-to-text similarity evaluation of our model and baselines. The
best results are highlighted with boldface.

As shown in Tab. 1, both ProxNPI and SDEdit perform poorly on this novel
problem, suggesting the challenging nature of generating egocentric action
frames. IP2P performs much better in almost all metrics by end-to-end training
and serves as the strongest baseline model in our experiments. However, our
LEGO model consistently outperforms IP2P by a large margin (1.8%, 3.5%, 1.6%,
0.901, 0.139 and 0.8% respectively) in all six metrics on Ego4D. LEGO also
exceeds IP2P notably (1.6%, 2.9%, 1.6%, 0.105 and 0.4% respectively) in five
metrics on Epic-Kitchens. Though, LEGO slightly lags behind IP2P in FID score,
it still achieves the second best performance.

With regard to image-text similarity, LEGO consistently outperforms prior
approaches on both datasets (Tab. 2). LEGO outperforms IP2P by 0.4% and 0.2%
on Ego4D and by 1.6% and 1.0% on Epic-Kitchens. The result suggests the
synthetic frames from our model can better align with the action descriptions.
However, the performance gain is rather limited. We emphasize that this is
because the domain gap still exists for the BLIP model.

Figure 4: User study of our model and baselines. Win rate is the percentage of
each model being picked as the best in all samples. Our model outperforms all
baselines by a significant margin.

For our user study, we shuffle the order of the results from our model and the
baselines while presenting them to the raters. We define the win rate as the
percentage of each model being picked as the best out of the total 300
samples. Results are illustrated in Fig. 4. Our model surpasses the second
best baseline by 44.0% and 38.3% respectively on Ego4D and Epic-Kitchens. The
prominent gap further validates the superiority of our model.

###  4.3 Visualization of Generated Frames

Figure 5: Visualization of the proposed LEGO model and all baselines on Ego4D
(the first three rows) and Epic-Kitchens (the last three rows). The action
frames generated by LEGO align with the user prompt better and preserve more
environment contexts in input frames.

We additionally showcase examples of generated image from LEGO and baseline
models in Fig. 5. ProxNPI and SDEdit fail to understand the user prompt and
thus generate frames of irrelevant actions (_e.g_. row3). They may also easily
synthesize the action in a different environment (_e.g_. row2).
InstructPix2Pix is able to preserve more contexts but still fails to
semantically align with the action in user prompt (_e.g_. row4). In contrast,
LEGO can synthesize the action frame that better aligns with the user prompt
as well as retains more contexts in the background.

###  4.4 Ablation Study

| Conditioning | CLIP | EgoVLP | EgoVLP+ | PSNR
---|---|---|---|---|---
Ego4D | Actions Labels | 0.788 | 0.622 | 0.788 | 12.155
Descriptions | 0.791 | 0.629 | 0.791 | 12.200
Desc. + Img Embed. | 0.807 | 0.654 | 0.801 | 12.239
Desc. + Txt Embed. | 0.792 | 0.633 | 0.794 | 12.203
Desc. + Joint Embed. | 0.806 | 0.657 | 0.804 | 12.294
Epic-Kitchens | Actions Labels | 0.770 | 0.430 | 0.611 | 11.229
Descriptions | 0.775 | 0.437 | 0.615 | 11.247
Desc. + Img Embed. | 0.785 | 0.458 | 0.623 | 11.254
Desc. + Txt Embed. | 0.777 | 0.447 | 0.620 | 11.305
Desc. + Joint Embed. | 0.786 | 0.459 | 0.627 | 11.334
Table 3: Analysis of egocentric action frame generation performance with
different conditionings. Joint Embed. refers to incorporating both VLLM image
and text embeddings. The best results are highlighted with boldface. The
green rows refer to our full LEGO model performance.

We present comprehensive ablation studies to investigate the contribution of
enriched action descriptions, the VLLM image embedding and the VLLM text
embedding separately. Results are demonstrated in Tab. 3. Notably,
conditioning on enriched action descriptions can moderately improve the model
performance, supporting our claim that expanding the action description can
facilitate the learning of the state change during an egocentric action.
Moreover, utilizing the image embedding and text embedding from VLLM as
additional conditions both improve the model performance by a notable margin.
Interestingly, the image embedding leads to larger performance boost on both
datasets (_e.g_. 1.6%/1.0% in CLIP score, and 2.5%/2.1% in EgoVLP score).
These results suggest the VLLM image embedding $\mathcal{H}_{i}$ incorporates
important high-level semantic meaning that are not captured in the auto-
encoder image embedding $\mathcal{E}(\mathcal{X})$ or VLLM text emebdding
$\mathcal{H}_{t}$. Finally, our full LEGO model uses both VLLM image and text
embeddings (Desc.+Joint Embed.) and thus achieves the best performance on both
datasets considering all metrics.

###  4.5 Analysis of Visual Instruction Tuning

| Conditioning | CLIP | EgoVLP | EgoVLP+ | PSNR
---|---|---|---|---|---
Ego4D | Descriptions | 0.791 | 0.629 | 0.791 | 12.200
Desc.+Img Embed.(w/o FT) | 0.804 | 0.645 | 0.797 | 12.238
$\Delta$ (w/o FT - w/ FT) | -0.003 | -0.009 | -0.004 | -0.001
Desc.+Txt Embed.(w/o FT) | 0.791 | 0.625 | 0.788 | 12.154
$\Delta$ (w/o FT - w/ FT) | -0.001 | -0.008 | -0.006 | -0.049
Desc.+Joint Embed.(w/o FT) | 0.802 | 0.646 | 0.797 | 12.222
$\Delta$ (w/o FT - w/ FT) | -0.004 | -0.011 | -0.007 | -0.072
Epic-Kitchens | Descriptions | 0.775 | 0.437 | 0.615 | 11.247
Desc.+Img Embed.(w/o FT) | 0.783 | 0.449 | 0.620 | 11.085
$\Delta$ (w/o FT - w/ FT) | -0.002 | -0.009 | -0.003 | -0.169
Desc.+Txt Embed.(w/o FT) | 0.774 | 0.433 | 0.610 | 11.134
$\Delta$ (w/o FT - w/ FT) | -0.003 | -0.014 | -0.010 | -0.171
Desc.+Joint Embed.(w/o FT) | 0.784 | 0.447 | 0.619 | 11.094
$\Delta$ (w/o FT - w/ FT) | -0.002 | -0.012 | -0.008 | -0.240
Table 4: Performance of LEGO without finetuning (w/o FT). The rows highlighted
with  gray indicate the performance drop when compared with results of the
counterparts finetuned using visual instruction tuning (w/ FT) from Tab. 3.

We further examine whether the visual instruction tuned embeddings can help
the diffusion model to better understand the state change of an egocentric
action. As shown in Tab. 4, without any finetuning, the image embedding from
the VLLM can still improve the baseline model (_Description_) performance on
both datasets. However, the model using off-the-shelf VLLM text embedding
actually decreases the baseline model performance. We speculate this is
because, the VLLM may generate hallucinated responses without visual
instruction tuning, which incurs additional barriers for the text encoder to
extract useful information. In addition, finetuned embeddings yield much
larger improvement (see the highlighted rows in Tab. 4) across all embeddings
types and all metrics (_i.e_. a relative improvement of 1.7%/2.7% in EgoVLP
when using Joint Emebed.) on Ego4D/Epic-Kitchens datasets. These results
suggest that visual instruction tuning can align the conditioned embeddings
with the instructional prompts and thereby significantly benefit the model
performance.

##  5 Conclusion

In this paper, we introduce the novel problem of egocentric action frame
generation. We also propose a novel model — LEGO, that leverages visual
instruction tuning and a diffusion model to address this problem. Our key
intuition is to use visual instruction tuning to generate informative
responses that depict the execution of the egocentric action, and then design
the conditioning of the U-Net to exploit the image and text embeddings from a
visual instruction tuned VLLM. Our experiments on two large-scale egocentric
action datasets validate the advantage of the proposed approach as well as the
contribution of each model component. We believe our work provides an
important step in understanding the action stage change and controllability of
diffusion models, and suggests future research directions in egocentric AI
systems, image generation, and human skill transfer.

## References

  * Alayrac et al. [2022] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al.  Flamingo: a visual language model for few-shot learning.  _Advances in Neural Information Processing Systems_ , 35:23716–23736, 2022.
  * Anonymity [2023] Anonymity.  Making multimodal generation easier: When diffusion models meet llms.  _openreview_ , 2023.
  * Ashutosh et al. [2023] Kumar Ashutosh, Rohit Girdhar, Lorenzo Torresani, and Kristen Grauman.  Hiervl: Learning hierarchical video-language embeddings.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 23066–23078, 2023.
  * Baskin et al. [2007] Joseph H Baskin, Judith G Edersheim, and Bruce H Price.  Is a picture worth a thousand words? neuroimaging in the courtroom.  _American Journal of Law & Medicine_, 33(2-3):239–269, 2007.
  * Brooks et al. [2023] Tim Brooks, Aleksander Holynski, and Alexei A Efros.  Instructpix2pix: Learning to follow image editing instructions.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 18392–18402, 2023.
  * Brown et al. [2020] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901, 2020.
  * Chakrabarty et al. [2023] Tuhin Chakrabarty, Kanishk Singh, Arkadiy Saakyan, and Smaranda Muresan.  Learning to follow object-centric image editing instructions faithfully.  _arXiv preprint arXiv:2310.19145_ , 2023.
  * Chen et al. [2023a] Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, et al.  Pixart-alpha: Fast training of diffusion transformer for photorealistic text-to-image synthesis.  _arXiv preprint arXiv:2310.00426_ , 2023a.
  * Chen et al. [2023b] Wei-Ge Chen, Irina Spiridonova, Jianwei Yang, Jianfeng Gao, and Chunyuan Li.  Llava-interactive: An all-in-one demo for image chat, segmentation, generation and editing.  _arXiv preprint arXiv:2311.00571_ , 2023b.
  * Chowdhery et al. [2022] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al.  Palm: Scaling language modeling with pathways.  _arXiv preprint arXiv:2204.02311_ , 2022.
  * Couairon et al. [2022] Guillaume Couairon, Jakob Verbeek, Holger Schwenk, and Matthieu Cord.  Diffedit: Diffusion-based semantic image editing with mask guidance.  _arXiv preprint arXiv:2210.11427_ , 2022.
  * Dai et al. [2023] Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi.  Instructblip: Towards general-purpose vision-language models with instruction tuning.  _Advances in Neural Information Processing Systems_ , 2023.
  * Damen et al. [2022] Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Antonino Furnari, Evangelos Kazakos, Jian Ma, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, et al.  Rescaling egocentric vision: Collection, pipeline and challenges for epic-kitchens-100.  _International Journal of Computer Vision_ , pages 1–23, 2022.
  * Darkhalil et al. [2022] Ahmad Darkhalil, Dandan Shan, Bin Zhu, Jian Ma, Amlan Kar, Richard Higgins, Sanja Fidler, David Fouhey, and Dima Damen.  Epic-kitchens visor benchmark: Video segmentations and object relations.  In _Proceedings of the Neural Information Processing Systems (NeurIPS) Track on Datasets and Benchmarks_ , 2022.
  * Du et al. [2023] Yilun Du, Sherry Yang, Bo Dai, Hanjun Dai, Ofir Nachum, Joshua B Tenenbaum, Dale Schuurmans, and Pieter Abbeel.  Learning universal policies via text-guided video generation.  In _Thirty-seventh Conference on Neural Information Processing Systems_ , 2023.
  * Epstein et al. [2023] Dave Epstein, Allan Jabri, Ben Poole, Alexei A Efros, and Aleksander Holynski.  Diffusion self-guidance for controllable image generation.  In _Advances in Neural Information Processing Systems_ , 2023.
  * Girdhar and Grauman [2021] Rohit Girdhar and Kristen Grauman.  Anticipative video transformer.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pages 13505–13515, 2021.
  * Glorot and Bengio [2010] Xavier Glorot and Yoshua Bengio.  Understanding the difficulty of training deep feedforward neural networks.  In _Proceedings of the thirteenth international conference on artificial intelligence and statistics_ , pages 249–256. JMLR Workshop and Conference Proceedings, 2010.
  * Goel et al. [2023] Vidit Goel, Elia Peruzzo, Yifan Jiang, Dejia Xu, Nicu Sebe, Trevor Darrell, Zhangyang Wang, and Humphrey Shi.  Pair-diffusion: Object-level image editing with structure-and-appearance paired diffusion models.  _arXiv preprint arXiv:2303.17546_ , 2023.
  * Goodfellow et al. [2020] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial networks.  _Communications of the ACM_ , 63(11):139–144, 2020.
  * Goyal et al. [2022] Mohit Goyal, Sahil Modi, Rishabh Goyal, and Saurabh Gupta.  Human hands as probes for interactive object understanding.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 3293–3303, 2022.
  * Grauman et al. [2022] Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al.  Ego4d: Around the world in 3,000 hours of egocentric video.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 18995–19012, 2022.
  * Hafri et al. [2017] Alon Hafri, John C Trueswell, and Russell A Epstein.  Neural representations of observed actions generalize across static and dynamic visual input.  _Journal of Neuroscience_ , 37(11):3056–3071, 2017.
  * Han et al. [2023] Jiaming Han, Renrui Zhang, Wenqi Shao, Peng Gao, Peng Xu, Han Xiao, Kaipeng Zhang, Chris Liu, Song Wen, Ziyu Guo, et al.  Imagebind-llm: Multi-modality instruction tuning.  _arXiv preprint arXiv:2309.03905_ , 2023.
  * Han et al. [2024] Ligong Han, Song Wen, Qi Chen, Zhixing Zhang, Kunpeng Song, Mengwei Ren, Ruijiang Gao, Yuxiao Chen, Di Liu, Qilong Zhangli, et al.  Improving negative-prompt inversion via proximal guidance.  _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , 2024.
  * Hertz et al. [2022] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or.  Prompt-to-prompt image editing with cross attention control.  _arXiv preprint arXiv:2208.01626_ , 2022.
  * Heusel et al. [2017] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.  Gans trained by a two time-scale update rule converge to a local nash equilibrium.  _Advances in neural information processing systems_ , 30, 2017.
  * Ho et al. [2020] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  _Advances in neural information processing systems_ , 33:6840–6851, 2020.
  * [29] https://github.com/christophschuhmann/improved-aesthetic-predictor, 2023.  GitHub.
  * Huang et al. [2023] Jiancheng Huang, Yifan Liu, Jin Qin, and Shifeng Chen.  Kv inversion: Kv embeddings learning for text-conditioned real image action editing.  _arXiv preprint arXiv:2309.16608_ , 2023.
  * Huang et al. [2018] Yifei Huang, Minjie Cai, Zhenqiang Li, and Yoichi Sato.  Predicting gaze in egocentric video by learning task-dependent attention transition.  In _Proceedings of the European conference on computer vision (ECCV)_ , pages 754–769, 2018.
  * Huang et al. [2020] Yifei Huang, Minjie Cai, Zhenqiang Li, Feng Lu, and Yoichi Sato.  Mutual context network for jointly estimating egocentric gaze and action.  _IEEE Transactions on Image Processing_ , 29:7795–7806, 2020.
  * Iandola et al. [2016] Forrest N Iandola, Song Han, Matthew W Moskewicz, Khalid Ashraf, William J Dally, and Kurt Keutzer.  Squeezenet: Alexnet-level accuracy with 50x fewer parameters and¡ 0.5 mb model size.  _arXiv preprint arXiv:1602.07360_ , 2016.
  * Jia et al. [2022] Wenqi Jia, Miao Liu, and James M Rehg.  Generative adversarial network for future hand segmentation from egocentric video.  In _European Conference on Computer Vision_ , pages 639–656. Springer, 2022.
  * Jiang et al. [2023] Yitong Jiang, Zhaoyang Zhang, Tianfan Xue, and Jinwei Gu.  Autodir: Automatic all-in-one image restoration with latent diffusion.  _arXiv preprint arXiv:2310.10123_ , 2023.
  * Joseph et al. [2024] KJ Joseph, Prateksha Udhayanan, Tripti Shukla, Aishwarya Agarwal, Srikrishna Karanam, Koustava Goswami, and Balaji Vasan Srinivasan.  Iterative multi-granular image editing using diffusion models.  _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , 2024.
  * Kawar et al. [2022] Bahjat Kawar, Michael Elad, Stefano Ermon, and Jiaming Song.  Denoising diffusion restoration models.  In _Advances in Neural Information Processing Systems_ , 2022.
  * Kawar et al. [2023] Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani.  Imagic: Text-based real image editing with diffusion models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 6007–6017, 2023.
  * Kazakos et al. [2019] Evangelos Kazakos, Arsha Nagrani, Andrew Zisserman, and Dima Damen.  Epic-fusion: Audio-visual temporal binding for egocentric action recognition.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 5492–5501, 2019.
  * Kim et al. [2023] Sunwoo Kim, Wooseok Jang, Hyunsu Kim, Junho Kim, Yunjey Choi, Seungryong Kim, and Gayeong Lee.  User-friendly image editing with minimal text input: Leveraging captioning and injection techniques.  _arXiv preprint arXiv:2306.02717_ , 2023.
  * Kirillov et al. [2023] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al.  Segment anything.  _arXiv preprint arXiv:2304.02643_ , 2023.
  * Koh et al. [2023] Jing Yu Koh, Daniel Fried, and Ruslan Salakhutdinov.  Generating images with multimodal language models.  _Advances in Neural Information Processing Systems_ , 2023.
  * Lai et al. [2022] Bolin Lai, Miao Liu, Fiona Ryan, and James M Rehg.  In the eye of transformer: Global-local correlation for egocentric gaze estimation.  _British Machine Vision Conference_ , 2022.
  * Lai et al. [2023] Bolin Lai, Fiona Ryan, Wenqi Jia, Miao Liu, and James M Rehg.  Listen to look into the future: Audio-visual egocentric gaze anticipation.  _arXiv preprint arXiv:2305.03907_ , 2023.
  * Li et al. [2023a] Dongxu Li, Junnan Li, and Steven CH Hoi.  Blip-diffusion: Pre-trained subject representation for controllable text-to-image generation and editing.  _Advances in Neural Information Processing Systems_ , 2023a.
  * Li et al. [2022] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi.  Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation.  In _International Conference on Machine Learning_ , pages 12888–12900. PMLR, 2022.
  * Li et al. [2023b] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi.  Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.  _International Conference on Machine Learning_ , 2023b.
  * Li et al. [2023c] Jiaman Li, Karen Liu, and Jiajun Wu.  Ego-body pose estimation via ego-head pose estimation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 17142–17151, 2023c.
  * Li et al. [2018] Yin Li, Miao Liu, and James M Rehg.  In the eye of beholder: Joint learning of gaze and actions in first person video.  In _Proceedings of the European conference on computer vision (ECCV)_ , pages 619–635, 2018.
  * Lian et al. [2023] Long Lian, Boyi Li, Adam Yala, and Trevor Darrell.  Llm-grounded diffusion: Enhancing prompt understanding of text-to-image diffusion models with large language models.  _arXiv preprint arXiv:2305.13655_ , 2023.
  * Lin et al. [2022] Kevin Qinghong Lin, Jinpeng Wang, Mattia Soldan, Michael Wray, Rui Yan, Eric Z XU, Difei Gao, Rong-Cheng Tu, Wenzhe Zhao, Weijie Kong, et al.  Egocentric video-language pretraining.  _Advances in Neural Information Processing Systems_ , 35:7575–7586, 2022.
  * Liu et al. [2023a] Bolong Liu, Hao Zhang, Jie Liu, and Qiang Wang.  Acigs: An automated large-scale crops image generation system based on large visual language multi-modal models.  In _2023 20th Annual IEEE International Conference on Sensing, Communication, and Networking (SECON)_ , pages 7–13. IEEE, 2023a.
  * Liu et al. [2023b] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee.  Visual instruction tuning.  _Advances in neural information processing systems_ , 2023b.
  * Liu et al. [2020] Miao Liu, Siyu Tang, Yin Li, and James M Rehg.  Forecasting human-object interaction: joint prediction of motor attention and actions in first person video.  In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part I 16_ , pages 704–721. Springer, 2020.
  * Liu et al. [2022] Shaowei Liu, Subarna Tripathi, Somdeb Majumdar, and Xiaolong Wang.  Joint hand motion and interaction hotspots prediction from egocentric videos.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 3282–3292, 2022.
  * Loshchilov and Hutter [2017] Ilya Loshchilov and Frank Hutter.  Sgdr: Stochastic gradient descent with warm restarts.  In _International Conference on Learning Representations_ , 2017.
  * Luo et al. [2021] Zhengyi Luo, Ryo Hachiuma, Ye Yuan, and Kris Kitani.  Dynamics-regulated kinematic policy for egocentric pose estimation.  _Advances in Neural Information Processing Systems_ , 34:25019–25032, 2021.
  * Meng et al. [2022] Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon.  SDEdit: Guided image synthesis and editing with stochastic differential equations.  In _International Conference on Learning Representations_ , 2022.
  * Mirzaei et al. [2023] Ashkan Mirzaei, Tristan Aumentado-Armstrong, Marcus A Brubaker, Jonathan Kelly, Alex Levinshtein, Konstantinos G Derpanis, and Igor Gilitschenski.  Watch your steps: Local image and scene editing by text instructions.  _arXiv preprint arXiv:2308.08947_ , 2023.
  * Mokady et al. [2023] Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or.  Null-text inversion for editing real images using guided diffusion models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 6038–6047, 2023.
  * Molad et al. [2023] Eyal Molad, Eliahu Horwitz, Dani Valevski, Alex Rav Acha, Yossi Matias, Yael Pritch, Yaniv Leviathan, and Yedid Hoshen.  Dreamix: Video diffusion models are general video editors.  _arXiv preprint arXiv:2302.01329_ , 2023.
  * Orgad et al. [2023] Hadas Orgad, Bahjat Kawar, and Yonatan Belinkov.  Editing implicit assumptions in text-to-image diffusion models.  _arXiv preprint arXiv:2303.08084_ , 2023.
  * Pan et al. [2023] Zhihong Pan, Riccardo Gherardi, Xiufeng Xie, and Stephen Huang.  Effective real image editing with accelerated iterative diffusion inversion.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 15912–15921, 2023.
  * Pramanick et al. [2023] Shraman Pramanick, Yale Song, Sayan Nag, Kevin Qinghong Lin, Hardik Shah, Mike Zheng Shou, Rama Chellappa, and Pengchuan Zhang.  Egovlpv2: Egocentric video-language pre-training with fusion in the backbone.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 5285–5297, 2023.
  * Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In _International conference on machine learning_ , pages 8748–8763. PMLR, 2021.
  * Raffel et al. [2020] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _The Journal of Machine Learning Research_ , 21(1):5485–5551, 2020.
  * Ragusa et al. [2023] Francesco Ragusa, Giovanni Maria Farinella, and Antonino Furnari.  Stillfast: An end-to-end approach for short-term object interaction anticipation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 3635–3644, 2023.
  * Ramakrishnan et al. [2023] Santhosh Kumar Ramakrishnan, Ziad Al-Halah, and Kristen Grauman.  Naq: Leveraging narrations as queries to supervise episodic memory.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 6694–6703, 2023.
  * Rombach et al. [2022] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 10684–10695, 2022.
  * Ruiz et al. [2023] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman.  Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 22500–22510, 2023.
  * Shen et al. [2023] Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang.  Hugginggpt: Solving ai tasks with chatgpt and its friends in huggingface.  _arXiv preprint arXiv:2303.17580_ , 2023.
  * Shi et al. [2023] Jing Shi, Wei Xiong, Zhe Lin, and Hyun Joon Jung.  Instantbooth: Personalized text-to-image generation without test-time finetuning.  _arXiv preprint arXiv:2304.03411_ , 2023.
  * Stein et al. [2023] George Stein, Jesse C Cresswell, Rasa Hosseinzadeh, Yi Sui, Brendan Leigh Ross, Valentin Villecroze, Zhaoyan Liu, Anthony L Caterini, J Eric T Taylor, and Gabriel Loaiza-Ganem.  Exposing flaws of generative model evaluation metrics and their unfair treatment of diffusion models.  _arXiv preprint arXiv:2306.04675_ , 2023.
  * Su et al. [2023] Yixuan Su, Tian Lan, Huayang Li, Jialu Xu, Yan Wang, and Deng Cai.  Pandagpt: One model to instruction-follow them all.  _arXiv preprint arXiv:2305.16355_ , 2023.
  * Sudhakaran et al. [2019] Swathikiran Sudhakaran, Sergio Escalera, and Oswald Lanz.  Lsta: Long short-term attention for egocentric action recognition.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 9954–9963, 2019.
  * Sun et al. [2023] Zhengwentai Sun, Yanghong Zhou, Honghong He, and PY Mok.  Sgdiff: A style guided diffusion model for fashion synthesis.  In _Proceedings of the 31st ACM International Conference on Multimedia_ , pages 8433–8442, 2023.
  * Thoppilan et al. [2022] Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al.  Lamda: Language models for dialog applications.  _arXiv preprint arXiv:2201.08239_ , 2022.
  * Tome et al. [2020] Denis Tome, Thiemo Alldieck, Patrick Peluse, Gerard Pons-Moll, Lourdes Agapito, Hernan Badino, and Fernando De la Torre.  Selfpose: 3d egocentric pose estimation from a headset mounted camera.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2020.
  * Touvron et al. [2023] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al.  Llama: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_ , 2023.
  * Tsaban and Passos [2023] Linoy Tsaban and Apolinário Passos.  Ledits: Real image editing with ddpm inversion and semantic guidance.  _arXiv preprint arXiv:2307.00522_ , 2023.
  * Wallace et al. [2023] Bram Wallace, Akash Gokul, and Nikhil Naik.  Edict: Exact diffusion inversion via coupled transformations.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 22532–22541, 2023.
  * Wang et al. [2023a] Jian Wang, Diogo Luvizon, Weipeng Xu, Lingjie Liu, Kripasindhu Sarkar, and Christian Theobalt.  Scene-aware egocentric 3d human pose estimation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 13031–13040, 2023a.
  * Wang et al. [2023b] Kai Wang, Fei Yang, Shiqi Yang, Muhammad Atif Butt, and Joost van de Weijer.  Dynamic prompt learning: Addressing cross-attention leakage for text-based image editing.  _arXiv preprint arXiv:2309.15664_ , 2023b.
  * Wang et al. [2023c] Qian Wang, Biao Zhang, Michael Birsak, and Peter Wonka.  Instructedit: Improving automatic masks for diffusion-based image editing with user instructions.  _arXiv preprint arXiv:2305.18047_ , 2023c.
  * Wang et al. [2023d] Qian Wang, Biao Zhang, Michael Birsak, and Peter Wonka.  Mdp: A generalized framework for text-guided image editing by manipulating the diffusion path.  _arXiv preprint arXiv:2303.16765_ , 2023d.
  * Wang et al. [2023e] Wen Wang, Kangyang Xie, Zide Liu, Hao Chen, Yue Cao, Xinlong Wang, and Chunhua Shen.  Zero-shot video editing using off-the-shelf image diffusion models.  _arXiv preprint arXiv:2303.17599_ , 2023e.
  * Wang et al. [2021] Xiaohan Wang, Linchao Zhu, Heng Wang, and Yi Yang.  Interactive prototype learning for egocentric action recognition.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 8168–8177, 2021.
  * Wei et al. [2023] Yuxiang Wei, Yabo Zhang, Zhilong Ji, Jinfeng Bai, Lei Zhang, and Wangmeng Zuo.  Elite: Encoding visual concepts into textual embeddings for customized text-to-image generation.  _arXiv preprint arXiv:2302.13848_ , 2023.
  * Wen et al. [2023] Song Wen, Guian Fang, Renrui Zhang, Peng Gao, Hao Dong, and Dimitris Metaxas.  Improving compositional text-to-image generation with large vision-language models.  _arXiv preprint arXiv:2310.06311_ , 2023.
  * Wu et al. [2023a] Chenfei Wu, Shengming Yin, Weizhen Qi, Xiaodong Wang, Zecheng Tang, and Nan Duan.  Visual chatgpt: Talking, drawing and editing with visual foundation models.  _arXiv preprint arXiv:2303.04671_ , 2023a.
  * Wu et al. [2023b] Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji, and Tat-Seng Chua.  Next-gpt: Any-to-any multimodal llm.  _arXiv preprint arXiv:2309.05519_ , 2023b.
  * Xu et al. [2023] Yue Xu, Yong-Lu Li, Zhemin Huang, Michael Xu Liu, Cewu Lu, Yu-Wing Tai, and Chi-Keung Tang.  Egopca: A new framework for egocentric hand-object interaction understanding.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 5273–5284, 2023.
  * Ye et al. [2023] Yufei Ye, Xueting Li, Abhinav Gupta, Shalini De Mello, Stan Birchfield, Jiaming Song, Shubham Tulsiani, and Sifei Liu.  Affordance diffusion: Synthesizing hand-object interactions.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 22479–22489, 2023.
  * Yu et al. [2023a] Qifan Yu, Juncheng Li, Wentao Ye, Siliang Tang, and Yueting Zhuang.  Interactive data synthesis for systematic vision adaptation via llms-aigcs collaboration.  _arXiv preprint arXiv:2305.12799_ , 2023a.
  * Yu et al. [2023b] Zihao Yu, Haoyang Li, Fangcheng Fu, Xupeng Miao, and Bin Cui.  Fisedit: Accelerating text-to-image editing via cache-enabled sparse diffusion inference.  _arXiv preprint arXiv:2305.17423_ , 2023b.
  * Zhang et al. [2023a] Hang Zhang, Xin Li, and Lidong Bing.  Video-llama: An instruction-tuned audio-visual language model for video understanding.  _arXiv preprint arXiv:2306.02858_ , 2023a.
  * Zhang et al. [2017] Mengmi Zhang, Keng Teck Ma, Joo Hwee Lim, Qi Zhao, and Jiashi Feng.  Deep future gaze: Gaze anticipation on egocentric videos using adversarial networks.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 4372–4381, 2017.
  * Zhang et al. [2018] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang.  The unreasonable effectiveness of deep features as a perceptual metric.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 586–595, 2018.
  * Zhang et al. [2022] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al.  Opt: Open pre-trained transformer language models.  _arXiv preprint arXiv:2205.01068_ , 2022.
  * Zhang et al. [2023b] Shu Zhang, Xinyi Yang, Yihao Feng, Can Qin, Chia-Chih Chen, Ning Yu, Zeyuan Chen, Huan Wang, Silvio Savarese, Stefano Ermon, et al.  Hive: Harnessing human feedback for instructional visual editing.  _arXiv preprint arXiv:2303.09618_ , 2023b.
  * Zhang et al. [2023c] Yuxin Zhang, Nisha Huang, Fan Tang, Haibin Huang, Chongyang Ma, Weiming Dong, and Changsheng Xu.  Inversion-based style transfer with diffusion models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 10146–10156, 2023c.
  * Zhang et al. [2023d] Zhixing Zhang, Ligong Han, Arnab Ghosh, Dimitris N Metaxas, and Jian Ren.  Sine: Single image editing with text-to-image diffusion models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 6027–6037, 2023d.
  * Zhu et al. [2023] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny.  Minigpt-4: Enhancing vision-language understanding with advanced large language models.  _arXiv preprint arXiv:2304.10592_ , 2023.

Supplementary Material

This is the supplementary material for the paper titled ”LEGO: Learning
EGOcentric Action Frame Generation via Visual Instruction Tuning”. We organize
the content as follows:

A – Analysis of Image-to-Text Metrics
B – Quality of the Enriched Action Descriptions
C – Additional Experiment Results
C.1 – Performance at Different Transition Time
C.2 – Effect of Dataset Scale
C.3 – Generation of New Actions
C.4 – Additional Visualization
D – More Implementation Details
D.1 – Prompt and Examples for Data Curation
D.2 – Examples of Data Preprocessing
D.3 – Visual Instruction Tuning
D.4 – Egocentric Action Frame Generation
D.5 – User Study of Generated Action Frames
E – Limitation and Future Work
F – Code and Data Release

##  Appendix A Analysis of Image-to-Text Metrics

In Tab. 5, we report the image-to-text CLIP score of InstructPix2Pix (IP2P)
[5] baseline and the ground truth. Ideally, the CLIP score between the ground
truth and the text prompt should serve as a performance uppperbound (UB).
However, the upperbound image-to-text CLIP score is very close to the baseline
on Ego4D, and even lower than the baseline on Epic-Kitchens. It suggests the
CLIP model fails to align action descriptions with corresponding egocentric
images in semantics, thus resulting in a quick saturation in CLIP score. In
our experiments, we use BLIP to caption the generated image and measure the
text-to-text similarity of captions and action descriptions (following [36]).
The two metrics (BLIP-B and BLIP-L) that use two different BLIP structures
both result in larger gap between the baseline model and the upperbound
(3.7%/3.0% vs. 0.9% and 1.6%/1.4% vs.$-$1.1%). Therefore, we adopt BLIP based
metrics and user study to measure image-text alignment. Note that, our model
still performs on-par or slightly better than IP2P in image-to-text CLIP
score, and exceeds IP2P notably when using BLIP based metrics and user study
(see Tab. 2 and Fig. 4 in the main paper).

Methods | Ego4D | Epic-Kitchens
---|---|---
CLIP | BLIP-B | BLIP-L | CLIP | BLIP-B | BLIP-L
IP2P [5] | 0.205 | 0.200 | 0.205 | 0.217 | 0.254 | 0.264
UpperBound (UB) | 0.214 | 0.237 | 0.235 | 0.206 | 0.270 | 0.278
$\Delta=$ UB$-$IP2P | 0.009 | 0.037 | 0.030 | -0.011 | 0.016 | 0.014
Table 5: Image-to-text metrics of the baseline model and uppperbound. The gray
row shows the gap of IP2P to the upperbound. The upperbound measured by CLIP
score is comparable with or even lower than the baseline model (highlighted by
red).

##  Appendix B Quality of the Enriched Action Descriptions

In our approach, we finetune a visual large language model (VLLM) using visual
instruction tuning to generate enriched action descriptions. Here, we provide
additional analysis on the quality of generated action descriptions using user
study. We randomly sample 100 examples from the test set of each dataset. For
each instance, we show the input frame, target frame and the action
descriptions from VLLM. The rater is asked to select whether the description
aligns with the two frames. We hire 5 raters for each instance on Amazon
Mechanical Turk and finally get 500 samples for each dataset. The interface
shown to raters is illustrated in Fig. 6. The percentage of samples with
aligned frames and action descriptions is 87.0% on Ego4D and 84.0% on Epic-
Kitchens. The results suggest the visual instruction tuned VLLM can
effectively expand action labels with details captured in the input frame.

Figure 6: The interface used for evaluation of enriched action descriptions. Both input and target frames are shown to the raters. Figure 7: Comparison with baselines at different action transition time. Our model outperforms all baselines across all transition time. Methods | Ego4D | Epic-Kitchens
---|---|---
CLIP | EgoVLP | EgoVLP+ |  FID $\downarrow$ | PSNR |  LPIPS $\downarrow$ | CLIP | EgoVLP | EgoVLP+ |  FID $\downarrow$ | PSNR |  LPIPS $\downarrow$
LEGO | 0.806 | 0.657 | 0.804 | 23.830 | 12.294 | 0.364 | 0.786 | 0.459 | 0.627 | 21.566 | 11.334 | 0.404
LEGO (scaleup) | 0.809 | 0.663 | 0.808 | 23.560 | 12.301 | 0.363 | 0.789 | 0.475 | 0.635 | 19.635 | 11.400 | 0.399
Table 6: Comparison of LEGO trained with single dataset and both datasets
(denoted as scaleup). $\downarrow$ means a lower score in this metric suggests
a better performance. The better results are highlighted with boldface. The
performance of LEGO model can be effectively improved by involving more
training data.

##  Appendix C Additional Experiment Results

###  C.1 Performance at Different Transition Time

As explained in Sec. 4.1 of main paper, for an action beginning at $t$, we
select the frame at $t-\delta_{i}$ as input and the frame at $t+\delta_{o}$ as
target. We divide the test data into four bins according to the action state
transition time from input frame to target frame
$\delta=\delta_{i}+\delta_{o}$. We establish the threshold for each bin to
ensure that the quantity of data samples in each bin is relatively similar.
The performance of LEGO and baselines at different transition time is
demonstrated in Fig. 7. The flat curves suggest the egocentric action frame
generation problem is equally challenging regardless of transition time. Our
model still surpasses all baselines by a notable margin across all transition
time. This fine-grained result further validates the superiority of our
approach.

###  C.2 Effect of Dataset Scale

We further analyze how the dataset scale affects our model performance by
merging the training set of Ego4D and Epic-Kitchens. The training strategy and
other hyper-parameters remain identical to separate training on each dataset.
We demonstrate the results in Tab. 6. The performance of our model is further
boosted by leveraging more training data (_i.e_. scaleup). Notably, the gains
on Epic-Kitchens are more prominent than gains on Ego4D (_e.g_. 1.6% vs. 0.6%
on EgoVLP score, 1.931 vs. 0.270 on FID, _etc_.). The possible reason is that
Ego4D dataset has more training data covering more diverse scenarios and
actions. Hence, it can greatly compensate for the low diversity of Epic-
Kitchens dataset after merging them. The improvement on two datasets suggests
our model can be effectively boosted by scaling up the training data.

###  C.3 Generation of New Actions

In addition to generating action frames based on the pre-defined action
descriptions in our datasets, we validate whether our approach is able to
synthesize the action frames conditioning on the same contexts yet different
actions (novel image-action pairs). Results are illustrated in Fig. 8. We feed
different actions with the same input frame to our model. The synthesized
frames correctly depict the execution of the actions in the same environment.
This result further indicates that our model can understand the action state
change and generalize to different user queries.

Figure 8: Visualization of generating various actions from the same input
frame. The first action (”open drawer”) is the existing label for this example
in the dataset. We fill another five actions into the user query and
synthesize the action frames using our model. All generated frames align well
with the actions and preserve most contexts provided by the user.

###  C.4 Additional Visualization

We demonstrate more results from our model and the ground truth in Fig. 11.
The generated frames are well aligned with the user query and the ground truth
except some typical failure cases (last two rows). Our approach may fail to
associate the action with the correct objects when the objects are not
distinct enough in the egocentric perspective, _e.g_. the marker and croissant
in the first row of failure cases. In addition, generating the action frame in
some scenarios needs more contexts than a static input frame. For example, the
model fails to understand which object is the furniture and incorrectly drives
the nails into the wood under it (_i.e_. the second failure case of Ego4D). It
also lacks the context that the user already holds a bag of noodles, so it
synthesizes a frame of taking out the noodles from a cupboard (_i.e_. the
second failure case of Epic-Kitchens). These weaknesses can inspire more
future studies in egocentric action frame generation. Please refer to Appendix
E for more discussions.

Figure 9: The structure of the prompt sent to GPT-3.5. We specify the
composition of the input query (highlighted in blue). Then we articulate the
requirements for action enrichment (highlighted in green) and extra demands
(highlighted in yellow). Example-1 to Example-12 consist of the action label,
object bounding boxes and manual annotation of detailed action descriptions.
The inference example consists of just action label and object bounding boxes.

##  Appendix D More Implementation Details

###  D.1 Prompt and Examples for Data Curation

In data curation, we randomly select 12 examples from each datasets covering
diverse scenarios. All examples are shown in Fig. 13 and Fig. 14. We also
clarify our requirements in the prompt sent to GPT-3.5. The complete prompt is
shown in Fig. 9. We specify the composition of input query, the expected
detailed information and extra requirements for the output in the system
information. Then we input the examples for in-context learning and the
example for inference in the user input.

###  D.2 Examples of Data Preprocessing

For each input frame or target frame, we calculate aesthetic scores [29] of
the current frame as well as 3 frames before and after (7 frames in total). As
demonstrated in Fig. 12(a), the frame of the highest aesthetic score usually
has the best image quality. We then use this frame as input or target frame.
We also calculate the similarity of input and target frame for each action. We
empirically filter out data whose similarity is lower than 0.81 or higher than
0.97. Some examples of abandoned data are shown in Fig. 12(b). A very low
similarity usually indicates a big change in the background due to the head
motion. A very high similarity implies the action involves very small hand
movements. It’s hard for generative models to learn action generation in these
data samples.

###  D.3 Visual Instruction Tuning

We train the model with a batch size of 128 and a learning rate of $2\times
10^{-5}$. Warm-up strategy and consine anneal [56] are also used in training.
It takes 24 hours to train the model on 8 NVIDIA A100-SXM4-40GB for 3 epochs.
Please refer to Fig. 8 for the example of incorporating action label into
action instruction query templates.

###  D.4 Egocentric Action Frame Generation

We finetune the latent diffusion model with a batch size of 256 and an initial
learning rate of $10^{-4}$ without warm-up strategy. Horizontal flipping is
used as data augmentation during training. We train the model for 20,000
iterations on 8 NVIDIA A100-SXM4-40GB over 38 hours. In inference, we apply
100 denoising steps for each instance.

We also use classifier-free guidance for two conditions (following [5]) by
sharing the same guidance scale across the enriched action descriptions, VLLM
image embeddings and VLLM text embeddings. As defined in Sec. 3.2 in main
paper, we use $\mathcal{C}$ to denote the three conditions and use
$\mathcal{X}$ to denote the input frame. Specifically, we randomly set only
the image conditioning $\mathcal{X}=\varnothing$ at a probability of 5%, only
the conditioning from VLLM $\mathcal{C}=\varnothing$ at a probability of 5%
and both $\mathcal{X}=\varnothing$ and $\mathcal{C}=\varnothing$ at a
probability of 5%. Then the score estimate of our model is

$\displaystyle\tilde{e}_{\theta}(z_{t},\mathcal{X},\mathcal{C})$
$\displaystyle=e_{\theta}(z_{t},\varnothing,\varnothing)$ (3)
$\displaystyle+s_{x}\cdot(e_{\theta}(z_{t},\mathcal{X},\varnothing)-e_{\theta}(z_{t},\varnothing,\varnothing))$
(4)
$\displaystyle+s_{c}\cdot(e_{\theta}(z_{t},\mathcal{X},\mathcal{C})-e_{\theta}(z_{t},\mathcal{X},\varnothing)),$
(5)

where $\theta$ refers to the parameters in the denoising UNet. $z_{t}$ is the
noisy latent at timestep $t$, which is obtained by diffusion process in
training and randomly initialized by a gaussian noise in inference. $s_{x}$
and $s_{c}$ are the guidance scales corresponding to the conditioning
$\mathcal{X}$ and $\mathcal{C}$ respectively. In inference, we use $s_{x}=7.5$
and $s_{c}=1.5$ which are identical to the settings in InstructPix2Pix [5].

###  D.5 User Study of Generated Action Frames

Figure 10: The interface used for evaluation of generated action frames. The
four generated frames are randomly shuffled to avoid potential bias.

The user interface for evaluation of generated action frames is illustrated in
Fig. 10. We show the input frame and shuffled outputs from four models to
raters. To make a fair comparison, we show the action label instead of the
enriched action description because the baseline models only take original
action labels as input.

##  Appendix E Limitation and Future Work

In this paper, we use an egocentric image to capture the user’s environment
contexts and generate the action frame to provide visual instructions.
However, there are still some problems that are not explicitly solved by our
method. We find it’s hard for our model to associate the action with correct
objects when there are too many irrelevant objects around. Synthesizing the
diverse and complex hand-object interactions is also a big challenge
especially when people are operating some machines. In addition, our work
indicates a few valuable directions for the future study.

  * •

The embeddings from visual large language model (VLLM) are fed into the UNet
together with the CLIP based text representation. How to leverage the VLLM
embeddings more effectively in diffusion models deserves future study.

  * •

Recognizing and localizing the objects that are relevant with the action
descriptions in a chaotic environment may be a bottleneck for the application
in real-world problems, which deserves more attention.

  * •

It’s difficult to synthesize correct interactions of hands and objects in some
professional work, such as using a wood cutter, operating a lawn mower and
sewing clothes on a sewing machine. How to combine affordance understanding
with generative models may be a key step to address this problem.

  * •

Existing automatic image-to-text similarity metric doesn’t generalize well to
the egocentric domain. We expect more investigation of better evaluation
metrics for image-text alignment.

##  Appendix F Code and Data Release

The code, pre-trained model weights, additional data annotations and our
train/test split will be made publicly available to the research community to
facilitate future studies.

Figure 11: Additional visualization of LEGO output on Ego4D (on the left of
the dash line) and Epic-Kitchens (on the right of the dash line). We also show
the ground truth for reference. In the last two rows, we demonstrate some
failure cases on the two datasets. Figure 12: Data preprocessing in our work.
(a) The fame with the highest aesthetic score (highlighted) is less blurry and
then used as the target frame of this action. (b) The actions with too low
($<$0.81) or too high similarity ($>$0.97) between input and target frames are
filtered out from the datasets. Figure 13: All Ego4D examples used for data
curation from GPT-3.5 via in-context learning. For simplicity, the bounding
boxes are only shown on images. We input the coordinates of bounding boxes to
GPT-3.5 in practice. Figure 14: All Epic-Kitchens examples used for data
curation from GPT-3.5 via in-context learning. For simplicity, the bounding
boxes are only shown on images. We input the coordinates of bounding boxes to
GPT-3.5 in practice.