# Unsupervised Learning of Graph from Recipes

Aïssatou Diallo, Antonis Bikakis, Luke Dickens, Rob Miller, Anthony Hunter
University College London, United Kingdom
Corresponding author<EMAIL_ADDRESS>

###### Abstract

Cooking recipes are one of the most readily available kinds of procedural
text. They consist of natural language instructions that can be challenging to
interpret. In this paper, we propose a model to identify relevant information
from recipes and generate a graph to represent the sequence of actions in the
recipe. In contrast with other approaches, we use an unsupervised approach. We
iteratively learn the graph structure and the parameters of a $\mathsf{GNN}$
encoding the texts (text-to-graph) one sequence at a time while providing the
supervision by decoding the graph into text (graph-to-text) and comparing the
generated text to the input. We evaluate the approach by comparing the
identified entities with annotated datasets, comparing the difference between
the input and output texts, and comparing our generated graphs with those
generated by state of the art methods.

##  1 Introduction

Procedural texts are a common type of natural language writing that provides
instructions on following a procedure. They are essential in everyday life as
they enable us to learn skills, complete tasks, and solve problems. Generally,
they are loosely structured in step-by-step format with temporal order (as
they would occur in real life) and logical flow (the results of previous steps
feed into later steps). A procedural text can be used to guide an intelligent
system for a wide range of applications, such as automated robotics, but the
representation must be in a form that a computer-based system can interpret
Papadopoulos et al. (2022). Given this, these representations must ideally
capture all relevant entities and actions from the original text, while
encoding all relationships that exist between them. A graph is a natural
choice for this purpose as it represents not only the attributes of the
entities but also their relationships. Moreover, being able to represent
procedural text within a graph structure can facilitate reasoning and
inference, such as adapting procedures when resources are unavailable or
actions are impossible, see Bikakis et al. (2023).

More precisely, procedural text understanding implies correctly identifying
the resources and actions effects on resources, as well as understanding the
flow between the procedures. For example, given the recipe "combine flour,
sugar and milk in a bowl. Then, add the chocolate chips to the mixture.
Finally, bake the cookies at 250 degrees.", an intelligent system must be able
to: (i) correctly identify entities and their types including implicit
entities, e.g. "mixture" as a combination of previously mentioned ingredients;
(ii) understand the temporal flow of the instructions, e.g. one must combine
the first three ingredients into a batter before adding the chocolate chips;
(iii) recognize the logical flow of information, e.g. the "bake" action acts
on the output of the"combine" and "add" actions. This presents a challenge not
present in factoid based reading comprehension Clark and Gardner (2018), in
that our putative procedural text interpreter must model the dynamics and
causal effects not explicitly mentioned in the text at hand Dalvi et al.
(2018); Bosselut et al. (2018).

Figure 1: Example of output graph with the associated recipe. The identified
entities are in bold.

This paper presents a model, trained in a self-supervised fashion to encode
cooking recipes as graphs, that identifies key actions, entities, including
those that are implicit, and their relationships. We conduct a series of
experiments showing that our proposed method yields meaningful graphs which
can be used to reason about the original recipe. Our key contributions can be
summarized as follows: (i) proposing a scalable approach to transform
procedural texts into graphs while learning discrete and sparse dependencies
among the entities using as a case study the cooking domain, and (ii)
presenting a new method for jointly learning the edge connectivity of a graph
in absence of explicit supervision at the graph-level.

##  2 Background

In this section, we introduce some preliminary notions on graph theory and
graph neural networks (GNNs).

#### Graph Theory Basics

A (heterogeneous) graph $\mathcal{G}$ can be described by the tuple
$\mathcal{G}=\\{\mathcal{V},\mathcal{E}\\}$, where
$\mathcal{V}=\\{v_{1},...,v_{N}\\}$ is a set of $N$ nodes and
$\mathcal{E}=\\{e_{1},...,e_{M}\\}$ is a set of $M$ edges. In the case of
heterogeneous graphs, there is a node type mapping function
$\phi:\mathcal{V}\rightarrow\mathcal{C}$. For a given graph $\mathcal{G}$, the
corresponding adjacency matrix $\mathbf{A}\in\\{0,1\\}^{N\times N}$ describes
the connectivity between the nodes. The connected nodes are called neighbour
nodes. $\mathbf{A}_{i,j}=1$ indicates that $v_{i}$ is adjacent to $v_{j}$,
otherwise $\mathbf{A}_{i,j}=0$. Following the notation of Franceschi et al.
(2019), we denote the set of all $N\times N$ adjacency matrices as
$\mathcal{H}_{N}$.

#### Graph Convolutional Networks

GNNs are machine learning models designed for reasoning with graph-structure
data while modelling relational and structural information by learning
neighbour-aware node representations. In this study, we use Graph
Convolutional Network ($\mathsf{GCN}$) Kipf and Welling (2017) to aggregate
the neighbour information for each entity node.

$\mathbf{h}^{l+1}_{i}=\sigma\left(\mathbf{W}\sum_{j\in
N(i)\cup\\{i\\}}\frac{1}{\sqrt{d(i)d(j)}}\mathbf{h}^{l}_{i}\right)$ (1)

Eq. 1 describes the propagation mechanism for the $\mathsf{GCN}$. At layer
$l$, each node $i$ aggregates the representation of itself and all its
neighbouring nodes $N(i)$ based on $\mathbf{A}$. Then, the updated
representation $\mathbf{h}^{l+1}_{i}$ is computed based on the weight matrix
of the layer $\mathbf{W}$, the activation function $\sigma$, normalized by the
degree of the source $d(i)$ and its connected nodes $d(j)$.

##  3 Model

Figure 2: Overview of the proposed model. The first part, in blue, encodes the
procedures and identifies the relevant entities (refer to Figure 3 for more
details). These are passed to the second part (in red) containing the Graph
Structure Encoder module, which learns the adjacency matrix and the graph
encoding (with cost $L_{gse}$) (refer to Figure 4) and the decoder that back-
translate the graph into a recipe (with cost $L_{gen}$).

In this paper, we address the challenge of producing graphs of cooking recipes
when no supervision in terms of graphs is available. We aim to discover
implicit entities and relationships between actions, ingredients and
locations, and for the relationships to carry a similar meaning, e.g. an
ingredient relates to an action that acts upon it without explicitly enforcing
them so that the output is loosely aligned with the semantics of flow-graphs
such as Mori et al. (2014b).

#### Task Definition

A recipe $R$ consists of $T$ sentences $R=\\{s_{0},s_{1},...,s_{T}\\}$. Each
sentence $s_{i}$ contains the action $a_{i}$ that operates on $i_{i}$, a set
of ingredients in (a set of) $l_{i}$ locations. All cooking actions,
ingredients and locations are part of a pre-specified set of entities. Often,
a cooking action can lead to an intermediate output such as batter, dough or
mixture. In the scope of this work, these intermediate outputs are considered
ingredients. The model aims to learn a graph describing the interactions
between actions, ingredients and locations for a given recipe. These entities
are represented as nodes in the output graph and the edges are the connections
between the $|\mathcal{C}|=3$ different types of nodes, namely actions,
ingredients and locations. In the absence of a dataset containing pairs of
recipes and their corresponding cooking graphs, we tackle this problem in an
unsupervised way.

#### Overview

For doing this, given a recipe we first parse it such that each instruction
contains only one cooking action (e.g., "sprinkle with salt // mix well // set
aside in a bowl"). Then, a previously trained model is applied to each
instruction to encode the sentence and to identify the relevant entities that
should be present in the cooking graph. Next, these elements are sequentially
passed to a graph encoder that jointly learns the graph structure as an
adjacency matrix, the node and graph representations. The resulting graph
encoding is used as a conditioning for a transformer-based module that
translates it back to textual cooking instructions. Finally, the output text
is compared to the original set of cooking instructions in a cyclic way to
guide the learning process.

###  3.1 Entity Identifier

Given a sentence $s_{t}$ with $L_{s}$ tokens such that
$s_{t}=\\{w_{i,1},...,w_{i,L_{s}\\}}$, we use $\mathsf{BERT}$ Devlin et al.
(2019) as a context encoder, which is based on a stack of multilayer
transformer blocks Vaswani et al. (2017). The instruction representation
$\mathbf{h}_{s_{t}}$ is obtained by extracting the hidden state
$\mathbf{h}^{s_{t}}_{0}$ corresponding to the special token $[\texttt{CLS}]$
so that $\mathbf{h}_{s_{t}}=\mathbf{h}^{s_{t}}_{[\texttt{CLS}]}$:

$\\{\mathbf{h}^{s_{t}}_{[\texttt{CLS}]},...,\mathbf{h}^{s_{t}}_{L_{s}}\\}=\texttt{BERT}\\{[\texttt{CLS}],...,w^{s_{t}}_{L_{s}}\\}$
(2)

The first block of our model takes as input one sentence at a time and
extracts which actions are executed on the ingredients while tracking the
changes in location and the transformation of the ingredients. Existing
procedural text understanding models rely on some type of name entity
recognition sub-task to obtain the entities the model should focus on such as
in Tang et al. (2022). This strategy would be sufficient if entities were
always explicitly mentioned, but natural language often elides these arguments
or uses pronouns. As such, the module must be able to consider entities
mentioned previously or under different names.

Figure 3: Details of the Entity Identifier. The recipe is parsed such that
each sentence contains only one action whereas multiple ingredients and
locations are permitted.

This problem can be seen as a multitask learning problem where we need to
learn to identify actions, ingredients and locations simultaneously.
Specifically, a feed-forward network (FFN) with a cross entropy loss is used
to identify the correct cooking action. Concurrently, two FFNs with a binary
cross entropy cost function are used to learn the subsets of ingredients and
locations for a given instruction $s_{t}$. Figure 3 illustrates this component
of our approach.

###  3.2 Graph Structure Encoder

We aim to learn the adjacency matrix and the graph representation of a graph
$G$ associated with a recipe $R$ and a set of (heterogeneous) entities. Our
encoder takes as input one instruction at a time until the end of the recipe.
It recursively generates the graph for the block of the cooking recipe seen up
to the $t$-th generation step. The output is conditioned on the already
generated graph representation at time step $t-1$. The process is divided into
two main parts: (1) generating the structure of the graph, described by the
adjacency matrix; (2) learning the parameters for the $\mathsf{GCN}$ as well
as obtaining the final graph representation. There are two main approaches for
generating the adjacency matrix. The first is to think of the problem as a
probabilistic one and to learn the parameters of a probability distribution
over the graphs as in Franceschi et al. (2019). The second method is to work
with a continuous relaxation of the adjacency matrix. We chose the latter
approach in this work.

We recall that the node features of the entities are not given and need to be
learned along the graph structure. For doing this, we initialize the feature
matrix $\mathbf{X}$ containing the node representations with a pre-trained
model Pellegrini et al. (2021) specific to the cooking domain, acting as node
encoder. More specifically, given a recipe with $T$ instructions each with an
action $a_{i}$, $i_{i}$ ingredients and $l_{i}$ locations with
$i\in\\{0,...,T\\}$, we first concatenate the indices denoting the entities
into a single list of indices. We impose a specific ordering for the
concatenation: [action; ingredients; locations] and within each subset, the
items are sorted in alphabetical order. This yield $\mathbf{X}^{(t)}$ for the
$t$-th sentence in the recipe with the initialized embedding layer described
earlier. We now explain the details of the graph generation step below.

#### Relation matrix

We extract a relation matrix $\mathbf{R}^{(t)}$ characterizing the probability
of a relation (an edge) between two nodes based on $\mathbf{X}^{(t)}$. This is
meant to capture the relationship between the nodes and propagate this
relation to the topological structure of the graph. We first apply two
different mappings to $\mathbf{X}^{(t)}$ to obtain a
$\mathbf{Z}^{(t)}=\textsf{FFN}(\mathbf{X}^{(t)})$ and
$\mathbf{M}^{(t)}=\textsf{FFN}(\mathbf{X}^{(t)})$. Finally, the relation
matrix is obtained as
$\mathbf{R}^{(t)}=\mathbf{Z}^{(t)}\mathbf{M}^{(t)^{\top}}$.

#### Adjacency Processor

The relation matrix $\mathbf{R}^{(t)}$ is not suited to be used as an
adjacency matrix, as it may contain both positive and negative values and it
is non-normalized. Let us define a function $\mathcal{S}$ that takes as input
the relation matrix $\mathbf{R}^{(t)}$ and outputs a continuous relaxation of
the adjacency matrix $\mathbf{A}^{(t)}$. $\mathcal{S}$ will take a matrix and
transform it into a low entropy one, meaning that most of its elements will be
equal to zero, hence augmenting its sparsity. A simple algorithm for this
purpose is the Sinkhorn-Knopp algorithm Knopp and Sinkhorn (1967), which
consists in iteratively rescaling the rows and columns of a square matrix with
positive entries. By applying the algorithm in the log domain, we can apply
the algorithm to the relation matrix and obtain an adjacency matrix by finally
applying the $\exp$ operator to the output for getting positive values. To sum
up, $\mathbf{A}^{(t)}=\mathcal{S}(\mathbf{R}^{(t)})$111For the sake of
brevity, we slightly abuse the notation because the
$\mathcal{S}(\mathbf{R}^{(t)})$ is not a $(0,1)$-matrix but a continuous
relaxation..

#### Node Classification

We choose a two-layer $\mathsf{GCN}$ Kipf and Welling (2017) that takes as
input the feature matrix $\mathbf{X}^{(t)}$ and the adjacency matrix
$\mathbf{A}^{(t)}$. The first layer of the GCN architecture projects the
inputs into an intermediate latent space and the second layer maps it to the
output embedding space. Due to the absence of labelled graphs to guide the
learning process, we include a pretext task, which is a pre-designed task for
the network to solve such that it can be trained by the learning objective
functions of the pretext task and the features and parameters are learned
through this process. A natural choice for the pretext task is node
classification and the goal is to predict the entity type of each node by
providing the logits for each class through $\mathrm{softmax}$ in this way
$\mathbf{\hat{y}}=\mathrm{softmax}(\mathsf{GCN}(\mathbf{X},\mathbf{A}))$.

$\displaystyle\mathcal{L}_{gse}=\ell(\mathbf{\hat{y}},\mathbf{y})$ (3)

where $\ell$ corresponds to a cross-entropy loss function, $\mathbf{y}$ are
the ground-truth labels known before-hand and $\mathbf{\hat{y}}$ are the
predicted labels. Eq. 3 will constitute the first term of the global loss.

1

Input : Trained model,

$R=\\{s_{0},...,s_{T}\\}$: the input recipe;

Output : $\mathbf{A}$: adjacency matrix;

$Y$: nodes;

$Y_{\mathcal{C}}$: nodes classes;

2 $t\leftarrow 0$

3 $graphSequence=[\ ]$

4for  _$s_{t}$ in $R$_ do

5 $\mathbf{X}^{(t)}$, ${Y}^{(t)}\leftarrow$ Entity Identifier$(s_{t})$

6 $\mathbf{Z}^{(t)}$, $\mathbf{M}^{(t)}\leftarrow$
$W_{1}\mathbf{X}^{(t)}+b_{1}$, $W_{2}\mathbf{X}^{(t)}+b_{2}$

7 $\mathbf{R}^{(t)}\leftarrow\mathbf{Z}^{(t)}\mathbf{M}^{(t)}\top$

8 $\mathbf{A}^{(t)}\leftarrow$ Adjacency Processor$(\mathbf{R}^{(t)})$

9 $\tilde{\mathbf{A}}^{(t)}\leftarrow$ Normalization of $\mathbf{A}^{(t)}$

10
$\tilde{\mathbf{G}}^{(t)}\leftarrow\mathsf{GCN}(\mathbf{X}^{(t)},\tilde{\mathbf{A}}^{(t)})$

11 $\tilde{\mathbf{g}}^{(t)}\leftarrow$ Graph pooling
$(\tilde{\mathbf{G}}^{(t)})$

12 ${graphSequence}.append(\tilde{\mathbf{g}}^{(t)})$

13

14 end for

15$\mathbf{G}\leftarrow\mathsf{GRU}(graphSequence)$

return _$\mathbf{G}$ , $\mathbf{A}^{(T)}$, ${Y}^{(T)}$,
${Y}^{(T)}_{\mathcal{C}}$_

Algorithm 1 Graph Structure Encoder

#### Recurrent Graph Embedding

We apply the steps detailed above iteratively until reaching the end of the
recipe. At each time step, the node vectors are the concatenation of the
unique node indices seen up to the $t$-th sentence of the recipe such that the
last iteration, the adjacency matrix $\mathbf{A}^{(t=T)}$ has $m$ unique nodes
with $m=a_{T}+i_{T}+l_{T}$. Each graph encoding $\mathbf{H}^{(t)}$ goes
through a mean pooling function in order to obtain a global representation as
a single graph embedding. These vectors are then stacked into a matrix
$\mathbf{\tilde{G}}^{R}\in\mathbb{R}^{T\times d}$. We concatenate
$\mathbf{\tilde{G}}^{R}$ to the sentence representations from eq. 2 to obtain
$\mathbf{{G}}^{R}\in\mathbb{R}^{T\times 2d}$.

The final sequence of graph representations is obtained using a two-layer
bidirectional $\mathsf{GRU}$ to update the input sequence as seen in Figure 4.
Algorithm 1 describes the graph generation process. Finally, this sequence is
used to condition the transformer-based decoder.

Figure 4: We exploit the ability of $\mathsf{GRU}$ at handling temporal
dependencies to increase the expressive power of the iteratively constructed
partial graphs. The $\mathsf{GRU}$ has two inputs: the current input (the
(partial) graph at $t$ concatenated with $\mathbf{h}_{s_{t}}$) and the
previous state (output of a $\mathsf{GRU}$ at $t-1$).

###  3.3 Cooking Instruction Decoder

Given a graph sequence representing a recipe with $T$ steps, the goal of the
cooking instruction decoder is to produce a sequence of textual instructions
$\hat{R}=\\{\hat{s}_{0},\hat{s}_{1},...,\hat{s}_{T}\\}$ by means of a
transformer-based architecture. This decoder is conditioned on the sequence of
graph embeddings provided by the graph encoder described earlier.

The instruction decoder is composed of transformer blocks each of them
containing a self-attention layer applying self-attention over previously
generated outputs, a cross-attention layer that attends to the model
conditioning in order to refine the self-attention output and a linear layer.
This process is the same as the original Transformer Vaswani et al. (2017) and
we omit it due to the limited space. A final $\mathrm{softmax}$ provides a
distribution over the words for each time step. We use the cross-entropy loss
$\mathcal{L}_{gen}$ between the generated instructions $\hat{s}_{t}$ and the
original instructions ${s}_{t}$.

###  3.4 Optimization

We train our model, outlined in Figure 2 in two stages. In the first phase, we
train the entity identifier described in section 3.1. Then, in the second
stage, we train the graph structure encoder from section 3.2 and the cooking
instruction decoder from section 3.3. The total loss used to perform the
learning is
$\mathcal{L}_{tot}=\mathcal{L}_{gse}+\mathcal{L}_{gen}+\lambda\|A\|_{1}$. The
last term is a regularization term that encourages the sparsity of the
adjacency matrix. We apply a bi-level optimization approach by alternatively
updating the weights involved in the learning of the adjacency matrix and the
weights for the $\mathsf{GNN}$. The cooking instruction decoder is trained
with teacher forcing Williams and Zipser (1989).

##  4 Experiments

Model | F1 | Recall
---|---|---
2-layer LSTM Entity Recognizer | 50.98 | 13.33
Adapted Gated Recurrent Entity | 45.94 | 7.74
EntNet | 48.57 | 9.87
Neural Process Networks | 55.39 | 20.45
Ours | 65.14 | 63.44

Table 1: Results for entity selection. The reported scores are averaged for
all three types of entities.

| NYC | EFC
---|---|---
| BLEU | ROUGEL | BLEU | ROUGEL
NPN | 0.37 | 0.35 | - | -
Ours w/o s. | 0.42 | 0.41 | 0.44 | 0.39
Ours | 0.48 | 0.45 | 0.49 | 0.38

Table 2: Text generation results for text $\rightarrow$ graph and graph
$\rightarrow$ text.

For learning and evaluation, we use the "Now You’re Cooking" (NYC) dataset
used in Bosselut et al. (2018) that contains 65816 recipes for training, 175
recipes for validation and 700 recipes for testing. This dataset contains
annotations for the ingredients, actions and locations for each sentence in a
given recipe. For evaluating the quality of the graphs produced, we rely on
the dataset "English Flow Corpus" (EFC) by Yamakata et al. (2020).

#### Implementation details

We use the following settings. We train the entity identifier for 5 epochs
with a learning rate of $2\cdot 10^{-5}$. The nodes’ hidden dimension is
$100$. The 2-layer $\mathsf{GCN}$ outputs a graph representation of hidden
dimension $768$. The graph structure encoder is trained with a learning rate
of $10^{-3}$ and the instruction decoder has a base learning rate of $10^{-4}$
and a step decay of $0.1$ every 5 epochs. We train for 10 epochs. All
experiments are conducted on a Nvidia Titan X GPU.

###  4.1 Entity Selection

In this section, we evaluate the model performance in identifying the correct
entities and modelling their interactions. This is framed as a classification
task. Following Bosselut et al. (2018), we rely on the macro F1 score and the
recall to measure the performance of the model. The Table 2 shows the results.
A selected entity is denoted as one whose sigmoid activation is greater than
0.5. The reported scores are averaged for all three types of entities.

#### Baselines

The main baseline for this stage of the framework consists of the Neural
Process Networks by Bosselut et al. (2018). For the sake of completeness, we
report the scores of their baseline to better position our model. The first
two baselines, represented by the first two lines of Table 2 consist of an
$\mathsf{LSTM}$ and $\mathsf{GRU}$ model trained to predict the entities
without any other external information. This is closer to our model. Adapted
Recurrent Entity Network Henaff et al. (2017) and Bosselut et al. (2018) are
both $\mathsf{RNN}$-based models. The first one is a network doted with
external “memory chains” with a delayed memory update mechanism to track
entities. The second one aims to simulate the effects of actions on entities
in procedural text. It embeds input sentences in a $\mathsf{GRU}$, then uses a
$\mathsf{FFN}$ to predict actions which occur in each sentence and sentence-
level and recurrent attention mechanisms to identify entities affected by
actions.

| Graph-edit distance
---|---
RandomGraph | 112.1
Ours w/o sent | 71.4
Ours | 67.1
Table 3: Graph edit distance. Due to the high computational cost for graph
matching, we report the result for 10 sample pairs of generated graphs and
ground truths.

#### Results

From the results, we can observe that a transformer-based classifier
outperforms all the baselines. However, it should be noted that in the
ingredient type entity selection, the model sometimes fails. Given the
sentence add sugar and remaining broth, the model selects chicken broth and
sugar as ingredients while ignoring most of the other ingredients from the
ground truth-set. This is probably due to the fact that there is a need to
inject external information or explicitly track the ingredients that have been
used and consumed throughout the recipe because the sole information contained
in the sentence is not sufficient to infer a mapping to previous ingredients.
The situation slightly improves, when including intermediates "ingredients" as
an entity that should be tracked. The model is not required to link the
construct dough to its single components but can continue to reason about
actions and locations related to this food item.

Reference text |  | add ingredients in the order suggested by your manufacturer.
---
set for dough setting and start the machine. when the unit signals remove
dough.
pat dough into a greased 30cm ( 12 in ) round pizza pan. let stand 10 minutes.
preheat oven to 200 c / gas mark 6. spread pizza sauce over dough.
sprinkle toppings over sauce ( such as basil and mozzarella for a simple
margherita pizza ).
bake 15 to 20 minutes, or until crust is golden brown.
Generated text |  | add the ingredients in the order listed.
---
set machine on, the dough cycle is best.
form the dough and pat into 12 - inch round pizza pan.
let stand 15 mins. preheat oven to 350f 200c. spread pizza sauce over dough.
sprinkle cheeses, then tomato sauce.
bake 10 to 15 minutes or until crust is golden brown

Table 4: Example of generated text from the graph representation of a recipe
for making pizza.

|  | Predicted | Ground truth
---|---|---|---
$s_{t-1}$ |  if you use canned chickpeas, reduce or omit the salt. |  garlic, yogurt, chili |  chili powder, curry powder, garlic, ginger, parsley, salt , yogurt
$s_{t}$ |  mix yogurt , garlic and spices.
$s_{t-1}$ |  heat the oil. |  beef |  cornstarch, garlic clove, oil, oyster sauce, pepper, sherry, soy sauce, sugar
$s_{t}$ |  stir-fry beef quickly, until the meat is medium rare.
$s_{t-1}$ |  dip bottom of balloon in melted chocolate and sit on cookie sheet. |  chocolate |  chocolate, raspberry
$s_{t}$ |  refrigerate until hard.
$s_{t-1}$ |  sift flour in a bowl. |  cornstarch, sugar |  cornstarch, sugar
$s_{t}$ |  combine the sugar and cornstarch in a saucepan.

Table 5: Examples of selected ingredients. We provide the sentence $s_{t}$ and
sentence $s_{t-1}$ for context. Ingredients are displayed in green if they are
present in the ground truth set of ingredients and red otherwise. Best viewed
in color.

###  4.2 Graph Generation

In this section, we test the graph generation performance of our model. The
goal is to validate the hypothesis that our proposed model is capable of
generating meaningful graphs. For doing this, we compare against two different
baselines and analyze the graphs obtained against the ground-truth graph-flow
from Yamakata et al. (2020).

#### Baselines

The lack of large datasets with recipes and graph pairs is the main reason
that motivates this study. In order to assess the validity of our method, we
evaluate two aspects of our method. The first one is the quality of the
generated text. This allows us to indirectly estimate whether the graph
encoder is capable of extracting meaningful relations. We report the scores of
this indirect evaluation using benchmark metrics for text generation.
Additionally, we want to evaluate the quality of the graphs. For this, we use
the dataset by Yamakata et al. (2020). We apply our model to the textual
recipes and evaluate the text generation performance and the output graph in
terms of graph edit distance (GED) Sanfeliu and Fu (1983). For a novel line of
work, the RandomGraph, a random graph structure, is an intuitive baseline that
gives us insights into our approach. We include as a comparison the
performance of our model without the concatenation of the sentence
representations but only the node representation learned by the graph encoder.

#### Metrics

For the graph $\rightarrow$ text task, we use BLEU Papineni et al. (2002) and
ROUGEL Lin (2004) to evaluate the closeness of our generated text to the input
recipe. These metrics measure the accuracy of the n-grams between the
predicted text and the ground truth. For evaluating the performance of the
text $\rightarrow$ graph task, we measure the GED between our predicted graphs
and the ground truth. This score is to find the best set of transformations
that can transform graph $g_{1}$ into graph $g_{2}$ by means of edit
operations on graph $g_{1}$. The allowed operations are inserting, deleting
and/or substituting vertices and their corresponding edges:

$\mathsf{GED}(g_{1},g_{2})=\min_{e_{1},...,e_{k}\in\gamma(g_{1},g_{2})}\sum_{i=1}^{k}c(e_{i})\vspace{-0.15cm}$
(4)

where $c$ is the cost function for an editing operation $e_{i}$ and
$\gamma(g_{1},g_{2})$ identifies the set of edit paths transforming $g_{1}$
into $g_{2}$.

#### Results

As shown in Table 2, our model achieves a good performance for the task of
unsupervised graph generation. The recipes generated (an example is given in
Table 4) are coherent and follow the flow of the input recipes. Moreover, the
analysis of some examples shows that in most cases the original ingredients
are recognized by the model and placed in the correct context. Considering the
graph generation, Table 3 shows the results of the GED score. We compare
against the ground-truth flow graphs from the "EFC" corpus. It is important to
clarify that these graphs are structurally different from ours. They are
generated from a pre-processing task involving tagging each word of the recipe
and predicting the edges between these nodes. Before the evaluation, we
aggregate the duplicate nodes and discard input recipes with less than two
steps. We then compute the $\mathsf{GED}$ by sampling 10 graphs and averaging
the scores. This compromise is due to the high computational cost of the
graph-matching problem. We compare the graphs obtained by our model to the
processed ground truth and additionally compare the $\mathsf{GED}$ score to
RandomGraph. We can notice how the complete model including the sentence
representation has a lower $\mathsf{GED}$ from the ground-truth set.

##  5 Related Work

#### Graph Generation

Graph generation is the task of modeling and generating real-world graphs.
Early works generated graphs with structural assumptions defined a priori
Albert and Barabási (2002); Leskovec et al. (2010) although in many domains,
the network properties and generation principles are largely unknown.
Motivated by the need to improve fidelity, researchers studied methods to
learn generative models from observed graphs. Recent works in graph generation
exploit deep generative models such as variational autoencoders Kingma and
Welling (2013) and generative adversarial modelsGoodfellow et al. (2014).
There are different approaches for generating graphs and they differ based on
the elements of the graph they generate at each step (or in one shot): (i)
node Su et al. (2019); Assouel et al. (2018); Liu et al. (2018); Kearnes et
al. (2019), edge Goyal et al. (2020); Bacciu et al. (2020, 2019) and motif
sequence Liao et al. (2019); Podda et al. (2020), respectively generate node,
edge or sequence of smaller graphs at each time step; (ii) adjacency matrix
based Ma et al. (2018); Pölsterl and Wachinger (2019); De Cao and Kipf (2018),
that directly learns a mapping to the latent embedding to the output graph in
terms of an adjacency matrix, generally with the addition of node/edge
attribute matrices/tensors; and (iii) methods generate the edge probability
based on pairwise relationships between the given nodes embedding Grover et
al. (2019); Kipf and Welling (2016); Shi et al. (2020) which is close to our
method. The intuition behind these models is that nodes that are close in the
embedding space should have a high probability of being connected. However, we
do not follow the standard method of enforcing semantic similarity between
nodes as a mean for generating the adjacency matrix as the relationship we
want to emerge from the graph is of the type action $\rightarrow$ ingredient
and location $\rightarrow$ action.

#### Text to graph

The literature on text-to-graph generation is predominantly based of knowledge
graphs generation methods Gardent et al. (2017); Flanigan et al. (2016);
Konstas and Lapata (2013); Li et al. (2021) although other works generate
flows graphs similar to ours. One specificity of our approach is that due to
the generation methods, we do not allow for nodes to repeat themselves which
is not the case for flow graphs. Additionally, our work relates to procedural
knowledge extraction task Qian et al. (2020); Hanga et al. (2020); Honkisz et
al. (2018) which is fundamental for transforming natural texts into structured
data like graphs. Guo et al. (2020) proposed an unsupervised training method
that can iteratively back-translate between the text and graph data in a
cyclic manner, such that the problem becomes similar to finding an alignment
between unpaired graphs and related texts. Guo et al. (2020). The fundamental
difference is that we do not have access to graph-structure data.

#### Food understanding

Large datasets focusing on the cooking domain have gathered a new interest
leading to advancements in food understanding. Most notable examples of these
datasets are Food-101Bossard et al. (2014) and Recipe1M Salvador et al.
(2017). Unlike our method, these benchmark datasets have been mostly exploited
as reference benchmarks for the computer vision literature. The text-based
studies in natural language processing have focused on tasks such as recipe
generation in the context of procedural text understanding in the form of
flow-graphs Hammond (1986); Mori et al. (2014a, b) and more generally recipe
parsing Chang et al. (2018); Jermsurawong and Habash (2015); Kiddon et al.
(2015). However, the datasets used in these studies are small (less than 300
recipes) and the supervision is limited. The edge connectivity is obtained
from several structural assumptions and the nodes are uniquely identified
within the graphs. Our work proposed an unsupervised approach that provides a
method to cope with scalability.

#### Conditional text generation

Conditional text generation with auto-regressive models has been widely
studied in the literature. There are different types of conditioning. The
neural machine translation task is an example of text-based conditioning
Gehring et al. (2017); Fan et al. (2018); Sutskever et al. (2014); Vaswani et
al. (2017) is The image captioning task Vinyals et al. (2015); Xu et al.
(2015); Lu et al. (2017) uses conditioning based on data in other modalities,
namely the input to the decoder is an image representation. Different
architectures have been studied for this task, such as $\mathsf{RNN}$ Fan et
al. (2018), $\mathsf{CNNs}$ Gehring et al. (2017) or attention-based methods
Vaswani et al. (2017). In this paper, we exploit the conditional text
generation task using the learned graph representation as a prompt to the text
decoder.

##  6 Conclusion

We proposed a model that transforms a procedural text into a graph condensing
the relevant information necessary to perform the procedures. This is
performed in an unsupervised way as no supervision at the graph-level is
provided. More specifically, only the raw recipe text is used to construct a
graph. Experimental results show the validity of our approach. The objective
relates to how faithful the recovered text is to the original recipe text.
These representations are meant to able to provide abstraction from the
expression in surface strings. The graphs produced can be integrated into
automated reasoning frameworks. In future work, we plan to generate richer
graphs containing more information in terms of qualifiers and quantifiers,
that can be used to teach automated agents how to execute and reason about
procedures.

##  7 Limitations

The scope of this work is to study an approach to produce a formal
representation in the form of graphs when no supervision data is available.
The lack of large publicly available datasets and work studies focusing on
this specific issue limits the depth of analysis that could be performed.
Moreover, in this work, we did some specific choices that may make the graphs
produced less directly human interpretable. This is highlighted by the choice
of having persistent entities in the recipe represented by a single node that
can have multiple connections whereas the flow graphs in the cooking domain
stress less this aspect by allowing duplicates in the graph. This reflects the
alternative intended use of the representations.

There are additional concerns that need to be considered when generating
graphs from cooking recipes. (i) bias: the model could be biased towards
certain cuisines. Our model autonomously extracts the cooking ingredients to
be included in the graphs. However, there is a possibility that some
ingredients might not be recognized as such due to not being part of the
dataset of origin used to train the entity selection module; (ii) inaccuracies
or misinformation: the graph generator may generate graphs that contain
inaccurate or misleading information. This is due to the self-supervised
nature of the framework.

Finally, future work involves an analysis of the semantic relations that are
represented within the learned graphs to investigate whether programs can be
predicted from this representation that could lead to a finished product.

##  8 Ethics Statement

#### Data sources

Now You’re Cooking dataset Bosselut et al. (2018) and corpora used in this
work is publicly available. The English Flow Corpus dataset Yamakata et al.
(2020) is made available upon requesting it from the original authors. To
quantify the performance of our model, we use $\mathsf{BLEU}$ and
$\mathsf{ROUGE_{L}}$ scoring implementations using the
$\mathsf{pycocoevalcap}$ tool from https://github.com/salaniz/pycocoevalcap.

#### Models

The output graphs are closer to abstract representations and are not intended
to be used by users to interact directly. The generative models are based on
pre-trained language models, which may generate misleading information if not
prompted correctly.

## References

  * Albert and Barabási (2002) Réka Albert and Albert-László Barabási. 2002.  Statistical mechanics of complex networks.  _Reviews of modern physics_ , 74(1):47.  Publisher: APS.
  * Assouel et al. (2018) Rim Assouel, Mohamed Ahmed, Marwin H. Segler, Amir Saffari, and Yoshua Bengio. 2018\.  Defactor: Differentiable edge factorization-based probabilistic graph generation.  _arXiv preprint arXiv:1811.09766_.
  * Bacciu et al. (2019) Davide Bacciu, Alessio Micheli, and Marco Podda. 2019.  Graph generation by sequential edge prediction.  In _27th European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning, ESANN 2019_ , pages 95–100. ESANN (i6doc. com).
  * Bacciu et al. (2020) Davide Bacciu, Alessio Micheli, and Marco Podda. 2020.  Edge-based sequential graph generation with recurrent neural networks.  _Neurocomputing_ , 416:177–189.  ISBN: 0925-2312 Publisher: Elsevier.
  * Bikakis et al. (2023) Antonis Bikakis, Aissatou Diallo, Luke Dickens, Anthony Hunter, and Rob Miller. 2023\.  A Graphical Formalism for Commonsense Reasoning with Recipes.  ArXiv:2306.09042 [cs].
  * Bossard et al. (2014) Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. 2014.  Food-101–mining discriminative components with random forests.  In _European conference on computer vision_ , pages 446–461. Springer.
  * Bosselut et al. (2018) Antoine Bosselut, Omer Levy, Ari Holtzman, Corin Ennis, Dieter Fox, and Yejin Choi. 2018.  Simulating Action Dynamics with Neural Process Networks.
  * Chang et al. (2018) Minsuk Chang, Léonore V. Guillain, Hyeungshik Jung, Vivian M. Hare, Juho Kim, and Maneesh Agrawala. 2018.  Recipescape: An interactive tool for analyzing cooking instructions at scale.  In _Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems_ , pages 1–12.
  * Clark and Gardner (2018) Christopher Clark and Matt Gardner. 2018.  Simple and Effective Multi-Paragraph Reading Comprehension.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 845–855.
  * Dalvi et al. (2018) Bhavana Dalvi, Lifu Huang, Niket Tandon, Wen-tau Yih, and Peter Clark. 2018.  Tracking State Changes in Procedural Text: a Challenge Dataset and Models for Process Paragraph Comprehension.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 1595–1604.
  * De Cao and Kipf (2018) Nicola De Cao and Thomas Kipf. 2018.  MolGAN: An implicit generative model for small molecular graphs.  _arXiv preprint arXiv:1805.11973_.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Fan et al. (2018) Angela Fan, Mike Lewis, and Yann Dauphin. 2018.  Hierarchical neural story generation.  _arXiv preprint arXiv:1805.04833_.
  * Flanigan et al. (2016) Jeffrey Flanigan, Chris Dyer, Noah A. Smith, and Jaime Carbonell. 2016.  Generation from Abstract Meaning Representation using Tree Transducers.  In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 731–739, San Diego, California. Association for Computational Linguistics.
  * Franceschi et al. (2019) Luca Franceschi, Mathias Niepert, Massimiliano Pontil, and Xiao He. 2019.  Learning Discrete Structures for Graph Neural Networks.  In _Proceedings of the 36th International Conference on Machine Learning_ , pages 1972–1982. PMLR.  ISSN: 2640-3498.
  * Gardent et al. (2017) Claire Gardent, Anastasia Shimorina, Shashi Narayan, and Laura Perez-Beltrachini. 2017.  The WebNLG Challenge: Generating Text from RDF Data.  In _Proceedings of the 10th International Conference on Natural Language Generation_ , pages 124–133, Santiago de Compostela, Spain. Association for Computational Linguistics.
  * Gehring et al. (2017) Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N. Dauphin. 2017\.  Convolutional sequence to sequence learning.  In _International conference on machine learning_ , pages 1243–1252. PMLR.
  * Goodfellow et al. (2014) Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. 2014.  Generative Adversarial Nets.  In _NIPS_.
  * Goyal et al. (2020) Nikhil Goyal, Harsh Vardhan Jain, and Sayan Ranu. 2020.  GraphGen: a scalable approach to domain-agnostic labeled graph generation.  In _Proceedings of The Web Conference 2020_ , pages 1253–1263.
  * Grover et al. (2019) Aditya Grover, Aaron Zweig, and Stefano Ermon. 2019.  Graphite: Iterative Generative Modeling of Graphs.  In _Proceedings of the 36th International Conference on Machine Learning_ , pages 2434–2444. PMLR.  ISSN: 2640-3498.
  * Guo et al. (2020) Qipeng Guo, Zhijing Jin, Xipeng Qiu, Weinan Zhang, David Wipf, and Zheng Zhang. 2020\.  CycleGT: Unsupervised Graph-to-Text and Text-to-Graph Generation via Cycle Training.  In _Proceedings of the 3rd International Workshop on Natural Language Generation from the Semantic Web (WebNLG+)_ , pages 77–88, Dublin, Ireland (Virtual). Association for Computational Linguistics.
  * Hammond (1986) Kristian J. Hammond. 1986.  CHEF: A Model of Case-based Planning.  In _AAAI_ , volume 86, pages 267–271.
  * Hanga et al. (2020) Khadijah M. Hanga, Yevgeniya Kovalchuk, and Mohamed Medhat Gaber. 2020.  A Graph-Based Approach to Interpreting Recurrent Neural Networks in Process Mining.  _IEEE Access_ , 8:172923–172938.
  * Henaff et al. (2017) Mikael Henaff, Jason Weston, Arthur Szlam, Antoine Bordes, and Yann LeCun. 2017\.  Tracking the World State with Recurrent Entity Networks.  In _5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings_. OpenReview.net.
  * Honkisz et al. (2018) Krzysztof Honkisz, Krzysztof Kluza, and Piotr Wisniewski. 2018.  A Concept for Generating Business Process Models from Natural Language Description.  In _Knowledge Science, Engineering and Management - 11th International Conference, KSEM 2018, Changchun, China, August 17-19, 2018, Proceedings, Part I_ , volume 11061 of _Lecture Notes in Computer Science_ , pages 91–103. Springer.
  * Jermsurawong and Habash (2015) Jermsak Jermsurawong and Nizar Habash. 2015.  Predicting the structure of cooking recipes.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_ , pages 781–786.
  * Kearnes et al. (2019) Steven Kearnes, Li Li, and Patrick Riley. 2019.  Decoding molecular graph embeddings with reinforcement learning.  _arXiv preprint arXiv:1904.08915_.
  * Kiddon et al. (2015) Chloé Kiddon, Ganesa Thandavam Ponnuraj, Luke Zettlemoyer, and Yejin Choi. 2015\.  Mise en place: Unsupervised interpretation of instructional recipes.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_ , pages 982–992.
  * Kingma and Welling (2013) Diederik P. Kingma and Max Welling. 2013.  Auto-encoding variational bayes.  _arXiv preprint arXiv:1312.6114_.
  * Kipf and Welling (2016) Thomas N. Kipf and Max Welling. 2016.  Variational graph auto-encoders.  _arXiv preprint arXiv:1611.07308_.
  * Kipf and Welling (2017) Thomas N. Kipf and Max Welling. 2017.  Semi-Supervised Classification with Graph Convolutional Networks.  ArXiv:1609.02907 [cs, stat].
  * Knopp and Sinkhorn (1967) Paul Knopp and Richard Sinkhorn. 1967.  Concerning nonnegative matrices and doubly stochastic matrices. _Pacific Journal of Mathematics_ , 21(2):343–348.  Publisher: Pacific Journal of Mathematics, A Non-profit Corporation.
  * Konstas and Lapata (2013) Ioannis Konstas and Mirella Lapata. 2013.  Inducing Document Plans for Concept-to-Text Generation.  In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, EMNLP 2013, 18-21 October 2013, Grand Hyatt Seattle, Seattle, Washington, USA, A meeting of SIGDAT, a Special Interest Group of the ACL_ , pages 1503–1514. ACL.
  * Leskovec et al. (2010) Jure Leskovec, Deepayan Chakrabarti, Jon Kleinberg, Christos Faloutsos, and Zoubin Ghahramani. 2010.  Kronecker graphs: an approach to modeling networks.  _Journal of Machine Learning Research_ , 11(2).  ISBN: 1532-4435.
  * Li et al. (2021) Junyi Li, Tianyi Tang, Wayne Xin Zhao, Zhicheng Wei, Nicholas Jing Yuan, and Ji-Rong Wen. 2021.  Few-shot Knowledge Graph-to-Text Generation with Pretrained Language Models.  In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , pages 1558–1568, Online. Association for Computational Linguistics.
  * Liao et al. (2019) Renjie Liao, Yujia Li, Yang Song, Shenlong Wang, Will Hamilton, David K. Duvenaud, Raquel Urtasun, and Richard Zemel. 2019.  Efficient graph generation with graph recurrent attention networks.  _Advances in neural information processing systems_ , 32.
  * Lin (2004) Chin-Yew Lin. 2004.  Rouge: A package for automatic evaluation of summaries.  In _Text summarization branches out_ , pages 74–81.
  * Liu et al. (2018) Qi Liu, Miltiadis Allamanis, Marc Brockschmidt, and Alexander Gaunt. 2018.  Constrained graph variational autoencoders for molecule design.  _Advances in neural information processing systems_ , 31.
  * Lu et al. (2017) Jiasen Lu, Caiming Xiong, Devi Parikh, and Richard Socher. 2017.  Knowing When to Look: Adaptive Attention via a Visual Sentinel for Image Captioning.  In _2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017_ , pages 3242–3250. IEEE Computer Society.
  * Ma et al. (2018) Tengfei Ma, Jie Chen, and Cao Xiao. 2018.  Constrained generation of semantically valid graphs via regularizing variational autoencoders.  _Advances in Neural Information Processing Systems_ , 31.
  * Mori et al. (2014a) Shinsuke Mori, Hirokuni Maeta, Tetsuro Sasada, Koichiro Yoshino, Atsushi Hashimoto, Takuya Funatomi, and Yoko Yamakata. 2014a.  Flowgraph2text: Automatic sentence skeleton compilation for procedural text generation.  In _Proceedings of the 8th International Natural Language Generation Conference (INLG)_ , pages 118–122.
  * Mori et al. (2014b) Shinsuke Mori, Hirokuni Maeta, Yoko Yamakata, and Tetsuro Sasada. 2014b.  Flow graph corpus from recipe texts.  In _Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14)_ , pages 2370–2377.
  * Papadopoulos et al. (2022) Dim P. Papadopoulos, Enrique Mora, Nadiia Chepurko, Kuan Wei Huang, Ferda Ofli, and Antonio Torralba. 2022.  Learning Program Representations for Food Images and Cooking Recipes.  ArXiv:2203.16071 [cs].
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  Bleu: a method for automatic evaluation of machine translation.  In _Proceedings of the 40th annual meeting of the Association for Computational Linguistics_ , pages 311–318.
  * Pellegrini et al. (2021) Chantal Pellegrini, Ege Özsoy, Monika Wintergerst, and Georg Groh. 2021.  Exploiting Food Embeddings for Ingredient Substitution.  In _HEALTHINF_ , pages 67–77.
  * Podda et al. (2020) Marco Podda, Davide Bacciu, and Alessio Micheli. 2020.  A deep generative model for fragment-based molecule generation.  In _International Conference on Artificial Intelligence and Statistics_ , pages 2240–2250. PMLR.
  * Pölsterl and Wachinger (2019) Sebastian Pölsterl and Christian Wachinger. 2019.  Likelihood-free inference and generation of molecular graphs.  _arXiv preprint arXiv:1905.10310_.
  * Qian et al. (2020) Chen Qian, Lijie Wen, Akhil Kumar, Leilei Lin, Li Lin, Zan Zong, Shuang Li, and Jianmin Wang. 2020.  An Approach for Process Model Extraction by Multi-grained Text Classification.  In _Advanced Information Systems Engineering - 32nd International Conference, CAiSE 2020, Grenoble, France, June 8-12, 2020, Proceedings_ , volume 12127 of _Lecture Notes in Computer Science_ , pages 268–282. Springer.
  * Salvador et al. (2017) Amaia Salvador, Nicholas Hynes, Yusuf Aytar, Javier Marin, Ferda Ofli, Ingmar Weber, and Antonio Torralba. 2017.  Learning cross-modal embeddings for cooking recipes and food images.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 3020–3028.
  * Sanfeliu and Fu (1983) Alberto Sanfeliu and King-Sun Fu. 1983.  A distance measure between attributed relational graphs for pattern recognition.  _IEEE transactions on systems, man, and cybernetics_ , (3):353–362.  ISBN: 0018-9472 Publisher: IEEE.
  * Shi et al. (2020) Chence Shi, Minkai Xu, Zhaocheng Zhu, Weinan Zhang, Ming Zhang, and Jian Tang. 2020\.  Graphaf: a flow-based autoregressive model for molecular graph generation.  _arXiv preprint arXiv:2001.09382_.
  * Su et al. (2019) Shih-Yang Su, Hossein Hajimirsadeghi, and Greg Mori. 2019.  Graph generation with variational recurrent neural network.  _arXiv preprint arXiv:1910.01743_.
  * Sutskever et al. (2014) Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014.  Sequence to sequence learning with neural networks.  _Advances in neural information processing systems_ , 27.
  * Tang et al. (2022) Jialong Tang, Hongyu Lin, Meng Liao, Yaojie Lu, Xianpei Han, Le Sun, Weijian Xie, and Jin Xu. 2022.  Procedural Text Understanding via Scene-Wise Evolution.  _arXiv preprint arXiv:2203.07600_.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is All you Need.  In _Advances in Neural Information Processing Systems_ , volume 30. Curran Associates, Inc.
  * Vinyals et al. (2015) Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. 2015.  Show and tell: A neural image caption generator.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 3156–3164.
  * Williams and Zipser (1989) Ronald J Williams and David Zipser. 1989.  A learning algorithm for continually running fully recurrent neural networks.  _Neural computation_ , 1(2):270–280.  Publisher: MIT Press One Rogers Street, Cambridge, MA 02142-1209, USA journals-info ….
  * Xu et al. (2015) Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron C. Courville, Ruslan Salakhutdinov, Richard S. Zemel, and Yoshua Bengio. 2015.  Show, Attend and Tell: Neural Image Caption Generation with Visual Attention.  In _Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015_ , volume 37 of _JMLR Workshop and Conference Proceedings_ , pages 2048–2057. JMLR.org.
  * Yamakata et al. (2020) Yoko Yamakata, Shinsuke Mori, and John Carroll. 2020.  English Recipe Flow Graph Corpus.  In _Proceedings of the Twelfth Language Resources and Evaluation Conference_ , pages 5187–5194, Marseille, France. European Language Resources Association.