# Spatially Resolved Gene Expression Prediction from H&E Histology Images via
Bi-modal Contrastive Learning

Ronald Xie1,2,3,4 Kuan Pang1,2,4 Sai W. Chung1,5 Catia T. Perciani1,5
Sonya A. MacParland1,5 Bo Wang1,2,3 Gary D. Bader1,3,4,611footnotemark: 1
1University of Toronto, 2Vector Institute, 3University Health Network,
4The Donnelly Centre, 5Toronto General Hospital Research Institute,
6Canadian Institute for Advanced Research (CIFAR)
{ronald.xie, kuan.pang, sai.chung,
catia.perciani<EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>Co-senior author

###### Abstract

Histology imaging is an important tool in medical diagnosis and research,
enabling the examination of tissue structure and composition at the
microscopic level. Understanding the underlying molecular mechanisms of tissue
architecture is critical in uncovering disease mechanisms and developing
effective treatments. Gene expression profiling provides insight into the
molecular processes underlying tissue architecture, but the process can be
time-consuming and expensive. We present BLEEP (Bi-modaL Embedding for
Expression Prediction), a bi-modal embedding framework capable of generating
spatially resolved gene expression profiles of whole-slide Hematoxylin and
eosin (H&E) stained histology images. BLEEP uses contrastive learning to
construct a low-dimensional joint embedding space from a reference dataset
using paired image and expression profiles at micrometer resolution. With this
approach, the gene expression of any query image patch can be imputed using
the expression profiles from the reference dataset. We demonstrate BLEEP’s
effectiveness in gene expression prediction by benchmarking its performance on
a human liver tissue dataset captured using the 10x Visium platform, where it
achieves significant improvements over existing methods. Our results
demonstrate the potential of BLEEP to provide insights into the molecular
mechanisms underlying tissue architecture, with important implications in
diagnosis and research of various diseases. The proposed approach can
significantly reduce the time and cost associated with gene expression
profiling, opening up new avenues for high-throughput analysis of histology
images for both research and clinical applications.
Code available at https://github.com/bowang-lab/BLEEP

##  1 Introduction

Histology imaging of whole-slide Hematoxylin and eosin (H&E) stained tissues
has long been used in academic and clinical settings for research and
diagnosis. It provides useful information pertaining to tissue architecture
and composition at the microscopic level, which are critical for understanding
disease mechanisms and developing effective treatments. Gene expression
profiling is a powerful tool that offers deeper insights into the molecular
processes underlying tissue architecture. However, bulk RNA sequencing does
not capture heterogeneity within one sample whereas single-cell RNA sequencing
(scRNA-seq) or single-nucleus RNA sequencing (snRNA-seq) captures
heterogeneity without spatial context.

In recent years, various spatial transcriptomics methods such as Visium[22],
MERFISH [4], seqFISH+[8], STARmap[23], smFISH[6], and Targeted ExSeq[1] have
emerged as a promising direction to bridge the gap between histology imaging
and gene expression profiling. However, these methods are often low throughput
or low content. They also tend to be time-consuming, expensive and require
specialized equipment and extensive domain expertise to optimize.

With the availability of these bi-modal datasets, a unique opportunity arises
to explore the possibility of predicting spatially resolved expression
profiles of whole tissues solely from their histology image. ST-Net and
HisToGene are two methods developed for this purpose [14, 9] but so far have
achieved limited success. Current progress has been limited due to three
significant challenges. Firstly, the problem is ill-posed. While the histology
images share some information with their paired spatial transcriptomics, it is
likely that the image features can not be used to predict the expression of
all genes and vice versa. However, expression of marker genes for cell types
or subtypes (MG), highly expressed genes (HEG), and highly variable genes
(HVG) should be prioritized as they tend to be the most biologically relevant
candidates for disease diagnosis and drug development. Secondly, the problem
is dimensionally cursed. It is estimated that typical mammalian cells express
around 5,000 to 15,000 genes. Existing solutions often only predict the
expression of a limited panel of genes (~200) and often fail to preserve the
variance and heterogeneity in the original dataset. This obfuscates the
biological signals intrinsic in the original data, rendering the predictions
ineffective for practical use. Thirdly, due to the developing landscape of
spatial transcriptomic methods, existing datasets are prone to experimental
artifacts both within one sample and across different samples, which
complicates the training of expression prediction models. Furthermore, the
measured gene expressions often show poor agreement with their protein
profiles, which could be reflected during H&E staining.

We present BLEEP (Bi-modaL Embedding for Expression Prediction), a novel bi-
modal embedding framework designed to address the aforementioned challenges
associated with predicting gene expression from histology images. BLEEP uses
contrastive learning, which effectively aligns paired image and expression
representations from a reference dataset in a low-dimensional joint embedding
space. Using this joint embedding space, BLEEP accurately imputes the gene
expression of any query image patch by leveraging the expression profiles from
the reference dataset.

We demonstrate the effectiveness of BLEEP by benchmarking its performance
using a challenging human liver tissue dataset captured via the 10x Visium
platform, where it significantly outperforms existing methods such as
HisToGene and ST-Net in terms of the average correlation to original
expressions across marker genes (MG), highly expressed genes (HEG) and highly
variable genes (HVG). BLEEP also preserves heterogeneity in the predicted
expression profiles and recaptures and denoises the gene-gene correlations
present in the original dataset.

The proposed approach alleviates the ill posed nature of the expression
prediction problem by implicitly encouraging shared features to be learned
between image and expression modalities via a contrastive objective which
could prevent modality specific information from disorienting the joint
embedding space. The novel query-reference imputation process from the learned
joint embedding serve to mitigate the curse of dimensionality of the
expression prediction problem as the gene expression profiles of query image
patches are no longer predicted independently, but rather calculated from $k$
closest reference expression profiles in the joint space. Lastly, we
demonstrate that the resulting expression prediction is resilient to
experimental artifacts both within one sample and across different samples, in
addition to outperforming existing solutions on the aforementioned metrics.

To our knowledge, this is the first bi-modal embedding-based framework
proposed for the task of expression prediction from histology images. The
proposed approach has the potential to significantly reduce the time and cost
associated with gene expression profiling, opening up new avenues for high-
throughput analysis of histology images for both research and clinical
applications.

##  2 Related Works

###  2.1 Existing histology expression prediction approaches

Several existing approaches have shown promising results in predicting
expression from histology images including HE2RNA[20], ST-Net[9],
HisToGene[14], hist2rna[13], Hist2ST[26] and others[7, 24].

ST-Net and HisToGene are two of the most popular methods for predicting
spatially resolved expression from H&E images. Both of these approaches frame
the task of expression prediction as regression tasks trained in a feed-
forward fashion. ST-Net uses a resnet50 image encoder followed by a fully
connected layer where as HisToGene leverages a vision transformer backbone and
an increased field of view.

Methods that predict tissue-level expression generally achieve good
correlation but lack the ability to generate spatially resolved expression
profiles (HE2RNA). Existing methods that are capable of generating spatially
resolved expression predictions were either not quantitatively evaluated
(hist2RNA), limited in terms of the predicted panel (ST-Net, Hist2ST,
HisToGene), or prone to overfitting [24].

Both HisToGene and Hist2ST utilize spot-spatial relations to improve
performance. However, our work challenges the necessity of this information,
particularly in tissues with distinct and repetitive spatial patterns like
human liver tissue. The implicit assumption that spatially adjacent regions
should have similar representations compared to spatially distant regions may
not be beneficial for performance in such cases. Hard coding position
information could also lead to overfitting in data-scarce scenarios.

###  2.2 Contrastive representation learning

Contrastive learning plays a pivotal role in advancing the capabilities of
deep learning models, particularly in recent visual language models [5, 18,
17, 15]. One notable application that has emerged from contrastive learning is
the Contrastive Language-Image Pretraining (CLIP) framework [15]. CLIP bridges
the gap between language and vision domains by learning joint representations
of paired images and textual descriptions, enabling cross-modal understanding
and reasoning.

BLEEP draws inspiration from CLIP with modifications to learn a similar joint
embedding between spot expression profiles captured by the 10x Visium platform
and their spatially paired image patch spanning roughly 55$\mu$m. However,
unlike CLIP’s usage setting, BLEEP directly interpolates in the joint
embedding space to produce expression predictions, which is unfeasible for
image and text domains where a domain-specific decoder is required to produce
the final prediction.

###  2.3 Query-reference imputation

The query-reference imputation of BLEEP is partly inspired by SeuratV3’s [19]
integration process, where the expression profiles are calculated from a
linear combination of the closest anchors in the reference dataset given a
query. However, Seurat requires a shared expression panel across modalities to
integrate, whereas BLEEP is able to make spatially resolved expression
predictions based on the morphological features present in the histology image
alone.

##  3 Methods

Figure 1: BLEEP achieves gene expression prediction from H&E image through (a)
BLEEP learns a bimodal embedding from expression profiles and H&E image
patches, (b) images patch queries are projected into the joint embedding space
to index the $k$ nearest reference expression profiles, and (c) the indexed
reference expression profiles are linearly combined to produce the imputed
gene expressions for queries.

###  3.1 Data and preprocessing

The dataset [2, 3] used to train and benchmark BLEEP, HisToGene, and ST-Net
consists of four consecutive 16 micrometer thick slices of human liver tissue
from neurologically deceased donor livers suitable for transplantation that
were OCT embedded, frozen, sliced with a cryostat and imaged using the 10x
Genomics Visium platform [3]. Samples were collected with institutional ethics
approval from the University Health Network (REB# 14-7425-AE). After quality
control, the slices contain 2378, 2349, 2277, and 2265 spots respectively. A
224$\times$224 image patch centered around each spot roughly 55$\mu$m each
side is extracted from the whole slide H&E image and paired with the
corresponding gene expression profile of each spot. The data is publicly
available for download at
https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE240429.

Each spot is normalized to the total count and log normalized before highly
variable genes are computed using the Scanpy[25] package. The union of the top
1000 most highly variable genes from each of the 4 slices was used for
training and prediction, amounting to 3467 genes in total. Finally, the
expression data of these four samples are batch corrected using Harmony [11]
before one of the slices (slice #3) is randomly selected to be held out for
testing.

###  3.2 Learning bimodal embedding for expression prediction

As illustrated in Figure 1a, the first step towards learning a bimodal
embedding for expression prediction is to extract features from the two
modalities respectively using the image encoder and the expression encoder.
Given a batch of $B$ paired image patches ($V\in\mathbb{N}^{B\times L\times
L}$) and normalized expression profiles ($X\in\mathbb{N}^{B\times C}$), where
$L$ is the image patch size and $C$ is the gene set size, we use an image
encoder $f_{img}$ and an expression encoder $f_{expr}$ to project the inputs
into $h$-dim image embeddings and expression embeddings through
$H_{v}=f_{img}(H)\in\mathbb{N}^{B\times h}$ and
$H_{x}=f_{expr}(X)\mathbb{N}^{B\times h}$. We use contrastive learning to
further align the latent space for $H_{v}$ and $H_{x}$.

Our contrastive learning approach is inspired by CLIP [16, 21], but the
adjusted loss function better smooths out the loss landscape for our specific
use case. Unlike natural images, multiple spots with similar expression
profiles or image morphology are often expected to be sampled in the same
batch. The modification will prevent the model from pulling apart spots with
similar expression profiles within the same batch, thereby increasing the
coherence of the resulting joint embedding. We use an existing implementation
of this loss variant[21]. In detail, we first generate the paired similarity
through $sim(H_{v},H_{x})=H_{x}H_{v}^{T}$. To take into account the pairs with
similar morphological features or expression landscapes, we begin with
calculating the internal similarities, $sim(H_{v},H_{v})=H_{v}H_{v}^{T}$ and
$sim(H_{x},H_{x})=H_{x}H_{x}^{T}$, then a similarity adjusted target matrix is
denoted as:

$target=(softmax(sim(H_{x},H_{x})+sim(H_{v},H_{v}))/2\cdot\tau)$

where $\tau$ is a temperature hyperparameter. Cross entropy(ce) loss is
applied to align the image features and expression features to produce the
final loss $\mathcal{L}$:

$\mathcal{L}={mean}({ce}(sim(H_{v},H_{x}),target)+{ce}(sim(H_{v},H_{x})^{T},target^{T}))$

For BLEEP, we use the pretrained ResNet50[10] as the image encoder and a fully
connected network (FCN) with an output dimension of 256 as the expression
encoder, which doubles as a projection head. The image features from the image
encoder are passed through a separate projection head to bring the two
modalities to the same dimension before applying the contrastive loss similar
to CLIP[15], where the model learns to pull the paired representations
together while pushing other representations apart. We find that the ResNet50
image encoder with fewer trainable parameters obtained more favorable results
compared to various pretrained vision-transformer (ViT) encoders
(Supplementary Table 1). Larger models in conjunction with a relatively small
training dataset may encourage information to be memorized in the weights of
the network rather than being encoded in the projections, therefore rendering
the learned joint embedding ineffective for downstream imputation for our use
case.

BLEEP is trained using 4 NVIDIA V100 GPUs with the AdamW optimizer[12], a
batch size of 512 and a learning rate of 0.001 for 150 epochs.

###  3.3 Query-Reference imputation

As illustrated in Figure 1b, the process starts by first splitting the H&E
image into $N$ small image patches to be encoded by the trained image encoder.
Once the image patches are represented in the joint embedding space, the $k$
nearest expression profiles from the reference are selected based on their
proximity (by Euclidean distance) in the joint embedding space to each patch.
Finally, the expression profiles of the query patches are imputed as a linear
combination of the selected expression profiles in the reference 1c. Refer to
supplementary materials for implementation details.

Algorithm 1 Bimodal Embedding for Expression Prediction

Input: Paired image patches ($V\in\mathbb{N}^{B\times L\times L}$), normalized
expression profiles ($X\in\mathbb{N}^{B\times C}$), image encoder $f_{img}$,
expression encoder $f_{expr}$, temperature $\tau$

Output: Joint embedding space ($H_{v},H_{x}$)

1:function BimodalEmbedding($V,X,f_{img},f_{expr},\tau$)

2: $H_{v},H_{x}\leftarrow f_{img}(V),f_{expr}(X)$ $\triangleright$ Image and
Expression embeddings

3: $sim(H_{v},H_{x})\leftarrow H_{x}\cdot H_{v}^{T}$ $\triangleright$ Paired
similarity

4: $sim(H_{v},H_{v})\leftarrow H_{v}\cdot H_{v}^{T}$ $\triangleright$ Internal
similarities (image)

5: $sim(H_{x},H_{x})\leftarrow H_{x}\cdot H_{x}^{T}$ $\triangleright$ Internal
similarities (expression)

6:
$target\leftarrow\text{softmax}((sim(H_{x},H_{x})+sim(H_{v},H_{v}))/2\cdot\tau)$
$\triangleright$ Similarity-adjusted target

7:
$\mathcal{L}\leftarrow\text{mean}(\text{ce}(sim(H_{v},H_{x}),target)+\text{ce}(sim(H_{v},H_{x})^{T},target^{T}))$
$\triangleright$ Cross entropy loss

8: return $H_{v},H_{x},\mathcal{L}$

9:function QueryReferenceImputation($\text{Image},f_{img},H_{v},X,k$)

10: $V^{\prime}\leftarrow$ split(Image) $\triangleright$ Split into image
patches

11: $Q^{\prime}_{v}\leftarrow f_{img}(V^{\prime})$ $\triangleright$ Project
patches(queries) into embedding space

12: distances $\leftarrow$ dist($q^{\prime}$, $H_{v}$ ,$l_{2}$) for
$q^{\prime}\in Q^{\prime}_{v}$ $\triangleright$ Compute l2 distance for each
query

13: indices $\leftarrow$ topk(distances) $\triangleright$ Get indices of top K
matched expression from reference

14: reference_profiles $\leftarrow X$[indices]

15: return weighted_avg(reference_profiles) $\triangleright$ Return prediction

##  4 Experiments

###  4.1 BLEEP predicts spatially resolved gene expression profiles that
correlate well with original expression

Table 1: Average correlation of predicted expression for 8 marker genes derived from Andrews et al. [2] (MG), top 50 most highly expressed genes (HEG) and top 50 most highly variable genes (HVG) compared to ground truth expressions on held out dataset.  Method | MG | HEG | HVG
---|---|---|---
HisToGene | 0.097$\pm$0.015 | 0.072$\pm$0.018 | 0.071$\pm$0.011
ST-Net | 0.099$\pm$0.020 | 0.126$\pm$0.005 | 0.091$\pm$0.007
BLEEP | 0.217$\pm 0.002$ |  0.175$\pm$0.016 |  0.173$\pm$0.011
Table 2: Predicted gene expression values with top 5 correlations with original profile for each method from one representative replicate. HisToGene | ST-Net | BLEEP
---|---|---
Gene Name | r | Gene Name | r | Gene Name | r
CYP3A4 | $0.549$ | CYP3A4 | $0.549$ | CYP3A4 | $0.741$
CYP1A2 | $0.542$ | CYP1A2 | $0.532$ | CYP1A2 | $0.681$
GLUL | $0.488$ | CYP2E1 | $0.530$ | CYP2E1 | $0.675$
CYP2E1 | $0.330$ | GLUL | $0.463$ | GLUL | $0.656$
FABP1 | $0.328$ | SLCO1B3 | $0.375$ | FABP1 | $0.503$

Table 1 shows the performance of HisToGene, ST-Net and BLEEP for predicting a
marker gene set (MG) derived from literature by Andrews et al. [2], the top 50
most highly expressed genes (HEG) and the top 50 most highly variable genes
(HVG). The BLEEP predicted expression profiles show the highest correlation
with ground truth across all three gene sets, achieving an increase of 120%,
39% and 90% in $r$ value across the three gene sets, respectively, compared to
the second scoring method.

Furthermore, Table 2 shows the top 5 predicted genes from each method. We
observed that the most well-predicted genes are relatively consistent across
methods. These genes are known to be spatially zonated, with CYP3A4, CYP1A2,
CYP2E1, GLUL being pericentrally zonated and FABP1 being periportally zonated.
We also observe that BLEEP consistently achieved higher correlation values for
these genes compared to HisToGene and ST-Net, further demonstrating the
effectiveness of BLEEP in expression prediction.

Despite the good prediction of select genes in Table 2, we note that the
overall absolute correlation in Table 1 remains low, highlighting the
difficulty of the prediction task for most genes. The low scores could be
attributed to several causes, including the expression of certain genes being
poorly correlated with morphological features; the poor detection of certain
genes by the Visium platform causing their expression to be less predictable;
and experimental artifacts that could introduce non-biological variance to the
data independent of the image.

###  4.2 BLEEP retains biological heterogeneity

Figure 2: Predicted expression profiles compare with reference expression
profiles normalized by gene count means (Upper) or gene count variance
(Lower).

Figure 2 highlights the key advantage of BLEEP compared to supervised
regression-based approaches such as HisToGene and ST-Net. We observe that
while BLEEP only narrowly outperforms HisToGene and ST-Net in correctly
predicting the mean expression of genes within one sample, both HisToGene and
ST-Net fail to recapitulate the variance of the genes being predicted.

While the variance of BLEEP expression predictions is in general
underestimated, they still maintain sufficient biological heterogeneity
particularly when the predicted expressions are plotted in fixed scale with
the original expression 3. Additional figures depicting the spatial expression
of other genes are available in (Supplementary Figure 1). Both HisToGene and
ST-Net fail to resemble the original expression, likely due to the curse of
dimensionality of the prediction task, resulting in the model learning mean
expressions as a shortcut, with each gene having roughly the same variance
regardless of their mean expression, which is undesirable.

Figure 4 demonstrates the effectiveness of BLEEP in preserving gene-gene
correlations (GGCs) as further testament to its ability to preserve relevant
biological heterogeneity.

Figure 3: Original and predicted spatially resolved expression levels for
CYP3A4 overlaying the H&E image, visualized with variable (Top) and fixed
(Bottom) color scale. Figure 4: Gene-gene correlation heatmap calculated using
the predicted expressions for each method.

###  4.3 BLEEP inferred expression is robust to experimental artifacts and
batch effects

Supplementary Table 2 illustrates the clustering statistics of the
unsupervised clusters resulting from the predicted expression profiles. When
the resulting unsupervised clusters are projected onto the histology image in
Figure 5 we make two observations. Firstly, perhaps unsurprisingly, all three
methods are robust to experimental artifacts within one slice (The red
regions) as the histology image surrounding the region was not affected. The
low-quality regions are characterized by a lower-than-normal number of counts
detected and a higher-than-normal fraction of mitochondrial genes. Their
expression profiles can be rescued by any method that predicts expression from
the image. Secondly, all three methods are capable of capturing the zonation
patterns across the tissue slice to a reasonable extent. While HisToGene has
the best clustering agreement out of the three methods according to Table
(Supplementary Table 2) with BLEEP and ST-Net following close behind, for this
particular dataset, the clustering metrics such as NMI and ARI are likely not
accurate measures of prediction quality because the hepatocytes already
closely resemble each other and lie along a continuous gradient. Hence, the
resulting clusters will be very sensitive to the choice of the clustering
method or the hyperparameters of the method.

However, unlike HisToGene and ST-Net, BLEEP is the least prone to potentially
introduce batch effects between samples during the prediction process owing to
the imputation strategy. We see from Supplementary Figure 1 that both ST-Net
and especially HisToGene predictions are a bit out of distribution when their
predicted expression profiles are plotted alongside that of all the reference
expression profiles.

Figure 5: Leiden clusterings for original and predicted expressions overlaying
the H&E image. Image-only expression predictions are invariant to low quality
regions (red) during actual experiment.

###  4.4 BLEEP ablation experiments

Here we present the ablation experiments conducted in Table 3. From these
experiments we make a few important observations:

The choice of K during the query-reference imputation process influences the
prediction quality quite negatively when a low value is selected (K = 10).
Values of K above our default value could provide some small improvements to
correlation of the resulting predicted expression values for the HVG and HEG
gene sets, but the differences were not pronounced. This is inline with what
one might expect from taking the pseudo-bulk of the top K most likely
expression profiles given any query image. However, the MG gene set did not
show much improvement with increasing K. Furthermore, doing so may carry a
trade off of further systematically deviating from the original variance of
the dataset due to the increased averaging effect. With this in mind, we feel
our default value of K = 50 remains adequate.

We also observe that the most similar match between query and reference is
usually not the best prediction (as seen from the 3rd row of the ablation
table and indirectly the 4th row when predictions are weighted by their
similarity). We suspect the gap may close to some degree as the reference
grows further in size, but in general some amount of averaging is desired for
query-reference imputation to remove some noise intrinsic in the Visium
platform. In addition, as no reference expression profile will perfectly match
that of the query image, averaging multiple profiles is required to more
appropriately describe the query expression profile. Nevertheless, we also
highlight the possibility of genuine biological signals being averaged out,
which is an important consideration to be further investigated.

Smoothing the contrastive loss objective to take into account patch similarity
showed modest increase in performance. The gain in performance may be due to
the fact that relaxing the contrastive objective is more compatible with the
similarity based inference strategy. The smoothing may help lessen the extent
similar references are pushed apart in embedding space during training,
resulting in improved querying of the top K most likely expression profiles
given a query image patch during the inference stage.

Table 3: BLEEP ablation experiments. Variables tested include: smoothed objective versus the original CLIP objective, choice of top K and the method of aggregation. Pearson correlation values are reported from the marker gene set (MG), the top 50 highly variable genes (HVG) and the top 50 highly expressed genes (HEG). The correlation values show the average of 3 replicates. Accompanied uncertainty values denote the maximum difference from the mean of the 3 replicates.The last line denote current BLEEP default settings.  |  |  | Average Pearson Correlation
---|---|---|---
Smoothed Obj. | K | Aggregation | MG | HVG | HEG
Yes | 10 | average | $0.179\pm 0.020$ | $0.146\pm 0.008$ | $0.148\pm 0.022$
Yes | 100 | average | $0.215\pm 0.011$ | $\mathbf{0.180\pm 0.012}$ | $\mathbf{0.181\pm 0.015}$
Yes | – | simple | $0.079\pm 0.032$ | $0.075\pm 0.016$ | $0.084\pm 0.023$
Yes | 50 | weighted | $0.186\pm 0.018$ | $0.161\pm 0.015$ | $0.157\pm 0.026$
No | 50 | average | $0.209\pm 0.017$ | $0.165\pm 0.007$ | $0.170\pm 0.005$
Yes | 50 | average | $\mathbf{0.217\pm 0.002}$ | $\mathbf{0.175\pm 0.016}$ | $\mathbf{0.173\pm 0.011}$

##  5 Discussion and Conclusion

In this study, we introduced BLEEP (Bi-modaL Embedding for Expression
Prediction), a novel framework for predicting gene expression from histology
images. BLEEP constructs a joint embedding space from paired image and
expression features in a reference dataset and subsequently utilizes the k
most similar expression profiles in the joint space to impute the expression
for any given image query. To the best of our knowledge, this is the first bi-
modal embedding-based framework proposed for the task of expression prediction
from histology images. We also show that the query-reference imputation
process is effective and well suited for the task of expression prediction
from the bi-modal joint embedding.

We demonstrated that BLEEP effectively addresses three major challenges in the
H&E image to expression prediction task. Firstly, the ill-posed nature of the
problem is alleviated by the proposed joint image and expression embedding
space, optimized using a contrastive learning objective inspired by CLIP. This
encourages the shared features between the two modalities to be preferentially
encoded in the joint space. Secondly, the curse of dimensionality is tackled
by the query-reference imputation process, which predicts the entire
expression profile jointly through linear combination rather than predicting
each individual gene separately using supervised regression tasks. Lastly, we
showed that BLEEP’s expression prediction is resilient to experimental
artifacts, both within a single sample and across different samples.

We further observed that the correlation matrix of BLEEP’s predicted
expressions not only captures existing patterns in gene-gene correlations
(GGCs) but also accentuates more subtle positive and negative GGCs. This could
be attributed to the ability of BLEEP’s imputation process to average out the
noise intrinsic in the 10x Visium platform, thereby increasing the absolute
values of these correlations. This is consistent with the results presented in
Figure 2, where the predicted gene expressions by BLEEP exhibit lower variance
compared to the original dataset. This highlights BLEEP’s capability to
combine expression profiles of similar-looking spots in the dataset and
generate expression profiles with reduced noise and enhanced biological signal
compared to the original dataset. Ongoing work is evaluating BLEEP’s ability
to detect new spatially resolved gene modules in response to this observation.

However, an alternate explanation for the observed results could be that
averaging during imputation removes genuine, abrupt biological signals,
resulting in artificially smoothed expression patterns. It is possible that
this may be true for certain genes, especially ones that correlate poorly with
image features. Increasing the size of the reference dataset may mitigate this
issue by reducing the distance between any query patch and the reference
expression profiles. In practice, once the expression encoder is trained, it
can be quickly used to integrate new datasets to the shared embedding space as
reference. Furthermore, imputed expressions from query datasets could be
subsequently integrated, making the process of improving BLEEP prediction
natural.

Nevertheless, the improvement offered by BLEEP over existing methods is
substantial. We achieved significantly higher correlation with actual
expression profiles across the marker gene set (MG), top 50 highly expressed
genes (HEG), and top 50 highly variable genes (HVG) (Figure 1), with
improvements ranging between 39% to 120% compared to the second highest
scoring method. The top predicted genes from BLEEP are consistent with those
from ST-Net and HisToGene, but the correlation values (r) are higher by up to
35% compared to the second-highest scoring method. BLEEP also excel in
retaining the biological heterogeneity of the original sample as shown in
Figures 2, 3, 4 and 5.

Overall, our proposed framework, BLEEP, has the potential to significantly
reduce the time and cost associated with gene expression profiling, opening up
new avenues for high-throughput analysis of histology images for both research
and clinical applications.

##  6 Acknowledgements

All authors thank the Vector Institute, Calcul Québec, and the Digital
Research Alliance of Canada for their support. This research has been made
possible in part by a grant to G.D.B. from the Chan Zuckerberg Initiative DAF,
an advised fund of Silicon Valley Community Foundation. B.W. is supported by
the NSERC discovery grant and CIFAR chair programs [RGPIN-2020-06189,
DGECR-2020-00294]. R.X. is supported by the Ontario Graduate Scholarship.

## References

  * [1] Shahar Alon, Daniel R Goodwin, Anubhav Sinha, Asmamaw T Wassie, Fei Chen, Evan R Daugharthy, Yosuke Bando, Atsushi Kajita, Andrew G Xue, Karl Marrett, et al.  Expansion sequencing: Spatially precise in situ transcriptomics in intact biological systems.  Science, 371(6528):eaax2656, 2021.
  * [2] Tallulah S Andrews, Jawairia Atif, Jeff C Liu, Catia T Perciani, Xue-Zhong Ma, Cornelia Thoeni, Michal Slyper, Gökcen Eraslan, Asa Segerstolpe, Justin Manuel, et al.  Single-cell, single-nucleus, and spatial rna sequencing of the human liver identifies cholangiocyte and mesenchymal heterogeneity.  Hepatology Communications, 6(4):821–840, 2022.
  * [3] Tallulah S Andrews, Diana Nakib, Catia Perciani, Xue Zhong Ma, Lewis Liu, Erin Winter, Damra Camat, Sai Chung, Justin Manuel, Shantel Mangroo, et al.  Single-cell and spatial transcriptomics reveals the human liver immunological landscape and myeloid dysfunction in psc.  bioRxiv, pages 2023–07, 2023.
  * [4] Kok Hao Chen, Alistair N Boettiger, Jeffrey R Moffitt, Siyuan Wang, and Xiaowei Zhuang.  Spatially resolved, highly multiplexed rna profiling in single cells.  Science, 348(6233):aaa6090, 2015.
  * [5] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.  A simple framework for contrastive learning of visual representations.  In International conference on machine learning, pages 1597–1607. PMLR, 2020.
  * [6] Simone Codeluppi, Lars E Borm, Amit Zeisel, Gioele La Manno, Josina A van Lunteren, Camilla I Svensson, and Sten Linnarsson.  Spatial organization of the somatosensory cortex revealed by osmfish.  Nature methods, 15(11):932–935, 2018.
  * [7] Charles Comiter, Eeshit Dhaval Vaishnav, Metamia Ciapmricotti, Bo Li, Yiming Yang, Scott J Rodig, Madison Turner, Kathleen L Pfaff, Judit Jané-Valbuena, Michal Slyper, et al.  Inference of single cell profiles from histology stains with the single-cell omics from histology analysis framework (schaf).  bioRxiv, pages 2023–03, 2023.
  * [8] Chee-Huat Linus Eng, Michael Lawson, Qian Zhu, Ruben Dries, Noushin Koulena, Yodai Takei, Jina Yun, Christopher Cronin, Christoph Karp, Guo-Cheng Yuan, et al.  Transcriptome-scale super-resolved imaging in tissues by rna seqfish+.  Nature, 568(7751):235–239, 2019.
  * [9] Bryan He, Ludvig Bergenstråhle, Linnea Stenbeck, Abubakar Abid, Alma Andersson, Åke Borg, Jonas Maaskola, Joakim Lundeberg, and James Zou.  Integrating spatial gene expression and breast tumour morphology via deep learning.  Nature biomedical engineering, 4(8):827–834, 2020.
  * [10] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [11] Ilya Korsunsky, Nghia Millard, Jean Fan, Kamil Slowikowski, Fan Zhang, Kevin Wei, Yuriy Baglaenko, Michael Brenner, Po-ru Loh, and Soumya Raychaudhuri.  Fast, sensitive and accurate integration of single-cell data with harmony.  Nature methods, 16(12):1289–1296, 2019.
  * [12] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  arXiv preprint arXiv:1711.05101, 2017.
  * [13] Raktim Kumar Mondol, Ewan KA Millar, Peter H Graham, Lois Browne, Arcot Sowmya, and Erik Meijering.  hist2rna: An efficient deep learning architecture to predict gene expression from breast cancer histopathology images.  Cancers, 15(9):2569, 2023.
  * [14] Minxing Pang, Kenong Su, and Mingyao Li.  Leveraging information in spatial transcriptomics to predict super-resolution gene expression from histology images in tumors.  bioRxiv, pages 2021–11, 2021.
  * [15] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In International conference on machine learning, pages 8748–8763. PMLR, 2021.
  * [16] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In International Conference on Machine Learning, 2021.
  * [17] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.  Hierarchical text-conditional image generation with clip latents.  arXiv preprint arXiv:2204.06125, 2022.
  * [18] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever.  Zero-shot text-to-image generation.  In International Conference on Machine Learning, pages 8821–8831. PMLR, 2021.
  * [19] Rahul Satija, Jeffrey A Farrell, David Gennert, Alexander F Schier, and Aviv Regev.  Spatial reconstruction of single-cell gene expression data.  Nature biotechnology, 33(5):495–502, 2015.
  * [20] Benoît Schmauch, Alberto Romagnoni, Elodie Pronier, Charlie Saillard, Pascale Maillé, Julien Calderaro, Aurélie Kamoun, Meriem Sefta, Sylvain Toldo, Mikhail Zaslavskiy, et al.  A deep learning model to predict rna-seq expression of tumours from whole slide images.  Nature communications, 11(1):3877, 2020.
  * [21] M. Moein Shariatnia.  Simple CLIP, 4 2021.
  * [22] Patrik L Ståhl, Fredrik Salmén, Sanja Vickovic, Anna Lundmark, José Fernández Navarro, Jens Magnusson, Stefania Giacomello, Michaela Asp, Jakub O Westholm, Mikael Huss, et al.  Visualization and analysis of gene expression in tissue sections by spatial transcriptomics.  Science, 353(6294):78–82, 2016.
  * [23] Xiao Wang, William E Allen, Matthew A Wright, Emily L Sylwestrak, Nikolay Samusik, Sam Vesuna, Kathryn Evans, Cindy Liu, Charu Ramakrishnan, Jia Liu, et al.  Three-dimensional intact-tissue sequencing of single-cell transcriptional states.  Science, 361(6400):eaat5691, 2018.
  * [24] Yinxi Wang, Kimmo Kartasalo, Philippe Weitz, Balazs Acs, Masi Valkonen, Christer Larsson, Pekka Ruusuvuori, Johan Hartman, and Mattias Rantalainen.  Predicting molecular phenotypes from histopathology images: a transcriptome-wide expression–morphology analysis in breast cancer.  Cancer Research, 81(19):5115–5126, 2021.
  * [25] F Alexander Wolf, Philipp Angerer, and Fabian J Theis.  Scanpy: large-scale single-cell gene expression data analysis.  Genome biology, 19:1–5, 2018.
  * [26] Yuansong Zeng, Zhuoyi Wei, Weijiang Yu, Rui Yin, Yuchen Yuan, Bingling Li, Zhonghui Tang, Yutong Lu, and Yuedong Yang.  Spatial transcriptomics prediction from histology jointly through transformer and graph neural networks.  Briefings in Bioinformatics, 23(5), 2022.


# Syntax-Guided Transformers: Elevating Compositional Generalization and
Grounding in Multimodal Environments

Danial Kamali
Michigan State University
<EMAIL_ADDRESS>
Parisa Kordjamshidi
Michigan State University
<EMAIL_ADDRESS>

###### Abstract

Compositional generalization, the ability of intelligent models to extrapolate
understanding of components to novel compositions, is a fundamental yet
challenging facet in AI research, especially within multimodal environments.
In this work, we address this challenge by exploiting the syntactic structure
of language to boost compositional generalization. This paper elevates the
importance of syntactic grounding, particularly through attention masking
techniques derived from text input parsing. We introduce and evaluate the
merits of using syntactic information in the multimodal grounding problem. Our
results on grounded compositional generalization underscore the positive
impact of dependency parsing across diverse tasks when utilized with Weight
Sharing across the Transformer encoder. The results push the state-of-the-art
in multimodal grounding and parameter-efficient modeling and provide insights
for future research.

##  1 Introduction

Compositional Generalization refers to the ability of an intelligent agent to
generalize its understanding of the underlying structure of a problem,
especially when it is faced with novel compositions of the previously seen
building blocks or components Chomsky (1957); Montague (1970). It is
fundamental for models to be able to extrapolate from their training
environment to novel situations, a common occurrence in real-world
applications. Hupkes et al. (2020) categorizes compositional generalization
capabilities into five categories, systematicity, substitutivity, localism &
globalism, and overgeneralization. These abilities are crucial for models to
achieve strong performance on tasks that require reasoning and understanding
of hierarchical structures, such as natural language understanding, object
classification, and robotics.

Humans understand new compositions of previously observed concepts and simpler
constructs. On the other hand, despite remarkable progress in the field of
Artificial Intelligence, even state-of-the-art language models demonstrate
limitations in this aspect Lake and Baroni (2018); Thomas McCoy et al. (2020);
Shaw et al. (2021). Especially, they often fail to effectively generalize in
the reasoning depth, which involves handling multi-turn reasoning about
entities and their properties in the world or even the co-occurrence of unseen
spatial relations Wu et al. (2021). These limitations indicate a crucial need
for innovative approaches to address these issues.

Input Command: pull the small blue object that is inside of the small green
box and in the same row as the red circle while zigzagging.
Action sequence: turn left, turn left, walk, turn right, walk, turn right,
walk, pull

Figure 1: This example is taken from the ReaSCAN dataset. Here, an agent is
provided with a command. Its objective is to generate/execute a series of
predefined actions to fulfill the task within the given environment.

In this research, our objective is to exploit the syntactic structure of
language to enhance compositional generalization. Our focus is mainly on the
multimodal problem setting that entangles vision and language. In this unique
setting, compositional linguistic descriptions must be accurately grounded in
the environment to devise coherent action plans or achieve specific goals. An
illustrative example of this scenario is shown in Figure 1.

The motivation behind leveraging syntax in our approach stems from the
inherent structure and compositionality of natural language. Syntactic parsing
provides crucial structural information about how words in a sentence relate
to each other. We hypothesize that syntactic structure can improve intelligent
agents’ ability to discern the applicable attributes and descriptions for each
object in its environment and better apprehend deeper levels of reasoning.

By imposing an understanding of language structure through syntactic parsing,
we aim to extend the capabilities of current multimodal language models. This
could potentially pave the way for more sophisticated models capable of
robustly interacting with dynamic and complex vision and language
environments. Apart from using structure, we equipped our end-to-end model
with weight sharing that has demonstrated improving the generalization
capabilities in single-modality tasks.

As a result, we reach state-of-the-art performance on the ReaSCAN
compositional generalization benchmark, showing improvement across all test
splits, especially ones requiring sentence structure comprehension. In
summary, our contributions include:

  * •

Enhancing grounded compositional generalization by integrating syntactic
parsing into our model.

  * •

Using syntax-guided attention masking along with weight sharing, we build a
highly parameter-efficient model compared to baselines.

  * •

Our model has shown marked improvement in performance across a variety of
tasks that are designed for compositional generalization evaluation while
enhancing computational efficacy.

##  2 Related Work

The machine learning research community primarily focused on understanding the
error bounds and the bias-variance trade-off Hastie et al. (2009) to
understand and improve the models’ generalizability. Later, techniques like
dropout Srivastava et al. (2014) were introduced to improve neural models’
generalization. Recently, studies have examined the generalizability of
various neural network architectures using specialized generalization
evaluation tasks Hupkes et al. (2020); Ontanon et al. (2022); Csordás et al.
(2021). Additionally, numerous datasets such as SCAN Lake and Baroni (2018),
CFQ Keysers et al. (2020), and COGS Kim and Linzen (2020) have been developed
to assess compositional generalization capabilities. Diverse strategies such
as data augmentation Andreas (2020); Shaw et al. (2021), innovative
architectural designs Korrel et al. (2019); Gao et al. (2020), and neuro-
symbolic methods Mao et al. (2019), have been proposed to enhance these
capabilities. Consequently, these advances in text-based generalization have
inspired research in multimodal compositional generalization, with
developments including complex benchmarks like gSCAN Ruis et al. (2020) and
ReaSCAN Wu et al. (2021), and advanced architectures applied to multimodal
grounding Kuo et al. (2021); Jiang and Bansal (2021); Qiu et al. (2021a);
Sikarwar et al. (2022); Shaw et al. (2021).

Furthermore, recent research highlights the significant role of syntactic
information in enhancing neural models’ compositional generalization
capability. Kuo et al. (2021) suggested aligning the compositional structure
of networks with the problem domain, resulting in a dynamic compositional
neural network. Moreover, Shaw et al. (2021) and Qiu et al. (2022) recommended
grammar induction-based data augmentation techniques to improve compositional
generalization. Unlike our work that focuses on input command structure, Kim
et al. (2021b) introduced the concept of using parse tree node annotations in
the target sequence of sequence-to-sequence tasks for enhancing compositional
generalization. Meanwhile, Kim et al. (2021a) incorporated parse tree nodes
into the ETC Ainslie et al. (2020) model. They employed attention masking
specific for ETC to symbolize the relations of tokens and aid this model in a
simplified classification task based on the CFQ dataset.

We are inspired by previous research Kim et al. (2021a) that employs a similar
technique with manually extracted parses for compositional generalization on
the single text modality. However, our model utilizes off-the-shelf parsers
instead of accurate manually generated parse trees, and it is generally
applicable independent of the underlying models.

##  3 Problem Setting

Various studies on compositional generalization have presented a range of
tasks and problem settings Lake and Baroni (2018); Keysers et al. (2020); Kim
and Linzen (2020); Wu et al. (2021); Ruis et al. (2020). These datasets are
comprised of a training set and several test sets. To ensure rigorous
evaluation, the test sets have been deliberately structured to differ from the
training set in a way that requires the compositional capability to succeed.
Our paper focuses on grounding natural language instructions in the visual
modality, where we map words to specific objects or actions in a multimodal
environment that provides a framework to evaluate an intelligent agent’s
compositional structures and spatial reasoning capabilities.

We use the most recent multimodal compositional generalization benchmarks to
assess our models comprehensively. In these benchmarks, an agent receives
natural language instruction to carry out an action or navigate specific
environments. These datasets are inherently synthetic, and they have been
carefully crafted to guarantee that the test sets are systematically different
from the training sets. By placing commands within a spatial context, these
benchmarks bridge the gap between abstract cognitive understanding and
practical action execution. Consequently, they stand as both a scholarly tool
for studying compositional generalization and a valuable resource for fields
like robotics that require comprehension of spatially anchored commands.

Among these benchmarks, our primary focus is ReaSCAN, owing to its heightened
complexity and recent introduction to the academic community. An example of
this dataset, depicted in Figure 1, consists of three main components: The
initial state of the world, the provided input command, and the corresponding
target command. Tasked with this information, the agent aims to infer the
target command by leveraging both the information from the input command and
the initial state. Structurally, the world’s representation in ReaSCAN is
formulated as a 6×6×17 matrix. Each matrix cell comprises a 17-dimensional
vector encapsulating information pertaining to an object’s attributes—namely,
color, shape, and size—along with indicators of the agent’s positioning and
orientation. The evaluation metric for this dataset is the percentage of exact
matches of the predicted action sequence. The ReaSCAN dataset includes one
random test split that mirrors the training’s component and compound
distribution, in addition to seven compositional generalization test splits.
Each of these splits is designed to probe a specific facet of a model’s
grounding generalization capability, as detailed in Table 1. Category A test
splits delve into novel attribute compositions at both the command and object
levels, drawing inspiration from gSCAN. Category B test splits assess the
model’s ability to generalize to unprecedented co-occurrences of concepts and
spatial relations. Meanwhile, Category C probes the model’s capacity to
extrapolate from simple command structures to more intricate structures with
higher levels of reasoning Wu et al. (2021). To illustrate the A1 split, all
examples with commands containing variations of "yellow square" (such as
"small yellow square" or "big yellow square") are excluded from the training
data. This prevents models from associating targets with that phrase. However,
the training set does include examples like "yellow cylinder" and "blue
square." As a result, during testing, models are expected to accurately
interpret the "yellow square" even without prior exposure to the actual
composition.

Split | Held-out Examples
---|---
Random | Random.
A1 |  yellow square referred with color & shape.
A2 |  red square referred in the command.
A3 |  small cylinder referred with size and shape
B1 | co-occur of small red circle and big blue
square.
B2 | co-occur of same size as and inside of
relations.
C1 | Additional conjunction clause depth added
to 2-relative-clause commands.
C2 |  2-relative-clause command with that is
instead of and.
Table 1: ReaSCAN dataset test splits.

##  4 Proposed Method

Figure 2: Overall architecture of the proposed model.

To address the challenge at hand, we implemented a multimodal transformer, as
illustrated in Figure 2. In this model, input commands are tokenized and then
supplemented with positional encoding before passing to the transformer.
Concurrently, the visual environment is segmented into 36 distinct cells, each
serving as a visual token. After passing the visual token to a linear layer,
these tokens receive positional encoding and are passed to the transformer.

We’ve employed a generic parser to seamlessly embed the structure of the
textual modality into our model, thereby shaping attention masks for the
encoder’s textual self-attention. Prioritizing efficiency, parsing each input
command is conducted during a preprocessing phase.

Our transformer is based on the GroCoT model Sikarwar et al. (2022). Each
encoder layer employs a cross-attention mechanism between modalities, followed
by modality-specific self-attention. Our computed input command masks are
utilized in the self-attention modules of the textual modality. Remarkably,
encoder layer weights are consistently shared across all layers.

In the end, we concatenate the encoded result of each modality and pass it to
the transformer’s auto-regressive decoder to generate the action sequence
corresponding to the input command given the environment.

Figure 3: Examples of parse trees.

###  4.1 Syntax-guided attention

One main component of our proposed model is exploiting the syntactic structure
of the command. For this aim, we investigate using both dependency and
constituency parsing. Dependency and constituency trees can be used to analyze
the grammatical structure of sentences. Dependency trees focus on the
grammatical relationships between individual words, where each word except the
root depends on another, and the edges of the tree signify these dependencies.
However, constituency trees emphasize the hierarchical organization of words
into larger syntactic units or constituents, with internal nodes representing
these groupings and leaves representing individual words. While dependency
trees are more concerned with identifying grammatical roles and relationships
between words, constituency trees aim to show how words group together into
larger syntactic units, often carrying syntactic labels like NP (noun phrase)
or VP (verb phrase) Foscarin et al. (2023); Hearne et al. (2008). Examples of
these parse trees are shown in Figure 3.

Syntax-guided attention masking. We use the syntactic information to guide the
self-attention module of transformer encoder layers as depicted in Figures 2
and 4(b). We force each token to only attend to the tokens connected in the
syntax tree. In this way, we avoid faulty attention patterns and overfitting
irrelevant parts of the sentence. In addition, by imposing the structure with
a parse tree, our model can capture the nesting structure of the command’s
meaning and the relationships between its components. By making the structural
information explicit, our model can potentially extrapolate the meaning of
novel combinations and nesting linguistic structures encompassing higher
reasoning depth.

###  4.2 Weight Sharing

Parameter sharing is a strategic approach where identical learned parameters
are applied across various positions or layers within a model. This technique
enables the reuse of the same encoder unit at each phase of the transformer
encoder Dehghani et al. (2019). Such an approach not only streamlines the
model but also nurtures the acquisition of more robust and adaptable
representations of the input Ontanon et al. (2022). The findings of Kim et al.
(2021a) demonstrate that a transformer employing attention masking requires
extended training epochs for convergence, potentially due to masking-induced
backpropagation constraints. In light of this, we hypothesize that introducing
weight sharing might counterbalance this challenge. Weight sharing reduces the
model’s complexity by decreasing the number of parameters, which could lead to
faster convergence. This method acts as a form of regularization, stabilizing
training and facilitating smoother optimization landscapes. In addition,
Ontanon et al. (2022) show that a transformer with shared weights across its
encoder layers is arguably endowed with a more suitable inductive bias that
allows the model to learn the primitive concepts. We hypothesize this will
positively affect learning spatial relations or object-property relations,
which are frequently used in our model’s input. Motivated by these advantages,
we incorporated this weight sharing technique into our transformer model to
evaluate its efficacy in a multimodal setting. Beyond the enhanced
generalizability, weight sharing serves as a computational benefit by reducing
the number of learnable parameters during the training phase.

(a) Self-attention w/o masking

(b) Self-Attention w/ masking

Figure 4: Self-Attention example from the A2 test set of ReaSCAN dataset.
Figures (a) and (b) depict the averaged self-attention map from our models’
over all encoder layers and heads. Rows and columns correspond to text tokens.
Brighter attention cells indicate higher attention weights

##  5 Experiments

Implementation Details. Our model architecture is founded on the GroCoT
framework as detailed by Sikarwar et al. (2022) and is implemented using the
PyTorch machine learning library Paszke et al. (2019). Also, we employed the
pre-trained stanza toolkit Qi et al. (2020) for constituency and dependency
parsing. We used 48 GB A6000 GPUs accompanied by 756GB RAM. On average, each
experiment took about 52 hours to train the models from scratch, with the Adam
optimizer Kingma and Ba (2017) parameter updates throughout the training
regimen. To ensure a rigorous evaluation, we used the same specialized
compositional validation set as Sikarwar et al. (2022), drawing 500 samples
from each compositional division of the primary dataset. Model proficiency was
assessed against this validation set, with the highest-performing model
designated as our optimal choice. Our results are presented as an average
derived from three independent runs, each initialized with a random seed. We
ran the models for the ReaSCAN benchmark for 120 epochs, and the models for
the gSCAN and GSRR benchmarks for 100 epochs. Hyperparameters used for the
experiments of each dataset are shown in Appendix A. The code and models
proposed in this work are all available in
GitHub111https://github.com/HLR/Syntax-Guided-Transformers.

Datasets. We used gSCAN Ruis et al. (2020), GSRR Qiu et al. (2021b), ReaSCAN
Wu et al. (2021) benchmarks for evaluation. The Grounded SCAN (gSCAN) dataset
is a benchmark tailored for examining compositional generalization in machine
learning models by translating natural language commands into actions in a
grid-world scenario. Its unique splits ensure models move beyond rote
memorization to deep compositional understanding of concepts. The Grounded
Systematic Relation Reasoning (GSRR) dataset extends gSCAN by aligning natural
language instructions intricately with visual elements, emphasizing spatial
relationships and object references. ReaSCAN, a further development, brings
the challenges of real-world reasoning into this environment by introducing
more challenging tasks and concept relations. Together, these datasets offer a
high-complexity framework for assessing the compositional and relational
understanding of machine learning models in visual environments. A detailed
explanation of both the gSCAN and Grounded Systematic Relation Reasoning
datasets can be found in Appendix B.

Baselines. We embarked on a series of experiments designed to evaluate our
model’s effectiveness compared to the most recent state-of-the-art models on
the mentioned multimodal compositional generalization datasets. We include the
following baselines. (a) Ruis et al. (2020) (Multimodal LSTM) is a fusion of
sequence-to-sequence (seq2seq) architecture with a visual encoder, employing a
recurrent ’command encoder’ to process the instructions. (b) Gao et al. (2020)
(GCN-LSTM) integrates a Graph Convolutional Neural (GCN) network with a
multimodal LSTM. The command encoding is achieved via a BiLSTM equipped with
multi-step textual attention, while the world is encoded through a GCN layer.
(c) Qiu et al. (2021b) (Multimodal Transformer) is a multimodal transformer
equipped with cross-attention for multimodal compositional generalization. (d)
Sikarwar et al. (2022) (GroCoT) is another transformer-based model that
incorporates interleaved self-attention into the multimodal transformer with
cross-attention.

Model | A1 | A2 | A3 | B1 | B2 | C1 | C2 | Avg
---|---|---|---|---|---|---|---|---
LSTM* | 50.4 | 14.7 | 50.9 | 52.2 | 39.4 | 49.7 | 25.7 | 40.40
GCN-LSTM | 92.3 | 42.1 | 87.5 | 69.7 | 52.8 | 57.0 | 22.1 | 60.50
Transformer* | 96.7 | 58.9 | 93.3 | 79.8 | 59.3 | 75.9 | 25.5 | 69.90
GroCoT | 99.6 | 93.1 | 98.9 | 93.9 | 86.0 | 76.3 | 27.3 | 82.2
Constituency† |  99.75$\pm$0.11 | 96.70$\pm$1.40 |  99.68$\pm$0.10 | 95.19$\pm$1.17 | 88.37$\pm$1.50 | 69.07$\pm$0.60 | 27.00$\pm$0.54 | 82.25$\pm$0.63
Dependency† | 99.65$\pm$0.9 |  97.37$\pm$0.48 | 99.62$\pm$0.07 |  95.46$\pm$2.01 |  90.15$\pm$3.88 |  92.55$\pm$1.51 | 21.77$\pm$5.25 |  85.22$\pm$0.87
Table 2: The result of our proposed model on the ReaSCAN dataset test splits. The results are an average of three runs. $\dagger$ denotes the models with masking. Models marked with * refer to the multimodal version of their implementation. Model | A | B | C | E | F | H | Comp. Avg
---|---|---|---|---|---|---|---
LSTM* | 97.7 | 54.9 | 23.5 | 35.0 | 92.5 | 22.7 | 32.7
GCN-LSTM | 98.6 | 99.1 | 80.3 | 87.3 | 99.3 | 33.6 | -
Transformer* | 99.9 | 99.9 | 99.3 | 99.0 | 99.9 | 22.2 | 60.0
GroCoT | 99.9 | 99.9 | 99.9 | 99.8 | 99.9 | 22.9 | 60.4
Constituency† |  99.95$\pm$0.07 |  99.92$\pm$0.06 |  99.88$\pm$0.11 | 99.88$\pm$0.09 |  100.00$\pm$0.00 | 22.84$\pm$0.93 | 60.36$\pm$0.11
Dependency† | 99.92$\pm$0.09 | 99.85$\pm$0.18 | 99.86$\pm$0.11 |  99.96$\pm$0.06 | 99.89$\pm$0.16 | 23.89$\pm$1.54 |  60.49$\pm$0.20
Table 3: The result of our proposed model on the gSCAN dataset test splits. The results are an average of three runs. We did not report the results on D and G splits since we achieved 0.00$\pm$0.00 % performance, But take them into account in the averaged result. $\dagger$ denotes the models with masking. Models marked with * refer to the multimodal version of their implementation. Model | I | II | III | IV | V | VI | Comp. Avg
---|---|---|---|---|---|---|---
LSTM* | 86.5 | 40.1 | 86.1 | 5.5 | 81.4 | 81.8 | 58.9
Transformer* | 94.7 | 64.4 | 94.9 | 49.6 | 59.3 | 49.5 | 63.5
GroCoT | 99.9 | 98.6 | 99.9 | 99.7 | 99.5 | 96.5 | 98.8
Constituency† | 99.85$\pm$0.00 | 99.90$\pm$0.03 | 99.16$\pm$0.26 | 99.88$\pm$0.03 | 96.73$\pm$2.16 |  97.85$\pm$0.46 | 98.58$\pm$0.39
Dependency† |  99.91$\pm$0.02 |  99.93$\pm$0.01 | 99.41$\pm$0.28 |  99.96$\pm$0.01 | 99.03$\pm$0.23 | 97.38$\pm$0.63 |  99.07$\pm$0.16
Table 4: The result of our proposed model on the GSRR dataset test splits. The
results are an average of three runs. $\dagger$ denotes the models with
masking. Models marked with * refer to the multimodal version of their
implementation.

Results. We comprehensively evaluated our approach across all the previously
mentioned benchmarks compared to the baselines. Alongside the accuracy and
efficacy metrics, we also provide insights into the computational overhead
associated with our method. Furthermore, a qualitative analysis is presented,
delving deeper into our approach’s performance nuances and strengths.

The benchmark results, presented in Tables 2, 3, and 4, demonstrate our
model’s superior performance over all reported models, with a notable 3%
improvement on the average of ReaSCAN benchmark splits. This substantiates our
hypothesis that incorporating syntactic parsing significantly boosts the
model’s generalization derived from grounded compositional training data.
Moreover, dependency parsing consistently outperformed constituency or marked
a very similar performance across multiple benchmarks, including GSRR and
gSCAN. Our model displayed improvements across nearly all ReaSCAN splits
except for C2. As per Sikarwar et al. (2022), the C2 split is "unfair,"
lacking the required information in training data for comprehensive model
training. Even including syntactic information could not improve the model’s
performance on this split and even caused a decrease in the performance. Our
methodology also showcased its merit in the object property test cases (A1-3),
effectively constraining attention to words pertinent to target object
descriptors. For instance, as shown in Figure 4, the attention weights from
the properties to the corresponding objects are high.

Notably, our model exhibited considerable strides in the C1 split, indicative
of the value added by syntactic information. For a more reliable comparison,
we applied a t-test to our C1 test split results. Using a significance level
($\alpha$) of 0.05, this statistical analysis provided further validation for
the observed enhancements in our model’s performance, particularly within the
context of the C1 test split. Furthermore, our model exhibits enhanced
performance on the GSRR dataset. As illustrated in Table 4, both variants of
our model demonstrate improvements in the II split. It is worth noting that
the II split shares the same challenge as the A2 split from the ReaSCAN
dataset but in a less complex environment.

While our proposed techniques effectively address splits A, B, C, E, and F,
mirroring the successes of previous works such as Sikarwar et al. (2022) and
Qiu et al. (2021b), they struggle with challenges presented by specific gSCAN
compositionality splits, notably D, G, and H. These particular splits are
designed to assess the model’s capacity for systematic generalization when
novel patterns should occur on the output sequence rather than in grounding
the input instruction Sikarwar et al. (2022), a facet that is not expected to
be captured by our proposed model.

W/S | Mask | A1 | A2 | A3 | B1 | B2 | C1 | C2 | Avg
---|---|---|---|---|---|---|---|---|---
- | - | 99.29$\pm$0.27 | 91.82$\pm$6.50 | 98.49$\pm$1.17 | 93.50$\pm$0.85 | 83.15$\pm$1.41 | 75.85$\pm$1.35 |  25.03$\pm$6.82 | 81.02$\pm$0.22
✓ | - | 99.68$\pm$0.22 | 97.09$\pm$1.72 | 99.64$\pm$0.20 | 94.86$\pm$0.77 | 81.49$\pm$4.27 | 66.30$\pm$6.65 | 21.66$\pm$1.83 | 80.10$\pm$1.08
- | Dep. | 98.09$\pm$0.27 | 85.21$\pm$6.85 | 97.35$\pm$0.75 | 93.61$\pm$2.75 | 90.62$\pm$1.59 | 75.27$\pm$1.77 | 21.91$\pm$1.63 | 80.29$\pm$1.43
✓ | Dep. |  99.65$\pm$0.9 |  97.37$\pm$0.48 |  99.62$\pm$0.07 |  95.46$\pm$2.01 |  90.15$\pm$3.88 |  92.55$\pm$1.51 | 21.77$\pm$5.25 |  85.22$\pm$0.87
Table 5: The ablation study result of our modifications on ReaSCAN dataset
test splits. Results are reported on an average of three runs. We evaluate
every combination of components from our best model. W/S stands for weight
sharing, and the ✓shows the presence of the module. Dep in this table refers
to the Dependency masking. We evaluate the model with or without dependency
masking in the masking part.

(a) Environment

(b) Cross-attention w/o masking

(c) Cross-Attention w/ masking

Figure 5: Cross-Attention from Text-to-Image. In Figure (a), the purple zone
indicates the model’s incorrect object selection, while the red zone
highlights the accurate choice. Figures (b) and (c) depict the averaged cross-
attention map from our models over encoder layers and attention heads. The
rows represent environment cells (the first element shows the row, and the
second shows the column index, both starting from 0), and the columns
correspond to text tokens. Brighter attention cells signify elevated attention
weights.

###  5.1 Ablation

For a granular understanding of the contributions from each alteration to the
baseline model, we undertook an ablation study. This involved the sequential
removal of each modification to measure its individual impact. As depicted in
Table 5, while individual modifications did not significantly change the
baseline, their collective integration enhanced the model’s generalization.
Remarkably, eliminating dependency parsing or weight sharing resulted in a
noticeable performance dip. The improvement upon integration posits that
weight sharing can potentially offset the masking prolonged convergence
challenge by reducing parameter count, thereby mitigating the convergence
issues.

###  5.2 Qualitative Analysis

In our previous discussions, we highlighted the significance of integrating
dependency parsing as a fundamental approach to understanding the complex
structures inherent in sentences. This integration is not a mere enhancement;
it critically enriches the model’s grounding capabilities, offering a more
robust bridge between raw textual sequences and their semantic structure.

To provide empirical evidence of our technique for guiding attention, we
conducted an analysis of the cross-attention module. We aimed to compare its
behavior before and after applying attention masking. The results, presented
in Figure 5, indicate a clear trend: in 86% of validation samples, the cross-
attention module exhibits a pronounced focus on the target object.

Figures 5(b) and 5(c) elucidate the impact of self-attention masking on these
weights. After using attention masking (see Figure 5(b)), the attention
distribution becomes notably sparser; instead of individual words attending in
isolation to every potentially relevant cell, they now form cohesive
compositional expressions, each attending to the corresponding cells as a
whole. For instance, in Figure 5(c), "and in the same," phrase’s tokens attend
to cells (1,2), (4,3), and (5,2) together with greater attention on the target
object in contrast to their attention pattern without masking.

Model | #Parameters
---|---
Multimodal LSTM | 74K
Multimodal Transformer | 3M
GroCoT | 4.6M
Dependency† (ours) | 1.9M
Table 6: Comparing model parameters: our model vs. current state-of-the-art
models. Dependency† refers to the model with dependency parsing for attention
masking.

###  5.3 Efficiency Analysis

In the realm of modern model design, the challenge lies in amplifying
capabilities while managing computational overhead. Our methodology adeptly
navigates this balance. A cornerstone of our model’s efficiency is the
strategic adoption of weight sharing within the transformer encoder. By
reusing weights across different components, we significantly reduce the
parameter space. This not only streamlines memory utilization and accelerates
training but also acts as an implicit regularizer, bolstering the model’s
generalization capabilities and reducing overfitting. Further enhancing this
is our implementation of attention masking, which refines computational
efficiency. By enabling the model to selectively bypass attention to certain
tokens, we can optimize the model to avoid redundant computational processes,
ensuring optimal resource allocation and superior performance.

As illustrated in Table 6, our model stands out in terms of efficiency.
Despite having fewer parameters (1.9M) than the models by Qiu et al. (2021a)
and Sikarwar et al. (2022), which have 3M and 4.6M parameters respectively,
our model consistently outperforms them across all benchmarks.

##  6 Conclusion

Our research demonstrated that exploiting the syntactic structure of
compositional and complex linguistic and spatial expressions improved the
grounding ability of the instruction-follower agent in multimodal
environments. Our results indicated improvements compared to the previous
state-of-the-art models. In particular, we show that our proposed model is
effective for generalization on tasks and test splits that require
generalization over unobserved reasoning depths, such as the C1 split in the
ReaSCAN dataset. By utilizing the syntactic-guided attention masking along
with the weight sharing, we achieved not only more accurate but also more
parameter-efficient models for grounded compositional generalization.

## Limitations

Despite the promising results achieved in our study, several limitations
warrant consideration:

Synthetic Data: Our experiments predominantly rely on synthetic datasets.
While these datasets provide a controlled environment for assessing model
performance, they might not capture the complexities and nuances of real-world
data. Evaluating the models on real-world datasets is crucial to ensure their
practical applicability.

Error Propagation from the Parser: The model’s performance is intrinsically
tied to the accuracy of the pre-trained parsers we utilized. Errors or
inaccuracies in parsing can lead to suboptimal model outputs. Additionally,
our synthetic data, being unambiguous, might not reveal the full extent of
potential parser-related issues.

Computational Constraints: Due to computational limitations, the
hyperparameter search might not have been exhaustive. A more comprehensive
exploration might yield better model configurations.

## Acknowledgement

This project is supported by the National Science Foundation (NSF) CAREER
award 2028626 and partially supported by the Office of Naval Research (ONR)
grant N00014-20-1-2005. Any opinions, findings, and conclusions or
recommendations expressed in this material are those of the authors and do not
necessarily reflect the views of the National Science Foundation nor the
Office of Naval Research. We thank all reviewers for their thoughtful comments
and suggestions.

## References

  * Ainslie et al. (2020) Joshua Ainslie, Santiago Ontanon, Chris Alberti, Vaclav Cvicek, Zachary Fisher, Philip Pham, Anirudh Ravula, Sumit Sanghai, Qifan Wang, and Li Yang. 2020.  ETC: Encoding long and structured inputs in transformers.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 268–284, Online. Association for Computational Linguistics.
  * Andreas (2020) Jacob Andreas. 2020.  Good-enough compositional data augmentation.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7556–7566, Online. Association for Computational Linguistics.
  * Chomsky (1957) Noam Chomsky. 1957.  _Syntactic Structures_.  Mouton, The Hague.
  * Csordás et al. (2021) Róbert Csordás, Kazuki Irie, and Juergen Schmidhuber. 2021.  The devil is in the detail: Simple tricks improve systematic generalization of transformers.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 619–634, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Dehghani et al. (2019) Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Łukasz Kaiser. 2019.  Universal transformers.
  * Foscarin et al. (2023) Francesco Foscarin, Daniel Harasim, and Gerhard Widmer. 2023.  Predicting music hierarchies with a graph-based neural decoder.
  * Gao et al. (2020) Tong Gao, Qi Huang, and Raymond Mooney. 2020.  Systematic generalization on gSCAN with language conditioned embedding.  In _Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing_ , pages 491–503, Suzhou, China. Association for Computational Linguistics.
  * Hastie et al. (2009) Trevor Hastie, Robert Tibshirani, and Jerome Friedman. 2009.  _The Elements of Statistical Learning_.  Springer New York.
  * Hearne et al. (2008) Mary Hearne, Sylwia Ozdowska, and John Tinsley. 2008.  Comparing constituency and dependency representations for SMT phrase-extraction.  In _Actes de la 15ème conférence sur le Traitement Automatique des Langues Naturelles. Articles courts, TALN 2008, Avignon, France, June 2008_ , pages 131–140. ATALA.
  * Hupkes et al. (2020) Dieuwke Hupkes, Verna Dankers, Mathijs Mul, and Elia Bruni. 2020.  Compositionality decomposed: How do neural networks generalise? (extended abstract).  In _Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20_ , pages 5065–5069. International Joint Conferences on Artificial Intelligence Organization.  Journal track.
  * Hupkes et al. (2023) Dieuwke Hupkes, Mario Giulianelli, Verna Dankers, Mikel Artetxe, Yanai Elazar, Tiago Pimentel, Christos Christodoulopoulos, Karim Lasri, Naomi Saphra, Arabella Sinclair, Dennis Ulmer, Florian Schottmann, Khuyagbaatar Batsuren, Kaiser Sun, Koustuv Sinha, Leila Khalatbari, Maria Ryskina, Rita Frieske, Ryan Cotterell, and Zhijing Jin. 2023.  State-of-the-art generalisation research in nlp: A taxonomy and review.
  * Jiang and Bansal (2021) Yichen Jiang and Mohit Bansal. 2021.  Inducing transformer’s compositional generalization ability via auxiliary sequence prediction tasks.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 6253–6265, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Keysers et al. (2020) Daniel Keysers, Nathanael Schärli, Nathan Scales, Hylke Buisman, Daniel Furrer, Sergii Kashubin, Nikola Momchev, Danila Sinopalnikov, Lukasz Stafiniak, Tibor Tihon, Dmitry Tsarkov, Xiao Wang, Marc van Zee, and Olivier Bousquet. 2020.  Measuring compositional generalization: A comprehensive method on realistic data.  In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net.
  * Kim et al. (2021a) Juyong Kim, Pradeep Ravikumar, Joshua Ainslie, and Santiago Ontañón. 2021a.  Improving compositional generalization in classification tasks via structure annotations.
  * Kim and Linzen (2020) Najoung Kim and Tal Linzen. 2020.  COGS: A compositional generalization challenge based on semantic interpretation.  _CoRR_ , abs/2010.05465.
  * Kim et al. (2021b) Segwang Kim, Joonyoung Kim, and Kyomin Jung. 2021b.  Compositional generalization via parsing tree annotation.  _IEEE Access_ , 9:24326–24333.
  * Kingma and Ba (2017) Diederik P. Kingma and Jimmy Ba. 2017.  Adam: A method for stochastic optimization.
  * Korrel et al. (2019) Kris Korrel, Dieuwke Hupkes, Verna Dankers, and Elia Bruni. 2019.  Transcoding compositionally: Using attention to find more generalizable solutions.  In _Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_ , pages 1–11, Florence, Italy. Association for Computational Linguistics.
  * Kuo et al. (2021) Yen-Ling Kuo, Boris Katz, and Andrei Barbu. 2021.  Compositional networks enable systematic generalization for grounded language understanding.  In _Findings of the Association for Computational Linguistics: EMNLP 2021_ , pages 216–226, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Lake and Baroni (2018) Brenden M. Lake and Marco Baroni. 2018.  Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks.
  * Mao et al. (2019) Jiayuan Mao, Chuang Gan, Pushmeet Kohli, Joshua B. Tenenbaum, and Jiajun Wu. 2019\.  The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervision.  In _International Conference on Learning Representations_.
  * Montague (1970) Richard Montague. 1970.  Universal grammar.  _Theoria_ , 36(3):373–398.
  * Ontanon et al. (2022) Santiago Ontanon, Joshua Ainslie, Zachary Fisher, and Vaclav Cvicek. 2022.  Making transformers solve compositional tasks.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 3591–3607, Dublin, Ireland. Association for Computational Linguistics.
  * Paszke et al. (2019) Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019.  Pytorch: An imperative style, high-performance deep learning library.
  * Qi et al. (2020) Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning. 2020\.  Stanza: A Python natural language processing toolkit for many human languages.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations_.
  * Qiu et al. (2021a) Linlu Qiu, Hexiang Hu, Bowen Zhang, Peter Shaw, and Fei Sha. 2021a.  Systematic generalization on gSCAN: What is nearly solved and what is next? In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 2180–2188, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Qiu et al. (2022) Linlu Qiu, Peter Shaw, Panupong Pasupat, Pawel Nowak, Tal Linzen, Fei Sha, and Kristina Toutanova. 2022.  Improving compositional generalization with latent structure and data augmentation.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 4341–4362, Seattle, United States. Association for Computational Linguistics.
  * Qiu et al. (2021b) Yao Qiu, Jinchao Zhang, and Jie Zhou. 2021b.  Improving gradient-based adversarial training for text classification by contrastive learning and auto-encoder.  In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , pages 1698–1707, Online. Association for Computational Linguistics.
  * Ruis et al. (2020) Laura Ruis, Jacob Andreas, Marco Baroni, Diane Bouchacourt, and Brenden M. Lake. 2020.  A benchmark for systematic generalization in grounded language understanding.
  * Shaw et al. (2021) Peter Shaw, Ming-Wei Chang, Panupong Pasupat, and Kristina Toutanova. 2021.  Compositional generalization and natural language variation: Can a semantic parsing approach handle both? In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 922–938, Online. Association for Computational Linguistics.
  * Sikarwar et al. (2022) Ankur Sikarwar, Arkil Patel, and Navin Goyal. 2022.  When can transformers ground and compose: Insights from compositional generalization benchmarks.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 648–669, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Srivastava et al. (2014) Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014.  Dropout: A simple way to prevent neural networks from overfitting.  _Journal of Machine Learning Research_ , 15:1929–1958.
  * Thomas McCoy et al. (2020) R. Thomas McCoy, Ellie Pavlick, and Tal Linzen. 2020.  Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference.  In _ACL 2019 - 57th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference_ , ACL 2019 - 57th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference, pages 3428–3448. Association for Computational Linguistics (ACL).
  * Wu et al. (2021) Zhengxuan Wu, Elisa Kreiss, Desmond C. Ong, and Christopher Potts. 2021.  ReaSCAN: Compositional reasoning in language grounding.  _NeurIPS 2021 Datasets and Benchmarks Track_.

##  Appendix A Hyperparameters

Here, we present the hyperparameters used in the models for every benchmark in
Table 7.

Hyperparameter | gSCAN | GSRR | ReaSCAN
---|---|---|---
Number of Vision Self-Attention Layers | $3$ | $3$ | $6$
Number of Text Self-Attention Layers | $3$ | $3$ | $6$
Number of Cross-Attention Layers | $3$ | $3$ | $6$
Number of Decoder Layers | $3$ | $3$ | $6$
Embedding Size | $128$ | $128$ | $128$
Hidden Layer Size | $256$ | $256$ | $256$
Number of Attention Heads | $8$ | $8$ | $8$
Learning Rate | $5\times 10^{-5}$ | $1\times 10^{-5}$ | $1\times 10^{-5}$
Batch Size | $64$ | $64$ | $32$
Dropout | $0.1$ | $0.1$ | $0.1$
Number of Epochs | $100$ | $100$ | $120$
Table 7: Hypterparameters used in the experiments.

##  Appendix B Datasets Description

###  B.1 Grounded SCAN dataset

The Grounded SCAN (gSCAN) dataset is a pivotal benchmark for assessing
compositional generalization in machine learning models. Evolving from the
foundational SCAN Lake and Baroni (2018) dataset, gSCAN is designed to
evaluate a model’s proficiency in translating command sequences into actions
within a grid world environment, emphasizing on compositional challenges.

This benchmark offers systematic test splits that rigorously examine a model’s
capability to generalize beyond its training data. These compositional splits
include:

  * •

A (Random): Random data with a similar distribution to the training data.

  * •

B (Color-Shape): Novel composition of object properties in the testing. Yellow
squares are referred to by color and shape.

  * •

C (Color Only): Red squares as target.

  * •

D (Novel Direction): Challenges a model’s spatial comprehension, with targets
set in unfamiliar directions, the southwest.

  * •

E (Novel Contextual References): Evaluates a model’s understanding of relative
sizes, with commands pointing to circles of size 2 described as "small."

  * •

F (Novel Composition of Actions and Arguments): Probes a model’s grasp of
object classes and their nuances, exemplified by squares of size 3
necessitating two pushes.

  * •

G (Adverb): Commands carrying the adverb "cautiously" test how well the model
interprets action modifiers after seeing limited training samples (k=1).

  * •

H (Adverb-Verb Combination): Generalizes to commands pairing actions and their
modifiers, like "while spinning" combined with "pull."

The compositional test splits of the gSCAN dataset ensure that models are not
indulging in learning statistical shortcuts but are genuinely mastering
compositional reasoning. In gSCAN, every command is mapped to an action
sequence for an agent in the grid world, whether moving to a particular spot
or interacting with a distinct described object.

###  B.2 Grounded Systematic Relation Reasoning dataset (GSRR)

The Grounded Systematic Relation Reasoning (GSRR) dataset, introduced by Qiu
et al. (2021b), extends the gSCAN benchmark. Their initial analyses of the
gSCAN dataset indicated its efficacy; the authors observed that several
remaining challenges might not be primarily tied to visual grounding. In light
of this, they proposed the GSRR task, characterized by an elevated complexity
in aligning natural language instructions with the visual environment.

In this dataset, language expressions specifically delineate target objects
and explicitly describe their relationships with a secondary referenced
object. They incorporate two types of relations into our dataset: immediate
adjacency ("next to") and cardinal directions such as "north" and "west." In
addition, they put visual distractors objects within the environment to
emphasize the critical role of spatial relations in identifying the target
objects.

The dataset is systematically divided into various splits to ensure a
comprehensive assessment:

  * •

I (Random): Similar distribution as the training.

  * •

II (Visual): Commands centering on "red squares" either as targets or
references.

  * •

III (Relation): Complex instructions involving combinations like "green
squares" and "blue circles."

  * •

IV (Referent) Emphasizing "yellow squares" as primary targets.

  * •

V (Relative Position 1): Commands where targets are situated to the "north" of
their reference points.

  * •

VI (Relative Position 2): Instructions where targets are located "southwest"
relative to their references.

##  Appendix C Evaluation Card

Here, we present the evaluation card of our compositional generalization
experiments based on Hupkes et al. (2023) taxonomy.

Motivation
---
Practical Cognitive Intrinsic Fairness $\square$ $\square$
Generalisation type
Compo- sitional Structural Cross Task Cross Language Cross Domain Robust- ness
$\square$ $\square$
Shift type
Covariate Label Full Assumed $\square$
Shift source
Naturally occuring Partitioned natural Generated shift Fully generated
$\square$
Shift locus
Train–test Finetune train–test Pretrain–train Pretrain–test $\square$