# Combining Attention with Flow for Person Image Synthesis

Yurui Ren  School of Electronic and Computer Engineering, Peking University
<EMAIL_ADDRESS>,  Yubo Wu  School of Electronic and Computer Engineering,
Peking University ,  Thomas H. Li  AIIT, Peking University
ITRDIT, Peking University ,  Shan Liu  Tencent America<EMAIL_ADDRESS>and
Ge Li  School of Electronic and Computer Engineering, Peking University
<EMAIL_ADDRESS>
(2021)

###### Abstract.

Pose-guided person image synthesis aims to synthesize person images by
transforming reference images into target poses. In this paper, we observe
that the commonly used spatial transformation blocks have complementary
advantages. We propose a novel model by combining the attention operation with
the flow-based operation. Our model not only takes the advantage of the
attention operation to generate accurate target structures but also uses the
flow-based operation to sample realistic source textures. Both objective and
subjective experiments demonstrate the superiority of our model. Meanwhile,
comprehensive ablation studies verify our hypotheses and show the efficacy of
the proposed modules. Besides, additional experiments on the portrait image
editing task demonstrate the versatility of the proposed combination.

Person Image Synthesis, Image Spatial Transformation, Correspondence
Estimation

††journalyear: 2021††copyright: acmcopyright††conference: Proceedings of the
29th ACM International Conference on Multimedia; October 20–24, 2021; Virtual
Event, China††booktitle: Proceedings of the 29th ACM International Conference
on Multimedia (MM ’21), October 20–24, 2021, Virtual Event, China††price:
15.00††doi: XX.XXXX/XXXXXXX.XXXXXXX††isbn: 978-1-4503-8651-7/21/10††ccs:
Computer systems organization Embedded systems††ccs: Computer systems
organization Redundancy††ccs: Computer systems organization Robotics††ccs:
Networks Network reliability

Figure 1. The generated examples and the corresponding visualizations of the
deformations obtained by the attention and flow-based operations. Our model
takes the complementary advantages of flow and attention.

The generated examples of our model.

##  1\. Introduction

Being able to synthesize person images by transforming the poses of given
persons is an important task with a large variety of applications. Industries
such as electronic commerce, virtual reality, film production, and next-
generation communication require such algorithms to generate content. In many
cases, however, these requirements are achieved by graphic technologies with
precise control over the image rendering. Editing images in this way needs
professionals to build fine-grained 3D models for each scene. This complex and
tedious process prevents ordinary users from these algorithms and meanwhile
increases the costs of the generated contents.

Recently, advances in computer vision fields have made tremendous progress in
generating realistic images (Goodfellow et al., 2014; Brock et al., 2018;
Karras et al., 2019; Karras et al., 2020). Some algorithms (Ma et al., 2017;
Siarohin et al., 2018; Ren et al., 2020; Men et al., 2020) are proposed to
automatically synthesize person images from references using learning-based
methods. Formally, the pose-guided person image synthesis task aims to
synthesize person images by transforming the poses of reference images
according to the given modifications while preserving the reference
identities. Examples are provided in Fig. 1. It can be seen that the reference
and target images in this task have clear mapping relationships: targets are
the spatial transformation versions of the reference images. Therefore, this
task can be tackled by reassembling the references in the spatial domain.
However, Convolutional Neural Networks (CNNs) lack the abilities to enable
efficient spatial transformation (Goodfellow et al., 2016; Vaswani et al.,
2017). Convolutional operations are building blocks that process one local
neighborhood at a time. To model long-term dependencies, stacks of
convolutional operations are required. Realistic source textures will be “wash
away” during these operations, which results in over-smoothed images.
Therefore, a fundamental challenge for this task is to design efficient
spatial transformation blocks to reassemble the reference images.

The attention operation has been proved as an effective method to extract non-
local dependencies (Vaswani et al., 2017; Wang et al., 2018; Zhang et al.,
2019). The response of a target query is calculated as the weighted sum of
source features. By using this operation, each target feature directly
communicates with all source features. Thus, targets can be expected to sample
specific source features by increasing the weights of the corresponding
regions and refusing the other features. However, to generate images with
realistic textures, each output position should only sample a very local
region of sources. This requires the attention correlation matrix to be a
sparse matrix to reject all the unsampled features, which is extremely
difficult for the standard attention operation. Meanwhile, this position-
irreverent operation cannot maintain the patterns of the reference images
(_e.g_. logos).

Another efficient spatial transformation operation is the flow-based
operation. This operation warps the source information by predicting 2D
coordinate offsets specifying the sampling positions. Different from the
attention operation, the flow-based operation can obtain photo-realistic
textures since it sampling a very local source patch for each target position.
However, it is hard for the networks to obtain stable gradients from the flow-
based operation because each output feature is only related to a local and
indeterminate source patch. This phenomenon will hinder the models to extract
accurate motions, which is more evident when complex deformations and severe
occlusions are observed.

Observing the complementary advantages of these two operations, in this paper,
we propose a novel model by combining the attention operation with the flow-
based operation. The architecture of the proposed model is shown in Fig. 2 and
Fig. 3. Specifically, a Deformation Estimation Module is first designed to
extract the deformations between the reference images and the desired targets.
Two types of deformations are estimated: the correlation matrices for the
attention operation and the flow fields for the flow-based operation. Then, we
generate a combination map that is responsible for selecting the better
deformation between the correlation matrices and the flow-fields for each
target position. Finally, an Image Synthesis Module is employed to synthesize
the target images by reassembling the source feature maps according to the
estimated deformations and combination maps.

We compare the proposed model with several state-of-the-art methods. The
experiment results show the superiority of our model. Meanwhile, comprehensive
ablation studies are conducted to verify the hypothesis and show the efficacy
of the proposed modules. Besides, we further apply our model to tackle the
portrait image editing task. We show that the proposed model can achieve
intuitive portrait image control by modifying the poses and expressions of
reference images according to the provided modifications. The main
contributions of our paper can be summarized as

  * •

We propose a novel model for person image synthesis by combining the attention
operation with the flow-based operation. Taking the complementary advantages
of these operations, our model can synthesize images with not only accurate
structures but also realistic details.

  * •

We demonstrate the versatility of the proposed model by further extending it
to tackle the portrait image editing task. Experiments show that our model can
synthesize portrait images with accurate movements.

Figure 2. The architecture of the deformation estimation module. The
deformations are estimated by the attention correlation estimator and the flow
field estimator. Then, the combination maps are generated using the
corresponding warped images. Our model estimates multi-scale deformations to
transform both global and local contexts.

The architecture of the deformation estimation module.

##  2\. Related Work

Recently, deep neural networks are starting to produce visually compelling
images conditioned on certain user specifications like segmentation and edge
map. (Isola et al., 2017; Choi et al., 2018; Huang et al., 2018; Liu et al.,
2019a; Yu et al., 2019). The pose-guided person image synthesis task is a
highly active topic in this field, where the target images are synthesized by
rendering the corresponding skeletons with the appearance of the reference
images. Ma _et al_. (Ma et al., 2017) tackle this task by proposing a coarse-
to-fine framework. Their framework first synthesizes coarse images with
accurate poses and then refine the results by adding vivid textures in an
adversarial way. Some follow-up works (Ma et al., 2018; Esser et al., 2018)
manage to disentangle the poses and appearance of the reference images to
improve the results. However, these methods use 1D embeddings to represent the
appearance, which hinders the generation of complex textures. Men _et al_.
(Men et al., 2020) alleviate this problem by extracting appearance from
different semantics separately. The extracted embeddings are then injected
into the feature maps of target skeletons to generate the final results.
Instead of using the same embeddings for all target positions, Zhang _et al_.
(Zhang et al., 2021) propose to inject the style of each semantic part into
the corresponding target semantic regions. With the representative appearance
embeddings, these methods can generate images with realistic textures.
However, they cannot maintain the patterns of the reference images. Meanwhile,
these methods rely on accurate human parsing maps. The performance may be
vulnerable to parsing errors.

Some other methods tackle this task by proposing efficient spatial
transformation modules (Siarohin et al., 2018; Wang et al., 2019; Liu et al.,
2019b; Zablotskaia et al., 2019; Ren et al., 2020; Tang et al., 2021).
Siarohin _et al_. (Siarohin et al., 2018) introduce deformable skip
connections to spatially transform the source neural textures with a set of
affine transformations. This method relieves the spatial misalignment caused
by pose difference and achieves good results. However, it requires one to
predefine a set of transformation components, which limits the application.
Zhu _et al_. (Zhu et al., 2019) use cascaded attention blocks to transfer the
source information progressively. This method can generate accurate structures
for target images. But complex textures are smoothed during multiple
transfers, which leads to the performance decline. Li _et al_. (Li et al.,
2019) propose a flow-based method for this task. To estimate accurate
deformations, they generate flow field labels with additional 3D human
reconstruction methods. However, the performance of this model is limited by
the accuracy of the 3D reconstruction model. Han _et al_. (Han et al., 2019)
propose a cascaded flow estimator to predict flow fields in an unsupervised
manner. However, this method warps the sources at the pixel level, which means
that additional refinement networks are required to fill the holes caused by
occlusions. Ren _et al_. (Ren et al., 2020) propose a global-flow local-
attention framework to reassemble the input image at the feature level. Tang
_et al_. (Tang et al., 2021) further improve the results by proposing a
structure-aware person image synthesis method that predicts flow fields of
different body semantics separately. Recently, some methods (Zhang et al.,
2020; Zhou et al., 2021) achieve spatial transformation by using the attention
operation. The correspondences between the sources and targets are extracted
by leaning attention correlation metrics. However, the attention operation
cannot maintain the spatial distributions of the reference images, which
hinders the methods remonstrating complex textures.

##  3\. Our Approach

We propose a novel model for the pose-guided person image synthesis task. The
motivation of our model comes from the observation that the commonly used
spatial transformation operations have complementary advantages. As shown in
Fig. 1, the flow-based operation can extract vivid source textures by
assigning a very local patch for each target position. However, it lacks the
ability to capture complex deformations between sources and targets. On the
contrary, the attention operation can extract accurate deformations and
synthesize targets with reasonable structures. However, it cannot maintain the
source textures. Therefore, by combining these two operations, our model can
synthesize images with not only accurate global structures but also realistic
local details. In the following, we first provide the details of our
deformation estimation module (Sec. 3.1). Then, the image synthesis module is
introduced for the target image synthesis (Sec. 3.2). Finally, we explain the
training functions (Sec. 3.3). Please note that we describe the model warping
source features at a single scale for the simplicity of the discussion. Our
model can be extended by warping multi-scale source features to transform both
global and local contexts.

###  3.1. Deformation Estimation Module

A fundamental challenge of the pose-guided person image synthesis task is to
accurately reassemble the source information according to the provided
modifications. This requires one to estimate the correspondence between the
reference image $\mathbf{I}_{r}$ and the desired target image
$\mathbf{I}_{t}$. We deal with this task using a deformation estimation
module. The architecture of this module is shown in Fig. 2. It consists of
three parts: the attention correlation estimator, the flow field estimator,
and the combination map generator.

The Attention Correlation Estimator is responsible for calculating the
attention correlation matrix $\mathbf{C}$ that contains the correlations of
all queries to all keys. This estimator first encodes the reference image
$\mathbf{x}_{r}$ and the target skeleton $\mathbf{s}_{t}$ to feature maps
using encoders $E_{x}$ and $E_{s}$ respectively.

(1) $\displaystyle\mathbf{k}$ $\displaystyle=E_{x}(\mathbf{x}_{r})$
$\displaystyle\mathbf{q}$ $\displaystyle=E_{s}(\mathbf{s}_{t})$

where $\mathbf{k}\in\mathbb{R}^{H\times W\times K}$ and
$\mathbf{q}\in\mathbb{R}^{H\times W\times K}$ are the extracted feature maps
representing keys and queries, respectively. $H$ and $W$ denote the spatial
size of the feature map and $K$ is the number of feature channels. Then, the
correlation matrix $\mathbf{C}\in\mathbb{R}^{HW\times HW}$ is obtained as

(2)
$\mathbf{C}^{i,j}=\frac{\text{exp}(\alpha\beta^{i,j})}{\sum_{i\in\Omega}\text{exp}(\alpha\beta^{i,j})},\text{where
}\beta^{i,j}={\mathbf{k}^{i}}^{T}\mathbf{q}^{j}$

where $\Omega$ is the coordinate set of the feature maps. Symbols
$\mathbf{k}^{i}\in\mathbb{R}^{K}$ and $\mathbf{q}^{j}\in\mathbb{R}^{K}$ denote
the feature located in $i$ of the reference feature map $\mathbf{k}$ and the
feature located in $j$ of the skeleton feature map $\mathbf{q}$ respectively.
Coefficient $\alpha$ is used to control the sharpness of the softmax
operation. In this paper, we set $\alpha=100$ for all experiments. With matrix
$\mathbf{C}$, we can calculate the attention results
$\mathbf{o}\in\mathbb{R}^{H\times W\times K^{\prime}}$ by weighted sum the
source inputs $\mathbf{x}\in\mathbb{R}^{H\times W\times K^{\prime}}$.

(3)
$\mathbf{o}^{j}=\mathcal{W}_{a}(\mathbf{x},\mathbf{C})^{j}=\sum_{i\in\Omega}\mathbf{C}^{i,j}\mathbf{x}^{i}$

where $\mathcal{W}_{a}(*,*)$ indicates the attention warping operation. By
using the attention operation, the source information can be integrated into
desired target positions according to matrix $\mathbf{C}$.

Figure 3. The architecture of the image synthesis module. The feature maps of
the reference images are warped using the estimated deformations. Then, the
network generates the final images by rendering the target skeletons with
aligned neural textures.

The architecture of the image generation module.

The Flow Field Estimator is responsible for estimating flow fields that
contain the relative movements between the sources and targets. Different from
the attention operation, the flow-based operation forces the correlation
matrices to be sparse matrices by sampling a local source region for each
output. Therefore, this operation can help with reconstructing vivid source
textures. To be specific, the estimator $F$ takes the reference image
$\mathbf{x}_{r}$, the reference skeleton $\mathbf{s}_{r}$, and the target
skeleton $\mathbf{s}_{t}$ as the inputs. The flow fields
$\mathbf{w}\in\mathbb{R}^{H\times W\times 2}$ are generated by analyzing the
difference between the reference images and the desired targets.

(4) $\mathbf{w}=F(\mathbf{s}_{t},\mathbf{s}_{r},\mathbf{x}_{r})$

We design $F$ with an auto-encoder structure. It first extracts features from
the inputs and then decodes them into flow fields according to the extracted
features. Several skip-connections are used for leveraging both local and
global contexts. After obtaining $\mathbf{w}$, the output results of the flow-
based operation can be obtained by

(5) $\mathbf{o}=\mathcal{W}_{f}(\mathbf{x},\mathbf{w})$

where $\mathcal{W}_{f}(*,*)$ indicates the flow-based warping operation which
samples the input $\mathbf{x}$ with flow fields $\mathbf{w}$ using the
Bilinear interpolation method.

The Combination Map Generator predicts the combination maps to select from the
warping results of the attention operation and the warping results of the
flow-based operation. As mentioned before, the attention operation and the
flow-based operation have complementary advantages. Therefore, reasonably
combining their results is important for exploiting their strengths and thus
improve the quality of the final images. Here, we tackle this task by
generating combination maps $\mathbf{m}\in\mathbb{R}^{H\times W\times 1}$
using a content-aware combination map generator $M$.

(6)
$\mathbf{m}=M(\mathcal{W}_{a}(\mathbf{x}_{r}^{\mathrel{\scalebox{0.8}[0.5]{$\downarrow$}}},\mathbf{C}),\mathcal{W}_{f}(\mathbf{x}_{r}^{\mathrel{\scalebox{0.8}[0.5]{$\downarrow$}}},\mathbf{w}))$

where $\mathbf{x}_{r}^{\mathrel{\scalebox{0.8}[0.5]{$\downarrow$}}}$ is the
resized image of the original reference image $\mathbf{x}_{r}$. We design
generator $M$ with several residual blocks. The Sigmoid activation function is
used as the non-linear function of the output layer. The combination maps
$\mathbf{m}$ have continuous values between $0$ and $1$. With the deformations
$\mathbf{C}$, $\mathbf{w}$ and the combination maps $\mathbf{m}$, we can
generate the final images by spatially transforming the source textures.

###  3.2. Image Synthesis Module

The image synthesis module $G$ is used to synthesize the final images by
rendering target skeletons with reference textures. The architecture of this
module is shown in Fig. 3. Specifically, this module takes $\mathbf{x}_{r}$,
$\mathbf{s}_{t}$, $\mathbf{C}$, $\mathbf{w}$, and $\mathbf{m}$ as inputs and
synthesizes the predicted images $\hat{\mathbf{x}}_{t}$.

(7)
$\hat{\mathbf{x}}_{t}=G(\mathbf{x}_{r},\mathbf{s}_{t},\mathbf{C},\mathbf{w},\mathbf{m})$

The warping block in module $G$ is responsible for reassembling the reference
neural textures according to the estimated deformations. Let $\mathbf{f}_{r}$
denotes the feature map extracted from the reference image $\mathbf{x}_{r}$.
This block generates the aligned feature map $\mathbf{f}_{a}$ by first warping
$\mathbf{f}_{r}$ with both the attention correlation matrix $\mathbf{C}$ and
the flow field $\mathbf{w}$ and then combining the warped results with the
combination map $\mathbf{m}$.

(8)
$\mathbf{f}_{a}=\mathbf{m}\otimes\mathcal{W}_{a}(\mathbf{f}_{r},\mathbf{C})+(1-\mathbf{m})\otimes\mathcal{W}_{f}(\mathbf{f}_{r},\mathbf{w})$

where $\otimes$ denotes the element-wise multiplication over the spatial
domain. After obtaining $\mathbf{f}_{a}$, the target image
$\hat{\mathbf{x}}_{t}$ is generated by adding vivid neural textures to the
feature map $\mathbf{f}_{t}$ extracted from the target skeleton
$\mathbf{s}_{t}$

(9) $\mathbf{f}_{o}=\mathbf{f}_{t}+\mathbf{f}_{a}$

where $\mathbf{f}_{o}$ is the output feature map containing both target
semantics and reference textures. We further decode $\mathbf{f}_{o}$ to
synthesize $\hat{\mathbf{x}}_{t}$

###  3.3. Training Losses

The proposed model is trained with several loss functions that fulfill special
tasks. These loss functions can be divided into two categories: losses for
accurate deformations and losses for realistic images.

Losses for Accurate Deformations. Estimating accurate deformations between
sources and targets is crucial for generating realistic images. However, using
losses at the end of the network (_e.g_. reconstruction loss) cannot guarantee
that the model learns meaningful deformations. Therefore, several losses are
designed to directly constrain the estimated deformations. For the attention
correlation matrix $\mathbf{C}$, we calculate the $\ell_{1}$ distance between
the warped image and the target image.

(10)
$\mathcal{L}_{attn}=\lVert\mathcal{W}_{a}(\mathbf{x}_{r}^{\mathrel{\scalebox{0.8}[0.5]{$\downarrow$}}},\mathbf{C}),\mathbf{x}_{t}^{\mathrel{\scalebox{0.8}[0.5]{$\downarrow$}}}\rVert_{1}$

This attention loss $\mathcal{L}_{attn}$ encourages the correlation matrix
$\mathbf{C}$ containing meaningful deformations to reduce the reconstruction
errors. To constrain the flow field $\mathbf{w}$, we employ the sampling
correctness loss $\mathcal{L}_{flow}$ and the regularization loss
$\mathcal{L}_{regu}$ proposed in paper (Ren et al., 2020). The sampling
correctness loss calculates the normalized cosine similarity between the
warped reference feature and the ground-truth target feature. We use VGG-19 to
extract the corresponding features $\mathbf{v}_{r}$, $\mathbf{v}_{t}$ from
images $\mathbf{x}_{r}$, $\mathbf{x}_{t}$. This loss is defined as

(11)
$\mathcal{L}_{flow}=\frac{1}{N}\sum_{l\in\Omega}exp[-\frac{\mu\big{(}\mathcal{W}_{f}(\mathbf{v}_{r},\mathbf{w})^{l},\mathbf{v}_{t}^{l}\big{)}}{\mu\big{(}\mathbf{v}_{r}^{l_{max}},\mathbf{v}_{t}^{l}\big{)}}]$

where $\mu(*)$ indicates the cosine similarity operation. Coordinate set
$\Omega$ contains all $N$ positions in the feature maps. For each position
$l$, this loss first calculates the similarity between the warped reference
feature $\mathcal{W}_{f}(\mathbf{v}_{r},\mathbf{w})^{l}$ and the ground-truth
target feature $\mathbf{v}_{t}^{l}$. Then, the similarity is normalized by
$\mu(\mathbf{v}_{r}^{l^{\prime}},\mathbf{v}_{t}^{l})$ to avoid the bias
brought by occlusion, where $\mathbf{v}_{r}^{l_{max}}$ is the most similar
feature of $\mathbf{v}_{t}^{l}$ in $\mathbf{v}_{r}$.

(12)
$l_{max}=\operatorname*{arg\,max}_{i\in\Omega}\mu(\mathbf{v}_{r}^{i},\mathbf{v}_{t}^{l})$

The regularization loss $\mathcal{L}_{regu}$ is used to extract the spatial
correlations of the flow fields $\mathbf{w}$. It assumes that each local
deformation estimated by $\mathbf{w}$ should be an affine transformation. Let
$\mathbf{R}_{l}=\left[{\begin{array}[]{cccc}x_{1}&x_{2}&...&x_{n\times n}\\\
y_{1}&y_{2}&...&y_{n\times n}\\\ 1&1&...&1\end{array}}\right]$ be the
reference coordinates of a local $n\times n$ patch centered at location $l$.
The coordinates of the sampling points can be calculated using the
corresponding flow field patch $\mathbf{w}_{l}$ with
$\mathbf{S}_{l}=\mathbf{R}_{l}+\mathbf{w}^{*}_{l}$, where $\mathbf{w}^{*}_{l}$
is the homogeneous coordinates of $\mathbf{w}_{l}$. This loss assumes a linear
relationship between $\mathbf{R}_{l}$ and $\mathbf{S}_{l}$ and calculates the
least-square error.

(13)
$\mathcal{L}_{regu}=\sum_{l\in\Omega}\left\lVert\mathbf{R}_{l}-\hat{\mathbf{A}}_{l}\mathbf{S}_{l}\right\lVert_{2}^{2}$

where the metric $\hat{\mathbf{A}}_{l}$ is the least-square solution of the
linear equation $\mathbf{R}_{l}=\mathbf{A}_{l}\mathbf{S}_{l}$. It can be
calculated as

(14)
$\hat{\mathbf{A}}_{l}=\mathbf{R}_{l}\mathbf{S}_{l}^{T}(\mathbf{S}_{l}\mathbf{S}_{l}^{T})^{-1}$

Losses for Realistic Images. After obtaining accurate deformations, our model
synthesizes the final images with the warped features. Several losses are
designed to obtain realistic images. The perceptual loss proposed in paper
(Johnson et al., 2016) is used to calculate the reconstruction error between
the predicted image $\hat{\mathbf{x}}_{t}$ and the ground-truth image
$\mathbf{x}_{t}$.

(15)
$\mathcal{L}_{perc}=\sum_{i}\lVert\phi_{i}(\mathbf{x}_{t}),\phi_{i}(\hat{\mathbf{x}}_{t})\rVert_{1}$

where $\phi_{i}$ denotes the $i$-th activation map of the VGG-19 network.
Besides, a face reconstruction loss is used for generating natural faces. This
loss calculates the difference between features of the cropped faces.

(16)
$\mathcal{L}_{face}=\sum_{i}\lVert\phi_{i}\big{(}C_{face}(\mathbf{x}_{t})\big{)},\phi_{i}\big{(}C_{face}(\hat{\mathbf{x}}_{t})\big{)}\rVert_{1}$

where $C_{face}(*)$ is the face cropping function. To generate vivid details,
we use a style loss to calculate the statistical error between the activation
maps.

(17) $\mathcal{L}_{style}=\sum_{j}\lVert
G^{\phi}_{j}(\mathbf{x}_{t}),G^{\phi}_{j}(\hat{\mathbf{x}}_{t})\rVert_{1}$

where $G^{\phi}_{j}$ represents the Gram matrix of activation map $\phi_{j}$.
In addition to the VGG-based losses, a generative adversarial loss is employed
to mimic the distribution of ground-truth images.

(18)
$\mathcal{L}_{adv}=\mathbb{E}(1-D(\hat{\mathbf{x}}_{t})+\mathbb{E}(D(\mathbf{x}_{t}))$

where $D$ is the discriminator. We use the following overall loss to train our
model.

(19) $\displaystyle\mathcal{L}$
$\displaystyle=\lambda_{attn}\mathcal{L}_{attn}+\lambda_{flow}\mathcal{L}_{flow}+\lambda_{regu}\mathcal{L}_{regu}$
$\displaystyle+\lambda_{perc}\mathcal{L}_{perc}+\lambda_{face}\mathcal{L}_{face}+\lambda_{style}\mathcal{L}_{style}$
$\displaystyle+\lambda_{adv}\mathcal{L}_{adv}$

##  4\. Experiment

Dataset. The In-shop Clothes Retrieval Benchmark of the DeepFashion dataset
(Liu et al., 2016) is used in our experiments. This dataset contains $52712$
high-resolution images ($256\times 256$) of fashion models with different
clothing items in different poses. Images of the same person in the same
clothes are paired for training and testing. We split the dataset according to
the personal identity so that the identities of the training and testing sets
do not overlap. A total of $101,966$ pairs are randomly selected for training
and $8,570$ pairs for testing.

Implementation Details. In our experiments, reference features with
resolutions as $32\times 32$ and $64\times 64$ are extracted and warped to
generate the final results. Considering the huge deformations between the
reference images and target images, we train the model in stages to avoid it
getting stuck in bad local minimas. The deformation estimation module is first
pre-trained. Then we train the whole model in an end-to-end manner. The batch
size is set to $24$ for all experiments. We use the historical average
technique (Salimans et al., 2016) to update the average model by weighted
averaging current parameters with previous parameters. More details can be
found in the Supplementary Materials.

|  VU-Net |  Def-GAN |  Pose-Attn |  Intr-Flow |  ADGAN |  GFLA |  Ours
---|---|---|---|---|---|---|---
|  |  |  |  |  |  |
SSIM $\uparrow$ |  0.6738 |  0.6836 |  0.6714 |  0.6968 |  0.6736 |  0.7074 |  0.7113
LPIPS $\downarrow$ |  0.2637 |  0.2330 |  0.2533 |  0.1875 |  0.2250 |  0.1962 |  0.1813
FID $\downarrow$ |  23.669 |  18.460 |  20.728 |  13.014 |  14.546 |  9.9125 |  9.4502
FR $\uparrow$ |  4.12 |  14.40 |  9.56 |  16.80 |  29.08 |  18.88 |  30.00
Table 1. The evaluation results compared with several state-of-the-art
methods. SSIM and LPIPS calculate the reconstruction errors. FID indicates the
realism of the generated images. Fooling rate (FR) is obtained by human
subjective studies. It represents the probability that the generated images
are mistaken for real images.

###  4.1. Comparisons with State-of-the-arts

We compare our model with several state-of-the-art methods including VU-Net
(Esser et al., 2018), Def-GAN (Siarohin et al., 2018), Pose-Attn(Zhu et al.,
2019), Intr-Flow (Li et al., 2019), ADGAN (Men et al., 2020), and GFLA (Ren et
al., 2020). The released weights of these methods are used for evaluation.

Figure 4. The qualitative comparisons with several state-of-the-art methods
including VU-Net (Esser et al., 2018), Def-GAN (Siarohin et al., 2018), Pose-
Attn(Zhu et al., 2019), Intr-Flow (Li et al., 2019), ADGAN (Men et al., 2020),
and GFLA (Ren et al., 2020)

The qualitative comparisons.

Figure 5. More results of our model. Deformations and combination maps
estimated for $64\times 64$ feature maps are provided. The visualizations are
resized for illustration.

More results of our model.

Figure 6. The qualitative results of the ablation study. We mark some typical
artifacts with red arrows.

The generated examples of the ablation models.

Qualitative Results. We provide the typical qualitative results in Fig. 4 for
intuitive comparisons. It can be seen that without using spatial
transformation blocks, VU-Net fails to reconstruct the details of the clothes.
Def-GAN and Pose-Attn can generate images with relatively reasonable
structures. However, obvious artifacts can be observed in images with complex
textures. ADGAN first extracts the semantic embeddings from the reference
images and then injects the extracted variables into target skeletons. This
model can generate realistic results for solid color clothes. However, since
the extracted 1D embeddings do not contain the image spatial information, it
struggles to reconstruct complex textures.

Using the flow-based operation as the spatial transformation modules, Intr-
Flow and GFLA can extract vivid neural textures from the reference images and
generate results with realistic details. However, their results may suffer
from inaccurate structures, which is more evident when complex deformations
and severe occlusions are observed. The main reason is that the poor gradients
provided by the warping operation hinder the model to estimate accurate
deformation for each position. By combining the flow-based operation with the
attention operation, our model can generate images with not only accurate
structures but also vivid textures. Besides, we provide more results of our
model in Fig. 5. The corresponding combination maps, as well as the
visualizations of the deformations are also given. It can be seen that the
deformations estimated by the flow fields are selected to generate complex
textures. For the smooth areas (_e.g_. arms, legs), the deformations estimated
by the attention correlation matrices are selected. Thus, our model can use
the reference information to synthesize realistic results.

Quantitative Results. The evaluation results are shown in Tab. 1. We use
Structure Similarity Index Measure (SSIM) (Wang et al., 2004) and Learned
Perceptual Image Patch Similarity (LPIPS) (Zhang et al., 2018) to evaluate the
reconstruction errors between the generated images and the ground-truth
images. SSIM is a commonly used pixel-level image similarity indicator.
However, as discussed in paper (Zhang et al., 2018), pixel-level distance
metrics may be insufficient for assessing perceptual quality. Therefore, LPIPS
is further employed to evaluate the perceptual distance by calculating feature
differences using a network trained on human judgments. It can be seen that
our model achieves competitive SSIM and LPIPS scores, which means that we can
generate images with better perceptual quality and fewer reconstruction
errors. Besides, Fréchet Inception Distance (FID) (Heusel et al., 2017) is
used to measure the distance between the distributions of synthesized images
and real images. This metric indicates the realism of the generated images.
Our model achieves the best FID score compared with the state-of-the-art
models.

Since the objective metrics have their own limitations, their results may
mismatch with the actual subjective perceptions (Zhang et al., 2018).
Therefore, we conduct a user study to compare the subjective quality. Image
pairs of the ground-truth and generated images are shown to volunteers who are
expected to select the more realistic image from each data pair. The Fooling
Rate (FR) is calculated as the final score of the user study. This test is
implemented on Amazon Mechanical Turk (MTurk). We randomly select $550$ images
as the test set. Each image pair is compared $5$ times by different
volunteers. A total of $350$ volunteers participate in this experiment where
each volunteer conducts $55$ comparisons. The evaluation results are shown in
Tab. 1. It can be seen that our model achieves the best FR score, which means
that we can generate more realistic images.

###  4.2. Ablation Study

In this section, we evaluate the efficacy of the proposed modules by comparing
our model with several variants.

Baseline. A baseline model is trained to prove the efficacy of the deformation
modules. An auto-encoder network is used to design this model. The reference
images and target skeletons are concatenated as the model inputs. We train
this model using the same loss functions as that of our image generation
module.

Attention Model. The attention model is used to evaluate the performance gain
brought by the attention correlation estimator. We remove the flow field
estimator and use the attention correlation matrices $\mathbf{C}$ as the final
deformations.

Flow Model. Similarly, the flow model is designed to evaluate the efficacy of
the flow field estimator. The attention correlation estimator is removed from
the proposed model. We take the flow fields $\mathbf{w}$ as the final
deformations to warp the reference features.

Without Face Loss. This model is used to show the efficacy of face
reconstruction loss $\mathcal{L}_{face}$ on the quality of the generated
images. We remove the face reconstruction loss when training this model.

Full Model. The proposed model with both attention correlation estimator and
flow field estimator is used in this model.

The qualitative results of the ablation models are provided in Fig. 6. It can
be seen that the baseline model generates images with correct poses and
identities. However, it cannot preserve the textures of the reference images.
By using the spatial transformation modules, both the attention model and the
flow model can efficiently deform the reference images and generate targets
with realistic details. However, different types of artifacts can be observed
in their results. The powerful attention operation helps the model extract
long-term correlations and generate accurate target structures. However, the
dense connections affect the model to benefit from the image locality. As
shown in the top three rows in Fig. 6, the complex textures of reference
images cannot be reconstructed well. Meanwhile, as this operation may destroy
the spatial distributions of the reference images, the special patterns
(_e.g_. logos) are not generated. The flow-based operation can build
correlations between adjacent deformations. Thus, it helps to reconstruct the
patterns and complex textures by extracting the whole local patches from
reference images. However, as shown in the bottom three rows in Fig. 6, this
operation fails to estimate accurate deformations for all target positions,
which may lead to inaccurate target structures. Our full model benefits from
both transformation modules and generates images with not only accurate
structures but also realistic details. Meanwhile, compared with the w/o face
model, the full model produces more realistic faces, which improves the
perceptual quality of the final images.

|  Baseline |  Attn Model |  Flow Model |  w/o Face |  Full Model
---|---|---|---|---|---
|  |  |  |  |
SSIM $\uparrow$ |  0.6997 |  0.7119 |  0.7084 |  0.7124 |  0.7113
LPIPS $\downarrow$ |  0.2189 |  0.1852 |  0.1911 |  0.1778 |  0.1813
FID $\downarrow$ |  16.283 |  10.636 |  10.535 |  9.9336 |  9.4502

Table 2. The evaluation results of the ablation study.

The quantitative evaluation results are shown in Tab. 2. It can be seen that
the baseline model fails to achieve good evaluation results. The main reason
is that this model lacks efficient spatial transformation blocks to deform the
reference information. Thus, the network cannot obtain aligned feature maps to
synthesize the final images. Compared with the baseline, both the attention
model and the flow model achieve significant performance gains. This indicates
that designing efficient spatial transformation modules is crucial for this
task. Meanwhile, the attention model achieves better reconstruction scores
than the flow model, which confirms that the attention operation helps to
generate accurate target structures. However, the poor FID score indicates
that the realism of the final images is degraded since it cannot preserve
vivid reference details. By combining the attention operation with the flow
operation, both the w/o face model and the full model obtain additional
performance improvements. Such combination helps the models take the
complementary advantages of the deformation blocks. Although the w/o face
model achieves better LPIPS and SSIM scores, as shown in Fig. 6, due to the
acuteness of the human visual system towards the faces, a dramatic improvement
in visual quality is obtained by using the face reconstruction loss
$\mathcal{L}_{face}$.

Figure 7. The results of the portrait image editing task. The top two rows
show the results of pose editing. The bottom row shows the results of
expression editing.

The face editing results.

###  4.3. Portrait Image Editing

In this subsection, we further apply our model to tackle the portrait image
editing task. To achieve intuitive portrait image control, we employ the
three-dimensional morphable face models (Blanz and Vetter, 1999; Paysan et
al., 2009) (3DMMs) to describe the motions of the faces. 3DMMs allow users to
control the 3D face meshes with fully disentangled semantic parameters (_e.g_.
shape, pose). The VoxCeleb dataset (Nagrani et al., 2017) which contains
$22496$ talking-head videos is used for training and testing. The face
reconstruction model proposed in paper (Deng et al., 2019) is employed to
extract the corresponding 3DMM parameters from the images. The face landmarks
are rendered from the extracted 3DMM parameters and used as the semantics
$\mathbf{s}_{r}$, $\mathbf{s}_{t}$ of the face images $\mathbf{x}_{r}$ and
$\mathbf{x}_{t}$. Images from the same video are randomly paired as the
reference and target images. After training, the users can control the motions
of portrait images by providing specific 3DMM parameters. Example results are
shown in Fig. 7. Our model enables intuitively editing the poses and
expressions of a given portrait image, which will find a large variety of
applications in the industries such as social media, virtual reality.
Meanwhile, our model can generate images with realistic details and accurate
motions. Please find more details in the Supplementary Materials.

##  5\. Conclusion

We have proposed a novel model for the pose-guided person image synthesis task
by combining the attention operation with the flow-based operation. We
empirically demonstrate the advantages and disadvantages of these two
operations for the spatial transformation. Ablation studies prove that our
model can take the complementary advantages to generate images with not only
accurate global structures but also realistic local details. Both subjective
and objective experiments show the superiority of the proposed model compared
with the state-of-the-arts. Besides, additional experiment on the portrait
image editing task proves that the proposed combination can be flexibly
applied to different types of data.

## References

  * (1)
  * Blanz and Vetter (1999) Volker Blanz and Thomas Vetter. 1999.  A morphable model for the synthesis of 3D faces. In _Proceedings of the 26th annual conference on Computer graphics and interactive techniques_. 187–194.
  * Brock et al. (2018) Andrew Brock, Jeff Donahue, and Karen Simonyan. 2018\.  Large scale GAN training for high fidelity natural image synthesis.  _arXiv preprint arXiv:1809.11096_ (2018).
  * Choi et al. (2018) Yunjey Choi, Minje Choi, Munyoung Kim, Jung-Woo Ha, Sunghun Kim, and Jaegul Choo. 2018\.  Stargan: Unified generative adversarial networks for multi-domain image-to-image translation. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 8789–8797.
  * Deng et al. (2019) Yu Deng, Jiaolong Yang, Sicheng Xu, Dong Chen, Yunde Jia, and Xin Tong. 2019\.  Accurate 3d face reconstruction with weakly-supervised learning: From single image to image set. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops_. 0–0.
  * Esser et al. (2018) Patrick Esser, Ekaterina Sutter, and Björn Ommer. 2018\.  A variational u-net for conditional appearance and shape generation. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. 8857–8866.
  * Goodfellow et al. (2016) Ian Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio. 2016.  _Deep learning_. Vol. 1.  MIT press Cambridge.
  * Goodfellow et al. (2014) Ian J Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014\.  Generative adversarial networks.  _arXiv preprint arXiv:1406.2661_ (2014).
  * Han et al. (2019) Xintong Han, Xiaojun Hu, Weilin Huang, and Matthew R Scott. 2019\.  Clothflow: A flow-based model for clothed person generation. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 10471–10480.
  * Heusel et al. (2017) Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. 2017\.  Gans trained by a two time-scale update rule converge to a local nash equilibrium. In _Advances in Neural Information Processing Systems_. 6626–6637.
  * Huang et al. (2018) Xun Huang, Ming-Yu Liu, Serge Belongie, and Jan Kautz. 2018\.  Multimodal unsupervised image-to-image translation. In _Proceedings of the European conference on computer vision (ECCV)_. 172–189.
  * Isola et al. (2017) Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. 2017.  Image-to-image translation with conditional adversarial networks. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 1125–1134.
  * Johnson et al. (2016) Justin Johnson, Alexandre Alahi, and Li Fei-Fei. 2016.  Perceptual losses for real-time style transfer and super-resolution. In _European conference on computer vision_. Springer, 694–711.
  * Karras et al. (2019) Tero Karras, Samuli Laine, and Timo Aila. 2019.  A style-based generator architecture for generative adversarial networks. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 4401–4410.
  * Karras et al. (2020) Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. 2020\.  Analyzing and improving the image quality of stylegan. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 8110–8119.
  * Li et al. (2019) Yining Li, Chen Huang, and Chen Change Loy. 2019.  Dense intrinsic appearance flow for human pose transfer. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 3693–3702.
  * Liu et al. (2019a) Ming-Yu Liu, Xun Huang, Arun Mallya, Tero Karras, Timo Aila, Jaakko Lehtinen, and Jan Kautz. 2019a.  Few-shot unsupervised image-to-image translation. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 10551–10560.
  * Liu et al. (2019b) Wen Liu, Zhixin Piao, Jie Min, Wenhan Luo, Lin Ma, and Shenghua Gao. 2019b.  Liquid warping gan: A unified framework for human motion imitation, appearance transfer and novel view synthesis. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 5904–5913.
  * Liu et al. (2016) Ziwei Liu, Ping Luo, Shi Qiu, Xiaogang Wang, and Xiaoou Tang. 2016.  Deepfashion: Powering robust clothes recognition and retrieval with rich annotations. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 1096–1104.
  * Ma et al. (2017) Liqian Ma, Xu Jia, Qianru Sun, Bernt Schiele, Tinne Tuytelaars, and Luc Van Gool. 2017\.  Pose guided person image generation.  _arXiv preprint arXiv:1705.09368_ (2017).
  * Ma et al. (2018) Liqian Ma, Qianru Sun, Stamatios Georgoulis, Luc Van Gool, Bernt Schiele, and Mario Fritz. 2018\.  Disentangled person image generation. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. 99–108.
  * Men et al. (2020) Yifang Men, Yiming Mao, Yuning Jiang, Wei-Ying Ma, and Zhouhui Lian. 2020.  Controllable person image synthesis with attribute-decomposed gan. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 5084–5093.
  * Nagrani et al. (2017) Arsha Nagrani, Joon Son Chung, and Andrew Zisserman. 2017\.  Voxceleb: a large-scale speaker identification dataset.  _arXiv preprint arXiv:1706.08612_ (2017).
  * Paysan et al. (2009) Pascal Paysan, Reinhard Knothe, Brian Amberg, Sami Romdhani, and Thomas Vetter. 2009.  A 3D face model for pose and illumination invariant face recognition. In _2009 sixth IEEE international conference on advanced video and signal based surveillance_. Ieee, 296–301.
  * Ren et al. (2020) Yurui Ren, Xiaoming Yu, Junming Chen, Thomas H Li, and Ge Li. 2020.  Deep image spatial transformation for person image generation. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 7690–7699.
  * Salimans et al. (2016) Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. 2016\.  Improved techniques for training gans.  _arXiv preprint arXiv:1606.03498_ (2016).
  * Siarohin et al. (2018) Aliaksandr Siarohin, Enver Sangineto, Stéphane Lathuiliere, and Nicu Sebe. 2018.  Deformable gans for pose-based human image generation. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. 3408–3416.
  * Tang et al. (2021) Jilin Tang, Yi Yuan, Tianjia Shao, Yong Liu, Mengmeng Wang, and Kun Zhou. 2021\.  Structure-aware Person Image Generation with Pose Decomposition and Semantic Correlation.  _arXiv preprint arXiv:2102.02972_ (2021).
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017\.  Attention is all you need.  _arXiv preprint arXiv:1706.03762_ (2017).
  * Wang et al. (2019) Ting-Chun Wang, Ming-Yu Liu, Andrew Tao, Guilin Liu, Jan Kautz, and Bryan Catanzaro. 2019\.  Few-shot video-to-video synthesis.  _arXiv preprint arXiv:1910.12713_ (2019).
  * Wang et al. (2018) Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. 2018.  Non-local neural networks. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 7794–7803.
  * Wang et al. (2004) Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. 2004\.  Image quality assessment: from error visibility to structural similarity.  _IEEE transactions on image processing_ 13, 4 (2004), 600–612.
  * Yu et al. (2019) Xiaoming Yu, Yuanqi Chen, Shan Liu, Thomas Li, and Ge Li. 2019.  Multi-mapping Image-to-Image Translation via Learning Disentanglement. In _Advances in Neural Information Processing Systems_.
  * Zablotskaia et al. (2019) Polina Zablotskaia, Aliaksandr Siarohin, Bo Zhao, and Leonid Sigal. 2019.  DwNet: Dense warp-based network for pose-guided human video generation.  _arXiv preprint arXiv:1910.09139_ (2019).
  * Zhang et al. (2019) Han Zhang, Ian Goodfellow, Dimitris Metaxas, and Augustus Odena. 2019.  Self-attention generative adversarial networks. In _International conference on machine learning_. PMLR, 7354–7363.
  * Zhang et al. (2021) Jinsong Zhang, Kun Li, Yu-Kun Lai, and Jingyu Yang. 2021\.  PISE: Person Image Synthesis and Editing with Decoupled GAN.  _arXiv preprint arXiv:2103.04023_ (2021).
  * Zhang et al. (2020) Pan Zhang, Bo Zhang, Dong Chen, Lu Yuan, and Fang Wen. 2020.  Cross-domain correspondence learning for exemplar-based image translation. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 5143–5153.
  * Zhang et al. (2018) Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. 2018.  The unreasonable effectiveness of deep features as a perceptual metric. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. 586–595.
  * Zhou et al. (2021) Xingran Zhou, Bo Zhang, Ting Zhang, Pan Zhang, Jianmin Bao, Dong Chen, Zhongfei Zhang, and Fang Wen. 2021\.  CoCosNet v2: Full-Resolution Correspondence Learning for Image Translation. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 11465–11475.
  * Zhu et al. (2019) Zhen Zhu, Tengteng Huang, Baoguang Shi, Miao Yu, Bofei Wang, and Xiang Bai. 2019\.  Progressive pose attention transfer for person image generation. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 2347–2356.


# Probabilistic Abduction for Visual Abstract Reasoning via Learning Rules in
Vector-symbolic Architectures

Michael Hersche1,2, Francesco di Stefano1,2 , Thomas Hofmann2,
Abu Sebastian1, Abbas Rahimi1
1IBM Research – Zurich, 2ETH Zurich Research conducted at IBM Research –
Zurich.Corresponding author<EMAIL_ADDRESS>
###### Abstract

Abstract reasoning is a cornerstone of human intelligence, and replicating it
with artificial intelligence (AI) presents an ongoing challenge. This study
focuses on efficiently solving Raven’s progressive matrices (RPM), a visual
test for assessing abstract reasoning abilities, by using distributed
computation and operators provided by vector-symbolic architectures (VSA).
Instead of hard-coding the rule formulations associated with RPMs, our
approach can learn the VSA rule formulations (hence the name Learn-VRF) with
just one pass through the training data. Yet, our approach, with compact
parameters, remains transparent and interpretable. Learn-VRF yields accurate
predictions on I-RAVEN’s in-distribution data, and exhibits strong out-of-
distribution capabilities concerning unseen attribute-rule pairs,
significantly outperforming pure connectionist baselines including large
language models. Our code is available at https://github.com/IBM/learn-vector-
symbolic-architectures-rule-formulations.

##  1 Introduction

The Raven’s progressive matrices (RPM) [1] test assesses human’s fluid
intelligence and abstract reasoning [2, 3]. It is a non-verbal test that
involves perceiving pattern continuation, elemental abstraction, and finding
relations between abstract elements based on underlying rules. Each RPM test
is an analogy problem presented as a 3×3 pictorial matrix of context panels.
Every panel in the matrix is filled with several geometric objects based on a
certain rule, except the last panel, which is left blank. The task is to
complete the missing panel by picking the correct answer from a set of
candidate answer panels that matches the implicit rule (see Fig. 1).

Recently, RPM has become a widely used task for effectively testing AI
capabilities in abstract reasoning, making analogies, and dealing with out-of-
distribution (OOD) data [4, 5, 6, 7, 8]. Large language models (LLMs) have
been recently shown zero-shot learning capability in solving RPMs when
perceptual visual information is parsed externally and provided as symbolic
inputs to LLMs [9, 10]. On the other hand, neuro-symbolic AI approaches
combine subsymbolic perception with various forms of symbolic reasoning
leading to state-of-the-art (SOTA) performance in visual [11, 12, 13, 14],
natural language [15], causal [16], mathematical [17], and RPM analogical [18,
19, 20] reasoning tasks. Specifically, for solving the RPM tests, as an
upgrade of pure logical reasoning, a neuro-symbolic approach [18] implemented
a _probabilistic_ abduction reasoning. Abduction refers to the process of
selectively inferring facts that give the best explaination to the perceptual
observations based on prior background knowledge which is represented in a
symbolic form [21]. The probabilistic abduction reasoning allowed for
perceptual uncertainties in the symbol grounding which is particularly useful
when connecting a trainable perception module to the reasoning module for the
end-to-end training.

However, the probabilistic abduction reasoning incurs exhaustive symbolic
searches. For instance, it requires to search over the symbols describing all
possible rule realizations that could govern the RPM tests to be able to
abduce the probability distribution of the rules [18]. A viable option to
reduce such search complexity is to use distributed representations provided
by vector-symbolic architectures (VSA) [22, 23, 24, 25], which is built on a
rich algebra supporting well-defined operations, including multiplicative
binding, unbinding, additive superposition, permutations, and similarity
search that allow performing symbolic computations on top of high-dimensional
distributed representations. Consequently, a neuro-vector symbolic
architecture (NVSA) [20] was proposed to perform _tractable_ probabilistic
abductive reasoning with distributed VSA representations and operators which
offer computation-in-superposition leading to two orders of magnitude faster
inference. Nevertheless, like other neuro-symbolic approaches, NVSA had to
carefully choose suitable operators and operands to formulate the individual
RPM rules. While NVSA allows the incorporation of domain knowledge into the
model, it requires complete knowledge of the problem.

We propose Learn-VRF, a novel probabilistic abduction reasoning approach that
_learns_ VSA rule formulations from examples by solving a convex optimization
problem. Besides its higher flexibility, better generality, and one-pass
learning, Learn-VRF maintains the advantages of neuro-symbolic approaches,
including transparency and interpretability. Learn-VRF transparently operates
in the rule space by abstracting away from the surface statistical patterns.
On the I-RAVEN dataset [5], with extracted visual attribute labels, Learn-VRF
achieves competitive accuracy on in-distribution data, and notably outperforms
LLMs [10] and neural MLP approaches on OOD tests while having orders of
magnitude lower trainable parameters.

##  2 Background

#### Vector-symbolic Architectures (VSA)

VSA manipulates high-dimensional distributed representations by
dimensionality-preserving operations, as opposed to tensor product
representations which yield higher-order tensors [17, 15, 26]. Many VSA
variants exist with different representations and operators (see [27] for a
review). In this work, we use binary sparse block codes (SBC) [28], which
induce a local blockwise structure that exhibits ideal variable binding
properties [29]. In binary SBC, the binding is defined as the blockwise
circular convolution ($\circledast$), and the approximate unbinding is the
blockwise circular correlation ($\otimes$). The VSA binding has properties
analogous to multiplication in the real number domain, including
commutativity, associativity, and the existence of a neutral element
($\mathbf{e}$). The cosine similarity ($\mathrm{cossim}(\cdot,\cdot)$)
measures the similarity between two vectors.

#### I-RAVEN Dataset

The I-RAVEN dataset [5] provides RPM tests with unbiased candidate sets. Fig.
1 shows an I-RAVEN example test. Each RPM test consists of nine context
panels, arranged in a 3x3 matrix, and eight candidate panels. The panels
contain objects that are arranged according to one of seven different
constellations. The object’s attributes (color, size, shape, number, position)
are governed by different underlying rules: constant, arithmetic, progression,
or distribute three.

##  3 Methods

This section presents Learn-VRF, which learns predictive VSA rule formulations
based on examples, depicted in Fig. 1. First, the attribute values (provided
by I-RAVEN metadata) are mapped to one-hot probability mass functions (PMFs),
and translated to a VSA vector. Learn-VRF learns $R$-many rules, which are
shared across the attributes. Each rule generates a prediction with a
confidence value. The predicted representation of the empty panel
($\mathbf{\hat{a}}^{(3,3)}$) is either based on selecting a prediction of a
rule with high confidence (sampling), or a weighted combination of all the
rules. Finally, the candidate panel with the largest sum of cosine
similarities to the predicted attribute vectors is the answer.

Figure 1: Learn-VRF for solving Raven’s progressive matrices.

#### Translating PMFs to VSA

This paper focuses exclusively on the RPM’s reasoning aspect; thus, we
directly extract the attribute labels provided by I-RAVEN metadata, as
practiced for LLMs [9, 10]. The attributes of each object in a panel are
mapped to attribute one-hot PMFs, which are further combined, yielding panel-
level PMFs describing the attribute’s distribution of a panel [18]. Each
attribute yields eight PMFs (one per context panel), represented by
$\mathbf{P}^{c}:=(\mathbf{p}^{(1,1)},\mathbf{p}^{(1,2)},...\mathbf{p}^{(3,2)})$.
Next, we translate the panel-level PMFs to high-dimensional VSA
representations:

$\displaystyle\mathbf{a}^{(i,j)}=\sum_{k=1}^{N}\mathbf{p}^{(i,j)}[k]\cdot\mathbf{b}_{k},$
(1)

where $N$ is the dimensionality of the panel-level PMF space. Similarly to
NVSA [20], we use a binary SBC dictionary $\\{\mathbf{b}_{k}\\}_{k=1}^{N}$,
where each basis vector ($\mathbf{b}_{i}$) has a dimension of 1024 and four
blocks. When combined with fractional power encoding [24], this basis permits
the representation of continuous attributes (e.g., color or size). The VSA
representation of different context panels are represented by
$\mathbf{A}^{c}:=(\mathbf{a}^{(1,1)},\mathbf{a}^{(1,2)},...\mathbf{a}^{(3,2)})$.

Table 1: Translation of RPM rules into VSA rule formulations.

Rule | VSA formulation
---|---
Constant | $\mathbf{\hat{a}}^{(3,3)}=\mathbf{a}^{(3,2)}=\mathbf{a}^{(3,1)}$
Arithmetic plus | $\mathbf{\hat{a}}^{(3,3)}=\mathbf{a}^{(3,1)}\circledast\mathbf{a}^{(3,2)}$
Progression positive | $\mathbf{\hat{a}}^{(3,3)}=\mathbf{a}^{(3,2)}\circledast(\mathbf{a}^{(3,2)}\otimes\mathbf{a}^{(3,1)})$
Distribute three | $\mathbf{\hat{a}}^{(3,3)}=(\mathbf{a}^{(1,1)}\circledast\mathbf{a}^{(1,2)}\circledast\mathbf{a}^{(1,3)})\otimes(\mathbf{a}^{(3,1)}\circledast\mathbf{a}^{(3,2)})$

#### Learning VSA Rule Formulations (Learn-VRF)

VSA allows us to predict the representation of the empty panel with a rule-
specific combination of operands (i.e., a selection of context panels) and
operations (binding and unbinding). Table 1 summarizes possible VSA
formulations for each RPM rule. All the rules are represented as a sequence of
binding and unbinding operations, which can be generalized to:

$\displaystyle\mathbf{\hat{a}}^{(3,3)}=(\mathbf{c}_{1}\circledast\mathbf{c}_{2}\circledast\mathbf{c}_{3})\otimes(\mathbf{c}_{4}\circledast\mathbf{c}_{5}\circledast\mathbf{c}_{6}),$
(2)

where $\mathbf{c}_{i}$ represents a context panel ($\mathbf{a}^{(i,j)}$) or
the identity ($\mathbf{e}$). Eq. (2) can be interpreted as a fractional form,
akin to a sequence of multiplicative and divisive operations among real
numbers.

While NVSA hard-coded the rules as in Table 1, we _learn_ the rule
formulations by finding, for each element in the fractional form, a convex
combination over the VSA vectors of the context panels’ attributes, augmented
with the neutral element:

$\displaystyle\mathbf{c}_{k}=\sum_{\text{panels
}(i,j)}w_{k}^{(i,j)}\cdot\mathbf{a}^{(i,j)}+v_{k}\cdot\mathbf{e},$ (3)

where the following constraints apply to the weights:

$\displaystyle\sum_{\text{panels }(i,j)}w_{k}^{(i,j)}+v_{k}=1,\quad\quad 0\leq
w_{k}^{(i,j)}\leq 1\,\forall i,j,\quad\quad 0\leq v_{k}\leq 1\,\forall k.$ (4)

Here, $w_{k}^{(i,j)}$ and $v_{k}$ are the trainable weights of the convex
combination. A rule in the VSA space can manifest in multiple functionally
equivalent representations within the proposed form. As a result, the
optimization problem will have multiple global minima corresponding to
different algebraic representations of the same rule.

Learn-VRF learns $R$-many rule formulations. Allowing more rules than actually
embedded in I-RAVEN (i.e., $R>R^{*}$) yields Learn-VRF to come up with
functionally equivalent duplications. Thus, by providing an overcomplete rule
space, Learn-VRF can still operate and does not need to have the exact
knowledge about the number of rules ($R^{*}$). In contrast, a related
learnable neuro-symbolic approach [30] performed a grid search for the optimal
number of attributes and rules in SATNet. For simplicity, we set the number of
rules equal to the rules in I-RAVEN ($R=R^{*}$).

#### Confidence value computation

Each rule ($r$) generates a VSA vector ($\mathbf{\hat{a}}^{(3,3)}_{r}$)
accompanied by a confidence value ($\hat{c}_{r}$). The confidence is computed
by deploying three formulations per rule, which predict all panels of the
third column. At inference, the confidence value is calculated as the sum of
the cosine similarities between the predicted VSA vectors of the first and
second row with their respective ground-truth vector:

$\displaystyle{c}_{r,\text{test}}=\mathrm{cossim}\left(\mathbf{a}^{(1,3)},\mathbf{\hat{a}}^{(1,3)}_{r}\right)+\mathrm{cossim}\left(\mathbf{a}^{(2,3)},\mathbf{\hat{a}}^{(2,3)}_{r}\right).$
(5)

During training, the ground-truth of the last row is accessible, enabling the
inclusion of the cosine similarity between the predicted panel and the ground-
truth answer ($\mathbf{a}_{y}$) into the confidence score:

$\displaystyle{c}_{r,\text{train}}=\mathrm{cossim}\left(\mathbf{a}^{(1,3)},\mathbf{\hat{a}}^{(1,3)}_{r}\right)+\mathrm{cossim}\left(\mathbf{a}^{(2,3)},\mathbf{\hat{a}}^{(2,3)}_{r}\right)+\mathrm{cossim}\left(\mathbf{a}_{y},\mathbf{\hat{a}}^{(3,3)}_{r}\right).$
(6)

A softmax with a temperature parameter is applied to the confidence scores,
producing a rule distribution.

#### Final estimation using sampling or weighted combination

We distinguish between two variants to determine the final estimation of the
empty panel. One variant is to sample a rule ($\hat{r}$) that has a high
confidence value. During training, we sample from the distribution of the
confidence values, whereas at inference, we select the rule with the
maximizing confidence value. The estimated panel is then defined as:

$\displaystyle\mathbf{\hat{a}}^{(3,3)}_{\text{sample}}=\mathbf{\hat{a}}^{(3,3)}_{\hat{r}}.$
(7)

Alternatively, we propose to create a convex weighted combination of all the
rules using the confidence values as weights:

$\displaystyle\mathbf{\hat{a}}^{(3,3)}_{\text{weighted
comb.}}=\sum_{r=1}^{R}c_{r}\cdot\mathbf{\hat{a}}^{(3,3)}_{r}.$ (8)

#### Training loss

The model weights are updated through stochastic gradient descent (SGD),
reducing the inverse cosine similarity of the selected rule as the loss:

$\displaystyle\mathcal{L}=1-\mathrm{cossim}\left(\mathbf{a}^{(1,3)},\mathbf{\hat{a}}^{(1,3)}\right)-\mathrm{cossim}\left(\mathbf{a}^{(2,3)},\mathbf{\hat{a}}^{(2,3)}\right)-\mathrm{cossim}\left(\mathbf{a}_{y},\mathbf{\hat{a}}^{(3,3)}\right),$
(9)

where $\mathbf{\hat{a}}^{(i,3)}$ results either from the sampling or the
weighted combination. The terms derived from the first and second rows refine
the rule selection mechanism, whereas the term associated with the ground-
truth answer adjusts the weights responsible for predicting the empty panel’s
VSA vector.

##  4 Experimental Results

#### I-RAVEN: in-distribution

Table 2 compares Learn-VRF’s accuracy with the SOTA deep neural nets SCL [31],
probabilistic abductive neuro-symbolic approaches such as PrAE [18] and NVSA
[20], and an LLM based on GPT-3 [10] on in-distribution (ID) I-RAVEN tasks. As
an additional baseline, we trained a model that predicts the PMF for each
attribute using a separate MLP. App. A describes the MLP baseline and App. B
our experimental setup.

Table 2: ID accuracy (%) on I-RAVEN. Selective approaches require the answer
panels as model inputs, while predictive approaches compare their prediction
against all possible answers.

|  |  |  | I-RAVEN accuracy (%)
---|---|---|---|---
Method |  | Predictive/
---
Selelective
#Parms | #Epochs | Avg. | C | 2x2 | 3x3 | L-R | U-D | O-IC | O-IG
SCL [31] | Selective | 961 k | 200 | 84.3 | 99.9 | 68.9 | 43.0 | 98.5 | 99.1 | 97.7 | 82.6
GPT-3 [10] | Selective | 175 B | n/a | 86.5 | 86.4 | 83.2 | 81.8 | 83.4 | 84.6 | 92.8 | 93.0
PrAE1 [18] | Predictive | - | - | 71.1 | 83.8 | 82.9 | 47.4 | 94.8 | 94.8 | 56.6 | 37.4
NVSA1 [20] | Predictive | - | - | 88.1 | 99.8 | 96.2 | 54.3 | 100 | 99.9 | 99.6 | 67.1
MLP baseline | Predictive | 300 k | 50 | 87.1 | 97.6 | 87.1 | 61.8 | 99.4 | 99.4 | 98.7 | 65.6
Learn-VRF (sampling) | Predictive | 5 k | 50 | 81.3 | 98.7 | 62.9 | 48.4 | 98.7 | 99.9 | 99.9 | 60.8
Learn-VRF (weighted comb.) | Predictive | 5 k | 50 | 84.1 | 99.1 | 73.5 | 50.9 | 99.9 | 99.9 | 100 | 65.2
Learn-VRF (sampling) | Predictive | 5 k | 1 | 80.0 | 99.1 | 52.3 | 51.1 | 98.7 | 99.9 | 99.9 | 59.5
Learn-VRF (weighted comb.) | Predictive | 5 k | 1 | 81.0 | 99.1 | 53.1 | 50.9 | 99.9 | 100 | 100 | 63.7

  * 1

No trainable parameters in the hard-coded reasoning; hence, learnable
parameters and epochs are zero for reasoning module.

Learn-VRF achieves an average accuracy of 81.3% when using a sampling-based
rule selection and 84.1% with the weighted rule combination, while only
requiring 5 k trainable parameters. The superior performance of the weighted
combination over the sampling approach may stem from its uniform update
mechanism, which adjusts all the rules (including the “correct” rule) instead
of only the sampled one. Learn-VRF remains accurate when training it with a
single pass over the training data, achieving 80.0% and 81.0% using the
sampling and weighted combination, respectively. Learn-VRF generates accurate
predictions in constellations with no position attribute (i.e., center, left-
right, up-down, out-in center). In these constellations, it outperforms GPT-3
and PrAE, and is on par with NVSA and SCL. Instead, the 2x2 grid, 3x3 grid,
and out-in grid require the position attribute that employs logical rules on
its bit-code vector, which the VSA rule formulations cannot support (see App.
D). While NVSA circumvented this limitation by treating some rules of the
position attribute in the original PMF domain, Learn-VRF uniformly implements
all rules in the VSA space, at some degree of accuracy degradation. The MLP
baseline can improve the accuracy in the position-related constellations, but
it requires 60$\times$ more parameters (300 k vs. 5 k) while also sacrificing
transparency and interpretability.

#### I-RAVEN: out-of-distribution

We test GPT-3, the MLP baseline, and the Learn-VRF on unseen rule-attribute
pairs in the center constellation, validating their OOD generalization
capabilities (see App. B). In this OOD task, Learn-VRF has been trained for 50
epochs. As shown in Table 3, the MLP baseline faces challenges in OOD even
though it was accurate on ID. Moreover, the GPT-3 shows a low accuracy,
particularly on the arithmetic rule, despite the potential for data
contamination [32], and our additional efforts to perform in-context learning
(see App. E). In contrast, Learn-VRF generalizes to OOD, achieving almost
perfect accuracy on most rule-attribute pairs thanks to Learn-VRF’s rule
sharing, which is enabled by the uniform, expressive PMF representation in the
VSA space using fractional power encoding.

Table 3: OOD accuracy (%) on unseen rule-attribute pairs on I-RAVEN in the
center constellation.

| Type | Size | Color
---|---|---|---
| Const. | Progr. | Dist.3 | Const. | Progr. | Dist.3 | Arith. | Const. | Progr. | Dist.3 | Arith.
GPT-3 zero-shot | 88.5 | 86.0 | 88.6 | 93.6 | 93.2 | 92.6 | 71.6 | 94.2 | 94.7 | 94.3 | 65.8
MLP baseline | 14.8 | 14.9 | 30.2 | 22.8 | 75.0 | 47.0 | 46.6 | 56.3 | 60.5 | 44.4 | 48.9
Learn-VRF (weighted comb.) | 100 | 100 | 99.7 | 100 | 100 | 99.8 | 99.8 | 100 | 98.8 | 100 | 100

#### Interpretation of learned rule formulations

Learn-VRF provides transparency and interpretability into the learned rules.
We can derive the most dominant formula by selecting the VSA vector that
exhibited the highest weight for each term within the Learn-VRF model’s
internal representations. Table 4 shows the learned rule formulations for the
center constellation. Analyzing these rules and performing straightforward
algebraic simplifications, we discern the equivalences shown in the second
column. App. C provides more details on the interpretation of the learned rule
formulations.

Table 4: Rule formulations learned by Learn-VRF with the respective rule
interpretation.

Extracted Learned Rules | Corresponding RPM Rule
---|---
$\mathbf{\hat{a}}^{(3,3)}=(\mathbf{a}^{(3,2)}\circledast\mathbf{a}^{(1,1)}\circledast\mathbf{a}^{(3,1)})\otimes(\mathbf{e}\circledast\mathbf{a}^{(1,2)}\circledast\mathbf{a}^{(3,2)})$ | dist. three right & constant
$\mathbf{\hat{a}}^{(3,3)}=(\mathbf{a}^{(3,2)}\circledast\mathbf{a}^{(1,2)}\circledast\mathbf{a}^{(2,3)})\otimes(\mathbf{a}^{(2,3)}\circledast\mathbf{e}\circledast\mathbf{a}^{(1,1)})$ | dist. three left & constant
$\mathbf{\hat{a}}^{(3,3)}=(\mathbf{a}^{(3,2)}\circledast\mathbf{a}^{(3,1)}\circledast\mathbf{a}^{(3,1)})\otimes(\mathbf{e}\circledast\mathbf{a}^{(3,1)}\circledast\mathbf{e})$ | arithmetic plus
$\mathbf{\hat{a}}^{(3,3)}=(\mathbf{a}^{(3,2)}\circledast\mathbf{a}^{(3,2)}\circledast\mathbf{a}^{(3,1)})\otimes(\mathbf{a}^{(3,2)}\circledast\mathbf{a}^{(3,2)}\circledast\mathbf{a}^{(3,2)})$ | arithmetic minus
$\mathbf{\hat{a}}^{(3,3)}=(\mathbf{a}^{(1,3)}\circledast\mathbf{a}^{(3,1)}\circledast\mathbf{a}^{(3,2)})\otimes(\mathbf{e}\circledast\mathbf{a}^{(3,2)}\circledast\mathbf{a}^{(1,1)})$ | progression

##  5 Conclusion

We proposed Learn-VRF, a novel VSA-based approach for probabilistic abduction
which can learn rule formulations. The model’s structure inherently provides
transparency: by identifying the index with the highest probability mass for
each term in the weight distribution, we could reconstruct the rule formula
chosen by the Learn-VRF. Moreover, Learn-VRF features a low parameter count,
single-pass learning, and OOD generalization with respect to unseen attribute-
rule combinations.

## Acknowledgement

This work is supported by the Swiss National Science foundation (SNF), grant
200800.

## References

  * [1] J. Raven, J. Court, and J. Raven, _Raven’s progressive matrices_. Oxford Psychologists Press, 1938.
  * [2] P. A. Carpenter, M. A. Just, and P. Shell, “What one intelligence test measures: a theoretical account of the processing in the Raven progressive matrices test,” _Psychological review_ , 1990.
  * [3] W. B. Bilker, J. A. Hansen, C. M. Brensinger, J. Richard, R. E. Gur, and R. C. Gur, “Development of abbreviated nine-item forms of the Raven’s standard progressive matrices test,” _Assessment_ , 2012.
  * [4] C. Zhang, F. Gao, B. Jia, Y. Zhu, and S.-C. Zhu, “Raven: A dataset for relational and analogical visual reasoning,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2019.
  * [5] S. Hu, Y. Ma, X. Liu, Y. Wei, and S. Bai, “Stratified rule-aware network for abstract visual reasoning,” in _Proceedings of the AAAI Conference on Artificial Intelligence (AAAI)_ , 2021.
  * [6] Y. Benny, N. Pekar, and L. Wolf, “Scale-localized abstract reasoning,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2021.
  * [7] M. Mitchell, “Abstraction and analogy-making in artificial intelligence,” _Annals of the New York Academy of Sciences_ , vol. 1505, no. 1, pp. 79–101, 2021.
  * [8] Małkiński, Mikołaj and Mańdziuk, Jacek, “Deep learning methods for abstract visual reasoning: A survey on Raven’s progressive matrices,” _arXiv preprint arXiv:2201.12382_ , 2022.
  * [9] T. Webb, K. J. Holyoak, and H. Lu, “Emergent analogical reasoning in large language models,” _Nature Human Behaviour_ , vol. 7, no. 9, pp. 1526–1541, Sep 2023.
  * [10] X. Hu, S. Storks, R. L. Lewis, and J. Chai, “In-context analogical reasoning with pre-trained language models,” in _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Long Paper)_ , 2023.
  * [11] K. Yi, J. Wu, C. Gan, A. Torralba, P. Kohli, and J. Tenenbaum, “Neural-symbolic VQA: Disentangling reasoning from vision and language understanding,” in _Advances in Neural Information Processing Systems (NeurIPS)_ , 2018.
  * [12] J. Mao, C. Gan, P. Kohli, J. B. Tenenbaum, and J. Wu, “The neuro-symbolic concept learner: Interpreting scenes, words, and sentences from natural supervision,” in _International Conference on Learning Representations (ICLR)_ , 2019.
  * [13] C. Han, J. Mao, C. Gan, J. Tenenbaum, and J. Wu, “Visual concept-metaconcept learning,” in _Advances in Neural Information Processing Systems (NeurIPS)_ , 2019.
  * [14] L. Mei, J. Mao, Z. Wang, C. Gan, and J. B. Tenenbaum, “FALCON: Fast visual concept learning by integrating images, linguistic descriptions, and conceptual relations,” in _International Conference on Learning Representations (ICLR)_ , 2022.
  * [15] I. Schlag and J. Schmidhuber, “Learning to reason with third-order tensor products,” in _Advances in Neural Information Processing Systems (NeurIPS)_ , 2018.
  * [16] K. Yi, C. Gan, Y. Li, P. Kohli, J. Wu, A. Torralba, and J. B. Tenenbaum, “CLEVRER: Collision events for video representation and reasoning,” in _International Conference on Learning Representations (ICLR)_ , 2020.
  * [17] I. Schlag, P. Smolensky, R. Fernandez, N. Jojic, J. Schmidhuber, and J. Gao, “Enhancing the transformer with explicit relational encoding for math problem solving,” _arXiv preprint arXiv:1910.06611_ , 2019.
  * [18] C. Zhang, B. Jia, S.-C. Zhu, and Y. Zhu, “Abstract spatial-temporal reasoning via probabilistic abduction and execution,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2021.
  * [19] V. Shah, A. Sharma, G. Shroff, L. Vig, T. Dash, and A. Srinivasan, “Knowledge-based analogical reasoning in neuro-symbolic latent spaces,” in _Proceedings of the 16th International Workshop on Neural-Symbolic Learning and Reasoning (NeSy)_ , 2022.
  * [20] M. Hersche, M. Zeqiri, L. Benini, A. Sebastian, and A. Rahimi, “A neuro-vector-symbolic architecture for solving Raven’s progressive matrices,” _Nature Machine Intelligence_ , vol. 5, no. 4, pp. 363–375, 2023.
  * [21] L. Magnani, _Abductive Cognition: The Epistemological and Eco-Cognitive Dimensions of Hypothetical Reasoning_. Springer Berlin, Heidelberg, 2009.
  * [22] R. W. Gayler, “Vector symbolic architectures answer Jackendoff’s challenges for cognitive neuroscience,” in _Joint International Conference on Cognitive Science (ICCS/ASCS)_ , 2003.
  * [23] T. A. Plate, “Holographic reduced representations,” _IEEE Transactions on Neural Networks_ , vol. 6, no. 3, pp. 623–641, 1995.
  * [24] ——, _Holographic Reduced Representations: Distributed Representation for Cognitive Structures_. Center for the Study of Language and Information, Stanford, 2003.
  * [25] P. Kanerva, “Hyperdimensional computing: An introduction to computing in distributed representation with high-dimensional random vectors,” _Cognitive Computation_ , vol. 1, no. 2, pp. 139–159, 2009.
  * [26] P. Smolensky, R. T. McCoy, R. Fernandez, M. Goldrick, and J. Gao, “Neurocompositional computing: From the central paradox of cognition to a new generation of ai systems,” _AI Magazine_ , vol. 43, no. 3, pp. 308–322, 2022.
  * [27] D. Kleyko, D. A. Rachkovskij, E. Osipov, and A. Rahimi, “A survey on hyperdimensional computing aka vector symbolic architectures, part I: models and data transformations,” _ACM Comput. Surv._ , 2022.
  * [28] M. Laiho, J. H. Poikonen, P. Kanerva, and E. Lehtonen, “High-dimensional computing with sparse vectors,” in _2015 IEEE Biomedical Circuits and Systems Conference (BioCAS)_ , 2015.
  * [29] E. P. Frady, D. Kleyko, and F. T. Sommer, “Variable binding for sparse distributed representations: Theory and applications,” _IEEE Transactions on Neural Networks and Learning Systems_ , 2021.
  * [30] S. Topan, D. Rolnick, and X. Si, “Techniques for symbol grounding with SATNet,” _Advances in Neural Information Processing Systems (NeurIPS)_ , vol. 34, pp. 20 733–20 744, 2021.
  * [31] Y. Wu, H. Dong, R. Grosse, and J. Ba, “The scattering compositional learner: Discovering objects, attributes, relationships in analogical reasoning,” _arXiv preprint arXiv:2007.04212_ , 2020.
  * [32] A. Saparov and H. He, “Language models are greedy reasoners: A systematic formal analysis of chain-of-thought,” in _The Eleventh International Conference on Learning Representations (ICLR)_ , 2023.
  * [33] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell _et al._ , “Language models are few-shot learners,” _Advances in Neural Information Processing Systems (NeurIPS)_ , vol. 33, pp. 1877–1901, 2020.
  * [34] L. Qiu, P. Shaw, P. Pasupat, T. Shi, J. Herzig, E. Pitler, F. Sha, and K. Toutanova, “Evaluating the impact of model scale for compositional generalization in semantic parsing,” in _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , 2022, pp. 9157–9179.

##  Appendix A MLP Baseline

Figure A1: MLP baseline which uses one MLP per each attribute.

This baseline, illustrated in Fig. A1, is built upon the use of $A$ instances
of MLPs, where $A$ denotes the total number of attributes. The MLP directly
models the transformations of the attribute PMFs. For every attribute, there
is a dedicated MLP, ensuring that each attribute is modeled independently.
Since the PMFs of the attributes have different dimensionality (i.e.,
different number of possible values), the MLPs cannot be shared across the
attributes, while it is possible with our Learn-VRF thanks to the use of VSA-
based fractional power encoding that allows to map any PMF vectors with
arbitrary sizes to the fixed-width, yet expressive, distributed
representation. Thanks to this _uniform_ interface, Learn-VRF can be plugged-
and-played from one attribute to another.

The first step of MLP baseline involves processing the derived attribute
values. These values are converted into one-hot PMFs. Once translated to the
PMF format, these representations are processed by a 3-layer MLP with 1024
hidden dimensions in each layer. The output from this MLP is another PMF
spanning the attribute values. For each attribute, its associated PMF vector
is compared against the PMF vectors of all candidate panels using the cross
entropy. For every attribute, we obtain eight scores corresponding to the
eight candidate panels. Lastly, we compute the average for each attribute,
resulting in eight overall candidate scores. The candidate panel with the
highest score is selected.

This baseline does not make any distinction between rule selection and
execution. Unlike Learn-VRF, MLPs can potentially capture non-linear
relationships, such as the logical rules present in the position attribute in
the 2x2 grid, 3x3 grid, and the out-in grid.

##  Appendix B Experimental Setup

### Training setup

In our experimental setup with the MLP baselines, we used three hidden layers,
each containing 1024 neurons. Layer normalization was applied between
consecutive layers. We set the learning rate to 1e-4, used a batchsize of 32,
and trained the model for 50 epochs. The model with the lowest validation loss
from these epochs was chosen for further evaluation on the test dataset.

In Learn-VRF, we used VSA vectors of 1024 dimensions with 4 blocks. Moreover,
we used a learning rate of 0.1 and batchsize of 4. We set the number of
learnable rules ($R$) equal to the number of rules in I-RAVEN. Finally, we
used a softmax temperature of 0.01, favoring exploitation over exploration,
allowing us to converge quickly.

### OOD test setup

In the out-of-distribution (OOD) experiments, we evaluate whether the model
can solve an unseen target attribute-rule pair (e.g., the constant rule on the
type attribute) when it has been trained on the examples containing all of the
attribute-rule pairs except the specific target one (e.g., the constant rule
on size and color, the progression rule on all attributes, and the distribute
rule on all attributes). We generate a new training and validation set
containing all examples except those with the target attribute-rule pair and a
test set containing examples exclusively with the target attribute-rule pair.

In these experiments, we focus on the center single constellation because it
encompasses a single panel. In contrast, configurations such as up-down, left-
right, and in-out entail two center single constellations, introducing
ambiguity in applying attribute-rule filtering.

The datasets are generated by filtering the existing splits I-RAVEN. The
training sets encompass between 2622 and 3437 samples, the validation sets
range from 841 to 1160 samples, while the test sets span from 803 to 1117
samples.

##  Appendix C Interpretation of Learned Rule Formulation in Learn-VRF

Within Learn-VRF, a singular rule in the VSA space can be depicted through
various synonymous representations. As an example, let us examine the
arithmetic plus rule for the last row:

$\displaystyle\mathbf{\hat{a}}^{(3,3)}=\mathbf{a}^{(3,1)}\circledast\mathbf{a}^{(3,2)}.$
(10)

This rule can be reformulated in the VSA fractional form in multiple manners,
an example subset of all the possible algebraic forms of this rule is the
following one:

$\displaystyle\mathbf{\hat{a}}^{(3,3)}$
$\displaystyle=(\mathbf{a}^{(3,1)}\circledast\mathbf{a}^{(3,2)}\circledast\mathbf{e})\otimes(\mathbf{e}\circledast\mathbf{e}\circledast\mathbf{e})$
(11) $\displaystyle\mathbf{\hat{a}}^{(3,3)}$
$\displaystyle=(\mathbf{a}^{(3,2)}\circledast\mathbf{a}^{(3,1)}\circledast\mathbf{e})\otimes(\mathbf{e}\circledast\mathbf{e}\circledast\mathbf{e})$
(12) $\displaystyle\mathbf{\hat{a}}^{(3,3)}$
$\displaystyle=(\mathbf{a}^{(3,1)}\circledast\mathbf{a}^{(3,2)}\circledast\mathbf{a}^{(3,2)})\otimes(\mathbf{a}^{(3,2)}\circledast\mathbf{e}\circledast\mathbf{e})$
(13)

When employing a regression loss function to the resultant VSA vector from the
Learn-VRF and to its related target VSA vector, the loss curve’s global minima
align with diverse algebraic interpretations of the identical rule.

##  Appendix D Model Limitations

The primary constraint of the Learn-VRF model lies in its expressiveness. To
understand this, consider that in the VSA space, the sum of two attribute
values is represented by the binding of their respective VSA vectors, while
the difference is depicted by the unbinding operation. Given the VSA
fractional form underpinning the Learn-VRF:

$\displaystyle\mathbf{\hat{a}}^{(3,3)}=(\mathbf{a}_{1}\circledast\mathbf{a}_{2}\circledast\mathbf{a}_{3})\otimes(\mathbf{a}_{4}\circledast\mathbf{a}_{5}\circledast\mathbf{a}_{6})$
(14)

In the numeric attribute space, this translates to:

$\displaystyle{\hat{a}}^{(3,3)}=(a_{1}+a_{2}+a_{3})-(a_{4}+a_{5}+a_{6})$ (15)

This implies that the model is limited to representing sums and differences of
attributes, precluding the possibility of modeling linear combinations. This
limitation becomes evident in complex constellations like the 2x2 grid, the
3x3 grid, or the out-in grid, where the position attribute employs logical
rules on its bit-code vector.

Table A1 and Table A2 show the number of rule occurrences of the
number/position and type/size/color attributes, respectively. On the position
attribute, the logical rules (progression and arithmetic) make up 23.9%,
26.7%, and 26.6% on the 2x2 grid, 3x3 grid, and out-in grid constellation,
respectively. Since Learn-VRF cannot represent these logical rules, an upper
bound for the accuracy can be determined by the share of non-logical rules
(i.e, 76.1%, 73.3%, and 73.4%) Indeed, Learn-VRF (weighted combination, 50
epochs) reaches almost the upper bound on the 2x2 grid (73.5% vs. 76.1%),
whereas the performances on the 3x3 grid (50.9% vs. 73.3%) and out-in grid
(65.2% vs. 73.4%) are slightly behind.

Table A1: Number of rule occurrences on the position and number attribute in
the I-RAVEN test set (2000 samples). The progression and arithmetic rule on
position attribute are logical rules, e.g., I-RAVEN implements the arithmetic
plus rule as logical OR.

| Number/Position | Number | Position |  Share of non-logical rules |  Acc. Learn-VRF (weighted comb., 50 epochs)
---|---|---|---|---|---
| Const. | Progr. | Arith. | Dist.3 |  | Progr.
---
(logical)
| Arith.
---
(logical)
Dist.3
2x2 grid | 320 | 178 | 335 | 326 | 163 | 316 | 362 | 76.1% | 73.5%
3x3 grid | 267 | 304 | 312 | 269 | 247 | 287 | 314 | 73.3% | 50.9%
O-IG (inner grid) | 325 | 158 | 321 | 342 | 175 | 357 | 322 | 73.4% | 65.2%

Table A2: Number of rule occurrences on the type, size, and color attribute in
the I-RAVEN test set (2000 samples). The sum of occurrences per attribute is
2000 for single constellations (center, 2x2 grid, 3x3 grid), and 4000 for dual
constellations (L-R, U-D, O-IC, O-IG).

| Type | Size | Color
---|---|---|---
| Const. | Progr. | Arith. | Dist.3 | Const. | Progr. | Arith. | Dist.3 | Const. | Progr. | Arith. | Dist.3
Center | 660 | 665 | 0 | 675 | 514 | 485 | 500 | 501 | 520 | 494 | 474 | 512
2x2 grid | 625 | 676 | 0 | 669 | 515 | 503 | 492 | 490 | 507 | 495 | 519 | 479
3x3 grid | 686 | 657 | 0 | 657 | 466 | 546 | 464 | 524 | 494 | 476 | 506 | 524
L-R | 1294 | 1306 | 0 | 1400 | 1031 | 983 | 972 | 1014 | 1006 | 1019 | 943 | 1032
U-D | 1332 | 1364 | 0 | 1304 | 989 | 1030 | 994 | 987 | 983 | 1015 | 983 | 1019
O-IC | 1278 | 1359 | 0 | 1363 | 1286 | 934 | 503 | 1277 | 2490 | 493 | 501 | 516
O-IG | 1334 | 1646 | 0 | 1320 | 1383 | 709 | 551 | 1357 | 2493 | 534 | 516 | 457

##  Appendix E LLM Experiments

This appendix describes our additional experiments testing the OOD
generalization of GPT-3 on I-RAVEN. Our implementation is based on [33], which
uses a text-davinci-002 model.

The attribute labels are transformed into prompts using two different
approaches. In the entangled setting (aka entity attribute naming), the
different attributes of an object are represented in one entity (e.g., packed
in brackets), yielding one prompt. Conversely, the attributes are fully
decomposed into one prompt per attribute in the disentangled setting (aka
attribute decomposition). Both approaches perform selective classification,
i.e., prompting the LLM with every answer panel in the context matrix and
selecting the candidate panel as the answer that yielded the highest sum of
log-probabilities in the answer part. On ID data, explicit disentanglement
improves LLM’s reasoning: in the center constellation, the GPT-3 achieves an
accuracy of 86.4% with disentanglement and 80.8% without disentanglement [33].

We tested the GPT-3 on our OOD test set using the disentangled setting. Note
that we do not have complete knowledge about the data used to train GPT-3.
Hence, data contamination is possible, i.e., the model could have been trained
on RPMs. As a result, there is no certainty that this test is truly OOD;
creating new synthetic datasets would be needed to have full control over the
OOD tests as it has been done in a synthetic question-answering dataset [32].
Table 3 shows that GPT-3 mainly faces challenges with the arithmetic rule,
achieving 71.6% on the size attribute and 65.8% on color.

As an additional experiment, we tested the effect of in-context learning on
the arithmetic rules. We randomly selected in-context examples that contained
the arithmetic rule on an attribute different from the one under test, e.g.,
for the OOD test with the arithmetic on the attribute size, we showed
additional examples where arithmetic was applied on color (but not size). As
no in-context examples could be shown in the disentangled configuration, we
switched to entangled setting. Fig. A2 shows an example. To reduce the number
of prompts, we only tested on a subset of 50 samples, which still gives an
indication of the trends. The results shown in Table A3 indicate that in-
context learning is rather detrimental in this application. Further efforts
such as prompt-tuning (e.g., the in-context selection and arrangement) may
improve the LLM’s in-context learning performance [34].

Figure A2: Prompt for GPT-3 experiment with three in-context examples in entangled setting. The single object in the center constellation is encoded into (type, size color). The OOD test focuses on the arithmetic rule on the attribute size. In-context examples show examples of arithmetic rule on attribute color. In this prompt, the empty panel in the text example is filled with the first candidate panel, which is not the correct one.  Table A3: Classification accuracy (%) on a subset (50 samples) of I-RAVEN OOD test set using GPT-3 (entangled setting) using a different number of in-context examples. #In-context samples | Size | Color
---|---|---
0 | 70.0 | 70.0
3 | 70.0 | 52.0
5 | 60.0 | 48.0
10 | 58.0 | 46.0