# Learning Human Mesh Recovery in 3D Scenes

Zehong Shen Zhi Cen Sida Peng Qing Shuai Hujun Bao Xiaowei Zhou†
State Key Lab of CAD&CG, Zhejiang University

###### Abstract

We present a novel method for recovering the absolute pose and shape of a
human in a pre-scanned scene given a single image. Unlike previous methods
that perform scene-aware mesh optimization, we propose to first estimate
absolute position and dense scene contacts with a sparse 3D CNN, and later
enhance a pretrained human mesh recovery network by cross-attention with the
derived 3D scene cues. Joint learning on images and scene geometry enables our
method to reduce the ambiguity caused by depth and occlusion, resulting in
more reasonable global postures and contacts. Encoding scene-aware cues in the
network also allows the proposed method to be optimization-free, and opens up
the opportunity for real-time applications. The experiments show that the
proposed network is capable of recovering accurate and physically-plausible
meshes by a single forward pass and outperforms state-of-the-art methods in
terms of both accuracy and speed. Code is available on our project page:
https://zju3dv.github.io/sahmr/.

†††Corresponding author.

##  1 Introduction

Figure 1: Comparison between the optimization-based method and the proposed
method. Optimization-based methods typically fit a parametric human model
iteratively by minimizing 2D reprojection error and scene conflicts. In
contrast, the proposed method utilizes a single forward pass of the network to
estimate the global position (blue ball), contact scene points (colored scene
points), and a scene-aware human mesh. This design leads to improvements in
both efficiency and accuracy.

Monocular human mesh recovery (HMR), i.e., estimating pose and shape
parameters of a parametric human model from a single image, has gained
significant attention in recent years. To better capture and understand human
behaviors, many recent works [1, 2, 3, 4, 5] propose to address the problem of
scene-aware HMR which involves human-scene interaction constraints when
recovering human meshes, given the 3D geometry of the scene scanned by range
sensors [5, 6, 7], as well as the camera pose of the input image relative to
the scene, which may enable more applications in video surveillance, household
robots, and motion analysis in gyms and clinics.

Most existing methods propose using scene-aware optimization to fit the human
mesh into a pre-scanned scene. They optimize a parametric human model
iteratively, e.g., SMPL [8], to minimize scene penetration, chamfer distance
of contact regions, and the 3D-2D re-projection error. However, optimization
tends to be slow at inference time and is sensitive to initialization and
hyperparameters, failing to respond in low-latency applications. As
illustrated in Fig. 1, an optimization-based method PROX [5] takes 18.4s to
fit a human model into the scene, while the incorrect position and pose still
occur.

Recent works [9, 10, 11, 12, 13] propose to recover human mesh with neural
networks trained on large-scale datasets [14, 15, 16, 17]. Specifically, the
networks learn a mapping from an input image to a human mesh in the canonical
coordinates. Applying these methods in the scene-aware HMR task still requires
post-processing optimization, where the global translation and the human poses
are refined in accordance with the given scene. However, the monocular
prediction is conditioned on the input image solely, omitting the joint
distribution of human pose and scene geometry, and therefore tends to suffer
from depth ambiguity and occlusion. As a result, the optimization-based post-
processing could be easily deteriorated by the erroneous initial poses and may
even worsen the initial prediction.

In this work, we propose a Scene-Aware Human Mesh Recovery network (SA-HMR),
the first learning-based approach that predicts the absolute position and mesh
of a human in the scene by a single forward pass. The overall pipeline is
illustrated in Fig. 2. Given the input image and scene point cloud, we first
use a sparse 3D CNN to estimate dense scene contacts and absolute human
position, where the scene contact estimation is treated as a point cloud
labeling task, and the human position prediction is presented as a voting
vector field refinement task. The predicted dense contact points are centered
by the human position and passed to a scene network in the human mesh recovery
step. Specifically, we enhance a pretrained monocular HMR network METRO [12]
by cross-attention with the proposed scene network in parallel. In this way,
SA-HMR learns a joint distribution of human pose and scene geometry, resulting
in more reasonable postures, contacts, and global positions, as illustrated in
Fig. 1. Learning scene-aware cues in the network also avoids scene-aware
optimization as post-processing and achieves fast inference speed.

We evaluate the proposed method on the RICH [6] and PROX [5] datasets of
indoor and outdoor scenes. The experimental results show that SA-HMR is not
only effective in recovering absolute positions and meshes that are in
accordance with the given scene, but also significantly faster than the
optimization-based baselines.

In summary, we make the following contributions:

  * •

The first optimization-free framework for scene-aware human mesh recovery from
a single image and a pre-scanned scene.

  * •

The cross-attention design for enhancing a pretrained HMR network with a
parallel scene network, enabling joint learning on the human pose and scene
geometry.

  * •

Superior performance compared to optimization-based baselines in terms of both
accuracy and speed.

##  2 Related Work

Monocular Human Mesh Recovery.  Most existing approaches formulate the
monocular HMR task as recovering the mesh of statistical human body models,
e.g. SMPL and SMPL-X [8, 18, 19], where recent works can be divided into
optimization-based and learning-based approaches.

The optimization-based approach fits a parametric human model by minimizing
the 3D-2D re-projection error of body joints and energy terms of heuristic
priors iteratively, which is represented by SMPLify [20] that fits SMPL [8].
More recently, SMPLify-X [19] proposes a variational pose prior and fits a
more expressive SMPL-X. Pose-NDF [21] proposes to represent the manifold of
plausible human poses with a neural field. While optimization-based methods
are general in their mathematical formulation, they are usually sensitive to
hyperparameters and require much time for inference.

The learning-based methods utilize deep neural networks to predict either the
parameters [9, 10, 22, 23] or the mesh vertices [11, 12, 13] of the SMPL
model. HMR [9] is the pioneering work in predicting SMPL parameters, and SPIN
[10] improves upon it using an optimization loop. For predicting SMPL mesh
vertices, GraphCMR [11] deforms a template human mesh using graph neural
network, while METRO [12] uses transformers, and [13] uses graph hierarchy to
further improve the performance. However, for scene-aware HMR, the vertices of
the human and scene meshes in contact are close in Euclidean space, making
methods that regress parameters unsuitable due to errors that accumulate along
the kinematic chains. Therefore, the proposed method is built on the works
that predict mesh vertices. More details can be found in Sec. 3.

Scene-aware Human Mesh Recovery.  PROX [5] is a seminal work that uses scene
constraints to reduce the depth and occlusion ambiguity in monocular HMR. It
achieves this by adding two energy terms of human-scene contact and
penetration in the optimization process [19]. In addition, scene-aware pose
generative models [2, 24] can also be used as prior terms in the scene-aware
HMR task. Other recent works in this area include MoCapDeform [4], which
considers deformable scene objects, LEMO [1], which uses temporal information,
and HULC [3], which uses consecutive frames and dense contacts prediction on
both the scene and human body. In contrast to these works, the proposed method
is optimization-free and requires only a single forward pass.

In a broader topic of capturing humans in a scene-aware manner, [25, 26, 27]
propose using a simulator and dynamic model, where a pre-defined agent is
controlled to interact with the scene, and [28, 29] consider human-object
arrangement by first predicting the human and object and then performing
global optimization.

Attention in Transformers.  Attention is a key mechanism in Transformers [30].
It allows a set of query features to fuse the most relevant information from
another set of key-value features. When query and key-value features come from
the same source, it is called self-attention, otherwise cross-attention. In
HMR, METRO [12] uses self-attention to reduce occlusion ambiguity by
establishing non-local feature exchange between visible and invisible parts of
a template human mesh. In feature matching, SuperGlue [31] uses cross-
attention to make the corresponding image features more similar. Predator [32]
uses cross-attention in matching two sets of point clouds.

Inspired by feature matching, the proposed method uses cross-attention to
potentially make the features of the human and scene that are in contact more
similar, resulting in better contact and more reasonable postures.

##  3 Methods

Figure 2:  Overview of the proposed SA-HMR. 1. The human root and scene
contact estimation module (Sec. 3.2) that first predicts the initial root and
then refines the root with 3D scene cues using a sparse 3D CNN. The module
also predicts contact labels [5] for each scene point. Please refer to Sec.
3.2 for a detailed definition of the 3D feature construction module. 2. The
scene-aware human mesh recovery module (Sec. 3.3) that enhances the pretrained
METRO [12] network with a parallel scene network. The scene network takes the
predicted contact scene points as input, and uses cross-attention to pass
messages to the intermediate features of the METRO network.

Given a calibrated image and a pre-scanned scene point cloud (Sec. 3.1), SA-
HMR first estimates the absolute human root position and scene contacts (Sec.
3.2), and then recovers the human mesh with the contact points by enhancing a
pretrained METRO network (Sec. 3.3). An overview of the proposed method is
presented in Fig. 2.

###  3.1 Preliminaries

Human Representation.  We use SMPL [8] as the human representation. The SMPL
is a parametric model that uses the body joint rotations, root translation,
and body shape coefficients to compute the body mesh. Following [11, 12], we
directly predict the SMPL mesh vertices $V=\mathbb{R}^{6890\times 3}$, and use
H36M [14] joint regression matrix $M\in\mathbb{R}^{14\times 6890}$ to compute
3D joints $J\in\mathbb{R}^{14\times 3}$ from the vertices for quantitative
evaluation, $J=MV$.

Scene and Image Representation.  We assume that the scene is pre-scanned with
range sensors, as in RICH [6] and PROX [5], and the image is calibrated and
localized in the scene, i.e. with known intrinsic and extrinsic parameters
$\\{(f,c_{x},c_{y}),(R_{c},t_{c})\\}$. Following METRO [12], we detect a
squared bounding box around the target human and resize the cropped region as
the input image $I\in\mathbb{R}^{224\times 224\times 3}$. Based on the camera
parameters and the bounding box, we select scene points that fall within the
visual frustum as the input scene point cloud $S\in\mathbb{R}^{N_{S}\times
3}$.

Human-Scene Contact.  Following PROX [5], we use 7 regions of the SMPL mesh
that are most likely to be contacted. The details are provided in the
supplementary material. Using these 7 categories and one for not being in
contact, we perform a segmentation task on the scene point cloud.

###  3.2 Human Root and Scene Contacts

Given an image $I$ bounding the human, we propose using a 2D convolutional
neural network (CNN) to extract image features $F$ and predict the initial
human root $r$. Based on the scene points $S$, we unproject the image features
$F$ to 3D, resulting in $\hat{F}$. Additionally, we calculate point-wise
offset vectors $O$ that point from a voxelized scene point cloud to the
initial root. By taking $\hat{F}$ and $O$ as input, a sparse 3D CNN predicts
the segmentation of scene contacts and the refined offsets, which are then
converted to the refined human root $r^{*}$. An overview of this process is
presented in the left column of Fig. 2.

Initial Root.  We predict the initial root $r\text{=}(X,Y,Z)$ in a 2.5D manner
following SMAP [33]. Specifically, we use a CNN to predict the 2D heatmap and
a normalized depth map of the root. Then, the 2D position $(x,y)$ is obtained
by applying argmax to the heatmap, and the corresponding normalized depth
value $\tilde{Z}$ is retrieved from the depth map. Finally, using the
intrinsic parameters ${f,c_{x},c_{y}}$ and image size $w$, the 3D root
position is computed:

$Z=\tilde{Z}\frac{f}{w}$ (1) $X=\frac{x-c_{x}}{f}\cdot Z,\quad
Y=\frac{y-c_{y}}{f}\cdot Z$ (2)

From our observation, estimating $(x,y)$ achieves good results with a mean
squared error of fewer than two pixels across datasets. However, estimating
$\tilde{Z}$ is relatively challenging, possibly due to the variations in human
shapes.

Figure 3:  3D feature construction. We voxelize a scene point cloud to a
sparse volume. The initial feature of each voxel consists of two parts, which
are the offset vector pointing from the voxel center to the human root and the
unprojected image features.

3D Feature Construction.  Based on the initial root $r$ and image features
$F$, we construct the 3D features on the voxelized scene point cloud, which is
illustrated in Fig. 3.

First, we select regions of interest around the initial root $r$ in the point
cloud. Specifically, we treat $r$ as an anchor and keep points within a radius
$\gamma_{1}$. Since $Z$ has more uncertainty than $(x,y)$, we sample two
additional anchors along the z-axis, whose distance to $r$ is $\gamma_{2}$.
Next, we construct a sparse volume $\bar{S}$ by voxelizing these points with
voxel size $s_{vox}$, where the center of each voxel is denoted as
$\bar{s}_{i}$.

For each voxel $i$, the feature consists of the offset vector ${o}_{i}$ and
the unprojected image feature $\hat{f}_{i}$. Specifically, $o_{i}$ is a vector
pointing from the voxel center to the human root:

${o}_{i}=r-\bar{s}_{i}$ (3)

$\hat{f}$ is computed by projecting voxel center $\bar{s}_{i}$ onto the image
using camera parameters and bilinearly sampling the image feature map $F$.

Estimating Refined Root and Scene Contacts.  We use sparse 3D CNN [34] to
process the constructed 3D features and learn to improve the root estimation
and predict the scene contacts. Specifically, the output of each voxel
includes an updated offset vector ${o}^{*}_{i}$, confidence $c_{i}$, and
segmentation indicating the contact category. We compute the refined root
$r^{*}$:

$r^{*}=\sum_{i}c_{i}\cdot(o^{*}_{i}+\bar{s}_{i}).$ (4)

There are 8 categories of contact points, including 7 most probable regions on
the body that would be contacted [5] and 1 category of not being in contact.
We take the category of the highest score as the prediction for the voxel and
set the dense point cloud belonging to the voxel with that category. The
contact points $\hat{S}_{seg3d}\in\mathbb{R}^{\hat{N}_{S}\times 3}$ serve as
the input for the mesh recovery module.

###  3.3 Scene-aware Human Mesh Recovery

Figure 4: Enhancing METRO with a parallel network. The orange parts are the
original METRO [12] network, where the residual connection and positional
encoding are omitted for simplicity. The blue parts are the proposed parallel
network which takes predicted contact scene points as input. The yellow parts
indicate feature interaction between METRO and the parallel network.

Since the training data of scene-aware human mesh recovery is limited, we
build our model upon a network named METRO [12] that is pre-trained on large-
scale data of monocular human mesh recovery. METRO processes feature based on
the self-attention mechanism, and our approach enhances METRO by adding a
parallel scene network, which provides a cross-attention-based mechanism that
enables METRO to notice important scene details and achieve scene-aware human
mesh recovery.

METRO  consists of a CNN backbone and multiple Transformer encoders. It first
extracts global CNN features, then combines the feature to the vertices of a
zero posed SMPL mesh, and finally predicts a posed mesh with shape through the
transformers. The part of the transformer is illustrated in the orange part of
Fig. 4.

Enhancing METRO with Cross-Attention.  We improve METRO with a scene network,
which makes the predicted human vertices to be close to the corresponding
contact scene points $\hat{S}_{seg3d}$ (Sec. 3.2).

As illustrated in Fig. 4, we add a parallel network, which has a similar
architecture as the transformer of METRO, to extract features of scene contact
points and output the point positions like an autoencoder. Specifically, we
first use METRO’s CNN backbone to extract the image feature and map it to a
set of vertex tokens by a fully-connected layer. In METRO, these tokens are
directly concatenated with the positions of initial human vertices. However,
the number of scene contact points is not the same as the tokens’, and there
is no one-to-one correspondence between them. To resolve this issue, we
perform the average pooling to vertex tokens based on the contact categories
defined on the SMPL mesh vertices [5], resulting in 7 tokens. Then, we append
each scene contact point with the corresponding aggregated token based on the
predicted category in Sec. 3.2. Intuitively, this helps the cross-attention to
focus on the semantically corresponding parts. To be invariant to the global
translation, the scene contact points are zero-centered by the predicted root
$r^{*}$.

Figure 5: Visualization of the cross-attention from a body vertex (blue point)
to the predicted dense scene contacts (white points). (a) The image and the
predicted dense scene contacts. (b) The hand vertex is in contact with the
scene according to the image, and its feature is similar to the nearby scene
point features, enabling the final vertex prediction to be close to the
corresponding scene surface. (c) The arm vertex is not in contact with the
scene, where feature similarities tend to be evenly distributed. The feature
similarities are normalized to the same range of 0$\to$1\.

Motivated by recent feature matching methods [31, 32], we propose to use
cross-attention to pass features from scene contact points to human vertices.
The cross-attention and self-attention share the same underlying mechanism,
both of which first compute the similarity of query and key, and then use the
weighted sum to fuse features. When the query and key are from the same source
features, it is self-attention, and otherwise is cross-attention. In practice,
we use linear attention operator [35] to improve efficiency. A visualization
of cross attention over human vertices and scene points is in Fig. 5. The
detailed network architecture is provided in the supplementary material.

Note that, we use a weight-sharing regressor layer with METRO, which regresses
from point-wise features to point position $(x,y,z)$. Therefore, to get a
similar $(x,y,z)$, the input features of this layer should also be similar.
This strategy implicitly aligns the features of human vertices and scene
points when their final predictions are near in 3D space, thus facilitating
the cross-attention to find correspondences between human vertices and scene
points.

###  3.4 Training Loss

Root and Contact.  The loss function $L_{\text{RC}}$ for the root and contact
estimation is defined as:

$L_{\text{RC}}=L_{\text{R2D}}+w_{\text{RZ}}\cdot
L_{\text{RZ}}+L_{\text{ROV}}+L_{\text{R3D}}+L_{\text{C}}$ (5)

where $L_{\text{R2D}}$ is the MSE loss on the root heatmap; $L_{\text{RZ}}$,
$L_{\text{ROV}}$, and $L_{\text{R3D}}$ are the L1 losses on the relative
depth, offset vectors, and the 3D root, respectively; $L_{\text{C}}$ is the
cross-entropy loss for contact categories of voxel points, where we
additionally train an auxiliary task of 2D contact segmentation similar to the
voxel points.

Human Mesh Recovery.  The loss function $L_{\text{HMR}}$ for the mesh recovery
is defined as:

$L_{\text{HMR}}=L_{\text{V}}+L_{\text{J}}+L_{\text{CP}}+L_{\text{GV}}$ (6)

where $L_{\text{V}}$, $L_{\text{J}}$, $L_{\text{CP}}$, and $L_{\text{GV}}$ are
the L1 losses on the translation-aligned human vertices, human joints,
reconstructed contact points, and global human vertices, respectively. More
details are in the supplementary material.

###  3.5 Implementation Details

We train two modules separately. For the root and contact module, the CNN is
HRNet-stride-4 [36] with METRO initialization, the sparse 3D CNN is SPVCNN
[34, 37] with random initialization. We use linear layers to align
intermediate feature dimensions. $\gamma_{1}$ is $1.25m$, $\gamma_{2}$ is
$0.5m$, $s_{vox}$ is $5^{3}cm^{3}$, and $w_{\text{RZ}}$ is 10. The contact
threshold is $7cm$. We flip images for augmentation. The module is trained
with an initial learning rate of 3.75e-5 and a batch size of 24. It converges
after 30 epochs of training on one V100 GPU. For the mesh recovery module, the
METRO network is initialized as pretrained and the scene network is randomly
initialized. The initial learning rate is 7.5e-6 and the batch size is 24. It
converges after 30 epochs of training.

##  4 Experiments

Method | Learning-based | Optimization | Scene-aware |  G-MPJPE$\downarrow$ |  G-MPVE$\downarrow$ |  PenE$\downarrow$ |  ConFE$\downarrow$ |  MPJPE$\downarrow$ |  MPVE$\downarrow$
---|---|---|---|---|---|---|---|---|---
Dataset GT [6] |  | ✓ | ✓ | / | / |  9.8 | 10.8 | / | /
SMPLify-X [19] |  | ✓ |  | 482.0 | 483.7 | 35.7 | 43.4 | 166.9 | 177.6
PROX [5] |  | ✓ | ✓ | 390.1 | 397.2 | 15.5 | 24.1 | 164.1 | 175.8
POSA [2] |  | ✓ | ✓ | 427.8 | 434.0 | 21.1 | 27.0 | 177.2 | 188.4
PLACE [38] |  | ✓ | ✓ | 395.9 | 403.0 | 16.1 | 24.8 | 163.8 | 175.4
METRO [12]† \+ SA-Opt [6, 29] | ✓ | ✓ | ✓ | 563.1 | 561.3 |  7.4 | 14.8 | 102.7 | 112.8
METRO [12] | ✓ |  |  | 678.6 | 679.4 | 52.2 | 56.9 | 129.6 | 134.5
METRO [12]† | ✓ |  |  | 511.7 | 509.7 | 33.6 | 37.6 |  98.8 | 107.9
SA-HMR | ✓ |  | ✓ | 264.6 | 272.7 | 14.9 | 19.0 |  93.9 | 103.0

Table 1:  Evaluation on the RICH [6] dataset. METRO† indicates that the model
is finetuned on the dataset. SA-Opt indicates scene-aware optimization, with
contact estimation from BSTRO [6] and loss formulation from PROX [5] and PHOSA
[29]. The proposed SA-HMR achieves the overall best results and is
significantly faster than the methods that require optimization.

###  4.1 Datasets

We train and evaluate the proposed method on RICH [6] and PROX [5] datasets
separately.

RICH [6] captures multi-view video sequences in 6 outdoor and 2 indoor
environments. It provides images, reconstructed bodies, scene scans, and
human-scene contact labels annotated on SMPL vertices. We skip frames
including multiple subjects, remove the first 45 frames of each video to avoid
static starting pose, and skip frames where the subjects’ 2D bounding boxes
are not inside the images. Then we downsample the train / val / test splits to
2 / 1 / 1 fps, resulting in 15360 / 3823 / 3316 frames.

PROX [5] captures monocular RGBD videos in 12 indoor environments. We use RGB
images and scene scans. It is a challenging dataset where severe occlusions
exist in most frames. We use the qualitative set for training and the
quantitative set for testing. In order to get better training annotations, we
additionally combine HuMoR [26] which utilizes motion prior and optimizes a
sequence of frames. Then, we manually remove the failed frames that are not
consistent with the images and scenes. Finally, the training split contains
4852 frames.

###  4.2 Metrics

We evaluate quantitatively in terms of human mesh recovery and human-scene
contact.

Human Mesh Recovery.  We report the Global Mean-Per-Joint-Position-Error
(G-MPJPE) and Global Mean-Per-Vertex-Error (G-MPVE) in scene coordinates,
which calculates the average L2 distances between predicted and ground truth
joints/vertices. Additionally, we report the translation-aligned metrics MPJPE
and MPVE.

Human-scene Contact.  We report the Penetration Error (PenE) and Contact
Failure Error (ConFE). PenE measures the total distance that SMPL vertices
penetrate the scene mesh:

$\text{PenE}=\sum_{i=1}^{V}\mathbbm{1}_{x<0}[sdf(v_{i},\mathcal{S})]\cdot|sdf(v_{i},\mathcal{S})|,$
(7)

where $V$ is the number of SMPL vertices, $sdf\left(v_{i},\mathcal{S}\right)$
is the signed distance of vertex $v$ to scene $\mathcal{S}$, and
$\mathbbm{1}_{x<0}[\cdot]$ is an indicator function that returns 1 when the
condition is met, and 0 otherwise. ConFE measures contact quality when the
ground-truth contact label is available:

$\begin{split}\text{ConFE}=&\sum_{i=1}^{V}(C_{gt}(v_{i})\cdot|sdf(v_{i},\mathcal{S})|\\\
+&(1-C_{gt}(v_{i}))\cdot\mathbbm{1}_{x<0}[sdf(v_{i},\mathcal{S})]\cdot|sdf(v_{i},\mathcal{S})|),\\\
\end{split}$ (8)

where $C_{gt}(v)$ equals 1 if $v$ is labeled as in contact, and 0 otherwise.
In order to obtain a good result of ConFE, the body vertices in contact should
be near the scene surface, while vertices not in contact should avoid
penetration.

###  4.3 Main Results

Method |  G-MPJPE$\downarrow$ |  G-MPVE$\downarrow$ |  PenE$\downarrow$ |  MPJPE$\downarrow$ |  MPVE$\downarrow$
---|---|---|---|---|---
Dataset GT [5] | / | / |  9.6 | / | /
SMPLify-X [19] | 216.0 | 222.6 | 49.3 | 100.7 | 112.8
PROX [5] | 172.0 | 178.5 | 10.7 | 101.1 | 114.0
POSA [2] | 172.3 | 180.9 | 16.6 | 108.5 | 119.4
PLACE [38] | 168.1 | 176.7 | 12.3 | 100.8 | 113.7
METRO [12] | 283.2 | 277.7 | 62.4 | 137.0 | 147.2
METRO [12]† | 265.6 | 262.7 | 67.5 | 117.1 | 128.5
SA-HMR | 150.4 | 160.0 | 26.9 | 111.1 | 122.5

Table 2: Evaluation on the PROX [6] dataset. The proposed method achieves the
best performance in global metrics.

Baselines.  Optimization: SMPLify-X [19] uses RGB only, PROX [5] extends it
with losses of human-scene contact and penetration, POSA [2] and PLACE [38]
extend PROX with scene-aware pose priors, and METRO [12]†+SA-Opt stands for
post-processing a finetuned METRO with scene-aware optimization, which will be
explained later. Learning-based: METRO predicts canonical human mesh vertices
and a weak-perspective camera. We solve the transformation from human to
camera coordinate by minimizing joint re-projection error with a PnP solver
[39]. The METRO† is finetuned with the same training protocol as SA-HMR. Since
METRO does not consider scenes, we additionally optimize the global pose and
scale by minimizing scene-aware losses, including re-projection error, human-
scene penetration, contact distance [6], and ordinal depth error [29],
following the key ideas of PROX and PHOSA [29].

Results.  For the RICH dataset, Tab. 1 shows that SA-HMR notably outperforms
other baselines in terms of the G-MPJPE and G-MPVE by a significant margin,
demonstrating the effectiveness of the proposed pipeline. The joint learning
on both image and scene geometry also improves the metrics of local pose and
human-scene contact. We use open-sourced code for SMPLify-X and PROX, and
implement POSA and PLACE upon PROX. Optimization methods approximately cost
18s for a single fitting, which is much slower compared to 0.2s of SA-HMR. We
provide qualitative results comparing to baselines in Fig. 6.

For the PROX dataset, SA-HMR outperforms all baselines in terms of global
accuracy as illustrated in Tab. 2. Since the pseudo ground truth is still not
of low quality for the training set of PROX, as well as a domain gap exists in
the test set where the subject wears a MoCap suit, our method falls a little
behind in local accuracy and scene penetration. And we do not report ConFE,
since the ground-truth contact label is not available. Nevertheless, the clear
improvement compared to the most relevant model METRO† has demonstrated the
effectiveness of the proposed method.

We also observe that while considering the scene geometry is critical for
estimating the global position and improving physical plausibility, it may not
fully resolve the ambiguity of the local pose, where multiple physically
plausible solutions may still exist. For example, the RGB-only method
SMPLify-X and the scene-aware method PROX perform similarly in MPJPE and MPVE.

###  4.4 Ablation Study

Dataset | Initial RtErr | Refined RtErr | Final RtErr
---|---|---|---
RICH | 510.8 | 284.7 | 246.5
PROX | 364.2 | 132.3 | 111.8

Table 3:  Ablation study of root estimation. The human root position errors
(RtErr) in mm are reported.

Root and Contact Module.  Tab. 3 shows that the predicted human root position
is improved progressively by the refinement and scene-aware HMR modules, where
the initial prediction [33] is improved 44%/52% in RICH, and 64%/69% in PROX.
The offset representation helps to improve erroneous initial root prediction
that is not consistent with the scene surface. For scene contact estimation,
the precision/recall is 0.57/0.53 on RICH, and 0.45/0.24 on PROX. We observe
that the contacts are difficult to predict, which aligns with the conclusion
of a recent work HULC [3]. More visualizations are presented in Fig. 7.

Method | G-MPJPE | G-MPVE | MPJPE | MPVE |  CErr$\downarrow$
---|---|---|---|---|---
w/o parallel | 304.8 | 312.9 | 98.5 | 108.9 | 10.2
Ours | 264.6 | 272.7 | 93.9 | 103.0 |  8.9

Table 4: Ablation study of the parallel network on the RICH dataset. The
compared variant fuses features of contact points at the early stage.

Root | Contact | MPJPE | MPVE |  CErr$\downarrow$
---|---|---|---|---
/ | / | 98.8 | 107.9 | 10.5
Est. | Est. | 93.9 | 103.0 |  8.9
GT | Est. | 89.2 | 98.1 |  7.9
Est. | GT | 90.4 | 99.2 |  8.3
GT | GT | 76.7 | 84.6 |  5.2

Table 5: Ablation study of the scene-aware mesh recovery module. We validate
the upper bound of the proposed method on the RICH dataset.

Mesh Recovery Module.  As shown in Tab. 4, the parallel network that uses
cross-attention outperforms a variant that fuses the pointnet features of the
contact points to the METRO network in the early stage. The CErr indicates the
error of contact mesh vertices in the translation-aligned coordinates. In Tab.
5, we validate the upper bound of the mesh recovery module. We replace the
intermediate estimation of root and contact, and find a steady improvement in
the pose and shape accuracy that outperform the baseline.

Running Time.  SA-HMR runs at 170ms with a peak memory cost of 1852 MB for a
$224\times 224$ image and a scene point cloud of $2cm$ resolution on a V100
GPU. Specifically, the root and contact module takes 92ms (CNN 50ms, SPVCNN
42ms), the mesh recovery module 75ms (CNN 49ms, Transformer 26ms), and 3ms for
the intermediate processing.

##  5 Conclusion

This work addressed the challenge of estimating the human mesh from an RGB
image with the consideration of the scene geometry. Our key idea is to inject
3D scene cues into a monocular human mesh recovery network to recover the
absolute human pose and shape in the scene. To this end, our approach first
predicts the 3D human location and then uses a sparse 3D CNN to estimate dense
human-scene contacts. We developed a transformer to extract features from
contact scene points and fed them into the pose estimation network using the
cross-attention scheme. Experiments demonstrated our approach achieves state-
of-the-art performance on the RICH and PROX datasets.

Acknowledgement.  This work was supported by NSFC (Grant 62172364) and the
Information Technology Center and State Key Lab of CAD&CG, Zhejiang
University.

Figure 6: Qualitative results on the RICH [6] dataset. We compare the proposed
method to PROX [5], finetuned METRO [12], finetuned METRO with scene-aware
optimization, and ground truth. The leftmost column shows the input images.
The proposed method recovers the global positions and human-scene contact more
accurately because of the 3D learning on human root refinement and dense scene
contact labeling tasks.  Figure 7: Qualitative visualization of the estimated
root locations and dense scene contacts. In both examples, the estimated
contact points provide accurate position and scene structure for the following
step of mesh recovery. The reconstructed human mesh is in good contact with
the corresponding scene regions.

## References

  * [1] Siwei Zhang, Yan Zhang, Federica Bogo, Marc Pollefeys, and Siyu Tang.  Learning motion priors for 4d human body capture in 3d scenes.  In ICCV, 2021.
  * [2] Mohamed Hassan, Partha Ghosh, Joachim Tesch, Dimitrios Tzionas, and Michael J. Black.  Populating 3d scenes by learning human-scene interaction.  In CVPR, 2021.
  * [3] Soshi Shimada, Vladislav Golyanik, Zhi Li, Patrick Pérez, Weipeng Xu, and Christian Theobalt.  HULC: 3d human motion capture with pose manifold sampling and dense contact guidance.  In ECCV, 2022.
  * [4] Zhi Li, Soshi Shimada, Bernt Schiele, Christian Theobalt, and Vladislav Golyanik.  MoCapDeform: Monocular 3d human motion capture in deformable scenes.  In 3DV, 2022.
  * [5] Mohamed Hassan, Vasileios Choutas, Dimitrios Tzionas, and Michael J. Black.  Resolving 3d human pose ambiguities with 3d scene constraints.  In ICCV, 2019.
  * [6] Chun-Hao P. Huang, Hongwei Yi, Markus Höschle, Matvey Safroshkin, Tsvetelina Alexiadis, Senya Polikovsky, Daniel Scharstein, and Michael J. Black.  Capturing and inferring dense full-body human-scene contact.  In CVPR, 2022.
  * [7] Siwei Zhang, Qianli Ma, Yan Zhang, Zhiyin Qian, Marc Pollefeys, Federica Bogo, and Siyu Tang.  Egobody: Human body shape, motion and social interactions from head-mounted devices.  2022\.
  * [8] Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J Black.  Smpl: A skinned multi-person linear model.  TOG, 2015.
  * [9] Angjoo Kanazawa, Michael J. Black, David W. Jacobs, and Jitendra Malik.  End-to-end recovery of human shape and pose.  In CVPR, 2018.
  * [10] Nikos Kolotouros, Georgios Pavlakos, Michael J. Black, and Kostas Daniilidis.  Learning to reconstruct 3d human pose and shape via model-fitting in the loop.  In ICCV, 2019.
  * [11] Nikos Kolotouros, Georgios Pavlakos, and Kostas Daniilidis.  Convolutional mesh regression for single-image human shape reconstruction.  In CVPR, 2019.
  * [12] Kevin Lin, Lijuan Wang, and Zicheng Liu.  End-to-end human pose and mesh reconstruction with transformers.  In CVPR, 2021.
  * [13] Kevin Lin, Lijuan Wang, and Zicheng Liu.  Mesh graphormer.  In ICCV, 2021.
  * [14] Catalin Ionescu, Dragos Papava, Vlad Olaru, and Cristian Sminchisescu.  Human3.6m: Large scale datasets and predictive methods for 3d human sensing in natural environments.  TPAMI, 2013.
  * [15] Timo Von Marcard, Roberto Henschel, Michael J Black, Bodo Rosenhahn, and Gerard Pons-Moll.  Recovering accurate 3d human pose in the wild using imus and a moving camera.  In ECCV, 2018.
  * [16] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In ECCV, 2014.
  * [17] Mykhaylo Andriluka, Leonid Pishchulin, Peter Gehler, and Bernt Schiele.  2d human pose estimation: New benchmark and state of the art analysis.  In CVPR, 2014.
  * [18] Javier Romero, Dimitrios Tzionas, and Michael J. Black.  Embodied hands: Modeling and capturing hands and bodies together.  TOG, 2017.
  * [19] Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed A. A. Osman, Dimitrios Tzionas, and Michael J. Black.  Expressive body capture: 3D hands, face, and body from a single image.  In CVPR, 2019.
  * [20] Federica Bogo, Angjoo Kanazawa, Christoph Lassner, Peter Gehler, Javier Romero, and Michael J Black.  Keep it smpl: Automatic estimation of 3d human pose and shape from a single image.  In ECCV, 2016.
  * [21] Garvita Tiwari, Dimitrije Antić, Jan Eric Lenssen, Nikolaos Sarafianos, Tony Tung, and Gerard Pons-Moll.  Pose-NDF: Modeling human pose manifolds with neural distance fields.  In ECCV, 2022.
  * [22] Hanbyul Joo, Natalia Neverova, and Andrea Vedaldi.  Exemplar fine-tuning for 3d human model fitting towards in-the-wild 3d human pose estimation.  In 3DV, 2021.
  * [23] Muhammed Kocabas, Chun-Hao P Huang, Joachim Tesch, Lea Müller, Otmar Hilliges, and Michael J Black.  Spec: Seeing people in the wild with an estimated camera.  In ICCV, 2021.
  * [24] Yan Zhang, Mohamed Hassan, Heiko Neumann, Michael J. Black, and Siyu Tang.  Generating 3d people in scenes without people.  In CVPR, 2020.
  * [25] Soshi Shimada, Vladislav Golyanik, Weipeng Xu, and Christian Theobalt.  Physcap: Physically plausible monocular 3d motion capture in real time.  TOG, 2020.
  * [26] Davis Rempe, Tolga Birdal, Aaron Hertzmann, Jimei Yang, Srinath Sridhar, and Leonidas J Guibas.  Humor: 3d human motion model for robust pose estimation.  In ICCV, 2021.
  * [27] Zhengyi Luo, Shun Iwase, Ye Yuan, and Kris Kitani.  Embodied scene-aware human pose estimation.  In NeurIPS, 2022.
  * [28] Zhenzhen Weng and Serena Yeung.  Holistic 3d human and scene mesh estimation from single view images.  In CVPR, 2021.
  * [29] Jason Y Zhang, Sam Pepose, Hanbyul Joo, Deva Ramanan, Jitendra Malik, and Angjoo Kanazawa.  Perceiving 3d human-object spatial arrangements from a single image in the wild.  In ECCV, 2020.
  * [30] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In NeurIPS, 2017.
  * [31] Paul-Edouard Sarlin, Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich.  Superglue: Learning feature matching with graph neural networks.  In CVPR, 2020.
  * [32] Shengyu Huang, Zan Gojcic, Mikhail Usvyatsov, Andreas Wieser, and Konrad Schindler.  Predator: Registration of 3d point clouds with low overlap.  In CVPR, 2021.
  * [33] Jianan Zhen, Qi Fang, Jiaming Sun, Wentao Liu, Wei Jiang, Hujun Bao, and Xiaowei Zhou.  SMAP: Single-shot multi-person absolute 3d pose estimation.  In ECCV, 2020.
  * [34] Haotian Tang, Zhijian Liu, Shengyu Zhao, Yujun Lin, Ji Lin, Hanrui Wang, and Song Han.  Searching efficient 3d architectures with sparse point-voxel convolution.  In ECCV, 2020.
  * [35] Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret.  Transformers are rnns: Fast autoregressive transformers with linear attention.  In ICML, 2020.
  * [36] Ke Sun, Bin Xiao, Dong Liu, and Jingdong Wang.  Deep high-resolution representation learning for human pose estimation.  In CVPR, 2019.
  * [37] Haotian Tang, Zhijian Liu, Xiuyu Li, Yujun Lin, and Song Han.  TorchSparse: Efficient Point Cloud Inference Engine.  In MLSys, 2022.
  * [38] Siwei Zhang, Yan Zhang, Qianli Ma, Michael J. Black, and Siyu Tang.  PLACE: Proximity learning of articulation and contact in 3d environments.  In 3DV, 2020.
  * [39] G. Bradski.  The OpenCV Library.  Dr. Dobb’s Journal of Software Tools, 2000.


# fruit-SALAD: A Style Aligned Artwork Dataset to reveal similarity perception
in image embeddings

Tillmann Ohm  Tallinn University, School of Digital Technologies
corresponding author: Tillmann Ohm<EMAIL_ADDRESS> Andres Karjus
Tallinn University, School of Humanities  Mikhail Tamm  Tallinn University,
School of Digital Technologies  Maximilian Schich  Tallinn University, Baltic
Film, Media and Arts School

###### Abstract

The notion of visual similarity is essential for computer vision, and in
applications and studies revolving around vector embeddings of images.
However, the scarcity of benchmark datasets poses a significant hurdle in
exploring how these models perceive similarity. Here we introduce Style
Aligned Artwork Datasets (SALADs), and an example of fruit-SALAD with 10,000
images of fruit depictions. This combined semantic category and style
benchmark comprises 100 instances each of 10 easy-to-recognize fruit
categories, across 10 easy distinguishable styles. Leveraging a systematic
pipeline of generative image synthesis, this visually diverse yet balanced
benchmark demonstrates salient differences in semantic category and style
similarity weights across various computational models, including machine
learning models, feature extraction algorithms, and complexity measures, as
well as conceptual models for reference. This meticulously designed dataset
offers a controlled and balanced platform for the comparative analysis of
similarity perception. The SALAD framework allows the comparison of how these
models perform semantic category and style recognition task to go beyond the
level of anecdotal knowledge, making it robustly quantifiable and
qualitatively interpretable.

Figure 1: Overview of one instance of 10 fruit categories in 10 styles. The
columns display fruit categories including from left to right: blueberries,
fig, strawberry, apple, orange, pineapple, bananas, pear, avocado, kiwi. The
rows display style labels trying to describe the style prompts from top to
bottom: ’Crayon’, ’Watercolor’, ’Comic’, ’Pixel’, ’Patch’, ’Cubic’, ’Oilio’,
’Glamour’, ’Lomo’, ’Pile’. The full dataset contains 100 instances for each
fruit category-style combination.

## Background & Summary

Similarity perception is an abstract and complex concept that differs widely
across mental and computational models, as explored in (computational)
neuroscience [1, 2], computer vision [3, 4, 5, 6], or (computational)
cognitive science [7, 8]. For mental and conceptual models, similarity refers
to resemblance or alikeness and describes groups with some shared properties,
as prominently outlined in Wittgenstein’s remarks on family resemblance [9,
10]. Conversely, in computational models, similarity denotes proximity and is
conventionally defined as inversely correlated with distance between data
points in a metric space.

Computer Vision applications heavily rely on such visual similarity, often
utilizing vector embeddings that set up a measurable multidimensional space to
index images. In similarity learning the goal is to train models that can
accurately capture the underlying similarities between data points, enabling
tasks such as image retrieval or classification based on similarity metrics
[11, 12, 13, 14, 15, 16, 17]. However, similarity in these contexts is often
implied to be understood in a singular notion, overlooking the multifaceted
nature of similarity perception crucial for informed decision-making in
selecting models or methods. For instance, Ref [18] utilizes CLIP [19] and
DINO [20] to evaluate subject fidelity of generated images, acknowledging the
varying importance of different similarity aspects. It is generally considered
that CLIP captures semantic relationships, while DINO focuses more on visual
features. Yet, validating such assumptions poses a significant challenge.

Research in quantitative and computational aesthetics [21, 22, 23], as well as
the interplay of computation and human cultures [24, 25], requires reliable
benchmark datasets that are interpretable by machines and humans. Previous
work has relied on embeddings of large amounts of well known artworks [26, 27]
or synthetic datasets of limited size [28, 29, 30, 31].

Benchmark image datasets for perceptual similarity judgment exist, with some
relying on annotated text captions of real-world images [32], while others
utilize synthetic image triplets designed to better align with mental models
[6]. However, these datasets primarily focus on specific tasks or aspects of
similarity perception and alignment, such as zero-shot evaluation or
similarity metric optimization.

Here we propose Style Aligned Artwork Datasets (SALADs), with the fruit-SALAD
serving as an exemplar. This synthetic image dataset comprises 10,000
generated images featuring 10 easily recognizable fruit categories, each
represented in 10 visually distinct styles, with 100 instances each. The
deliberate control over semantic and stylistic properties inherent to each
image facilitates comparative analysis of different image embedding and
complexity models, enabling an exploration of their similarity perception.

We characterize the dataset through various machine learning models and
complexity measures, showcasing how simple pairwise comparisons of image
vectors can yield robust inter-comparable measures. Our examples reveal
significant differences in similarity awareness across these methods and
models, shedding light on anecdotal considerations stemming from differences
in model or algorithm design, training data, parameter configuration, or
similarity measures. In turn, this approach can be used to guide model
training and alignment.

The fruit-SALAD offers opportunities for joint robust quantification and
qualitative human interpretation, enhancing algorithmic and human perception
regarding differences in measuring vector similarity and visual resemblance
across computational, and statistical models. This approach allows for a more
comprehensive assessment of similarity perception, beyond the scope of
existing benchmark datasets, ultimately contributing to a deeper understanding
of computational and human similarity perception mechanisms.

## Methods

### Image Generation

We used Stable Diffusion XL (SDXL)[33] and StyleAligned [34] to create the
fruit-SALAD by carefully crafting image generation prompts and supervising the
automation process. Diffusion probabilistic models [35] are typically trained
with the objective of denoising blurred images. By leveraging their ability to
iteratively refine images by processing random noise, these models can be used
in conjunction with text prompts to generate images. Such Text-to-Image
models, exemplified by DALL-E [36, 37], Midjourney [38] and StableDiffusion
[39], have recently gained significant attention in various creative and
commercial domains. These models and services have simplified the synthesis of
high-quality individual images, enabling unprecedented ease of use through
natural language. However, scaling the generation process or achieving
stylistically consistent images remains challenging but can be improved by
style alignment method [34] to coordinate shared attention across multiple
generations based on a reference style image.

Figure 2: Overview of the image generation process. (a) Image generation
pipeline. 1. Style reference image generation with Stable Diffusion XL [33] in
manual trial-and-error fashion using text prompts of style description in
combination with ‘an apple’. 2. Style aligned image generation [34] based on
each style reference image using diffusion inversion and text prompts
iterating over 10 fruit categories generating 100 instances each, resulting in
10,000 images. (b) Examples of tolerated and rejected results. Left: tolerated
minor issues which do not impact recognition of category or style; right:
rejected major issues which are either unrecognizable or inconsistent across
the style.

We utilized a computational approach to scale the image generation process.
Initially, we experimented in a trial-and-error fashion with different style
prompts in conjunction with different fruit categories, using SDXL [33] for
image generation. Successful results were selected as style references. We
then used style alignment [34] to generate multiple instances of different
fruits within the same style using diffusion inversion [40] of the reference
image. Through several iterations and adjustments to the prompts, we refined
the process and eventually automated the generation to produce 100 instances
for each fruit-style combination.

The fruit prompts and stylistic references we selected were carefully curated
to improve the robustness of the style alignment generation method. Among the
fruit prompts, we balanced between fruit prototypicality and variability
across different stylistic prompts to ensure compatibility with generation on
scale, while simultaneously covering a wide range of fruit shapes and colors.
Similarly, our selection of stylistic references was based on their
effectiveness in aligning with the generation space, focusing on those that
demonstrated superior performance in achieving stylistic coherence (see
examples in Fig. 1 and Fig. 2b).

We maintained dataset quality by visually assessing the entire dataset in 100
batches of 10 by 10 image grids (see example in Fig. 1) and manually replaced
images that were inconsistent across all instances. Therefore, the final
dataset with category and style classes may be biased by our own aesthetic
arbitration, which is akin to the inherent specificity of a chosen set of
handwritten digits [41].

### Image Embeddings

Our exemplary vector embeddings are derived from machine learning models and
compression algorithms through various commonly employed methods (Table 1).
For [42, 20, 43, 19, 44, 45] we extracted feature vectors using the flattened
last hidden states. For [46, 47, 48] we used average pooling from the second
to last layer.

model short name | type | training set | dimensions
---|---|---|---
ViT-B-16_IN21k | Vision Transformer [42] (base, 16x16) | ImageNet-21k [49] | 768
ViT-B-32_IN21k | Vision Transformer [42] (base, 32x32) | ImageNet-21k [49] | 768
ViT-H-14_IN21k | Vision Transformer [42] (huge, 14x14) | ImageNet-21k [49] | 1280
DINO_IN1k | DINO [20], Vision Transformer [42] (base, 16x16) | ImageNet-1k [49] | 768
DINOv2_B_LVD | DINOv2 [43] (base) | LVD-142M [43] | 768
ResNet50_IN1k | ResNet [46] | ImageNet-1k [49] | 2,048
VGG19_IN1k | VGG [47] | ImageNet-1k [49] | 512
Xception_IN1k | Xception [48] | ImageNet-1k [49] | 2,048
ConvNeXt_L400M | ConvNeXt [45] (base) | LAION-400M [50] | 512
ConvNeXt-v2_L400M | ConvNeXt-V2 [44] | LAION-400M [50] | 320
CLIP-ViT-B-16_L2B | CLIP [19], Vision Transformer [42] (base, 16x16) | LAION-2B [51] | 512
CLIP-ViT-B-32_L2B | CLIP [19], Vision Transformer [42] (base, 32x32) | LAION-2B [51] | 512
CLIP-ViT-H-14_L2B | CLIP [19], Vision Transformer [42] (huge, 14x14) | LAION-2B [51] | 1,024
CLIP-ViT-B-16_L400M | CLIP [19], Vision Transformer [42] (base, 16x16) | LAION-400M [50] | 512
CLIP-ViT-B-16_OA | CLIP [19], Vision Transformer [42] (base, 16x16) | OpenAI (undisclosed) | 512
CLIP-RN50_OA | CLIP [19], ResNet50 [46] | OpenAI (undisclosed) | 1,024
CLIP-RN101_OA | CLIP [19], ResNet101 [46] | OpenAI (undisclosed) | 512
Table 1:  Pre-trained Machine Learning models used for feature extraction. model short name | method | dimensions
---|---|---
CompressionEnsembles | Compression Ensembles [21] | 87
GIF_compression | LZW [52] to PNG file size ratios | 1
PNG_filesizes | original PNG file sizes | 1
style_blind | one-hot encoding of fruit category only, ignoring styles | 10
category_blind | one-hot encoding of styles only, ignoring fruit category | 10
balanced | one-hot encoding of fruit category and styles | 20
Table 2: Other methods used for feature extraction.

As an example of a quantitative aesthetics measure, we used the Compression
Ensembles method [21], which captures polymorphic family resemblance via a
number of transformations (87 in our implementation). We used GIF image
compression ratios, taking advantage of the Lempel–Ziv–Welch algorithm [52].
We also provide the PNG file sizes as comparison (Table 2).

To provide simple conceptual models for reference, we used binary, one-hot
encoded vectors. In this encoding scheme, each vector represents a fruit
category or style, with a value of 1 indicating the presence and 0 indicating
the absence of the corresponding category or style (Table 2). We are
consciously providing a simple conceptual reference, to avoid the
complications of full blown conceptual reference models, such as the CIDOC-CRM
[53].

## Technical Validation

### Self Recognition Test

One expects that, despite inevitable variation in similarity perception, the
similarity of images from the same category-style combination should be
systematically larger than between images of different categories and/or
styles. To assess this, we conduct a self-recognition test on the fruit-
SALAD_10k dataset. This test involves retrieving the top 100 Nearest Neighbors
for each image and counting how many instances of the same category-style
combination are found within this set. The average number of successful
retrievals across all 100 instances per model is then calculated. To validate
the self-recognition of image instances, we select the maximum values across
all computational models (Fig. 3).

Figure 3: Self-recognition Tests. Each cell represents the mean number of same
instances in the top 100 nearest neighbors of its fruit category (column) and
style (row) combination images. White cells without values have a perfect
score of 100 out of 100 correctly recognized instances. Left: Maximum values
from all computational models, taking into account that high scores within 100
out of 10,000 images reflect higher than chance results. Right: ResNet50_IN21k
as an example model.

If a category-style combination cannot be sufficiently recognized in any of
the computational models, we consider the self-recognition test failed.
Notably, we found that "apples" and "oranges" in the "Watercolor" style pose
the greatest challenge, achieving sufficient accuracy only after various
iterations of image generation.

### Model Heatmaps

We characterize the dataset, and concurrently exemplify its possible future
use by a set of category- and style-ordered distance matrices, which
demonstrate salient differences in category and style similarity weights,
across various computational models (Fig. 4). As a measure of similarity
between two sets of images we calculate the average distances between all
pairs of elements. To better generalize standardization, we use Mahalanobis
distance[54, 55], which normalizes and decorrelates the coordinates.

### Model Comparison

Each of the multiple embedding models can be characterized by a set of
distances between images in this embedding. One can consider this set of
distances as a multidimensional vector, characterizing a model. Thus, the
different models are represented as vectors in a shared space, which enables
their direct comparison. As coordinates we used standardized pairwise
distances between all unique pairs of 100 fruit category-style combinations,
i.e., all entries of the model heatmaps in Fig. 5. The principle components of
the resulting embedding are shown in Fig. 6.

Figure 4: DINO-ViT-B-16_IN1k Heatmap indicating the mutual Mahalanobis
distances of fruit-SALAD images. The matrix cells correspond to the mean of
all 10,000 distance pairs of 100 by 100 instances of fruit-SALAD images. Below
the diagonal: sorted by style first and fruit category second. Above the
diagonal: sorted by fruit category first and style second. The color indicates
the pairwise Mahalanobis distance of image embedding vectors obtained from the
respective model or algorithm, from low to high (blue to yellow) while low
values indicate higher similarity. The figure construction is comprehensive as
the matrices are symmetric; diagonal cells can be left out.

Investigating the differences in similarity perception can also be
accomplished by examining fruit categories and styles through the image
embeddings of individual models (Fig. 7).

Figure 5: Heatmaps indicating the mutual Mahalanobis distance of fruit-SALAD
images according to different models (see Fig. 4). Top row from left to right:
CLIP-ViT-B-16_L400M, DINOv2-B_LVD, CompressionEnsembles. Bottom row from left
to right: VGG19_IN1k, ViT-B-32_IN21, style_blind. The matrix ordering is
identical. Figure 6: Relative model comparison using principal component
analysis (PCA) based on 23 standardized model vectors of 4,950 dimensions.
These dimensions encompass the mutual Mahalanobis distances of all unique
category-style combinations of the fruit-SALAD images, excluding self-pairing.
Each fruit category-style combination is the mean of all 10,000 mutual
distances of 100 by 100 fruit-SALAD image instances.  Figure 7: Scatter plots
of apples and oranges using multidimensional scaling (MDS) based on normalized
image embedding vectors from two different models. Left: CLIP-ViT-B-16_L400M;
right: DINO-ViT-B-16_IN1k. Colors indicate fruit categories and dot shapes
indicate styles.

## Data Records & Code Availability

The fruit-SALAD consists of 10,000 PNG images of 1024x1024 pixels; 10 fruit
categories in 10 styles with 100 instances each.

The fruit-SALAD_10k image dataset, style reference images, used text prompts,
100 grid overview plots (10x10 images per instance), vector files and model
heatmaps are available at https://doi.org/10.5281/zenodo.11158522.

The image filenames adhere to the following format: fruit_style_instance.png.
For example, an image with the filename 8_1_42.png signifies fruit category 8
(avocado) rendered in style category 1 (Watercolor), and it represents
generation number 42.

For accessibility, we provide all vector files as comma-separated values
(.csv) with image file names as indices.

The data repository also includes all style reference images, used text
prompts, 100 grid overview plots (10x10 images per instance), and model
heatmaps.

Code performed to generate the fruit images is available at
https://github.com/tillmannohm/fruit-SALAD.

## Acknowledgements

All authors are supported by the CUDAN ERA Chair project for Cultural Data
Analytics, funded through the European Union’s Horizon 2020 research and
innovation program (Grant No. 810961).

## Author contributions

TO co-designed the research, generated the dataset, co-wrote the text, and
created the figures. MT, AK and MS also co-designed the research, co-wrote the
text, and provided conceptual guidance. All authors read and approved the
final manuscript.

## References

  * [1] Kaiser, D., Jacobs, A. M. & Cichy, R. M.  Modelling brain representations of abstract concepts.  _PLoS Computational Biology_ 18, e1009837 (2022).
  * [2] Charest, I., Kievit, R. A., Schmitz, T. W., Deca, D. & Kriegeskorte, N.  Unique semantic space in the brain of each beholder predicts perceived similarity.  _Proceedings of the National Academy of Sciences_ 111, 14565–14570 (2014).
  * [3] Lang, S. & Ommer, B.  Attesting similarity: Supporting the organization and study of art image collections with computer vision.  _Digital Scholarship in the Humanities_ 33, 845–856 (2018).
  * [4] Wei, Z., Wang, S. & Thawonmas, R.  Difference in perceived similarity between humans and machines.  _Art Research_ 22, 2 (2022).
  * [5] Muttenthaler, L. _et al._ Improving neural network representations using human similarity judgments.  _Advances in Neural Information Processing Systems_ 36 (2024).
  * [6] Fu, S. _et al._ Dreamsim: Learning new dimensions of human visual similarity using synthetic data.  _arXiv preprint arXiv:2306.09344_ (2023).
  * [7] Hummel, J. E. & Doumas, L. A. A.  Analogy and similarity.  In _The Cambridge Handbook of Computational Cognitive Sciences_ , 451–473, 10.1017/9781108755610.018 (Cambridge University Press, 2023).
  * [8] Richie, R. & Bhatia, S.  Similarity judgment within and across categories: A comprehensive model comparison.  _Cognitive science_ 45, e13030 (2021).
  * [9] Wittgenstein, L.  _Philosophical Investigations_ (Basil Blackwell, 1968).
  * [10] Rosch, E. & Mervis, C. B.  Family resemblances: Studies in the internal structure of categories.  _Cognitive psychology_ 7, 573–605 (1975).
  * [11] Mishra, S. _et al._ Effectively leveraging attributes for visual similarity.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 1015–1024 (2021).
  * [12] Liu, W., Liu, Z., Rehg, J. M. & Song, L.  Neural similarity learning.  _Advances in Neural Information Processing Systems_ 32 (2019).
  * [13] Cheng, X., Zhang, L. & Zheng, Y.  Deep similarity learning for multimodal medical images.  _Computer Methods in Biomechanics and Biomedical Engineering: Imaging & Visualization_ 6, 248–252 (2018).
  * [14] Veit, A., Belongie, S. & Karaletsos, T.  Conditional similarity networks.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 830–838 (2017).
  * [15] Mathisen, B. M., Aamodt, A., Bach, K. & Langseth, H.  Learning similarity measures from data.  _Progress in Artificial Intelligence_ 9, 129–143 (2020).
  * [16] Oh Song, H., Xiang, Y., Jegelka, S. & Savarese, S.  Deep metric learning via lifted structured feature embedding.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 4004–4012 (2016).
  * [17] Ma, W.-Y. & Manjunath, B. S.  Texture features and learning similarity.  In _Proceedings CVPR IEEE computer society conference on computer vision and pattern recognition_ , 425–430 (IEEE, 1996).
  * [18] Ruiz, N. _et al._ Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 22500–22510 (2023).
  * [19] Radford, A. _et al._ Learning transferable visual models from natural language supervision.  In _International conference on machine learning_ , 8748–8763 (PMLR, 2021).
  * [20] Caron, M. _et al._ Emerging properties in self-supervised vision transformers.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , 9650–9660 (2021).
  * [21] Karjus, A., Solà, M. C., Ohm, T., Ahnert, S. E. & Schich, M.  Compression ensembles quantify aesthetic complexity and the evolution of visual art.  _EPJ Data Science_ 12, 21 (2023).
  * [22] Forsythe, A., Nadal, M., Sheehy, N., Cela-Conde, C. J. & Sawey, M.  Predicting beauty: Fractal dimension and visual complexity in art.  _British journal of psychology_ 102, 49–70 (2011).
  * [23] Zhang, J., Miao, Y., Zhang, J. & Yu, J.  Inkthetics: a comprehensive computational model for aesthetic evaluation of chinese ink paintings.  _IEEE Access_ 8, 225857–225871 (2020).
  * [24] Brinkmann, L. _et al._ Machine culture.  _Nature Human Behaviour_ 7, 1855–1868 (2023).
  * [25] McCormack, J. & Cruz Gambardella, C.  Complexity and aesthetics in generative and evolutionary art.  _Genetic Programming and Evolvable Machines_ 23, 535–556 (2022).
  * [26] Srinivasa Desikan, B., Shimao, H. & Miton, H.  Wikiartvectors: Style and color representations of artworks for cultural analysis via information theoretic measures.  _Entropy_ 24, 1175 (2022).
  * [27] Mao, H., Cheung, M. & She, J.  Deepart: Learning joint representations of visual arts.  In _Proceedings of the 25th ACM international conference on Multimedia_ , 1183–1191 (2017).
  * [28] Lakhal, S., Darmon, A., Bouchaud, J.-P. & Benzaquen, M.  Beauty and structural complexity.  _Physical Review Research_ 2, 022058 (2020).
  * [29] Ostmeyer, J. _et al._ Synthetic images aid the recognition of human-made art forgeries.  _Plos one_ 19, e0295967 (2024).
  * [30] Duñabeitia, J. A. _et al._ Multipic: A standardized set of 750 drawings with norms for six european languages.  _Quarterly journal of experimental psychology_ 71, 808–816 (2018).
  * [31] Ovalle-Fresa, R., Di Pietro, S. V., Reber, T. P., Balbi, E. & Rothen, N.  Standardized database of 400 complex abstract fractals.  _Behavior Research Methods_ 54, 2302–2317 (2022).
  * [32] Liao, P., Li, X., Liu, X. & Keutzer, K.  The artbench dataset: Benchmarking generative models with artworks.  _arXiv preprint arXiv:2206.11404_ (2022).
  * [33] Podell, D. _et al._ Sdxl: Improving latent diffusion models for high-resolution image synthesis.  _arXiv preprint arXiv:2307.01952_ (2023).
  * [34] Hertz, A., Voynov, A., Fruchter, S. & Cohen-Or, D.  Style aligned image generation via shared attention.  _arXiv preprint arXiv:2312.02133_ (2023).
  * [35] Ho, J., Jain, A. & Abbeel, P.  Denoising diffusion probabilistic models.  _Advances in neural information processing systems_ 33, 6840–6851 (2020).
  * [36] Ramesh, A. _et al._ Zero-shot text-to-image generation.  In _International conference on machine learning_ , 8821–8831 (Pmlr, 2021).
  * [37] DALL·E 3 — openai.com.  https://openai.com/dall-e-3.  [Accessed 25-04-2024].
  * [38] Midjourney Website.  https://www.midjourney.com/website.  [Accessed 25-04-2024].
  * [39] Rombach, R., Blattmann, A., Lorenz, D., Esser, P. & Ommer, B.  High-resolution image synthesis with latent diffusion models.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 10684–10695 (2022).
  * [40] Song, J., Meng, C. & Ermon, S.  Denoising diffusion implicit models.  _arXiv preprint arXiv:2010.02502_ (2020).
  * [41] LeCun, Y., Bottou, L., Bengio, Y. & Haffner, P.  Gradient-based learning applied to document recognition.  _Proceedings of the IEEE_ 86, 2278–2324 (1998).
  * [42] Dosovitskiy, A. _et al._ An image is worth 16x16 words: Transformers for image recognition at scale.  _arXiv preprint arXiv:2010.11929_ (2020).
  * [43] Oquab, M. _et al._ Dinov2: Learning robust visual features without supervision.  _arXiv preprint arXiv:2304.07193_ (2023).
  * [44] Woo, S. _et al._ Convnext v2: Co-designing and scaling convnets with masked autoencoders.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 16133–16142 (2023).
  * [45] Liu, Z. _et al._ A convnet for the 2020s.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 11976–11986 (2022).
  * [46] He, K., Zhang, X., Ren, S. & Sun, J.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 770–778 (2016).
  * [47] Simonyan, K. & Zisserman, A.  Very deep convolutional networks for large-scale image recognition.  _arXiv preprint arXiv:1409.1556_ (2014).
  * [48] Chollet, F.  Xception: Deep learning with depthwise separable convolutions.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 1251–1258 (2017).
  * [49] Deng, J. _et al._ Imagenet: A large-scale hierarchical image database.  In _2009 IEEE conference on computer vision and pattern recognition_ , 248–255 (Ieee, 2009).
  * [50] Schuhmann, C. _et al._ Laion-400m: Open dataset of clip-filtered 400 million image-text pairs.  _arXiv preprint arXiv:2111.02114_ (2021).
  * [51] Schuhmann, C. _et al._ Laion-5b: An open large-scale dataset for training next generation image-text models.  _Advances in Neural Information Processing Systems_ 35, 25278–25294 (2022).
  * [52] Welch, T. A.  A technique for high-performance data compression.  _Computer_ 17, 8–19 (1984).
  * [53] Doerr, M.  The cidoc conceptual reference module: an ontological approach to semantic interoperability of metadata.  _AI magazine_ 24, 75–75 (2003).
  * [54] Mahalanobis, P.  On the generalised distance in statistics (reprint, 2018).  _Sankhya A_ 80, 1–7 (1936).
  * [55] De Maesschalck, R., Jouan-Rimbaud, D. & Massart, D. L.  The mahalanobis distance.  _Chemometrics and intelligent laboratory systems_ 50, 1–18 (2000).