# A note on The maximal operators of the Nörlund logaritmic means of Vilenkin-
Fourier series

George Tephnadze and Giorgi Tutberidze  George Tephnadze, The University of
Georgia, School of Science and Technology, 77a Merab Kostava St, Tbilisi,
0128, Georgia<EMAIL_ADDRESS>G.Tutberidze, The University of Georgia,
IV, 77a Merab Kostava St, Tbilisi 0128, Georgia and Department of Engineering
Sciences and Mathematics, Luleå University of Technology, SE-971 87 Luleå,
Sweden<EMAIL_ADDRESS>
###### Abstract.

The main aim of this paper is to investigate $\left(H_{p},L_{p}\right)$\- type
inequalities for the the maximal operators of Nörlund logaritmic means, for
$0<p<1.$

The research was supported by Shota Rustaveli National Science Foundation
grant YS-18-043.

2000 Mathematics Subject Classification. 42C10.

Key words and phrases: Vilenkin system, partial sums, Logarithmic means,
martingale Hardy space.

##  1\. INTRODUCTION

It is well-known that (see e.g. [1], [8] and [16]) Vilenkin systems do not
form bases in the Lebesgue space $L_{1}\left(G_{m}\right).$ Moreover, there
exists a function in the Hardy space $H_{1},$ such that the partial sums of
$f$ are not bounded in $L_{1}$-norm.

In [17] (see also [21]) it was proved that the following is true:

Theorem T1. Let $0<p<1.$ Then the maximal operator

$\overset{\sim}{S}_{p}^{*}f:=\sup_{n\in\mathbb{N}}\frac{\left|S_{n}f\right|}{\left(n+1\right)^{1/p-1}}$

is bounded from the Hardy space $H_{p}\left(G_{m}\right)$ to the space
$L_{p}\left(G_{m}\right).$ Here $S_{n}$ denotes the $n$-th partial sum with
respect to the Vilenkin system. Moreover, it was proved that the rate of the
factor $(n+1)^{1/p-1}$ is in a sense sharp.

In the case $p=1$ it was proved that the maximal operator
$\widetilde{S}^{\ast}$ defined by

$\widetilde{S}^{\ast}:=\sup_{n\in\mathbb{N}}\frac{\left|S_{n}\right|}{\log\left(n+1\right)}$

is bounded from the Hardy space $H_{1}\left(G_{m}\right)$ to the space
$L_{1}\left(G_{m}\right).$ Moreover, the rate of the factor $\log(n+1)$ is in
a sense sharp. Similar problems for Nörlund logarithmic means in the case when
$p=1$ was considered in [15].

Móricz and Siddiqi [9] investigate the approximation properties of some
special Nörlund means of Walsh-Fourier series of $L_{p}\left(G_{m}\right)$
functions in $L_{p}$-norm. Fridli, Manchanda and Siddiqi [5] improved and
extended the results of Móricz and Siddiqi [9] to Martingale Hardy spaces.
However, the case when $\left\\{q_{k}=1/k:k\in\mathbb{N}_{+}\right\\}$ was
excluded, since the methods are not applicable to Nörlund logarithmic means.
In [6] Gát and Goginava proved some convergence and divergence properties of
Walsh-Fourier series of the Nörlund logarithmic means of functions in the
Lebesgue space $L_{1}\left(G_{m}\right).$ In particular, they proved that
there exists an function in the space $L_{1}\left(G_{m}\right),$ such that

$\sup_{n\in\mathbb{N}}\left\|L_{n}f\right\|_{1}=\infty.$

In [2] (see also [15, 20]) it was proved that there exists a martingale $f\in
H_{p}\left(G_{m}\right),\quad(0<p<1)$ such that

$\sup_{n\in\mathbb{N}}\left\|L_{n}f\right\|_{p}=\infty.$

Analogical problems for Nörlund means with respect to Walsh, Kaczmarz and
unbounded Vilenkin systems were considered in [7], [3, 4], [10, 11, 12], [13,
14], [18, 19, 21], [22].

In this paper we discuss boundedness of weighted maximal operators from the
Hardy space $H_{p}\left(G_{m}\right)$ to the Lebesgue space
$L_{p}\left(G_{m}\right)$ for $0<p<1.$

##  2\. Definitions and Notation

Let $\mathbb{N}_{+}$ denote the set of the positive integers,
$\mathbb{N}:=\mathbb{N}_{+}\cup\\{0\\}.$

Let $m:=(m_{0},m_{1},...)$ denote a sequence of the positive integers not less
than $2.$

Denote by

$Z_{m_{k}}:=\\{0,1,...m_{k}-1\\}$

the additive group of integers modulo $m_{k}.$

Define the group $G_{m}$ as the complete direct product of the group
$Z_{m_{j}}$ with the product of the discrete topologies of $Z_{m_{j}}$ ,s.

The direct product $\mu$ of the measures

$\mu_{k}\left(\\{j\\}\right):=1/m_{k}\text{ \qquad}(j\in Z_{m_{k}})$

is the Haar measure on $G_{m_{\text{ }}}$with $\mu\left(G_{m}\right)=1.$

If $\sup\limits_{n\in\mathbb{N}}m_{n}<\infty$, then we call $G_{m}$ a bounded
Vilenkin group. If the generating sequence $m$ is not bounded then $G_{m}$ is
said to be an unbounded one. In this paper we discuss bounded Vilenkin groups
only.

The elements of $G_{m}$ are represented by sequences

$x:=(x_{0},x_{1},...x_{j},...)\qquad\left(x_{k}\in Z_{m_{k}}\right).$

It is easy to give a base for the neighborhood of $G_{m}$

$I_{0}\left(x\right):=G_{m},$ $I_{n}(x):=\\{y\in G_{m}\mid
y_{0}=x_{0},...y_{n-1}=x_{n-1}\\}\text{ }(x\in G_{m},\text{ }n\in\mathbb{N})$

Denote $I_{n}:=I_{n}\left(0\right),$ for $n\in\mathbb{N}$ and
$\overset{-}{I_{n}}:=G_{m}$ $\backslash$ $I_{n}$ .

If we define the so-called generalized number system based on $m$ in the
following way :

$M_{0}:=1,\text{ \qquad}M_{k+1}:=m_{k}M_{k}\qquad(k\in\mathbb{N})$

then every $n\in\mathbb{N}$ can be uniquely expressed as
$n=\overset{\infty}{\underset{k=0}{\sum}}n_{j}M_{j}$ where $n_{j}\in
Z_{m_{j}}$ $(j\in\mathbb{N})$ and only a finite number of $n_{j}`$s differ
from zero. Let $\left|n\right|:=\max\\{j\in\mathbb{N};\quad n_{j}\neq 0\\}.$

The norm (or quasi-norm) of the space $L_{p}(G_{m})$ is defined by

$\left\|f\right\|_{p}:=\left(\int_{G_{m}}\left|f\right|^{p}d\mu\right)^{1/p}\qquad\left(0<p<\infty\right).$

The space $weak-L_{p}\left(G_{m}\right)$ consists of all measurable functions
$f$ for which

$\left\|f\right\|_{weak-
L_{p}(G_{m})}:=\underset{\lambda>0}{\sup}\lambda^{p}\mu\left(x:\left|f\left(x\right)\right|>\lambda\right)<+\infty.$

Next, we introduce on $G_{m}$ an ortonormal system which is called the
Vilenkin system.

At first define the complex valued function
$r_{k}\left(x\right):G_{m}\rightarrow C,$ the generalized Rademacher functions
as

$r_{k}\left(x\right):=\exp\left(2\pi
ix_{k}/m_{k}\right)\qquad\left(i^{2}=-1,\qquad x\in G_{m},\qquad
k\in\mathbb{N}\right).$

Now define the Vilenkin system $\psi:=(\psi_{n}:n\in\mathbb{N})$ on $G_{m}$
as:

$\psi_{n}:=\overset{\infty}{\underset{k=0}{\Pi}}r_{k}^{n_{k}},\qquad\left(n\in\mathbb{N}\right).$

Specifically, we call this system the Walsh-Paley one if m=2.

The Vilenkin system is ortonormal and complete in
$L_{2}\left(G_{m}\right)\,$[1, 23].

Now we introduce analogues of the usual definitions in Fourier-analysis.

If $f\in L_{1}\left(G_{m}\right)$ we can establish the the Fourier
coefficients, the partial sums of the Fourier series, the Dirichlet kernels
with respect to the Vilenkin sistem $\psi$ in the usual manner:

$\displaystyle\widehat{f}$ $\displaystyle:$
$\displaystyle=\int_{G_{m}}f\overline{\psi}_{k}d\mu,\qquad\left(k\in\mathbb{N}\right),$
$\displaystyle S_{n}f$ $\displaystyle:$
$\displaystyle=\sum_{k=0}^{n-1}\widehat{f}\left(k\right)\psi_{k},\qquad\left(n\in\mathbb{N}_{+},\quad
S_{0}f:=0\right),$ $\displaystyle D_{n}$ $\displaystyle:$
$\displaystyle=\sum_{k=0}^{n-1}\psi_{k},\qquad\left(n\in\mathbb{N}_{+}\right).$

Recall that (for details see e.g. [1])

(1) $D_{M_{n}}\left(x\right)=\left\\{\begin{array}[]{l}M_{n}\quad x\in
I_{n}\\\ 0\qquad x\notin I_{n}\end{array}\right.$

The $\sigma$-algebra generated by the intervals
$\left\\{I_{n}\left(x\right):x\in G_{m}\right\\}$ will be denoted by
$\digamma_{n}$ $\left(n\in\mathbb{N}\right).$ Denote by
$f=\left(f_{n}:n\in\mathbb{N}\right)$ a martingale with respect to
$\digamma_{n}$ $\left(n\in\mathbb{N}\right).$ (for details see e.g. [24, 25]).
The maximal function of a martingale $f$ is defend by

$f^{*}=\sup_{n\in\mathbb{N}}\left|f_{n}\right|.$

In the case when $f\in L_{1},$ the maximal function is also be given by

$f^{*}\left(x\right)=\sup_{n\in\mathbb{N}}\frac{1}{\left|I_{n}\left(x\right)\right|}\left|\int_{I_{n}\left(x\right)}f\left(u\right)\mu\left(u\right)\right|.$

For $0<p<\infty$ the Hardy martingale spaces $H_{p}$ $\left(G_{m}\right)$
consist of all martingales for which

$\left\|f\right\|_{H_{p}}:=\left\|f^{*}\right\|_{p}<\infty.$

If $f\in L_{1},$ then it is easy to show that the sequence
$\left(S_{M_{n}}f:n\in\mathbb{N}\right)$ is a martingale. If
$f=\left(f_{n}:n\in\mathbb{N}\right)$ is martingale then the Vilenkin-Fourier
coefficients must be defined in a slightly different manner:

$\widehat{f}\left(i\right):=\lim_{k\rightarrow\infty}\int_{G_{m}}f_{k}\overline{\psi}_{i}d\mu.$

The Vilenkin-Fourier coefficients of $f\in L_{1}\left(G_{m}\right)$ are the
same as those of the martingale $\left(S_{M_{n}}f:n\in\mathbb{N}\right)$
obtained from $f$ .

Let {$q_{k}:k>0$} be a sequence of nonnegative numbers. The $n$-th Nörlund
means for the Fourier series of $f$ is defined by

$\frac{1}{Q_{n}}\overset{n}{\underset{k=1}{\sum}}q_{n-k}S_{k}f\qquad\text{where}\qquad
Q_{n}:=\overset{n}{\underset{k=1}{\sum}}q_{k}.$

If $q_{k}=1/k,$ then we get Nörlund logarithmic means

$L_{n}f:=\frac{1}{l_{n}}\overset{n-1}{\underset{k=0}{\sum}}\frac{S_{k}f}{n-k}\qquad\text{where}\qquad
l_{n}=\overset{n-1}{\underset{k=0}{\sum}}\frac{1}{n-k}=\overset{n}{\underset{j=1}{\sum}}\frac{1}{j}.$

A bounded measurable function $a$ is $p$-atom, if there exist a dyadic
interval $I,$ such that

$\int_{I}ad\mu=0,\ \ \left\|a\right\|_{\infty}\leq\mu\left(I\right)^{-1/p},\ \
\text{supp}\left(a\right)\subset I.$

##  3\. Formulation of Main Results

######  Theorem 1.

a) Let $0<p<1.$ Then the maximal operator

$\overset{\sim}{L}_{p}^{*}f:=\sup_{n\in\mathbb{N}}\frac{\left|L_{n}f\right|}{\left(n+1\right)^{1/p-1}}$

is bounded from the Hardy space $H_{p}\left(G_{m}\right)$ to the space
$L_{p}\left(G_{m}\right).$

b) Let $0<p<1$ and $\varphi:\mathbb{N}_{+}\rightarrow[1,\infty)$ be a non-
decreasing function satisfying the condition

(2) $\overline{\lim_{n\rightarrow\infty}}\frac{n^{1/p-1}}{\log
n\varphi\left(n\right)}=+\infty.$

Then there exists a martingale $f\in H_{p}\left(G_{m}\right),$ such that the
maximal operator

$\text{
}\sup_{n\in\mathbb{N}}\frac{\left|L_{n}f\right|}{\varphi\left(n+1\right)}$

is not bounded from the Hardy space $H_{p}\left(G_{m}\right)$ to the space
$L_{p}\left(G_{m}\right).$

##  4\. Proof of the Theorem

###### Proof.

Since

$\frac{\left|L_{n}f\right|}{\left(n+1\right)^{1/p-1}}\leq\frac{1}{\left(n+1\right)^{1/p-1}}\underset{1\leq
k\leq n}{\sup}\left|S_{k}f\right|\leq\underset{1\leq k\leq
n}{\sup}\frac{\left|S_{k}f\right|}{\left(k+1\right)^{1/p-1}}\leq\underset{n\in\mathbb{N}}{\sup}\frac{\left|S_{n}f\right|}{\left(n+1\right)^{1/p-1}}$

if we use Theorem T1 we obtain that

$\underset{n\in\mathbb{N}}{\sup}\frac{\left|L_{n}f\right|}{\left(n+1\right)^{1/p-1}}\leq\underset{n\in\mathbb{N}}{\sup}\frac{\left|S_{n}f\right|}{\left(n+1\right)^{1/p-1}}$

and

$\left\|\underset{n\in\mathbb{N}}{\sup}\frac{\left|L_{n}f\right|}{\left(n+1\right)^{1/p-1}}\right\|_{p}\leq\left\|\underset{n\in\mathbb{N}}{\sup}\frac{\left|S_{n}f\right|}{\left(n+1\right)^{1/p-1}}\right\|_{p}\leq
c_{p}\left\|f\right\|_{H_{p}}.$

Now, prove part b) of the Theorem. Let

$f_{n_{k}}=D_{M_{2n_{k}+1}}-D_{M_{2n_{k}}}.$

It is evident

(3) $\widehat{f}_{n_{k}}\left(i\right)=\left\\{\begin{array}[]{l}\text{
}1,\qquad\text{ if }\qquad i=M_{2n_{k}},...,M_{2n_{k}+1}-1,\\\ \text{
}0,\qquad\text{otherwise.}\end{array}\right.$

Then we can write

(4) $S_{i}f_{n_{k}}=\left\\{\begin{array}[]{l}D_{i}-D_{M_{2n_{k}}},\quad\text{
if }\quad i=M_{2n_{k}}+1,...,M_{2n_{k}+1}-1,\\\
f_{n_{k}},\qquad\qquad\quad\text{ if }\quad i\geq M_{2n_{k}+1},\\\
0,\qquad\qquad\qquad\text{ otherwise.}\end{array}\right.$

From (1) we get

$\displaystyle\left\|f_{n_{k}}\right\|_{H_{p}}$ $\displaystyle=$
$\displaystyle\left\|\sup\limits_{n\in\mathbb{N}}S_{M_{n}}f_{n_{k}}\right\|_{p}=\left\|D_{M_{2n_{k}+1}}-D_{M_{{}_{2n_{k}}}}\right\|_{p}$
$\displaystyle\leq$
$\displaystyle\left\|D_{M_{2n_{k}+1}}\right\|_{p}+\left\|D_{M_{{}_{2n_{k}}}}\right\|_{p}\leq
cM_{{}_{2n_{k}}}^{1-1/p}<c<\infty.$

Let $0<p<1$ and $\left\\{\lambda_{k}:k\in\mathbb{N}_{+}\right\\}$ be an
increasing sequence of the positive integers such that

$\lim_{k\rightarrow\infty}\frac{\lambda_{k}^{1/p-1}}{\varphi\left(\lambda_{k}\right)}=\infty.$

Let$\
\left\\{n_{k}:k\in\mathbb{N}_{+}\right\\}\subset\left\\{\lambda_{k}:k\in\mathbb{N}_{+}\right\\}$
such that

$\displaystyle\underset{k\rightarrow\infty}{\lim}\frac{\left(M_{{}_{2n_{k}}}+2\right)^{1/p-1}}{\log{\left(M_{2n_{k}}+2\right)}\varphi\left(M_{2n_{k}+2}\right)}\geq
c\lim_{k\rightarrow\infty}\frac{\lambda_{k}^{1/p-1}}{\varphi\left(\lambda_{k}\right)}=\infty.$

According to (4) we can conclude that

$\displaystyle\left|\frac{L_{M_{2n_{k}}+2}f_{n_{k}}}{\varphi\left(M_{2n_{k}+2}\right)}\right|=\frac{\left|D_{M_{{}_{2n_{k}}}+1}-D_{M_{{}_{2n_{k}}}}\right|}{l_{M_{2n_{k}}+1}\varphi\left(M_{2n_{k}+1}\right)}$
$\displaystyle=$
$\displaystyle\frac{\left|\psi_{M_{{}_{2n_{k}}}}\right|}{l_{M_{2n_{k}}+2}\varphi\left(M_{2n_{k}+1}\right)}=\frac{1}{l_{M_{2n_{k}}+1}\varphi\left(M_{2n_{k}+2}\right)}.$

Hence,

(6) $\mu\left\\{x\in
G_{m}:\left|L_{M_{2n_{k}}+2}f_{n_{k}}\right|\geq\frac{1}{l_{M_{2n_{k}}+2}\varphi\left(M_{2n_{k}+2}\right)}\right\\}=\mu\left(G_{m}\right)=1.$

By combining (4) and (6) we get that

$\displaystyle\frac{\frac{1}{l_{M_{2n_{k}}+2}\varphi\left(M_{2n_{k}+2}\right)}\left(\mu\left\\{x\in
G_{m}:\quad\left|L_{M_{2n_{k}}+2}f_{n_{k}}\right|\geq\frac{1}{l_{M_{2n_{k}}+2}\varphi\left(M_{2n_{k}+2}\right)}\right\\}\right)^{1/p}}{\left\|f_{n_{k}}\right\|_{p}}$
$\displaystyle\geq$
$\displaystyle\frac{M_{{}_{2n_{k}}}^{1/p-1}}{l_{M_{2n_{k}}+2}\varphi\left(M_{2n_{k}+2}\right)}\geq\frac{c\left(M_{{}_{2n_{k}}}+2\right)^{1/p-1}}{\log{\left(M_{2n_{k}}+2\right)}\varphi\left(M_{2n_{k}+2}\right)}\rightarrow\infty,\text{\quad
as \quad}k\rightarrow\infty.$

Theorem is proved.

∎

Open Problem. For any $0<p<1$ let find non-decreasing function
$\Theta:\mathbb{N}_{+}\rightarrow[1,\infty)$ suth that the following maximal
operator

$\overset{\sim}{L}_{p}^{*}f:=\sup_{n\in\mathbb{N}}\frac{\left|L_{n}f\right|}{\Theta\left(n+1\right)}$

is bounded from the Hardy space $H_{p}\left(G_{m}\right)$ to the Lebesgue
space $L_{p}\left(G_{m}\right)$ and the rate of
$\Theta:\mathbb{N}_{+}\rightarrow[1,\infty)$ is sharp, that is, for any non-
decreasing function $\varphi:\mathbb{N}_{+}\rightarrow[1,\infty)$ satisfying
the condition

$\overline{\lim_{n\rightarrow\infty}}\frac{\Theta\left(n\right)}{\varphi\left(n\right)}=+\infty,$

then there exists a martingale $f\in H_{p}\left(G_{m}\right),$ such that the
maximal operator

$\sup_{n\in\mathbb{N}}\frac{\left|L_{n}f\right|}{\varphi\left(n+1\right)}$

is not bounded from the Hardy space $H_{p}\left(G_{m}\right)$ to the space
$L_{p}\left(G_{m}\right).$

######  Remark 1.

According to Theorem 1 we can conclude that there exist absolute constants
$C_{1}$ and $C_{2}$ such that

$\frac{C_{1}n^{1/p-1}}{\log(n+1)}\leq\Theta\left(n\right)\leq C_{2}n^{1/p-1}.$

## References

  * [1] G. N. Agaev, N. Ya. Vilenkin, G. M. Dzhafarly and A. I. Rubinstein, Multiplicative systems of functions and harmonic analysis on zero-dimensional groups, Baku, Ehim, 1981 (in Russian).
  * [2] I. Blahota and G. Gàt, Norm summability of Nörlund logarithmic means on unbounded Vilenkin groups, Anal. Theory Appl., 24 (2008) no 1, 1-17.
  * [3] Blahota, G. Tephnadze, On the $(C,\alpha)$-means with respect to the Walsh system, Anal. Math., 40 (2014), 161-174.
  * [4] Blahota, G. Tephnadze, Strong convergence theorem for Vilenkin-Fejér means, Publ. Math. Debrecen, 85 (1-2) (2014), 181-196.
  * [5] S. Fridli, P. Manchanda and A.H. Siddiqi, Approximation by Walsh-Nörlund means, Acta Sci. Math.(Szeged) 74 (2008), no. 3-4, 593-608.
  * [6] G. Gát and U. Goginava, Uniform and $L$ -convergence of logarithmic means of Walsh-Fourier series, Acta Math. Sin. 22 (2006), no. 2, 497–506.
  * [7] U. Goginava and K. Nagy, On the maximal operator of Walsh-Kaczmarz-Fejér means, Czechoslovak Math. J. 61 (136) (2011), no. 3, 673-686.
  * [8] B. I. Golubov, A. V. Efimov and V. A. Skvortsov, Walsh series and transforms. (Russian) Nauka, Moscow, 1987, English transl. in Mathematics and its Applications (Soviet Series), 64. Kluwer Academic Publishers Group, Dordrecht, 1991.
  * [9] F. Móricz and A. Siddiqi, Approximation by Nörlund means of Walsh-Fourier series, J. Approx. Theory 70 (1992), no. 3, 375–389.
  * [10] K. Nagy, G. Tephnadze, Kaczmarz-Marcinkiewicz means and Hardy spaces, Acta math. Hung., 149, 2 (2016), 346-374.
  * [11] K. Nagy, G. Tephnadze, On the Walsh-Marcinkiewicz means on the Hardy space, Cent. Eur. J. Math., 12, 8 (2014), 1214-1228.
  * [12] K. Nagy, G. Tephnadze, Approximation by Walsh-Marcinkiewicz means on the Hardy space , Kyoto J. Math., 54, 3 (2014), 641-652.
  * [13] L. E. Persson, G. Tephnadze and P. Wall, On the maximal operators of Vilenkin-Nörlund means, J. Fourier Anal. Appl., 21, 1 (2015), 76-94.
  * [14] L. E. Persson, G. Tephnadze and P. Wall, Some new $(H_{p},L_{p})$ type inequalities of maximal operators of Vilenkin-Nörlund means with non-decreasing coefficients, J. Math. Inequal, 9, 4 (2015), 1055-1069.
  * [15] L. E. Persson, G. Tephnadze and P. Wall, On the Nörlund logarithmic means with respect to Vilenkin system in the martingale Hardy space $H_{1}$, Acta math. Hung., 154, 2 (2018) 289-301.
  * [16] F. Schipp, W.R. Wade, P. Simon and J. Pál, Walsh series, An Introduction to Dyadic Harmonic Analysis, Akadémiai Kiadó, (Budapest-Adam-Hilger (Bristol-New-York)), 1990.
  * [17] G. Tephnadze, On the partial sums of Vilenkin-Fourier series, 49 (2014), no. 1, 60-72, J. Contemp. Math. Anal. 49 (2014), no. 1, 23-32.
  * [18] G. Tephnadze, On the maximal operators of Walsh-Kaczmarz-Fejér means, Period. Math. Hung., 67, 1 (2013), 33-45.
  * [19] G. Tephnadze, Approximation by Walsh-Kaczmarz-Fejér means on the Hardy space, Acta Math. Sci., 34, 5 (2014), 1593-1602.
  * [20] G. Tephnadze, The maximal operators of logarithmic means of one-dimensional Vilenkin-Fourier series, Acta Math. Acad. Paed. Nyíreg., 27 (2011), 245-256.
  * [21] G. Tephnadze, Martingale Hardy Spaces and Summability of the One Dimensional Vilenkin-Fourier Series, PhD thesis, Department of Engineering Sciences and Mathematics, Luleå University of Technology, Oct. 2015 (ISSN 1402-1544).
  * [22] G. Tutberidze, A note on the strong convergence of partial sums with respect to Vilenkin system, J. Contemp. Math. Anal., (to appear).
  * [23] N. Ya. Vilenkin, A class of complete orthonormal systems, Izv. Akad. Nauk. U.S.S.R., Ser. Mat., 11 (1947), 363-400.
  * [24] F. Weisz, Martingale Hardy spaces and their applications in Fourier Analysis, Springer, Berlin-Heidelberg-New York, 1994.
  * [25] F. Weisz, Hardy spaces and Cesàro means of two-dimensional Fourier series, Bolyai Soc. Math. Studies, (1996), 353-367.


# DegustaBot: Zero-Shot Visual Preference Estimation for Personalized Multi-
Object Rearrangement

Benjamin A. Newman1∗ Pranay Gupta1 Kris M. Kitani1 Yonatan Bisk1
Henny Admoni1 Chris Paxton2
1 Carnegie Mellon University 2 Hello Robot
∗Corresponding Author: Benjamin Newman<EMAIL_ADDRESS>
###### Abstract

De gustibus non est disputandum (“there is no accounting for others’ tastes”)
is a common Latin maxim describing how many solutions in life are determined
by people’s personal preferences. Many household tasks, in particular, can
only be considered fully successful when they account for personal preferences
such as the visual aesthetic of the scene. For example, setting a table
_could_ be optimized by arranging utensils according to traditional rules of
Western table setting decorum, without considering the color, shape, or
material of each object, but this may not be a completely satisfying solution
for a given person. Toward this end, we present DegustaBot, an algorithm for
visual preference learning that solves household multi-object rearrangement
tasks according to personal preference. To do this, we use internet-scale pre-
trained vision-and-language foundation models (VLMs) with novel zero-shot
visual prompting techniques. To evaluate our method, we collect a large
dataset of naturalistic personal preferences in a simulated table-setting
task, and conduct a user study in order to develop two novel metrics for
determining success based on personal preference. This is a challenging
problem and we find that 50% of our model’s predictions are likely to be found
acceptable by at least 20% of people.

> Keywords: personalization, visual prompting, multi-object rearrangement

##  1 Introduction

Assistive robots in homes should complete tasks in ways that align with user
preferences [1]. Consider the task of setting out dishes and utensils on a
table for dinner at your home. The correct solution for this task is subject
to your preferences for the type of utensils you want to use and how you
prefer to arrange them. However, modeling individual preferences, especially
those that consider fine-grained features such as the real-valued location of
objects and their relative placements, is challenging as such preferences are
subjective, difficult to specify explicitly, and vary from person to person.
In this work, we explore this problem through the example of personalized
dinner table arrangement, and determine a personalized task plan for setting a
dinner table given examples of a person’s preference.

Prior research on multi-object rearrangement collects problem-specific
datasets of simulated or human demonstrations that represent personal
preferences for completing a household task. They hypothesize that collecting
large, task-specific preference datasets is sufficient to train a feature
space that can generalize to novel participants who were excluded from the
training data [2, 3, 4, 5]. These methods, however, often yield poor
generalization to these held-out participants due to large effects from
individual differences. These challenges indicate that is it difficult to
collect a large dedicated dataset to train a feature space that covers the
unbounded space of personal preferences for even a single task and suggests
that generalized pretraining is a potentially fruitful path of research. These
ideas are further compounded when considering underspecified preferences and
preferences that are influenced by cultural norms or decorum.

Figure 1: DegustaBot takes in a single person’s preferred table arrangements
(shown here as the visual context $k_{i}$ and order of object placement
$\mathcal{T}_{i}$), the objects $O$ from which the algorithm can select, the
table to set $a_{0}$ and the task prompt $\ell$. The robot then produces a
task plan in the form of an object arrangement (visualized as an image
$a_{T}$) and the order of object placement $\mathcal{T}$. This predicted
arrangement, Pred, should match a held-out preference created by the user, GT.

Large-language models (LLMs) may provide a solution. They are pretrained on
internet scale data, and have shown the ability to solve tasks upon which they
weren’t explicitly trained. They can leverage knowledge about the abstract
external concepts that often guide personal preferences. In fact, it has been
shown they can infer generalizable user preferences from few examples with in-
context learning [6]. Representing multi-object preference learning solely in
language, however, removes important visual information about a task. At the
same time, these problems cannot be solved purely visually, as they do not
have a grounded vocabulary of actions over which to reason. Therefore, we
developed a new prompting technique that provides such a vocabulary. We built
a virtual tabletop setting task, and collected a large dataset of human
preferences for different arrangements in order to test this technique.
Concretely, in this paper, we make the following contributions:

  * •

DegustaBot: a novel method for implicit visual preference learning to find
personalized solutions to fine-grained multi-object rearrangement tasks.

  * •

A novel human evaluation used to evaluate personalized multi-object
rearrangement agents.

  * •

A dataset of naturalistic preference data in a simulated multi-object
rearrangement task.

##  2 Related Work

Multi Object Rearrangement: Recent advances in embodies AI have led to a
flurry of benchmarks where an embodied agent is tasked with rearranging
objects in a real or simulated home environment [7, 8, 9, 10, 11, 12, 13, 14].
However, these benchmarks have specified goal states, as compared to our work
where the preferred object arrangement has to be inferred from context
examples.

Another line of work, focused on object rearrangement, where-in the target
objects are specified via pointing gestures [15], eye gaze [16], or target
layouts [17] during task execution. However these works require manually
specified target location for every object, and are hence human effort
intensive. A line of follow-up works address this issue by modeling
preferences using learnt priors about where objects typically go [18, 19, 20].
These preferences however are generic are not personalized.

Prior works on personalized object rearrangement, rely on simulated or large
crowd sourced datasets of human preferences to learn fixed latent preference
vectors [4] or latent preferences that can be adapted online [21], model
spatial relationships [3] or perform collaborative filtering [2]. In contrast,
our approach leverages in-context learning with large-scale pretrained VLMs to
perform personalized object rearrangement. While ours is not the first work
which leverages foundational models to perform few shot personalized
rearrangement, we are first ones to perform fine-grained preference alignment,
i.e, spatial preferences. As compared to prior works [6], which operate over a
discrete preference space, i.e, identifying the correct receptacle for each
object.

Foundation Models for Robotics: VLMs pre-trained on large scale datasets have
shown commonsense reasoning abilities. Researchers have leveraged these
abilities to perform planning and control for robotics [22, 23]. Many prior
works [24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35] have used pre-trained
LLMs to generate actionable natural language plans for robots. VLMs have also
been used to generate subgoals for navigation [36, 37, 38, 39, 40, 41] and
manipulation [42, 43] tasks. Additionally, prior works have also leveraged
LLMs to directly generate low-level executable policy code for robots [44,
45]. Another line of works, has also used LLMs to generate rewards, which can
be for RL [46, 47, 48]. In our work, we use a VLM to generate the policy code
to accomplish a continuous preference aligned novel goal state.

Visual Prompting in Vision Language Models The development of in-context
learning for few shot adaption of LLMs [49], was followed up by a flurry of
prompt optimization approaches. While one line of work focuses on prompt-
tuning [50] or prefix-tuning [51] through backpropagation using numerical
gradients, others rely on the generated answer scores to perform optimization
with textual gradients [52], prompt search using genetic algorithms [53], or
iterative prompt refinement [54]. More recently, approaches have tried to use
visual prompting to improve VLM performance by leveraging a VLM’s ability to
solve multiple choice problems. [55, 56]. Our approach adapts these methods to
a complex visual preference learning task using these insights to prompt VLMs
to combine information from multiple images to reason about a person’s
preference.

##  3 Problem Specification

People have widely varying preferences, making it difficult to develop one-
size fits all solutions, even for problems that at first glance seem to be
well-specified. One person may consider a table to be arranged with a simple
fork and a bowl, while another won’t sit down to eat until their flower
arrangement is perfect and a candle is lit. Furthermore, these preferences
cannot easily be specified in natural language: language is often either
ambiguous or requires cumbersome description.

Figure 2: Details of a table arrangement. An arrangement is described by the
objects within the arrangement, the order in which they are placed, their
features (such as color, shape and material, and their location and
orientation.

Instead, we argue that preferences can be specified implicitly by visual
representations of the final state of a multi-object rearrangement task.

To investigate this, we choose table setting as an example application.
Concretely, the goal of a preference model $\mathcal{M}$ is to maximize the
likelihood of seeing a table arrangement $a\in A$ given an initial state
$a_{0}$, a representation of a preference
$K=\left[k_{0},k_{1},\dots,k_{C}\right]$ each element of which is a
concatenation of the arrangement and object placement order
$k_{c}=\left[a_{c}^{T},\mathcal{T}_{c}\right]$, a set of objects that can be
used for the arrangement $O$ and a language prompt $\ell$ that describes the
general task: $\max_{a\in A}\mathcal{M}(a~{}|~{}K,O,a_{0},\ell)$.

An arrangement $a$ of length $T$ is a sequence of tuples containing
positioning $\mathbf{x}$ and object $\mathbf{o}$, that is,
$a_{1:T}=[(\mathbf{x}_{1},\mathbf{o}_{1}),\cdots,(\mathbf{x}_{T},\mathbf{o}_{T})]$.
$\mathbf{x}~{}\in~{}\mathbb{R}^{3}$ represents the continuous elements of an
arrangement (x and y-positions of an object on the table’s surface and r, the
object’s rotation), and $\mathbf{o}$ represents the discrete elements of an
arrangement (an object’s color $c$, material $m$, and type $u$). When
represented in language, an object can be described by a multi-label vector of
up to two colors (beige, black, blue, brown, gold, gray, green, pink, purple,
red, silver, white, and yellow), two materials (ceramic, glass, metal,
plastic, wood), and a single type (cup, fork, knife, plate, or spoon).

As in prior work, [55], we consider a family of functions $\Omega$ that lift
non-visual information into the image domain. An arrangement can be lifted
into the visual domain with $\Omega_{a}:a\rightarrow
I_{a}\times\mathcal{T}_{a}$, where $I_{a}$ is a visual representation of the
arrangement and $\mathcal{T}$ is a language description of the order in which
objects were placed. We also consider $\Omega_{O}:O\rightarrow I_{O}$, which
jointly maps all objects into the visual domain.

Finally, we consider a preference to be implicitly defined by a history of
previous arrangements completed by a single person. In a slight abuse of
notation, we assume that
$\Omega_{a}(K)=[\Omega_{a}(a_{0}),\Omega_{a}(a_{1}),\cdots,\Omega_{a}(a_{C})]$.

Objective. In this work, we want to investigate whether foundation models
trained on internet scale data and conditioned on implicitly defined
preferences can produce personalized task plans. In other words, we consider
the following problem:

$\mathcal{M}_{\theta}(K,O,a_{0},\ell)\stackrel{{\scriptstyle?}}{{=}}\mathcal{M}_{H}(O,a_{o},\ell)~{}\text{where}~{}K~{}\sim\mathcal{M}_{H},$

where $\mathcal{M_{\theta}}$ is a model parameterized by parameters $\theta$,
and $\mathcal{M}_{H_{k}}$ is a human with preference $k$. We parameterize
$M_{\theta}$ as a vision and language model. While there have been a surfeit
of large foundation models recently becoming available, we choose to
investigate three high performing, recent models: OpenAI’s GPT-4o [57],
Anthropic’s Claude-3 Haiku [58], and Google’s Gemini 1.5 Pro [59].

##  4 Estimating Simulated Table Arrangements using VLMs

Solving our implicit, visually represented table-top arrangement task requires
incorporating visual information across several images (e.g. $k$, $o$,
$a_{0}$). We present and evaluate several different zero-shot methods for
prompting VLMs to produce task plans to that solve table-top arrangement
problems. We do this through lifting functions which add extra information to
an image before sending it to a VLM like GPT4v alongside a query, such as
adding a grid or highlighting a point in space.

###  4.1 Producing Task Plans through Visual Prompting

We evaluate different lifting functions $\Omega_{a}$ and $\Omega_{O}$ for
solving zero-shot, personalized table-top arrangement. In addition to lifted
images, we include a text prompt that describes the problem and the desired
format of the output. Prompting specifics are left to the appendix.

Figure 3: Object and arrangement lifting functions, from left to right: OaL, a
representation of objects as language, for this we use the json
representation; GoMO, grid of marked objects, which represents objects
visually with referential marks overlaid on each object; UmA, Unmarked
Arrangement lifts the arrangement into the image domain; and finally GMA,
Grid-Marked Arrangement, which overlays a spatial reference grid on the
continuous table top space.

####  4.1.1 Selecting Objects

We first describe the object lifting functions, $\Omega_{O}$:

Objects as Language We take each object in our dataset and represent it as a
dictionary of features, which are concatenated to create a dictionary of all
objects and their representations (see Fig. 3, left).

Grid-of-Marked Objects. We represent objects visually by creating grids of
objects within a category. Each object is overlaid with a spatial reference
marker to aid in object detection [60] and the image is labeled with object
type and preference number (see Fig. 3, second column).

####  4.1.2 Selecting Locations and Rotations

We represent arrangements visually through $\Omega_{a}$ in two ways:

Unmarked Arrangements. We assume that positions x represent an object’s
centroid. Then, representative images are resized to fit within an initial
arrangement (e.g. an empty table) and are pasted at x. They are then rotated
according to the rotation r.

Grid-Marked Arrangements. We turn predicting a continuous location into a
multiple choice question over grid cells that serve as spatial reference
markers. First, we overlay a spatial grid on each preference image and then
ask the model to produce all the grid cells that an object will intersect with
on a table after it is placed. We average the centroids of the responses over
multiple responses to get a continuous position prediction.

####  4.1.3 Combining Lifting Functions into Visual Prompting Methods

We combine these lifting functions into four methods that test each models
ability to predict task plans in table-top arrangement tasks as follows:

  * •

DegustaBot-LOUMA. uses Language Objects with UnMarked Arrangements

  * •

DegustaBot-LOGMA. uses Language Objects with Grid-Marked Arrangements

  * •

DegustaBot-MOUMA. uses Marked Objects with UnMarked Arrangements

  * •

DegustaBot-MOGMA. uses Object Images with Grid-Marked Arrangements.

###  4.2 Simulating Preferences for Table Arrangement

We develop a small dataset of simulated preferences for controlled evaluation.
These preferences are drawn from three positional preferences, three object
preferences, and a single order preference. We design these preferences to
approximate a standard, a semi-standard and a non-standard table setting. To
test our model over a distribution of positions, we add positional and
rotational noise to these table settings. The object preferences are also
drawn from three color preferences: a preference for all red, all blue, or all
yellow objects.

We displayed arrangements on two visually distinct tables. In total, we
simulated 18 preferences on two table tops for a total of 36 preferences. We
tested these in both a reconstruction task (predict on the same table) and a
generalization task (predict on a different table) for 72 total experiments.

###  4.3 Validation Metrics

We test two main hypotheses: (1) that large vision and language models develop
a feature space that is useful for estimating preferences for real-valued
multi-object rearrangement tasks; and (2) that visually grounding preference
information aids in estimating complex preferences.

Object selection: this measures a method’s ability to select objects
consistent with the preference. We compute this as the accuracy of object
selection: did the model select exactly the right object?

Position Selection: We take a geometric approach to comparing arrangements.
While a point-to-point comparison seems immediately intuitive, the geometric
relationship between points in a particular table setting are important, as
well. For example, a table settings performing poorly when measured with
point-to-point correspondences may still be similar geometrically to the
preferred arrangement, only displaced slightly. To account for these issues,
we introduce another evaluation metric: root mean squared deviation (RMSD).

We use the Kabsch algorithm [61] to register matched points between predicted
and ground truth arrangements, and find the translation vector $g$ and
rotation matrix $r$ that minimize the root mean squared deviation between the
two paired frames:

$\min_{r\in R,g\in
G}\sqrt{\frac{1}{N}\sum_{n}^{N}(\bar{A}^{gt}_{n}-R(\bar{A}^{pred}_{n}-g))^{2}}.$

We report this minimized RMSD as our geometric evaluation metric.

###  4.4 Results

We test each model with each combination of visual lifting functions and
report results on how well the predicted image matches the ground truth
scene’s geometry, as reported through RMSD, and chosen utensils, as reported
through accuracy. These results are shown in Fig. 4.

First, we see that GPT-4o outperforms other choices of VLM in both RMSD and
object prediction, under all methods. We can also see that grid-marked methods
tend to improve a model’s ability to match scene geometry. Finally, we see
that representing utensils visually aids in correctly predicting utensils.
Results for Gemini-1.5-Pro, LOUMA are missing due to a 97% failure rate on
this method. Taking these results together, we see that DegustaBot-MOGMA using
GPT-4o outperforms all other methods on this task. We will use this method to
analyze naturalistic preferences in the next section.

##  5 Naturalistic Tabletop Arrangement Dataset

To understand if this approach works on naturalistic data, we collected a
dataset of preferences for table arrangements through an online study. We then
assessed model performance, as in Sec. 4.4.

Figure 4: Quantitative results for evaluating DegustaBot on simulated
preferences. Left shows the performance of each model’s ability to capture the
geometry of the table arrangement, as measured by RMSD. Right shows each model
and method’s accuracy when choosing items to place in the arrangement. GPT-4o
and MOGMA perform the best on both metrics.

###  5.1 Data Collection

Our user study consisted of a task with two stages: Preference Elicitation and
Self-Evaluation. During Preference Elicitation, we prompt participants to
create their preferred table-top arrangements. In Self-Evaluation,
participants rate these arrangements, as well as several transformations of
their arrangements. We used these ratings to develop a subjective analysis of
model performance. Participants saw a total of six rounds of this task: a
practice round, followed by five experimental trials.

We collected a dataset of 995 table arrangements drawn from 199 participants.
Participants were paid USD$4.00 for completing the 20 minute long task. All
participants were located within the United States at the time of task
completion. We did not collect further demographic information about our
participants. This study was approved by the IRB of record.

####  5.1.1 Preference Elicitation

Participants were first presented with an image of one of five different empty
tables, and asked to to set the table by selecting objects from five object
categories: plates, spoons, forks, knives, and cups. After selecting an object
category, participants were shown all instances in that category
simultaneously. Each category contained 25 objects of varying shapes, colors,
and materials, yielding an object dataset of 125 creative commons licensed and
public domain images of table setting utensils.

RMSD Threshold | 0.01 | 0.05 | 0.10
---|---|---|---
% of difference ratings $<0.2$ | 0.727 | 0.333 | 0.199
% of model responses | 0.145 | 0.332 | 0.532
Table 1: We compare our method’s performance against subjective acceptance at
several RMSD thresholds. When RMSD is below 0.01, 72.7% of people are likely
to find a scene acceptable. 14.5% of the scenes our model produced fell into
this category. At 0.05 RMSD, 33.3% of people find scenes acceptable, while
33.2% of scenes our model produced met this threshold. Finally at 0.10 RMSD,
19.9% of scenes were found to be acceptable, while our model produce such
scenes 53.2% of the time.

Participants were then presented with a simple drag and drop interface through
which they could place the object anywhere on the table’s surface and rotate
it object to their liking. Participants continued this procedure until they
were satisfied with the arrangement. The only restriction placed on table
arrangements was that participants were required to set a minimum of three
objects. After completing a table arrangement, participants moved on to the
self-evaluation portion of the task.

####  5.1.2 Subjective Acceptability

We then asked participants to evaluate several arrangements.

Rating the initial scene. After creating an arrangement, $a_{intial}$
participants were prompted to answer the question, Please rank your agreement
to the following statement: This is my preferred table arrangement, sliding
scale from 0-100 to offer their baseline acceptability score $b_{0}$.

Jittering the initial scene. To understand what errors affect people’s sense
of an acceptable arrangement we randomly transformed $a_{intial}$. A
transformation magnitude between 0 and 1 was randomly selected as $t_{j}$. The
scene $a_{intial}$ was then translated and rotated by this amount to create
$a_{jitter}$. Participants then gave a second acceptability rating
$b_{jitter}$.

Figure 5: Qualitative results. On the top line we see ground truth images from
our naturalistic preference data and on the bottom we see DegustaBot’s
predictions, lifted into the image domain. On the left side of the image we
see three examples where DegustaBot predicts similar arrangements to the
ground truth image. On the right we see some failure cases.

Threshold of Acceptability. We take the difference in ratings
$b_{initial}-b_{jitter}$, and correlate this with the distance between
$a_{initial}$ and $a_{jitter}$, giving us a subjective acceptability measure.
We choose a ratings difference threshold of 0.2, which is interpreted as the
threshold at which people find $a_{jitter}$ to be at least as acceptable as
$a_{intitial}$. We then report the percentage of people who meet this
acceptability threshold at various levels of RMSD. This tells us the
percentage of people who are likely to find a scene acceptable below a given
RMSD threshold.

###  5.2 Results

We evaluated our method on naturalistic preference data. In Tab. 1 we report
the rate at which our model produces acceptable tables as measured by the RMSD
distance between the scene our model produces and a held out arrangement. From
these data we can see that 1) people are highly sensitive the the jittered
scenes we produced, making this a strict metric and 2) our model produces
scenes that people would likely find acceptable under this metric between
14.5% and 53.2% of the time.

In Tab. 2 we report object prediction accuracy and RMSD as the number of
preferences images increases from 0, 2, to 4. We see that increased context
length improves performance metrics, but potentially with diminishing returns.
This indicates that 1) human preferences are diverse and not easily predicted
in a context-free setting and 2) our method makes efficient use of few context
examples to predict a person’s preference. We also see that the performance
here is well below the performance on the synthetic data, indicating that
visual preference learning from naturalistic preference data is a difficult
task.

Length of k | 0 | 2 | 4
---|---|---|---
Object Prediction Accuracy ($\uparrow$) | 0.037 | 0.090 | 0.089
RMSD ($\downarrow$) | 0.174 | 0.116 | 0.108
Table 2: We report object prediction accuracy and RMSD of our method on
naturalistic preferences. We see that with increased context length our method
achieves higher performance, though perhaps with diminishing returns.

Finally in Fig. 5 we show several qualitative results. Each column depicts a
different participant within our dataset. The top row displays ground truth
images, and the bottom rows depict our method’s prediction. When geometric
patterns are regular, our method does well at predicting them.

##  6 Limitations

In this work, we developed a method that can serve as a preference perception
pipeline for a robot that performs household multi-object rearrangement tasks,
DegustaBot. While we do not perform this experiment on a real robot, this
setting emulates suction cup pick-and-place routines that have been shown to
work reliably [62]. We instead focus on generating the task plan for such a
task from realistic preference data.

Additionally, we assumed that a robot has access to a person’s prior table
setting arrangements. Part of our contribution is to release a large dataset
of realistic table-top arrangements for others to train from. In reality, the
amount of data that we require from an individual to achieve reasonable
performance is quite small compared to prior preference learning approaches
[4].

A final limitation is that VLMs are not trained for fine-grained rearrangement
tasks like ours. We presented several prompting methods that achieve
reasonable performance even under this limitation, but our method would
improve substantially if the underlying models were fine-tuned for fine-
grained image reasoning tasks or even trained from the ground up for this kind
of problem [63].

##  7 Conclusion

We introduced DegustaBot, a novel method for prompting VLMs to perform the
challenging task of visual preference learning in multi-object rearrangement
tasks. We formalize this task, and then analyze several methods’ performance
over simulated preferences. We then collect a large dataset of naturalistic
preferences and evaluate our best performing method, finding that although
this is a challenging task, our method produces acceptable table arrangements.

#### Acknowledgments

If a paper is accepted, the final camera-ready version will (and probably
should) include acknowledgments. All acknowledgments go at the end of the
paper, including thanks to reviewers who gave useful comments, to colleagues
who contributed to the ideas, and to funding agencies and corporate sponsors
that provided financial support.

## References

  * Newman et al. [2022] B. A. Newman, R. M. Aronson, K. Kitani, and H. Admoni.  Helping people through space and time: Assistance as a perspective on human-robot interaction.  _Frontiers in Robotics and AI_ , 8, 2022.  ISSN 2296-9144.  doi:10.3389/frobt.2021.720319.  URL https://www.frontiersin.org/articles/10.3389/frobt.2021.720319.
  * Abdo et al. [2015] N. Abdo, C. Stachniss, L. Spinello, and W. Burgard.  Robot, organize my shelves! tidying up objects by predicting user preferences.  In _2015 IEEE international conference on robotics and automation (ICRA)_ , pages 1557–1564. IEEE, 2015.
  * Kang et al. [2018] M. Kang, Y. Kwon, and S.-E. Yoon.  Automated task planning using object arrangement optimization.  In _2018 15th International Conference on Ubiquitous Robots (UR)_ , pages 334–341. IEEE, 2018.
  * Kapelyukh and Johns [2022] I. Kapelyukh and E. Johns.  My house, my rules: Learning tidying preferences with graph neural networks.  In _Conference on Robot Learning_ , pages 740–749. PMLR, 2022.
  * Newman et al. [2023] B. A. Newman, C. J. Paxton, K. Kitani, and H. Admoni.  Towards online adaptation for autonomous household assistants.  In _Companion of the 2023 ACM/IEEE International Conference on Human-Robot Interaction_ , HRI ’23, page 506–510, New York, NY, USA, 2023. Association for Computing Machinery.  ISBN 9781450399708.  doi:10.1145/3568294.3580136.  URL https://doi.org/10.1145/3568294.3580136.
  * Wu et al. [2023] J. Wu, R. Antonova, A. Kan, M. Lepert, A. Zeng, S. Song, J. Bohg, S. Rusinkiewicz, and T. Funkhouser.  Tidybot: Personalized robot assistance with large language models.  _arXiv preprint arXiv:2305.05658_ , 2023.
  * Puig et al. [2018] X. Puig, K. Ra, M. Boben, J. Li, T. Wang, S. Fidler, and A. Torralba.  Virtualhome: Simulating household activities via programs.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 8494–8502, 2018.
  * Batra et al. [2020] D. Batra, A. X. Chang, S. Chernova, A. J. Davison, J. Deng, V. Koltun, S. Levine, J. Malik, I. Mordatch, R. Mottaghi, et al.  Rearrangement: A challenge for embodied ai.  _arXiv preprint arXiv:2011.01975_ , 2020.
  * Newman et al. [2020] B. Newman, K. Carlberg, and R. Desai.  Optimal assistance for object-rearrangement tasks in augmented reality, 2020.  URL https://arxiv.org/abs/2010.07358.
  * Szot et al. [2021] A. Szot, A. Clegg, E. Undersander, E. Wijmans, Y. Zhao, J. Turner, N. Maestre, M. Mukadam, D. S. Chaplot, O. Maksymets, et al.  Habitat 2.0: Training home assistants to rearrange their habitat.  _Advances in neural information processing systems_ , 34:251–266, 2021.
  * Ehsani et al. [2021] K. Ehsani, W. Han, A. Herrasti, E. VanderBilt, L. Weihs, E. Kolve, A. Kembhavi, and R. Mottaghi.  Manipulathor: A framework for visual object manipulation.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 4497–4506, 2021.
  * Weihs et al. [2021] L. Weihs, M. Deitke, A. Kembhavi, and R. Mottaghi.  Visual room rearrangement.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 5922–5931, 2021.
  * Gan et al. [2021] C. Gan, S. Zhou, J. Schwartz, S. Alter, A. Bhandwaldar, D. Gutfreund, D. L. Yamins, J. J. DiCarlo, J. McDermott, A. Torralba, et al.  The threedworld transport challenge: A visually guided task-and-motion planning benchmark for physically realistic embodied ai.  _arXiv preprint arXiv:2103.14025_ , 2021.
  * Newman et al. [2022] B. A. Newman, R. M. Aronson, S. S. Srinivasa, K. Kitani, and H. Admoni.  Harmonic: A multimodal dataset of assistive human–robot collaboration.  _The International Journal of Robotics Research_ , 41(1):3–11, 2022.  doi:10.1177/02783649211050677.  URL https://doi.org/10.1177/02783649211050677.
  * Rasch et al. [2019] R. Rasch, D. Sprute, A. Pörtner, S. Battermann, and M. König.  Tidy up my room: Multi-agent cooperation for service tasks in smart environments.  _Journal of Ambient Intelligence and Smart Environments_ , 11(3):261–275, 2019.
  * Newman et al. [2020] B. A. Newman, A. Biswas, S. Ahuja, S. Girdhar, K. K. Kitani, and H. Admoni.  Examining the effects of anticipatory robot assistance on human decision making.  In _Proceedings of the 12th International Conference on Social Robotics, (ICSR 2024), November 14–18_. LNAI, 2020.
  * Yan et al. [2021] Z. Yan, N. Crombez, J. Buisson, Y. Ruichck, T. Krajnik, and L. Sun.  A quantifiable stratification strategy for tidy-up in service robotics.  In _2021 IEEE international conference on advanced robotics and its social impacts (ARSO)_ , pages 182–187. IEEE, 2021.
  * Taniguchi et al. [2021] A. Taniguchi, S. Isobe, L. El Hafi, Y. Hagiwara, and T. Taniguchi.  Autonomous planning based on spatial concepts to tidy up home environments with service robots.  _Advanced Robotics_ , 35(8):471–489, 2021.
  * Kant et al. [2022] Y. Kant, A. Ramachandran, S. Yenamandra, I. Gilitschenski, D. Batra, A. Szot, and H. Agrawal.  Housekeep: Tidying virtual households using commonsense reasoning.  In _European Conference on Computer Vision_ , pages 355–373. Springer, 2022.
  * Sarch et al. [2022] G. Sarch, Z. Fang, A. W. Harley, P. Schydlo, M. J. Tarr, S. Gupta, and K. Fragkiadaki.  Tidee: Tidying up novel rooms using visuo-semantic commonsense priors.  In _European conference on computer vision_ , pages 480–496. Springer, 2022.
  * Newman et al. [2024] B. A. Newman, C. J. Paxton, K. Kitani, and H. Admoni.  Bootstrapping linear models for fast online adaptation in human-agent collaboration.  In _Proc. of the 23rd International Conference on Autonomous Agents and Multiagent Systems), Auckland, New Zealand, May 6 – 10, 2024_. IFAAMAS, 2024.
  * Firoozi et al. [2023] R. Firoozi, J. Tucker, S. Tian, A. Majumdar, J. Sun, W. Liu, Y. Zhu, S. Song, A. Kapoor, K. Hausman, et al.  Foundation models in robotics: Applications, challenges, and the future.  _arXiv preprint arXiv:2312.07843_ , 2023.
  * Hu et al. [2023] Y. Hu, Q. Xie, V. Jain, J. Francis, J. Patrikar, N. Keetha, S. Kim, Y. Xie, T. Zhang, Z. Zhao, et al.  Toward general-purpose robots via foundation models: A survey and meta-analysis.  _arXiv preprint arXiv:2312.08782_ , 2023.
  * Song et al. [2023] C. H. Song, J. Wu, C. Washington, B. M. Sadler, W.-L. Chao, and Y. Su.  Llm-planner: Few-shot grounded planning for embodied agents with large language models.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 2998–3009, 2023.
  * Huang et al. [2022] W. Huang, P. Abbeel, D. Pathak, and I. Mordatch.  Language models as zero-shot planners: Extracting actionable knowledge for embodied agents.  In _International Conference on Machine Learning_ , pages 9118–9147. PMLR, 2022.
  * Ahn et al. [2022] M. Ahn, A. Brohan, N. Brown, Y. Chebotar, O. Cortes, B. David, C. Finn, C. Fu, K. Gopalakrishnan, K. Hausman, A. Herzog, D. Ho, J. Hsu, J. Ibarz, B. Ichter, A. Irpan, E. Jang, R. J. Ruano, K. Jeffrey, S. Jesmonth, N. Joshi, R. Julian, D. Kalashnikov, Y. Kuang, K.-H. Lee, S. Levine, Y. Lu, L. Luu, C. Parada, P. Pastor, J. Quiambao, K. Rao, J. Rettinghouse, D. Reyes, P. Sermanet, N. Sievers, C. Tan, A. Toshev, V. Vanhoucke, F. Xia, T. Xiao, P. Xu, S. Xu, M. Yan, and A. Zeng.  Do as i can and not as i say: Grounding language in robotic affordances.  In _arXiv preprint arXiv:2204.01691_ , 2022.
  * Huang et al. [2022] W. Huang, F. Xia, T. Xiao, H. Chan, J. Liang, P. Florence, A. Zeng, J. Tompson, I. Mordatch, Y. Chebotar, et al.  Inner monologue: Embodied reasoning through planning with language models.  _arXiv preprint arXiv:2207.05608_ , 2022.
  * Zeng et al. [2022] A. Zeng, M. Attarian, B. Ichter, K. Choromanski, A. Wong, S. Welker, F. Tombari, A. Purohit, M. Ryoo, V. Sindhwani, et al.  Socratic models: Composing zero-shot multimodal reasoning with language.  _arXiv preprint arXiv:2204.00598_ , 2022.
  * Liu et al. [2023] Z. Liu, A. Bahety, and S. Song.  Reflect: Summarizing robot experiences for failure explanation and correction.  _arXiv preprint arXiv:2306.15724_ , 2023.
  * Raman et al. [2022] S. S. Raman, V. Cohen, E. Rosen, I. Idrees, D. Paulius, and S. Tellex.  Planning with large language models via corrective re-prompting.  In _NeurIPS 2022 Foundation Models for Decision Making Workshop_ , 2022.
  * Lin et al. [2023] K. Lin, C. Agia, T. Migimatsu, M. Pavone, and J. Bohg.  Text2motion: From natural language instructions to feasible plans.  _arXiv preprint arXiv:2303.12153_ , 2023.
  * Liu et al. [2023] B. Liu, Y. Jiang, X. Zhang, Q. Liu, S. Zhang, J. Biswas, and P. Stone.  Llm+ p: Empowering large language models with optimal planning proficiency.  _arXiv preprint arXiv:2304.11477_ , 2023.
  * Wang et al. [2023] Z. Wang, S. Cai, G. Chen, A. Liu, X. Ma, and Y. Liang.  Describe, explain, plan and select: Interactive planning with large language models enables open-world multi-task agents.  _arXiv preprint arXiv:2302.01560_ , 2023.
  * Silver et al. [2022] T. Silver, V. Hariprasad, R. S. Shuttleworth, N. Kumar, T. Lozano-Pérez, and L. P. Kaelbling.  Pddl planning with pretrained large language models.  In _NeurIPS 2022 foundation models for decision making workshop_ , 2022.
  * Newman et al. [2024] B. A. Newman, P. Gupta, Y. Bisk, K. Kitani, H. Admoni, and C. Paxton.  Leveraging vision and language models for zero-shot, personalization of household multi-object rearrangement tasks.  In _Proceedings of Human-Large Language Model Interaction Workshop HRI’24_. ACM, 2024.
  * Dorbala et al. [2022] V. S. Dorbala, G. Sigurdsson, R. Piramuthu, J. Thomason, and G. S. Sukhatme.  Clip-nav: Using clip for zero-shot vision-and-language navigation.  _arXiv preprint arXiv:2211.16649_ , 2022.
  * Chen et al. [2023] B. Chen, F. Xia, B. Ichter, K. Rao, K. Gopalakrishnan, M. S. Ryoo, A. Stone, and D. Kappler.  Open-vocabulary queryable scene representations for real world planning.  In _2023 IEEE International Conference on Robotics and Automation (ICRA)_ , pages 11509–11522. IEEE, 2023.
  * Shah et al. [2023a] D. Shah, B. Osiński, S. Levine, et al.  Lm-nav: Robotic navigation with large pre-trained models of language, vision, and action.  In _Conference on Robot Learning_ , pages 492–504. PMLR, 2023a.
  * Shah et al. [2023b] D. Shah, M. R. Equi, B. Osiński, F. Xia, B. Ichter, and S. Levine.  Navigation with large language models: Semantic guesswork as a heuristic for planning.  In _Conference on Robot Learning_ , pages 2683–2699. PMLR, 2023b.
  * Gadre et al. [2023] S. Y. Gadre, M. Wortsman, G. Ilharco, L. Schmidt, and S. Song.  Cows on pasture: Baselines and benchmarks for language-driven zero-shot object navigation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 23171–23181, 2023.
  * Huang et al. [2023] C. Huang, O. Mees, A. Zeng, and W. Burgard.  Visual language maps for robot navigation.  In _2023 IEEE International Conference on Robotics and Automation (ICRA)_ , pages 10608–10615. IEEE, 2023.
  * Cui et al. [2022] Y. Cui, S. Niekum, A. Gupta, V. Kumar, and A. Rajeswaran.  Can foundation models perform zero-shot task specification for robot manipulation?  In _Learning for Dynamics and Control Conference_ , pages 893–905. PMLR, 2022.
  * Shridhar et al. [2022] M. Shridhar, L. Manuelli, and D. Fox.  Cliport: What and where pathways for robotic manipulation.  In _Conference on Robot Learning_ , pages 894–906. PMLR, 2022.
  * Liang et al. [2023] J. Liang, W. Huang, F. Xia, P. Xu, K. Hausman, B. Ichter, P. Florence, and A. Zeng.  Code as policies: Language model programs for embodied control.  In _2023 IEEE International Conference on Robotics and Automation (ICRA)_ , pages 9493–9500. IEEE, 2023.
  * Singh et al. [2023] I. Singh, V. Blukis, A. Mousavian, A. Goyal, D. Xu, J. Tremblay, D. Fox, J. Thomason, and A. Garg.  Progprompt: Generating situated robot task plans using large language models.  In _2023 IEEE International Conference on Robotics and Automation (ICRA)_ , pages 11523–11530. IEEE, 2023.
  * Huang et al. [2023] W. Huang, C. Wang, R. Zhang, Y. Li, J. Wu, and L. Fei-Fei.  Voxposer: Composable 3d value maps for robotic manipulation with language models.  _arXiv preprint arXiv:2307.05973_ , 2023.
  * Yu et al. [2023] W. Yu, N. Gileadi, C. Fu, S. Kirmani, K.-H. Lee, M. G. Arenas, H.-T. L. Chiang, T. Erez, L. Hasenclever, J. Humplik, et al.  Language to rewards for robotic skill synthesis.  _arXiv preprint arXiv:2306.08647_ , 2023.
  * Ma et al. [2023] Y. J. Ma, W. Liang, G. Wang, D.-A. Huang, O. Bastani, D. Jayaraman, Y. Zhu, L. Fan, and A. Anandkumar.  Eureka: Human-level reward design via coding large language models.  _arXiv preprint arXiv:2310.12931_ , 2023.
  * Brown et al. [2020] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901, 2020.
  * Lester et al. [2021] B. Lester, R. Al-Rfou, and N. Constant.  The power of scale for parameter-efficient prompt tuning.  _arXiv preprint arXiv:2104.08691_ , 2021.
  * Li and Liang [2021] X. L. Li and P. Liang.  Prefix-tuning: Optimizing continuous prompts for generation.  _arXiv preprint arXiv:2101.00190_ , 2021.
  * Pryzant et al. [2023] R. Pryzant, D. Iter, J. Li, Y. T. Lee, C. Zhu, and M. Zeng.  Automatic prompt optimization with” gradient descent” and beam search.  _arXiv preprint arXiv:2305.03495_ , 2023.
  * Xu et al. [2022] H. Xu, Y. Chen, Y. Du, N. Shao, Y. Wang, H. Li, and Z. Yang.  Gps: Genetic prompt search for efficient few-shot learning.  _arXiv preprint arXiv:2210.17041_ , 2022.
  * Yang et al. [2023] C. Yang, X. Wang, Y. Lu, H. Liu, Q. V. Le, D. Zhou, and X. Chen.  Large language models as optimizers.  _arXiv preprint arXiv:2309.03409_ , 2023.
  * Nasiriany et al. [2024] S. Nasiriany, F. Xia, W. Yu, T. Xiao, J. Liang, I. Dasgupta, A. Xie, D. Driess, A. Wahid, Z. Xu, et al.  Pivot: Iterative visual prompting elicits actionable knowledge for vlms.  _arXiv preprint arXiv:2402.07872_ , 2024.
  * Liu et al. [2024] F. Liu, K. Fang, P. Abbeel, and S. Levine.  Moka: Open-vocabulary robotic manipulation through mark-based visual prompting.  _arXiv preprint arXiv:2403.03174_ , 2024.
  * et. al. [2023] O. et. al.  Gpt-4 technical report, 2023.
  * et. al. [2024a] A. T. et. al.  Claude 3 haiku, 2024a.
  * et. al. [2024b] G. T. et. al.  Gemini: A family of highly capable multimodal models, 2024b.
  * Yang et al. [2023] J. Yang, H. Zhang, F. Li, X. Zou, C. Li, and J. Gao.  Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v.  _arXiv preprint arXiv:2310.11441_ , 2023.
  * Kabsch [1976] W. Kabsch.  A solution for the best rotation to relate two sets of vectors.  _Acta Crystallographica Section A_ , 32(5):922–923, Sep 1976.  doi:10.1107/S0567739476001873.  URL https://doi.org/10.1107/S0567739476001873.
  * Kapelyukh et al. [2023] I. Kapelyukh, V. Vosylius, and E. Johns.  Dall-e-bot: Introducing web-scale diffusion models to robotics.  2023\.
  * Chen et al. [2024] B. Chen, Z. Xu, S. Kirmani, B. Ichter, D. Driess, P. Florence, D. Sadigh, L. Guibas, and F. Xia.  Spatialvlm: Endowing vision-language models with spatial reasoning capabilities.  _arXiv preprint arXiv:2401.12168_ , 2024.  URL https://arxiv.org/abs/2401.12168.

##  Appendix A Further Details on Prompting and Predicting

Here we present further details on how we prompt VLMs to obtain task plans for
complex, fine-grained preference tasks. Specifically, complex, fine-grained
preference tasks include those in which a preference can be held over multiple
different features of the task simultaneously and these features can be
discrete, such as the color or material of an object, or continuous, such as
the precise location of an object on a table’s surface.

###  A.1 Selecting Objects

Objects as Language. Our first lifting function represents objects as
language. We do this by taking each object in our dataset, and coding them for
color, material, and type of utensil. We take these features and create an
easy to parse json file. Then, we give each object an arbitrary ID between 0
and 125 and concatenate the ID and language representation into a json file
with 125 elements, each of which is a different object representation. We
include this json as a part of the prompt to the VLM. To make a prediction
about which object to choose, the VLM predicts an object ID directly. When we
request multiple responses from the model and choose the mode of the model
responses as the predicted object.

    example_json = {
        "color1": "blue", "material1": "ceramic",
        "shape": "round", "pattern": "solid",
        "texture": "smooth", "class": "cup",
        "material2": "ceramic", "color2": "blue",
    }

Grid-of-Marked Objects. We turn object selection into a multiple-choice
question answering task. We do this by displaying all the objects that the
model could use to arrange the table simultaneously in images separated by
object category. Each object instance is overlaid with a spatial reference
mark. The model uses these marks to identify which object to place. Again,
since we ask for multiple responses, we take the mode of the responses as our
method’s prediction.

###  A.2 Predicting Locations and Rotations

Unmarked Arrangements. To produce an arrangement using this lifting function,
we prompt the model to produce directly an x-position, y-position and rotation
in the normalized image coordinate frame, e.g. (0,1). The final answer is an
average over model responses.

Grid-Marked Arrangements. Our second arrangement lifting function again
borrows the idea of inserting spatial reference markers into a scene for both
localization and vocabulary grounding. We turn this task into a visual
multiple choice question-answering task by overlaying a spatially marked grid
using grid cells labelled with numbers and letters. To make a prediction, We
then take the weighted average of the locations of the centers of the grid
cells and use this as our prediction of $\mathbf{x}$. To predict a rotation
$r$, we discretize the rotation space into eight cardinal directions and ask
the model to produce the one that is closest to the direction the object
should be pointing, assuming the top of the image is North. Again, we average
over all model responses to produce a final answer.

###  A.3 Predicting Placement Order

We don’t consider any visual lifting functions to predict object order. We
simply include the placement order for each arrangement in $k$ in the problem
definition text prompt passed to the VLM. In this problem definition, we also
ask the model to produce its response list ordered by the placement order of
each object. To find the ultimate task plan, we first take the mode of the
lengths of each response to get the task plan length. Then, for each step in
the task plan, we take the mode over the utensil type.

###  A.4 Prompts

We use the following prompts for each of our model variants:

Objects as Language My preferences for setting a table are shown in the first
images. You are helping me set the table in the final image according to my
preferences. What objects should I place in this image, where should I place
them, and how should they be oriented? Give a position in [x,y] where each
value is a number between 0 and 1. Give rotation as the number of degrees
clockwise from image north. Give your answer as a python list formatted as
follows: [’type’:object_type, ’id’: object_reference_id, ’position’: [x, y],
‘rotation’: degrees]. Give object_id as an integer. Include only this list in
your response.

Unmarked Arrangements My preferences for setting a table are shown in the
images named final_state_K.jpg. You are helping me set the table in
initial_state_1.jpg according to my preferences. What objects should I place
in this image, where should I place them, and how should they be oriented?
Give a position in [x,y] where each value is a number between 0 and 1. Give
rotation as the number of degrees clockwise from image north. Give your answer
as a python list formatted as follows: [‘type’:object_type, ‘id’:
object_reference_id, ‘position’: [x, y], ‘rotation’: degrees]. Give object_id
as an integer. Include only this list in your response.

Grid Marked Arrangements My preferences for setting a table are shown in the
images named final_state_N.jpg. You are helping me set the table in
initial_state_1.jpg according to my preferences. What objects should I place
in this image, where should I place them, and how should they be oriented?
Please list all grid cells the object will intersect after it is placed. Give
the orientation as the closest cardinal direction [N, NE, E, SE, S, SW, W, NW]
the object is pointing in after it is placed. The top of the image is N. Give
your answer as a python list formatted as follows: [‘type’:object_type, ‘id’:
object_reference_id, ‘position’:[grid_cell_ids],
‘cardinal_direction’:direction]. Give grid_cell_ids as a list of string. Give
object_id as an integer. Include only this list in your response.

##  Appendix B Further Details on the Kabsch Algorithm

To compute the RMSD between two arragnements, we first define each arrangement
as a frame of points: $A^{gt}$ and $A^{pred}$. We match points between the
predicted and ground truth frames by computing their similarity matrix $D$,
using the Hamming distance over features as our distance function. Then for
each object in the ground truth frame, we find the most similar object in the
predicted setting, ensuring that each object only matches with one object from
the other setting. This yields two orderings $\bar{A}^{gt}$ and
$\bar{A}^{pred}$ over the points in the respective frames. We ensure that each
ordering is of the same length, $N$.

##  Appendix C VLM Model Choices and Costs

We factor in several considerations when choosing the VLMs to evaluate. First,
we want to optimize for the performance of the model; we want to test the best
available models to test our hypothesis that they can effectively featurize
complex preferences. However, technical limitations do play a role in our
eventual decisions. For example, the token request limit and pricing for
larger Anthropic models (Sonnet and Opus) made them prohibitive to use for
this work.

| Input | Output | Total
---|---|---|---
| Tokens | USD$ | Tokens | USD$ | Tokens | USD$
GPT-4o | 8035 | 0.05 | 6250 | 0.10 | 14285 | 0.15
Claude Haiku | 40175 | 0.02 | 6250 | 0.01 | 46425 | 0.03
Gemini 1.5 Pro Vision | $-$ | 0.08 | $-$ | 0.02 | $-$ | 0.10
Table 3: We compare the costs of running various models for a single
experiment in our dataset. A single experiment consists of 4 arrangement
images, 4 text labels for the placement order, and a text prompt. We ask each
model for 5 total responses. This is handled automatically by OpenAIs API. For
Anthropic and Google models, we manually pass 5 requests. Google’s pricing
model prices per image and character, rather than tokens, so this column is
omitted. Costs are rounded up to the nearest whole cent.

In Tab. 3 we show the various token and dollar costs to run our method on a
single experiment in our dataset. While we ran these experiments prior to
Google’s implementation of a pricing model, we include estimated costs here,
too.

##  Appendix D Examples of the Simulated Task

Figure 6: Examples of the simulated preferences over a variety of colors prior
to being transformed (top) and after being transformed, bottom.

Figure 6 shows three examples of our simulated preferences before (top) and
after (bottom) their transformations. We simulate a traditional Western table
arrangement (center), a mirrored arrangement (center), and an abstract
arrangement (right) to capture a variety of different difficulties. We add
further difficulty by adding random transformations and rotations to the
arrangements (bottom).

##  Appendix E Data Collection Interface

Figure 7: Images from the data collection interface. Participants are first
presented with a n empty table and a set of object icons (left). After
selecting an icon they are presented with the objects they can use to create
the arrangement (center). After selecting an object, they can place it on the
table. They continue this loop until they are satisfied with the arrangement
(right).

Figure 8 shows images from the data collection interface. Participants are
presented with a randomized order of empty tables they are asked to set. They
must set a minimum of three objects. They first choose an object icon that
represents the object they would like to place, then they are presented with
all object instances of that category. After choosing an instance, they drag,
drop and rotate the object on the table’s surface. They continue until they
are satisfied with the arrangement. We filter out participants who do not
complete teh self evaluation portion of the task, or those who do not complete
all 5 trials.

Figure 8: Images from the self evaluation portion of the user study.
Participants rated their scene, left, then the scene was jittered and rated,
center, finally the participants corrected the scene and rated again.

Figure 8 shows the self evaluation portion of the user study, where
participants rated their scene and possible transformations of their scene
that could represent positional errors our model could make. The goal of this
section of the user study was to correlate user acceptability to random
(jitters) and nonrandom (corrections) errors as measured in RMSD. This gives
us an upper (i.e. if all model errors were nonrandom) and lower bound (i.e. if
all model errors were random) against which to measure our models performance.
We report the stricter (e.g. assuming all model errors are random) of these
two measures in the paper.

Correcting the jittered scene. Finally, we seek an objective measure of the
types of errors that people find acceptable. To do this, we ask participants
to correct the jittered scene back to its pre-jitter state. This creates an
arrangement $a_{correc}$. Finally, we ask for an acceptability rating of
$a_{correct}$, $b_{correct}$.

To measure objective acceptability we perform a similar process by taking the
difference in acceptability ratings between the post-correction arrangement
and the initial arrangement: $b_{correct}-b_{initial}$ and correlating this
value to the distance between the corrected scene and initial scenes:
$d(a_{correct},a_{initial})$.

###  E.1 Descriptive Self Evaluation Results

Figure 9: Plots of the difference between jitter rating and baseline rating
(left) and correction rating and baseline rating (right).

Figure 9 shows the correlation between ratings differences and RMSD for both
the jitter condition (representing random errors), left, and the correction
condition (representing nonrandom errors), right. From the graph on the left,
we see a cluster of participants close to the origin, and then a quick slope
up, with many participants distributed througout the top of the graph. This
shows that participants are very sensitive to random errors, which is the
metric against with we report. We see from the graph on the right that
participants typically correct the scene back to an accptable range, even with
some error from the initial scene.

###  E.2 Example Arrangements

Figure 10: Example arrangements from three different participants.

Figure 10 shows five arrangements examples from three different participants.
We can see a diversity across the arrangements with which objects they choose,
where they place the objects on the table, and where they place the objects in
relation to one another. We can also see some consistency within an
arrangement across those same features. Our data set contains 995 of such
arrangements drawn from 199 different people.