# Congruence Properties of Indices of Triangular Numbers Multiple of Other
Triangular Numbers

Vladimir PLETSER  European Space Agency (ret<EMAIL_ADDRESS>
###### Abstract.

It is known that, for any positive non-square integer multiplier $k$, there is
an infinity of multiples of triangular numbers which are triangular numbers.
We analyze the congruence properties of the indices $\xi$ of triangular
numbers that are multiples of other triangular numbers. We show that the
remainders in the congruence relations of $\xi$ modulo $k$ come always in
pairs whose sum always equal $\left(k-1\right)$, always include 0 and
$\left(k-1\right)$, and only 0 and $\left(k-1\right)$ if $k$ is prime, or an
odd power of a prime, or an even square plus one or an odd square minus one or
minus two. If the multiplier $k$ is twice the triangular number of $n$, the
set of remainders includes also $n$ and $\left(n^{2}-1\right)$ and if $k$ has
integer factors, the set of remainders include multiples of a factor following
certain rules. Finally, algebraic expressions are found for remainders in
function of $k$ and its factors. Several exceptions are noticed and
superseding rules exist between various rules and expressions of remainders.
This approach allows to eliminate in numerical searches those
$\left(k-\upsilon\right)$ values of $\xi_{i}$ that are known not to provide
solutions, where $\upsilon$ is the even number of remainders. The gain is
typically in the order of $k/\upsilon$, with $\upsilon\ll k$ for large values
of $k$.

###### Key words and phrases:

Triangular Numbers, Multiple of Triangular Numbers, Recurrent Relations,
Congruence Properties

AMS 2010 Mathematics Subject Classification: Primary 11A25; Secondary 11D09

##  1\. Introduction

Triangular numbers $T_{t}=\frac{t\left(t+1\right)}{2}$ are one of the figurate
numbers enjoying many properties; see, e.g., [1, 2] for relations and
formulas. Triangular numbers $T_{\xi}$ that are multiples of other triangular
number $T_{t}$

(1.1) $T_{\xi}=kT_{t}$

are investigated. Only solutions for $k>1$ are considered as the cases $k=0$
and $k=1$ yield respectively $\xi=0$ and $\xi=t,\forall t$. Accounts of
previous attempts to characterize these triangular numbers multiple of other
triangular numbers can be found in [3, 4, 5, 6, 7, 8, 9]. Recently, Pletser
showed [9] that, for non-square integer values of $k$, there are infinitely
many solutions that can be represented simply by recurrent relations of the
four variables $t,\xi,Tt$ and $T_{\xi}$, involving a rank $r$ and parameters
$\kappa$ and $\gamma$, which are respectively the sum and the product of the
$\left(r-1\right)^{\text{th}}$ and the $r^{\text{th}}$ values of $t$. The rank
$r$ is being defined as the number of successive values of $t$ solutions of
(1.1) such that their successive ratios are slowly decreasing without jumps.

In this paper, we present a method based on the congruent properties of
$\xi\left(\text{mod\,}k\right)$, searching for expressions of the remainders
in function of $k$ or of its factors. This approach accelerates the numerical
search of the values of $t_{n}$ and $\xi_{n}$ that solve (1.1), as it
eliminates values of $\xi$ that are known not to provide solutions to (1.1).
The gain is typically in the order of $k/\upsilon$ where $\upsilon$ is the
number of remainders, which is usually such that $\upsilon\ll k$.

##  2\. Rank and Recurrent Equations

Sequences of solutions of (1.1) are known for $k=2,3,5,6,7,8$ and are listed
in the Online Encyclopedia of Integer Sequences (OEIS) [10], with references
given in Table 1.

Table 1. OEIS [10] references of sequences of integer solutions of (1.1) for $k=2,3,5,6,7,8$ $k$ | 2 | 3 | 5 | 6 | 7 | 8
---|---|---|---|---|---|---
$t$ | A053141 | A061278 | A077259 | A077288 | A077398 | A336623
$\xi$ | A001652 | A001571 | A077262 | A077291 | A077401 | A336625
$T_{t}$ | A075528 | A076139 | A077260 | A077289 | A077399 | A336624
$T_{\xi}$ | A029549 | A076140 | A077261 | A077290 | A077400 | A336626

Among all solutions, $t=0$ is always a first solution of (1.1) for all non-
square integer value of $k$, yielding $\xi=0$.

Let’s consider the two cases of $k=2$ and $k=7$ yielding the successive
solution pairs as shown in Table 2. We indicate also the ratios
$t_{n}/t_{n-1}$ for both cases and $t_{n}/t_{n-2}$ for $k=7$. It is seen that
for $k=2$, the ratio $t_{n}/t_{n-1}$ varies between close values, from 7 down
to 5.829, while for $k=7$, the ratio $t_{n}/t_{n-1}$ alternates between values
2.5 … 2.216 and 7.8 … 7.23, while the ratio $t_{n}/t_{n-2}$ decreases
regularly from 19.5 to 16.023 (corresponding approximately to the product of
the alternating values of the ratio $t_{n}/t_{n-1}$). We call rank $r$ the
integer value such that $t_{n}/t_{n-r}$ is approximately constant or, better,
decreases regularly without jumps (a more precise definition is given
further). So, here, the case $k=2$ has rank $r=1$ and the case $k=7$ has rank
$r=2$.

Table 2. Solutions of (1.1) for $k=2,7$ $n$ | $k=2$ | $k=7$
---|---|---
| $t_{n}$ | $\xi_{n}$ | $\frac{t_{n}}{t_{n-1}}$ | $t_{n}$ | $\xi_{n}$ | $\frac{t_{n}}{t_{n-1}}$ | $\frac{t_{n}}{t_{n-2}}$
0 | 0 | 0 |  | 0 | 0 |  |
1 | 2 | 3 | – | 2 | 6 | – | –
2 | 14 | 20 | 7 | 5 | 14 | 2.5 | –
3 | 84 | 119 | 6 | 39 | 104 | 7.8 | 19.5
4 | 492 | 696 | 5.857 | 87 | 231 | 2.231 | 17.4
5 | 2870 | 4059 | 5.833 | 629 | 1665 | 7.230 | 16.128
6 | 16730 | 23660 | 5.829 | 1394 | 3689 | 2.216 | 16.023

In [9],we showed that the rank $r$ is the index of $t_{r}$ and $\xi_{r}$
solutions of (1.1) such that

(2.1) $\kappa=t_{r}+t_{r-1}=\xi_{r}-\xi_{r-1}-1$

and that the ratio $t_{2r}/t_{r}$, corrected by the ratio $t_{r-1}/t_{r}$, is
equal to a constant $2\kappa+3$

(2.2) $\frac{t_{2r}-t_{r-1}}{t_{r}}=2\kappa+3$

For example, for $k=7$ and $r=2$, (2.1) and (2.2) yield respectively,
$\kappa=7$ and $2\kappa+3=17$.

Four recurrent equations for $t_{n},\xi_{n},T_{t_{n}}$ and $T_{\xi_{n}}$ are
given in [9] for each non-square integer value of $k$

(2.3) $\displaystyle t_{n}$
$\displaystyle=2\left(\kappa+1\right)t_{n-r}-t_{n-2r}+\kappa$ (2.4)
$\displaystyle\xi_{n}$
$\displaystyle=2\left(\kappa+1\right)\xi_{n-r}-\xi_{n-2r}+\kappa$ (2.5)
$\displaystyle T_{t_{n}}$
$\displaystyle=\left(4\left(\kappa+1\right)^{2}-2\right)T_{t_{n-r}}-T_{t_{n-2r}}+\left(T_{\kappa}-\gamma\right)$
(2.6) $\displaystyle T_{\xi_{n}}$
$\displaystyle=\left(4\left(\kappa+1\right)^{2}-2\right)T_{\xi_{n-r}}-T_{\xi_{n-2r}}+k\left(T_{\kappa}-\gamma\right)$

where coefficients are functions of two constants $\kappa$ and $\gamma$,
respectively the sum $\kappa$ and the product $\gamma=t_{r-1}t_{r}$ of the
first two sequential values of $t_{r}$ and $t_{r-1}$. Note that the first
three relations (2.3) to (2.5) are independent from the value of $k$.

##  3\. Congruence of $\xi$ modulo $k$

We use the following notations: for $A,B,C\in\mathbb{Z},B<C,C>1$, $A\equiv
B\left(\text{mod\,}C\right)$ means that $\exists D\in\mathbb{Z}$ such that
$A=DC+B$, where $B$ and $C$ are called respectively the remainder and the
modulus. To search numerically for the values of $t_{n}$ and $\xi_{n}$ that
solve (1.1), one can use the congruent properties of
$\xi\left(\text{mod\,}k\right)$ given in the following propositions. In other
words, we search in the following propositions for expressions of the
remainders in function of $k$ or of its factors.

######  Proposition 1.

For $\forall s,k\in\mathbb{Z}^{+}$, $k$ non-square,
$\exists\xi,\mu,\upsilon,i,j\in\mathbb{Z}^{+}$, such that if $\xi_{i}$ are
solutions of (1.1), then for $\xi_{i}\equiv\mu_{j}\left(\text{mod\,}k\right)$
with $1\leq j\leq\upsilon$, the number $\upsilon$ of remainders is always
even, $\upsilon\equiv 0\left(\text{mod\,}2\right)$, the remainders come in
pairs whose sum is always equal to $\left(k-1\right)$, and the sum of all
remainders is always equal to the product of $\left(k-1\right)$ and the number
of remainder pairs, $\sum_{j=1}^{\upsilon}\mu_{j}=\left(k-1\right)\upsilon/2$.

###### Proof.

Let $s,i,j,k,\xi,\mu,\upsilon,\alpha,\beta\in\mathbb{Z}^{+}$, $k$ non-square,
and $\xi_{i}$ solutions of (1.1). Rewriting (1.1) as
$T_{t_{i}}=T_{\xi_{i}}/k$, for $T_{t_{i}}$ to be integer, $k$ must divide
exactly $T_{\xi_{i}}=\xi_{i}\left(\xi_{i}+1\right)/2$, i.e., among all
possibilities, $k$ divides either $\xi_{i}$ or $\left(\xi_{i}+1\right)$,
yielding two possible solutions $\xi_{i}\equiv 0\left(\text{mod\,}k\right)$ or
$\xi_{i}\equiv-1\left(\text{mod}\,k\right)$, i.e. $\upsilon=2$ and the set of
$\mu_{j}$ includes $\left\\{0,\left(k-1\right)\right\\}$. This means that
$\xi_{i}$ are always congruent to either $0$ or $\left(k-1\right)$ modulo $k$
for all non-square values of $k$.

Furthermore, if some $\xi_{i}$ are congruent to $\alpha$ modulo $k$, then
other $\xi_{i}$ are also congruent to $\beta$ modulo $k$ with
$\beta=\left(k-\alpha-1\right)$. As
$\xi_{i}\equiv\alpha\left(\text{mod}\,k\right)$, then
$\xi_{i}\left(\xi_{i}+1\right)/2\equiv\left(\alpha\left(\alpha+1\right)/2\right)\left(\text{mod\,}k\right)$
and replacing $\alpha$ by $\alpha=\left(k-\beta-1\right)$ yields
$\left(\alpha\left(\alpha+1\right)/2\right)=\left(\left(k-\beta-1\right)\left(k-\beta\right)/2\right)$,
giving
$\xi_{i}\left(\xi_{i}+1\right)/2\equiv\left(\left(k-\beta-1\right)\left(k-\beta\right)/2\right)\left(\text{mod\,}k\right)\equiv$

$\left(\beta\left(\beta+1\right)/2\right)\left(\text{mod\,}k\right)$. In this
case, $\upsilon=4$ and the set of $\mu_{j}$ includes, but not necessarily
limits to,
$\left\\{0,\alpha,\left(k-\alpha-1\right),\left(k-1\right)\right\\}$. ∎

Note that in some cases, $\upsilon>4$, as for $k=66,70,78,105,...$ , $\nu=8$.
However, in some other cases, $\upsilon=2$ only and the set of $\mu_{j}$
contains only $\left\\{0,\left(k-1\right)\right\\}$, as shown in the next
proposition. In this proposition, several rules (R) are given constraining the
congruence characteristics of $\xi_{i}$.

######  Proposition 2.

For $\forall s,k,\alpha,n\in\mathbb{Z}^{+}$, $k$ non-square, $\alpha>1$,
$\exists\xi,\mu,\upsilon,i\in\mathbb{Z}^{+}$, such that if $\xi_{i}$ are
solutions of (1.1), then $\xi_{i}$ are always only congruent to $0$ and
$\left(k-1\right)$ modulo $k$ , and $\upsilon=2$ if either (R1) $k$ is prime,
or (R2) $k=\alpha^{n}$ with $\alpha$ prime and $n$ odd, or (R3) $k=s^{2}+1$
with $s$ even, or (R4) $k=s^{\prime 2}-1$ or (R5) $k=s^{\prime 2}-2$ with
$s^{\prime}$ odd.

###### Proof.

Let $s,s^{\prime},k,\alpha>1,n,i,\xi,\mu,\upsilon\in\mathbb{Z}^{+}$, $k$ non-
square, and $\xi_{i}$ are solutions of (1.1).

(R1)+(R2): If $k$ is prime or if $k=\alpha^{n}$ (with $\alpha$ prime and $n$
odd as $k$ is non-square), then, in both cases, $k$ can only divide either
$\xi_{i}$ or $\left(\xi_{i}+1\right)$, yielding the two congruences
$\xi_{i}\equiv 0\left(\text{mod\,}k\right)$ and
$\xi_{i}\equiv-1\left(\text{mod\,}k\right)$.

(R3): If $k=s^{2}+1$ with $s$ even, the rank $r$ is always $r=2$ [11], and the
only two sets of solutions are

(3.1) $\displaystyle\left(t_{1},\xi_{1}\right)$
$\displaystyle=\left(s\left(s-1\right),\left(s^{2}+1\right)\left(s-1\right)\right)$
(3.2) $\displaystyle\left(t_{2},\xi_{2}\right)$
$\displaystyle=\left(s\left(s+1\right),\left(s^{2}+1\right)\left(s+1\right)-1\right)$

as can be easily shown. For $t_{1}$, forming

$\displaystyle kT_{t_{1}}$
$\displaystyle=\frac{1}{2}\left(s^{2}+1\right)\left(s\left(s-1\right)\right)\left(s\left(s-1\right)+1\right)$
$\displaystyle=\frac{1}{2}\left[\left(s^{2}+1\right)\left(s-1\right)\right]\left[\left(s^{2}+1\right)\left(s-1\right)+1\right]=T_{\xi_{1}}$

which is the triangular number of $\xi_{1}$. One obtains similarly $\xi_{2}$
from $t_{2}$. These two relations (3.1) and (3.2) show respectively that
$\xi_{1}$ is congruent to $0$ modulo $k$ and $\xi_{2}$ is congruent to
$\left(k-1\right)$ modulo $k$.

(R4) For $k=s^{\prime 2}-1$ with $s^{\prime}$ odd, the rank $r=2$ [11], and
the only two sets of solutions are

(3.3) $\displaystyle\left(t_{1},\xi_{1}\right)$
$\displaystyle=\left(\left(s^{\prime}-1\right)s^{\prime}-1,\left(s^{\prime
2}-1\right)\left(s^{\prime}-1\right)-1\right)$ (3.4)
$\displaystyle\left(t_{2},\xi_{2}\right)$
$\displaystyle=\left(\left(s^{\prime}-1\right)\left(s^{\prime}+2\right)+1,\left(s^{\prime
2}-1\right)\left(s^{\prime}+1\right)\right)$

as can be easily demonstrated as above. These two relations (3.3) and (3.4)
show that $\xi_{1}$ and $\xi_{2}$ are congruent respectively to
$\left(k-1\right)$ and $0$ modulo $k$.

(R5) For $k=s^{\prime 2}-2$ with $s^{\prime}$ odd, the rank $r=2$ [11], and
the only two sets of solutions are

(3.5) $\displaystyle\left(t_{1},\xi_{1}\right)$
$\displaystyle=\left(\frac{1}{2}\left(s^{\prime}-2\right)\left(s^{\prime}+1\right),\frac{1}{2}\left(s^{\prime
2}-2\right)\left(s^{\prime}-1\right)-1\right)$ (3.6)
$\displaystyle\left(t_{2},\xi_{2}\right)$
$\displaystyle=\left(\frac{s^{\prime}}{2}\left(s^{\prime}+1\right)-1,\frac{1}{2}\left(s^{\prime
2}-2\right)\left(s^{\prime}+1\right)\right)$

as can easily be shown as above. These two relations (3.5) and (3.6) show that
$\xi_{1}$ and $\xi_{2}$ are congruent respectively to $\left(k-1\right)$ and
$0$ modulo $k$. ∎

There are other cases of interest as shown in the next two Propositions

######  Proposition 3.

For $\forall n\in\mathbb{Z}^{+}$, $\exists k,\xi,\mu<k,i,j\in\mathbb{Z}^{+}$,
$k$ non-square, such that if $\xi_{i}$ are solutions of (1.1) with
$\xi_{i}\equiv\mu_{j}\left(\text{mod}\,k\right)$, and (R6) if $k$ is twice a
triangular number $k=n\left(n+1\right)=2T_{n}$, then the set of $\mu_{j}$
includes $\left\\{0,n,\left(n^{2}-1\right),\left(k-1\right)\right\\}$, with
$1\leq j\leq\upsilon$.

###### Proof.

Let $n,k,\xi,\mu<k,i,j\in\mathbb{Z}^{+}$, $k$ non-square, and $\xi_{i}$
solutions of (1.1). Let $\xi_{i}\equiv\mu_{j}\left(\text{mod\,}k\right)$ with
$1\leq j\leq\upsilon$. As the ratio $\xi_{i}\left(\xi_{i}+1\right)/k$ must be
integer, $\xi_{i}\left(\xi_{i}+1\right)\equiv 0\left(\text{mod\,}k\right)$ or
$\mu_{j}\left(\mu_{j}+1\right)\equiv
0\left(\text{mod}\,n\left(n+1\right)\right)$ which is obviously satisfied if
$\mu_{j}=n$ or $\mu_{j}=\left(n^{2}-1\right)$. ∎

Finally, this last proposition gives a general expression of the congruence
$\xi_{i}\left(\text{mod\,}k\right)$ for most cases to find the remainders
$\mu_{j}$ other than $0$ and $\left(k-1\right)$.

######  Proposition 4.

For $\forall n>1\in\mathbb{Z}^{+}$, $\exists
k,f,\xi,\nu<n<k,\mu<k,m<n,i,j\in\mathbb{Z}^{+}$, $k$ non-square, let $\xi_{i}$
be solutions of (1.1) with $\xi_{i}\equiv\mu_{j}\left(\text{mod}\,k\right)$,
let $f$ be a factor of $k$ such that $f=k/n$ with
$f\equiv\nu\left(\text{mod\,}n\right)$ and $k\equiv\nu
n\left(\text{mod}\,n^{2}\right)$, then the set of $\mu_{j}$ includes either
$\left\\{0,mf,\left(\left(n-m\right)f-1\right),\left(k-1\right)\right\\}$ or
$\left\\{0,\left(mf-1\right),\left(n-m\right)f,\left(k-1\right)\right\\}$,
where $m$ is an integer multiplier of $f$ in the congruence relation and such
that $m<n/2$ or $m<\left(n+1\right)/2$ for $n$ being even or odd respectively,
and $1\leq j\leq\upsilon$.

###### Proof.

Let $n>1,k,f,\xi,\mu<k,m<n,i,j<n<k\in\mathbb{Z}^{+}$, $k$ non-square, and
$\xi_{i}$ a solution of (1.1). Let
$\xi_{i}\equiv\mu_{j}\left(\text{mod\,}k\right)$ with $1\leq j\leq\upsilon$.
As the ratio $\xi_{i}\left(\xi_{i}+1\right)/k$ must be integer,
$\xi_{i}\left(\xi_{i}+1\right)\equiv 0\left(\text{mod\,}k\right)$ or
$\mu_{j}\left(\mu_{k}+1\right)\equiv 0\left(\text{mod\,}fn\right)$. For a
proper choice of the factor $f$ of $k$, let $\mu_{j}$ be a multiple of $f$,
$\mu_{j}=mf$, then $m\left(mf+1\right)\equiv 0\left(\text{mod\,}n\right)$. As
$f\equiv\nu\left(\text{mod\,}n\right)$, one has

(3.7) $m\left(m\nu+1\right)\equiv 0\left(\text{mod}\,n\right)$

. Let now $\left(\mu_{j}+1\right)$ be a multiple of $f$, $\mu_{j}+1=mf$, then
$m\left(mf-1\right)\equiv 0\left(\text{mod\,}n\right)$ or

(3.8) $m\left(m\nu-1\right)\equiv 0\left(\text{mod\,}n\right)$

An appropriate combination of integer parameters $m$ and $\nu$ guarantees that
(3.7) and (3.8) are satisfied. Proposition 1 yields the other remainder value
as $mf+\left(n-m\right)f-1=k-1$ and $\left(mf-1\right)+\left(n-m\right)f=k-1$.
∎

The appropriate combinations of integer parameters $m$ and $\nu$ are given in
Table 3 for $2\leq n\leq 12$. The sign $-$ in subscript corresponds to the
remainder $\left(mf-1\right)$; the sign $/$ indicates an absence of
combination.

Table 3. Combination of parameters $m$ and $\nu$ for $2\leq n\leq 12$ $m$ |  | $\nu$
---|---|---
| $\searrow$ | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11
$n$ | 2 | 1_ |  |  |  |  |  |  |  |  |  |
3 | 1_ | 1 |  |  |  |  |  |  |  |  |
4 | 1_ | / | 1 |  |  |  |  |  |  |  |
5 | 1_ | 2 | 2_ | 1 |  |  |  |  |  |  |
6 | 1_ | / | / | / | 1 |  |  |  |  |  |
7 | 1_ | 3 | 2 | 2_ | 3_ | 1 |  |  |  |  |
8 | 1_ | / | 3_ | / | 3 | / | 1 |  |  |  |
9 | 1_ | 4 | / | 2 | 2_ | / | 4_ | 1 |  |  |
10 | 1_ | / | 3 | / | 5_ | / | 3_ | / | 1 |  |
11 | 1_ | 5 | 4_ | 3_ | 2 | 2_ | 3 | 4 | 5_ | 1 |
12 | 1_ | / | / | / | 3 | / | 4_ | / | / | / | 1

One deduces from Table 3 the following simple rules:

1) $\forall n\in\mathbb{Z}^{+}$, only those values of $\nu$ that are co-prime
with $n$ must be kept, all other combinations (indicated by $/$ in Table 3)
must be discarded as they correspond to combinations with smaller values of
$n$ and $\nu$; for $n$ even, this means that all even values of $\nu$ must be
discarded. For example, $\nu=2$ and $n=4$ are not co-prime and their
combination obviously corresponds to $\nu=1$ and $n=2$.

2) For $\nu=1$ and $\nu=n-1$, all values of $m$ are $m=1$ with respectively
the remainders $\left(mf-1\right)$ and $mf$.

3) For $\forall n,i\in\mathbb{Z}^{+}$, $n$ odd, $2\leq
i\leq\left(n-1\right)/2$, and for $\nu=\left(n-\left(2i-3\right)\right)/2$ and
$\nu=\left(n+\left(2i-3\right)\right)/2$, all the values of $m$ are $m=i$.

4) For $\forall n\in\mathbb{Z}^{+}$, $n$ odd, and for $\nu=2$ and $\nu=n-2$,
the remainders are respectively $mf$ and $\left(mf-1\right)$.

5) For $\forall n,i\in\mathbb{Z}^{+}$, $n$ even, $2\leq i\leq n/2$, and for
$\nu=\left(n-\left(2i-3\right)\right)/2$ and
$\nu=\left(n+\left(2i-3\right)\right)/2$, all the values of $m$ are $m=i$.

Expressions of $\mu_{i}$ are given in Table 4 for $2\leq n\leq 12$ (with codes
E$n\nu$). For example, for $k\equiv 12\nu\left(\text{mod}\,12^{2}\right)$ and
$\nu=5$ (code E125), i.e. $k=60,204,348,...$,
$\xi_{i}\equiv\mu_{j}\left(\text{mod\,}k\right)$ with the set of remainders
$\mu_{j}$ including
$\left\\{0,mf,\left(\left(n-m\right)f-1\right),\left(k-1\right)\right\\}$ with
$m=3$ (see Table 3) and $f=k/12=5,17,29...$respectively.

Table 4. Expressions of $\mu_{j}$ for $2\leq n\leq 12$ $n$ | $\nu$ | $m$ | $k\equiv$ | $f$ | $\mu_{j}$ | Code
---|---|---|---|---|---|---
2 | 1 | 1 | $2\left(\text{mod\,}4\right)$ | $k/2$ | $0,(k/2)-1,k/2,k-1$ | E21
3 | 1 | 1 | $3\left(\text{mod\,}9\right)$ | $k/3$ | $0,\left(k/3\right)-1,2k/3,k-1$ | E31
| 2 | 1 | $6\left(\text{mod}\,9\right)$ |  | $0,k/3,\left(2k/3\right)-1,k-1$ | E32
4 | 1 | 1 | $4\left(\text{mod\,}16\right)$ | $k/4$ | $0,\left(k/4\right)-1,3k/4,k-1$ | E41
| 3 | 1 | $12\left(\text{mod\,}16\right)$ |  | $0,k/4,\left(3k/4\right)-1,k-1$ | E43
5 | 1 | 1 | $5\left(\text{mod\,}25\right)$ | $k/5$ | $0,\left(k/5\right)-1,4k/5,k-1$ | E51
| 2 | 2 | $10\left(\text{mod\,}25\right)$ |  | $0,2k/5,\left(3k/5\right)-1,k-1$ | E52
| 3 | 2 | $15\left(\text{mod\,}25\right)$ |  | $0,\left(2k/5\right)-1,3k/5,k-1$ | E53
| 4 | 1 | $20\left(\text{mod\,}25\right)$ |  | $0,k/5,\left(4k/5\right)-1,k-1$ | E54
6 | 1 | 1 | $6\left(\text{mod\,}36\right)$ | $k/6$ | $0,\left(k/6\right)-1,5k/6,k-1$ | E61
| 5 | 1 | $30\left(\text{mod\,}36\right)$ |  | $0,k/6,\left(5k/6\right)-1,k-1$ | E65
7 | 1 | 1 | $7\left(\text{mod\,}49\right)$ | $k/7$ | $0,\left(k/7\right)-1,6k/7,k-1$ | E71
| 2 | 2 | $14\left(\text{mod\,}49\right)$ |  | $0,3k/7,\left(4k/7\right)-1,k-1$ | E72
| 3 | 3 | $21\left(\text{mod\,}49\right)$ |  | $0,2k/7,\left(5k/7\right)-1,k-1$ | E73
| 4 | 3 | $28\left(\text{mod\,}49\right)$ |  | $0,\left(2k/7\right)-1,5k/7,k-1$ | E74
| 5 | 2 | $35\left(\text{mod\,}49\right)$ |  | $0,\left(3k/7\right)-1,4k/7,k-1$ | E75
| 6 | 1 | $42\left(\text{mod\,}49\right)$ |  | $0,k/7,\left(6k/7\right)-1,k-1$ | E76
8 | 1 | 1 | $8\left(\text{mod\,}64\right)$ | $k/8$ | $0,\left(k/8\right)-1,7k/8,k-1$ | E81
| 3 | 3 | $24\left(\text{mod\,}64\right)$ |  | $0,\left(3k/8\right)-1,5k/8,k-1$ | E83
| 5 | 3 | $40\left(\text{mod\,}64\right)$ |  | $0,3k/8,\left(5k/8\right)-1,k-1$ | E85
| 7 | 1 | $56\left(\text{mod\,}64\right)$ |  | $0,k/8,\left(7k/8\right)-1,k-1$ | E87
9 | 1 | 1 | $9\left(\text{mod\,}81\right)$ | $k/9$ | $0,(k/9)-1,8k/9,k-1$ | E91
| 2 | 4 | $18\left(\text{mod\,}81\right)$ |  | $0,4k/9,(5k/9)-1,k-1$ | E92
| 4 | 2 | $36\left(\text{mod\,}81\right)$ |  | $0,2k/9,(7k/9)-1,k-1$ | E94
| 5 | 2 | $45\left(\text{mod\,}81\right)$ |  | $0,(2k/9)-1,7k/9,k-1$ | E95
| 7 | 4 | $63\left(\text{mod\,}81\right)$ |  | $0,(4k/9)-1,5k/9,k-1$ | E97
| 8 | 1 | $72\left(\text{mod\,}81\right)$ |  | $0,k/9,(8k/9)-1,k-1$ | E98
10 | 1 | 1 | $10\left(\text{mod\,}100\right)$ | $k/10$ | $0,(k/10)-1,9k/10,k-1$ | E101
| 3 | 3 | $30\left(\text{mod\,}100\right)$ |  | $0,3k/10,(7k/10)-1,k-1$ | E103
| 7 | 3 | $70\left(\text{mod\,}100\right)$ |  | $0,(3k/10)-1,7k/10,k-1$ | E107
| 9 | 1 | $90\left(\text{mod}\,100\right)$ |  | $0,k/10,(9k/10)-1,k-1$ | E109
11 | 1 | 1 | $11\left(\text{mod\,}121\right)$ | $k/11$ | $0,(k/11)-1,10k/11,k-1$ | E111
| 2 | 5 | $22\left(\text{mod\,}121\right)$ |  | $0,5k/11,(6k/11)-1,k-1$ | E112
| 3 | 4 | $33\left(\text{mod\,}121\right)$ |  | $0,(4k/11)-1,7k/11,k-1$ | E113
| 4 | 3 | $44\left(\text{mod\,}121\right)$ |  | $0,(3k/11)-1,8k/11,k-1$ | E114
| 5 | 2 | $55\left(\text{mod\,}121\right)$ |  | $0,2k/11,(9k/11)-1,k-1$ | E115
| 6 | 2 | $66\left(\text{mod\,}121\right)$ |  | $0,(2k/11)-1,9k/11,k-1$ | E116
| 7 | 3 | $77\left(\text{mod\,}121\right)$ |  | $0,3k/11,(8k/11)-1,k-1$ | E117
| 8 | 4 | $88\left(\text{mod\,}121\right)$ |  | $0,4k/11,(7k/11)-1,k-1$ | E118
| 9 | 5 | $99\left(\text{mod\,}121\right)$ |  | $0,(5k/11)-1,6k/11,k-1$ | E119
| 10 | 1 | $110\left(\text{mod\,}121\right)$ |  | $0,k/11,(10k/11)-1,k-1$ | E1110
12 | 1 | 1 | $12\left(\text{mod\,}144\right)$ | $k/12$ | $0,(k/12)-1,11k/12,k-1$ | E121
| 5 | 3 | $60\left(\text{mod\,}144\right)$ |  | $0,3k/12,(9k/12)-1,k-1$ | E125
| 7 | 4 | $84\left(\text{mod\,}144\right)$ |  | $0,(4k/12)-1,8k/12,k-1$ | E127
| 11 | 1 | $132\left(\text{mod\,}144\right)$ |  | $0,k/12,(11k/12)-1,k-1$ | E1211

Values of the remainders $\mu_{j}$ are given in Table 5 for $2\leq k\leq 120$,
with rule (R) and expression (E) codes as references. R and E codes separated
by comas imply that all references apply simultaneously to the case; E codes
separated by + mean that all expressions are applicable to the case; some
expression references are sometimes missing. One observes that in two cases
(for $k=74$ and 104), expressions could not be found (indicated by question
marks).

Table 5. Values of $\mu_{j}$ for $2\leq k\leq 120$ $k$ | $\mu_{j}$ | References | $k$ | $\mu_{j}$ | References
---|---|---|---|---|---
2 | 0,1 | R1,R6,E21 | 63 | 0,27,35,62 | E72,E97
3 | 0,2 | R1,E31 | 65 | 0,64 | R3
5 | 0,4 | R1,R3,E51 | 66 | 0,11,21,32,33,44,54,65 | E21+E31+E65+E116
6 | 0,2,3,5 | R6,E21,E32,E61 | 67 | 0,66 | R1
7 | 0,6 | R1,R5,E71 | 68 | 0,16,51,67 | E41
8 | 0,7 | R2,R4,E81 | 69 | 0,23,45,68 | E32
10 | 0,4,5,9 | E21,E52,E101 | 70 | 0,14,20,34,35,49,55,69 | E21+E54+E73+E107
11 | 0,10 | R1,E111 | 71 | 0,70 | R1
12 | 0,3,8,11 | R6,E31,E43,E121 | 72 | 0,8,63,71 | R6,E81,E98
13 | 0,12 | R1 | 73 | 0,72 | R1
14 | 0,6,7,13 | E21,E72 | 74 | 0,73 | ?
15 | 0,5,9,14 | E32,E53 | 75 | 0,24,50,74 | E31
17 | 0,16 | R1,R3 | 76 | 0,19,56,75 | E43
18 | 0,8,9,17 | E21,E92 | 77 | 0,21,55,76 | E74,E117
19 | 0,18 | R1 | 78 | 0,12,26,38,39,51,65,77 | E21+E32+E61
20 | 0,4,15,19 | R6,E41,E54 | 79 | 0,78 | R1,R5
21 | 0,6,14,20 | E31,E73 | 80 | 0,79 | R4
22 | 0,10,11,21 | E21,E112 | 82 | 0,40,41,81 | E21
23 | 0,22 | R1,R5 | 83 | 0,82 | R1
24 | 0,23 | R4 | 84 | 0,27,56,83 | E31,E127
26 | 0,12,13,25 | E21 | 85 | 0,34,50,84 | E52
27 | 0,26 | R2 | 86 | 0,42,43,85 | E21
28 | 0,7,20,27 | E43,E74 | 87 | 0,29,57,86 | E32
29 | 0,28 | R1 | 88 | 0,32,55,87 | E83,E118
30 | 0,5,24,29 | R6,E51,E65 | 89 | 0,88 | R1
31 | 0,30 | R1 | 90 | 0,9,80,89 | R6,E91,E109
32 | 0,31 | R2 | 91 | 0,13,77,90 | E75
33 | 0,11,21,32 | E32,E113 | 92 | 0,23,68,91 | E43
34 | 0,16,17,33 | E21 | 93 | 0,30,62,92 | E31
35 | 0,14,20,34 | E52,E75 | 94 | 0,46,47,93 | E21
37 | 0,36 | R1,R3 | 95 | 0,19,75,94 | E54
38 | 0,18,19,37 | E21 | 96 | 0,32,63,95 | E32
39 | 0,12,26,38 | E31 | 97 | 0,96 | R1
40 | 0,15,24,39 | E53,E85 | 98 | 0,48,49,97 | E21
41 | 0,40 | R1 | 99 | 0,44,54,98 | E92,E119
42 | 0,6,35,41 | R6,E61,E76 | 101 | 0,100 | R1,R3
43 | 0,42 | R1 | 102 | 0,50,51,102 | E21
44 | 0,11,32,43 | E43,E114 | 103 | 0,102 | R1
45 | 0,9,35,44 | E54,E95 | 104 | 0,103 | ?
46 | 0,22,23,245 | E21 | 105 | 0,14,20,35,69,84,90,104 | E32+E51+E71
47 | 0,46 | R1,R5 | 106 | 0,52,53,105 | E21
48 | 0,47 | R4 | 107 | 0,106 | R1
50 | 0,24,25,49 | E21 | 108 | 0,27,80,107 | E43
51 | 0,17,33,50 | E32 | 109 | 0,108 | R1
52 | 0,12,39,51 | E41 | 110 | 0,10,99,109 | R6,E101,E1110
53 | 0,52 | R1 | 111 | 0,36,74,110 | E31
54 | 0,26,27,53 | E21 | 112 | 0,48,63,111 | E72
55 | 0,10,44,54 | E51,E115 | 113 | 0,112 | R1
56 | 0,7,48,55 | R6,E71,E87 | 114 | 0,56,57,113 | E21
57 | 0,18,38,56 | E31 | 115 | 0,45,69,114 | E53
58 | 0,28,29,57 | E21 | 116 | 0,28,87,115 | E41
59 | 0,58 | R1 | 117 | 0,26,90,116 | E94
60 | 0,15,44,59 | E43,E125 | 118 | 0,58,59,117 | E21
61 | 0,60 | R1 | 119 | 0,118 | R1,R5
62 | 0,30,31,61 | E21 | 120 | 0,15,104,119 | E87

This Table 5 gives correctly the values of the remainder pairs in most of the
cases. There are although some exceptions and some values missing.

Among the exceptions to the values given in Table 5, for $n=2$, remainders
values for $k=30,42,74,90,110,\ldots$ are different from the theoretical ones
in Table 4. Furthermore, for $k=66,70,78,105,...$, additional remainders
exist. Expressions are missing for $k=74$ (E21) and 104 (E85). Finally, one
observes also that for 16 cases, some Rules or Expressions supersede some
other Expressions (indicated by Ra > Exy or Exy > Ezt), as reported in Table
6. For example, Rule 6 supersedes Expression 21 (R6 > E21) for
$k=30,42,90,110$, i.e., $k=2T_{5},2T_{6},2T_{9},2T_{10},...$ and more
generally for all $k=2T_{i}$ for $i\equiv 1,2\left(\text{mod}4\right)$.

Table 6. Rules and Expressions superseding other Rules and Expressions $k$ |
---|---
24 | R4 > E32; R4 > E83
30 | R6 > E21; R6 > E31; R6 > E103; E51 > E103; E65 > E103
42 | R6 > E21; R6 > E32
48 | R4 > E31
56 | R6 > E43
60 | E43 > E32; E43 > E52
65 | R3 > E53
72 | R6 > E43
80 | R4 > E51
84 | E31 > E41; E31 > E75
90 | R6 > E21; R6 > E53
102 | E21 > E31; E21 > E65
110 | R6 > E21; R6 > E52
114 | E21 > E32; E21 > E61
119 | R1 > E73; R5 > E73
120 | E87 > R4; E87 > E31; E87 > E54

Note that 11 of these 16 values of $k$ are multiple of 6, the others are 2 mod
6 and 5 mod 6 for, respectively three and two cases. One notices as well, that
generally, Ra and Exy supersede Ezt with $x<z$ and $t<y$, except for $k=60$
and $120$.

##  4\. Conclusions

We have shown that, for indices $\xi$ of triangular numbers multiples of other
triangular numbers, the remainders in the congruence relations of $\xi$ modulo
$k$ come always in pairs whose sum always equal $\left(k-1\right)$, always
include 0 and $\left(k-1\right)$, and only 0 and $\left(k-1\right)$ if $k$ is
prime, or an odd power of a prime, or an even square plus one or an odd square
minus one or minus two. If the multiplier $k$ is twice a triangular number of
$n,$the set of remainders includes also $n$ and $\left(n^{2}-1\right)$ and if
$k$ has integer factors, the set of remainders include multiple of a factor
following certain rules. Finally, algebraic expressions are found for
remainders in function of $k$ and its factors. Several exceptions are noticed
as well and it appears that there are superseding rules between the various
rules and expressions.

This approach allows to eliminate in numerical searches those
$\left(k-\upsilon\right)$ values of $\xi_{i}$ that are known not to provide
solutions of (1.1), where $\upsilon$ is the even number of remainders. The
gain is typically in the order of $k/\upsilon$, with $\upsilon\ll k$ for large
values of $k$.

## References

  * [1] Andrews, G.E. Number Theory, Dover, New York, 1971.
  * [2] Weisstein, E, W. "Triangular Number." From MathWorld–A Wolfram Web Resource. http://mathworld.wolfram.com/TriangularNumber.html. Last accessed 14 February 2021.
  * [3] Cunningham A., Mathematical Questions and Solutions in Continuation of the Mathematical Columns of "the Educational Times"., Volume 75, F. Hodgson, 1901, 87-88.
  * [4] de Joncourt E., The nature and notable use of the most simple trigonal numbers. The Hague: Husson, 1762.
  * [5] Roegel D., A reconstruction of Joncourt’s table of triangular numbers (1762). LOCOMAT project, https://locomat.loria.fr/joncourt1762/joncourt1762doc.pdf, 2013\. Last accessed 14 February 2021.
  * [6] Dickson L.E., History of the Theory of Numbers, Vol. II: Diophantine Analysis, Dover Publ., New York, 2005, p. 587.
  * [7] Chahal J.S. and D’Souza H., Some remarks on Triangular Numbers, in Number Theory with an Emphasis on the Markoff Spectrum, A.D. Pollington and W. Mran, eds., Marcel Dekker Inc., New York, 1993.
  * [8] Breiteig T., Quotients of triangular numbers. The Mathematical Gazette,Vol. 99, 545, 243-255, July 2015. DOI: 10.1017/mag.2015.33
  * [9] V. Pletser, Recurrent Relations for Multiple of Triangular Numbers being Triangular Numbers, ArXiv 2101.00998, 2021, http://arxiv.org/abs/2101.00998, last accessed 14 February 2021.
  * [10] N. J. A. Sloane, editor, The On-Line Encyclopedia of Integer Sequences, published electronically at https://oeis.org, Last accessed 14 February 2021.
  * [11] V. Pletser, Searching for Multiple of Triangular Numbers being Triangular Numbers, Preprint, to be submitted, February 2021\.


# WYTIWYR: A User Intent-Aware Framework with Multi-modal Inputs for
Visualization Retrieval

Shishi Xiao10009-0008-0262-5289, Yihan Hou10000-0002-1459-8766, Cheng
Jin20000-0002-3522-3592, and Wei Zeng1,20000-0002-5600-8824
1The Hong Kong University of Science and Technology (Guangzhou), Guangzhou,
China 2 The Hong Kong University of Science and Technology, Hong Kong SAR,
China Wei Zeng is the corresponding author. Email<EMAIL_ADDRESS>
###### Abstract

Retrieving charts from a large corpus is a fundamental task that can benefit
numerous applications such as visualization recommendations. The retrieved
results are expected to conform to both explicit visual attributes (_e.g._ ,
chart type, colormap) and implicit user intents (_e.g._ , design style,
context information) that vary upon application scenarios. However, existing
example-based chart retrieval methods are built upon non-decoupled and low-
level visual features that are hard to interpret, while definition-based ones
are constrained to pre-defined attributes that are hard to extend. In this
work, we propose a new framework, namely _WYTIWYR (What-You-Think-Is-What-You-
Retrieve)_ , that integrates user intents into the chart retrieval process.
The framework consists of two stages: first, the _Annotation_ stage
disentangles the visual attributes within the query chart; and second, the
_Retrieval_ stage embeds the user’s intent with customized text prompt as well
as bitmap query chart, to recall targeted retrieval result. We develop a
prototype _WYTIWYR_ system leveraging a contrastive language-image pre-
training (CLIP) model to achieve zero-shot classification as well as multi-
modal input encoding, and test the prototype on a large corpus with charts
crawled from the Internet. Quantitative experiments, case studies, and
qualitative interviews are conducted. The results demonstrate the usability
and effectiveness of our proposed framework.

<ccs2012> <concept> <concept_id>10003120.10003145</concept_id>
<concept_desc>Human-centered computing Visualization</concept_desc>
<concept_significance>500</concept_significance> </concept> <concept>
<concept_id>10002951.10003317.10003325.10003327</concept_id>
<concept_desc>Information systems Query intent</concept_desc>
<concept_significance>500</concept_significance> </concept> <concept>
<concept_id>10010147.10010178</concept_id> <concept_desc>Computing
methodologies Artificial intelligence</concept_desc>
<concept_significance>500</concept_significance> </concept> </ccs2012>

[500]Human-centered computing Visualization [500]Information systems Query
intent [500]Computing methodologies Artificial intelligence

††volume: 42††issue: 3

##  1 Introduction

Data visualization can empower users’ understanding of data and facilitate
communication. Therefore, enormous charts are flourishing on the Internet.
Designing an appropriate chart is a time-consuming and labor-intensive process
that needs to consider various visual attributes (_e.g._ , chart type, color)
[Shn96], as well as the design style (_e.g._ , context information, 2D/3D
effect) [MTW∗12]. Therefore, designing based on existing examples, rather than
starting with sketches, is a more preferred approach [BFW21, PSP21]. Chart
retrieval, as an approach to return the ranked examples with respect to the
corresponding query, has gained much interest in both industry and academia
[LWW∗22, HA19, SHL∗16].

Figure 1: Comparison of pipelines of existing and our proposed chart retrieval
frameworks: example-based retrieval (left-top), definition-based retrieval
(left-bottom), and our proposed intent-aware retrieval (right).

However, existing works mainly focus on improving the similarity between query
charts and retrieval results while neglecting implicit user intent. As
illustrated in Figure 1, existing chart retrieval frameworks might be divided
into two categories: example-based (Figure 1 (left-top)) and definition-based
(Figure 1 (left-bottom)) approaches. Example-based methods (_e.g._ , [SSK06,
DW14, BBK∗18]) characterize the relevance criteria by the visual similarity of
charts, whilst definition-based methods (_e.g._ , [CCA15, HA19]) measure
similarity based on attributes extracted from the input charts. However, these
methods may suffer from both input and target shifts problems. First, many
existing works take only specific formats (_e.g._ , scalable vector graphics
(SVG) [HA19, LWW∗22]) of visualization as inputs, which are often synthesized
with monotonous data distributions and limited chart types that incurs
overfitting problem of models embedded in the framework. In contrast, most
online charts are generally in bitmap formats, and they are highly diverse in
terms of chart types and visual styles[BDM∗18]. In addition, online charts
contain noisy and redundant information, putting higher demands on
generalizability and robustness of the retrieval framework. Second, example-
based methods would indiscriminately recall pixel-wise visual similarity in
the input chart, while definition-based methods consider only a small number
of pre-defined and fixed chart attributes. Such rigid similarity measurements
would return unwanted target charts. Furthermore, it may tend to limited the
retrieval to a specific chart attribute, a specific combination of chart
attributes, or even chart attributes that are not necessarily enclosed by the
query chart.

To mitigate such shifts, we propose a novel chart retrieval framework named
WYTIWYR. It explicitly extracts visual attributes for the query chart, and
offers a set of target charts through a customized retrieval procedure that
considers flexible combinations of the visual attributes and users’ intents.
As Figure 1 (right) illustrates, our method tears down the query chart
flexibly with customizable attributes and actively injects user intent as
auxiliary inputs to seek better chart retrieval outputs.

Our intent-aware framework consists of two essential stages, the first stage
is Annotation, which highlights explicitly disentangling the visual attributes
for query charts. The disentangled attributes would enable the flexibly for
users to combine the attributes representing their intent. Because these
attributes differ amongst chart visualizations, we performed preliminary
research with 18 visualization categories to establish the branching of each
attribute and what users would anticipate when retrieving charts. Based on the
study, we utilize deep neural networks to train several independent attribute
classifiers tailored to four primary visual attributes, namely {Type, Trend,
Color, Layout}. To enable the resilience to user intent, we leverage the
state-of-the-art contrastive language-image pretraining (CLIP) [RKH∗21] model
famous for zero-shot classification. The model allows users for their own
creation of the attribute classifier to identify the extended attributes.
These attributes are excluded from our preliminary set of classifiers but are
still within the query process.

The second stage is Retrieval, which highlights the role of human-in-the-loop
through intent attributes as prompts. The intent attributes would condition
intent-aware filter, narrow the search scope from a huge collection of charts,
and finally generate multiple candidate charts. The multi-modal encoder fuses
context information from both the query chart and user intent prompt into a
joint representation. After that, similarity modeling is conducted between
such joint representation and candidate charts. Ultimately, we rank the
similarity scores in decreasing order and output the top-$K$ results as the
target charts. In summary, our main contributions are summarized as follows:

  * •

Intent-aware Retrieval. We propose a novel framework integrating user intent
into chart retrieval process. The Annotation stage disentangles attributes and
enables a flexible combination of attributes. The Retrieval stage digests both
query chart and user intent as multi-modal inputs to get target retrieval
results.

  * •

Prototype Construction. We implement a prototype system combining CLIP model
with visual interface to support intent-aware retrieval. Dataset, code,
pretrained model are released at https://github.com/SerendipitysX/WYTIWYR.

  * •

Extensive Evaluation. We conduct extensive quantitative experiments, case
studies as well as expert interviews to validate the effectiveness of our
approach.

##  2 Related Works

Chart Attributes. Visualization is the process of mapping data to images. The
data can be encoded in different ways, yielding various types of chart
attributes that can be categorized in several levels. Low-level visual stimuli
of charts include color, position, and shape [RTOT06], while high-level
taxonomy includes consideration of the object of study, data, design model,
and user model [TM04]. The importance of choosing appropriate chart attributes
has been highlighted. Recent studies (_e.g._ , [MMA18, KRS∗21, LCF∗15,
SLC∗23]) stress the effects of chart attributes on users’ comprehension and
cognitive loads. Specifically, Li et al.[LCF∗15] point out that two charts of
the same data but with different layouts can cause perceptional imbalance,
even though they are informatively equivalent. Despite the importance, the
design space of chart attributes is too large for designers to choose from,
resulting in various types of charts in designers’ own style[SLC∗23]. Most of
these works follow a coarse-to-fine strategy to first identify chart types and
then extract visual marks and channels [SKC∗11, JKS∗17, PMH17, YZF∗22].

Existing methods often mix visual attributes as global style for similarity
estimation [MTW∗18, SHL∗16, JKS∗17], in which not all attributes are of
interest to users [LWW∗22]. To fill the gap, the Annotation stage of the
proposed WYTIWYR framework allows users to disentangle attributes in a given
chart.

Chart Retrieval. The core for effective retrieval is to delineate the
similarity between the query input and retrieval candidates in the database.
Based on the object of similarity measurement in the retrieval process,
existing chart retrieval strategies can be mainly categorized into two
classes: definition- and example-based approaches. Definition-based approaches
(_e.g._ , [CCA15, SHL∗16, HA19, ZDCC21, ZFF22, LWW∗22]) characterize the
criteria of similarity by explicit chart attributes, making it preferable for
well-configured chart formats, including SVG-style and Vega-Lite grammars. For
instance, Hoque and Agrawala[HA19] developed a search engine for D3
visualizations by using a JSON-like configuration that dictates query visual
encoding. Moreover, some studies (_e.g._ , [HMSA08, SGP∗18]) further use user
interaction records and visualization states represented in a provenance graph
to match the previous exploration states. Recent example-based methods regard
visualization charts as images and leverage deep learning models [MTW∗18,
ZFC∗23, SHL∗16, JKS∗17, YHZ23] to automatically extract visual features via an
end-to-end manner. A set of implicit low-level features are leveraged to
estimate the global similarity. However, definition-based approaches only
consider partially predetermined visual attributes, while example-based
approaches tend to match the query with all indiscriminate attributes.

Our proposed WYTIWYR framework combines the advantages of example-based
approaches in terms of their capability of capturing implicit attributes and
also definition-based approaches that leverage explicit attributes with high
interpretability. Moreover, the Retrieval stage enables a flexible combination
of these attributes with respect to user intent.

Vision-Language Pretraining. Large-scale pretrained models have shown
promising performance in both natural language processing and vision tasks
these years, such as BERT [DCLT18], RoBerTa [LOG∗19], and GPT series [BMR∗20,
RNS∗18, RWC∗19]. Following the paradigm of pretrain and finetune, many
downstream tasks [GKSS∗19, XBK∗15, LGR∗20] transfer the knowledge from the
pretrained model without training a new model from scratch by utilizing it.
Prompt, a text sequence in the form of natural language, links the pretrained
model and downstream tasks as a bridge. Strobelt _et al._ [SWS∗22] develops a
visual interface to provide a promising prompt evaluation. Combined with a
high-capacity text encoder and visual encoder, contrastive language-image
pretraining (CLIP) [RKH∗21] learns heterogeneous multi-modality
representations from 400 million image-text pairs by resorting to semantic
supervision from the embedding space of CLIP. Several studies [PWS∗21, GPM∗21,
GLKC22] extend its applicability in a zero-shot manner, which means the model
can predict samples whose class is not observed during training.

In this work, we take advantage of vision-language pretraining models, to
encode user intents as a prompt to collaborate with the decisive process of
retrieval. The CLIP-driven model is leveraged to align user intent with
corresponding visual attributes in both Annotation and Retrieval stages. In
this way, we can offer highly customizable attributes for chart annotation and
multi-modal representation extraction for chart retrieval.

##  3 WYTIWYR Framework

To build the WYTIWYR framework, we first formulate chart attributes based on a
preliminary study (Sec. 3.1), then present an overview of the WYTIWYR
framework (Sec. 3.2).

Figure 2: Results of our preliminary study. _Type, color, trend,_ and _layout_
are voted by most participants and identified as _primary_ attributes, whilst
the others are categorized as _extended_ attributes.

###  3.1 Attribute Formulation

There are two categories of chart attributes for chart retrieval: 1) _primary
attributes_ that are generally considered for chart retrieval, and 2)
_extended attributes_ that meet the specific needs of different users. To
better understand user intents, we conducted a comprehensive study to
categorize chart attributes.

Study design. Our preliminary study was conducted online and involved 40
participants whose ages ranged from 20 to 52 (mean = 24.3). Before the study,
we first introduced the background and purpose of the experiment to the
participants. Then we tested their knowledge on the chart through a set of
verification questions. Only the participants who could identify typical
charts, such as bar and pie charts, were allowed to participate in the study.
Fortunately, all participants were familiar with charts since their popularity
on social media. Next, the participants were shown multiple real-world charts
and asked to indicate what attributes were of interest. To eliminate any
potential subjective bias during the chart selection process for testing, a
random sample of five images (three in the Beagle dataset[BDM∗18], two in
other real-world examples) was taken for 18 types of chart from the collected
data. The testing charts were carefully chosen according to chart type
taxonomy as in the study by Borkin et al.[BVB∗13]. The detail of sampled
testing chart can be seen at supplementary material. Their answers were
recorded in text and summarized in different perspectives after the
experiments. The perspectives were cross-validated by two of our co-authors to
ensure correctness.

Figure 3: Illustration of attribute options of primary attributes for several
chart types. Some options may not be available for certain chart types, such
as _Layout_ for the point chart.  Figure 4: Overall pipeline of attributes-
aware retrieval. The pipeline consists of two stages: Annotation for
extracting the attributes and Retrieval for modeling the similarity between
query charts and charts in the database.

Result. As shown in Figure 2, we identify a total of eight perspectives of
attributes are of interest from the feedback, which are also consistent with
previous studies [BVB∗13, LWW∗22]. Among them, _Type, Color, Trend_ , and
_Layout_ are the four attributes mostly frequently mentioned by the
participants and can be applied to most types of charts. Although the _Form_
attribute was also frequently mentioned by participants, it is primarily
associated with line charts and less frequently mentioned in the other chart
types. Therefore, to ensure generalization and expressivity, we decided not to
combine _Form_ with the other four attributes. We dub primary attributes as
${{\color[rgb]{0,0.2,0.572}\definecolor[named]{pgfstrokecolor}{rgb}{0,0.2,0.572}{\mathcal{T}_{p}}}}:=\left\\{Type,\text{
}Color,\text{ }Trend,\text{ }Layout\right\\}$. In addition, the participants
also brought up some other chart attributes, including _form, background,
context_ , and _2D/3D_ , which we dubbed as extended attributes
$\mathcal{T}_{e}$ that includes the other attributes. For these chart
attributes, the variation between different users is large, making it
difficult to be comprehensively enumerated.

Moreover, the study also revealed a set of design options for each chart
attribute. Specifically, we choose 10 primary categories and 18 subcategories
of chart types from the visualization taxonomy[BVB∗13]. Figure 3 illustrates
the options in the attribute type of {_color_ , _trend_ , _layout_} for
several chart types including bar, line, and point charts. Notice that some
chart types may not have all options mentioned above. For example, _layout_
for point charts are not available, and maps only have _color_ options.

###  3.2 Overall Pipeline

The workflow of our proposed WYTIWYR framework is depicted in Figure 4. For
the Annotation stage, we employ several classifiers based on robust neural
network architecture to disentangle
$\\{{\color[rgb]{0,0.2,0.572}\definecolor[named]{pgfstrokecolor}{rgb}{0,0.2,0.572}{\mathcal{T}_{p}}},{\color[rgb]{0.33,0.8,0.725}\definecolor[named]{pgfstrokecolor}{rgb}{0.33,0.8,0.725}{\mathcal{T}_{e}}}\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\\}\in\mathcal{T}}$
embedded in the query chart $\mathcal{Q}$. After that, users can select the
disentangled attributes $\mathcal{I}_{A}$ as their intents based on their
needs. Also, $\mathcal{I}_{A}$ would condition the components of the filter in
the retrieval stage. For the Retrieval stage, we take a dual-path similarity
modeling between the query feature and the candidate info with the guidance of
intent attributes to yield the return charts. For the path of the query
feature, bitmap query chart $\mathcal{Q}$ with user intent text prompt
$\mathcal{I}_{P}$ is fed into a multi-modal encoder to obtain the joint
representation. For the candidate info path, the feature is produced by charts
in the database filtered by the intent-aware filter built by
$\mathcal{I}_{A}$. Throughout the overall pipeline, user intent can be
injected and guide the chart retrieval process by the following three ways:

  * •

Classifier Customization. Users can tailor the usage of classifiers depending
on their needs in order to determine the presence of an attribute in the query
chart.

  * •

Disentangled Attributes Selection. The attributes adopted in the Retrieval
stage can be selected and combined by the users.

  * •

User Prompt Tuning. Users can add specific implicit intent as the text prompt
to guide the retrieval process, which is independent of the query chart.

##  4 WYTIWYR Prototype System

###  4.1 Stage1: Annotation

As shown in Figure 5, the task of the Annotation stage is to disentangle and
generate the visual attributes $\mathcal{T}$ adhered to the given query chart
$\mathcal{Q}$. For the primary attributes
${\color[rgb]{0,0.2,0.572}\definecolor[named]{pgfstrokecolor}{rgb}{0,0.2,0.572}{\mathcal{T}_{p}}}$,
we adopt four classifiers for separate extraction in a supervised learning
manner. For the extended attributes
${\color[rgb]{0.33,0.8,0.725}\definecolor[named]{pgfstrokecolor}{rgb}{0.33,0.8,0.725}{\mathcal{T}_{e}}}$,
as they are absent from our training, we conduct the extraction process in the
fashion of zero-shot learning.

####  4.1.1 Annotation for Primary Attributes

Redundant Information Removal. Several practical scenario-based raw data are
shown in Figure 6 (a), including fancy decoration, text information,
background, and legends. This work focuses on the intrinsic attributes of
$\mathcal{Q}$ instead of this redundant information. Hence we remove the
redundant information using ISNet [QDH∗22], a segmentation network with
pretrained weights.

Nevertheless, there exists context beyond the segmentation capability of
ISNet, such as the emojis in the left-most example in Figure 6 (a) & (b). The
redundant emojis in red and yellow colors would degrade the performance of
colormap classification. Instead of being categorized as categorical colormap,
the chart would be mistakenly recognized to diverging colormap. To mitigate
such issue, we ignore the colors that share less than 10% pixels of non-
transparent regions of the image after segmentation.

Figure 5: The Annotation stage of our pipeline. To disentangle various
attributes in the query chart, four classifiers are built for primary
attributes. In addition, one CLIP-based classifier is employed to optionally
annotate the extended attribute via user customization. Figure 6: Redundant
information removal and main color extraction. (a) Raw images. (b) The chart
after removing redundant information. (c) The extracted color after filtering
redundant colors.

Unified Classifiers. For annotation of the primary attributes, we employ
ResNet50[HZRS16] architecture as the backbone for all four attribute
classifiers. Thanks to the residual connection among the convolution layers,
the network is both lightweight and performance-guaranteed. During the
training process, the network will learn the complex features under the
supervision of given attributes. In the reference process, the trained
classifier would identify the corresponding attribute in the query chart, as
in Figure 5.

The only difference in the four classifiers is the number of output channels
in the last fully connection layer. For instance, the number of $Type$
classifiers is set to $18$ since there are $18$ types of charts under
consideration. Similarly, based on the specific task, we set this number to
$3$ for the $Layout$, $Trend$, and $Color$ classifiers with the respect to
their attribute options; see Figure 3 for detail. Note that $\mathcal{Q}$ may
not have the coverage of all attributes included in the classifiers. For
instance,
${\color[rgb]{0,0.2,0.572}\definecolor[named]{pgfstrokecolor}{rgb}{0,0.2,0.572}{\mathcal{T}_{p}}}$
of a heatmap chart is $\left\\{Type,\text{ }Color\right\\}$ while
$\left\\{Trend,\text{ }Layout\right\\}$ attributes are absent.

Loss Function. There are two challenges to overcome when formulating the loss
function: 1) imbalanced samples in each class of the dataset (see Table 1),
and 2) the existence of noisy samples that are hard to classify. In order to
improve the accuracy for
${\color[rgb]{0,0.2,0.572}\definecolor[named]{pgfstrokecolor}{rgb}{0,0.2,0.572}{\mathcal{T}_{p}}}$
annotation, we introduce Focal Loss [LGG∗17], which modifies the standard
cross entropy loss to overcome the above-mentioned challenges. The Focal Loss
is defined as:

$\mathcal{L_{\mathrm{FL}}}(p_{t})=-\alpha_{t}(1-p_{t})^{\gamma}\operatorname{log}(p_{t}),$
(1)

where $p_{t}\in[0,1]$ represents the estimated probability of class $t$,
$\alpha_{t}$ represents the scaling factor, and $\gamma$ represents the
modulating factor. Among them, $\alpha_{t}$ is set by inverse class frequency,
thus learning parameters tend to contribute to classes with fewer samples, and
$\gamma$ assists in up-weight the loss assigned to poor-classified examples,
avoiding the possibility that the training process is dominated by the amount
of well-classified samples.

####  4.1.2 Annotation for Extended Attributes

Since the primary attributes only consider the general needs in chart
retrieval, we offer an optional classifier manipulated by the users to
distinguish $\mathcal{T}_{e}$ in this query chart, as shown in Figure 5. The
input of $\mathcal{T}_{e}$ classifier requires the user to provide several
labels of the attribute apart from $\mathcal{Q}$ (_e.g._ , style). We denote
these labels as $\\{{t}_{1},{t}_{2},\ldots,{t}_{m}\\}\in{T}$. As these labels
are out of our dataset, a task of zero-shot classification is naturally
formed. In the following, we will briefly introduce the mechanism of the CLIP
model at a general level with a toy example as in Figure 5, where the user
sets the text labels as [“3D style”, “Flat style”, “Sketch style”].

The CLIP model aligns $T$ embeddings with $\mathcal{Q}$ embeddings in a multi-
modality embedding space in a contrastive manner. Specifically, in the
embedding space of the example in Figure 5, the distance of $\mathcal{Q}$ and
“3D style” would be less than the other two irrelevant labels. As such, the
query results tend to be “3D style” bar charts. The CLIP model balances well
between the performance and computational cost in the prototype system.
Nevertheless, the model in our framework can be feasibly replaced with other
advanced language-image models, _e.g._ ,[LLXH22, LSG∗21, ZZF∗22].

###  4.2 Stage2: Retrieval

As the Annotation stage completes the disentanglement of the attributes
$\mathcal{T}$ in the query chart, users in the Retrieval stage can select the
intent attributes $\mathcal{I}_{A}$ based on their intents. Depicted in Figure
7, there are two branches with different modules for our Retrieval stage,
namely multi-modal encoder and intent-aware filter. The intent-aware filter
rules out charts whose attributes are beyond the range of $\mathcal{I}_{A}$.
The multi-modal encoder integrates $\mathcal{Q}$ and $\mathcal{I}_{P}$ as a
joint representation. Then, similarity modeling is conducted between results
of these two branches, yielding the final retrieval results based on the
modeling score.

Figure 7: The Retrieval stage consists of two branches. The intent-aware
filter branch (bottom) filters out charts based on intent attributes selected
by the user, generating multiple chart candidates. The multi-modal encoder
branch (top) produces the query feature based on the multi-modal inputs.
Similarity modeling is conducted between results of these two branches.

####  4.2.1 Multi-Modal Encoder

This branch would form a CLIP-generated query feature with inputs
$\mathcal{Q}\text{, }\mathcal{I}_{P}$. As introduced in Sec. 4.1.2, the CLIP
embedding space has heterogeneous aligned text and visual features from the
multi-modal input. Hence, $\mathcal{Q}$ and $\mathcal{I}_{P}$ are encoded into
features by their respective CLIP encoders, and fused as a joint multi-modal
feature $f_{\mathcal{M}}$ that can be denoted as:

$f_{\mathcal{M}}=\frac{F_{\theta}(\mathcal{Q})+G_{\phi}(\mathcal{I}_{P})}{2},$
(2)

where $F_{\theta}$ and $G_{\phi}$ are denoted as the image encoder and text
encoder of the CLIP model, respectively.

####  4.2.2 Intent-Aware Filter

The branch of Intent-Aware Filter rules out irrelevant charts in the database
and forms the candidate feature. Previous works (_e.g._ , [HA19, MTW∗18]) on
chart filtering primarily rely on fixed settings of the model, which are
coarse and neglect the user’s intents. For this shortcoming, we propose an
intent-aware filter constructed by $\mathcal{I}_{A}$, a set of disentangled
attributes selected by the user. For example, with $\mathcal{Q}$ illustrated
in Figure 7, a user may select the [Type, Layout, Trend] as the target
attributes for the retrieval, then the filter retains $n$ charts containing
these target attributes in the chart database. Then, the retained charts would
serve as candidates $C:=\\{c_{1},c_{2},\cdots,c_{n}\\}$ that are encoded as
candidate feature $f_{C}:=\\{f_{c_{1}},f_{c_{2}},\cdots,f_{c_{n}}\\}$ by image
encoder in the CLIP model.

####  4.2.3 Similarity Modeling

As the core of retrieval, similarity modeling is comprehensively performed
with triplets $\\{\mathcal{Q}$, $\mathcal{I}_{A}$, $f_{\mathcal{M}}\\}$. Among
them, $\mathcal{Q}$ provides the global perception composed of implicit
features. The explicit intent attributes $\mathcal{I}_{A}$ and implicit query
feature $f_{\mathcal{M}}$ that include user intent prompt and query chart,
work together to recall user-intended examples throughout the retrieval
process. The overall similarity score $\mathcal{S}$ can be denoted as:

$\mathcal{S}=S_{\mathcal{Q}}\cdot\text{e}^{\nu S_{\mathcal{I}_{A}}+\mu
S_{\mathcal{M}}},$ (3)

where $S_{\mathcal{Q}}$ demotes the global perception at pixel level, and
$S_{\mathcal{I}_{A}}$ and $S_{\mathcal{M}}$ denotes the similarity score of
user-selected attributes and the multi-modality feature, respectively. The
scaling factors $\nu$ and $\mu$ are empirically set as $1$ and $5$,
respectively. All $S_{\mathcal{Q}}$, $S_{\mathcal{I}_{A}}$ and
$S_{\mathcal{M}}$ are normalized with range of $[0,1]$. In the following, we
will introduce these similarity scores in detail.

Global Perception Score. Although salient attributes with respect to user
intent are disentangled and selected, a multitude of implicit features are
intertwined within the chart image, which forms the concept of global
perception. We adopt the classic cosine similarity and compute it at the pixel
level between $\mathcal{Q}$ and the feature of every candidate
$f_{C_{i}},i=1,2,\ldots,n$ as follows:

$S_{\mathcal{Q}}=\frac{{F_{\theta}({\mathcal{Q}}})\cdot{f_{C_{i}}}}{\|{F_{\theta}({\mathcal{Q}}})\|\|{f_{C_{i}}}\|}.$
(4)

Intent Attributes Score. Extended attribute
${\color[rgb]{0.33,0.8,0.725}\definecolor[named]{pgfstrokecolor}{rgb}{0.33,0.8,0.725}{\mathcal{T}_{e}}}$
contains a user’s expected attributes of retrieval results charts apart from
four primary attributes. Among the attributes, Type and Layout define core
properties of a chart, which is easy to distinguish from $\mathcal{Q}$.
Therefore, we neglect them in the similarity crafting. The other attributes
serve as the miscellaneous variant, and the change of them can be minuscule.
This motivates us to build another score $S_{\mathcal{I}_{A}}$ to further
enhance the retrieval process. Therefore, we mainly consider the following
three attributes: Trend $\mathcal{N}$, Color $\mathcal{C}$ and extended
attribute
${\color[rgb]{0.33,0.8,0.725}\definecolor[named]{pgfstrokecolor}{rgb}{0.33,0.8,0.725}{\mathcal{T}_{e}}}$.
$S_{\mathcal{I}_{A}}$ is formulated as:

$S_{\mathcal{I}_{A}}=S_{\mathcal{N}}+S_{\mathcal{C}}+S_{\mathcal{T}_{e}}.$ (5)

For $S_{\mathcal{N}}$, we build an extractor to extract the trend feature from
the query chart $\mathcal{Q}$ and every candidate chart
$C_{i},i=1,2,\ldots,n$. The extractor shares the parameters with the trend
classifier in the Annotation stage except the last fully connected layer.
Then, $S_{\mathcal{N}}$ can be estimated by cosine similarity between the
extracted features.

For $S_{\mathcal{C}}$, we follow the scheme in Figure 6 to transform
$\mathcal{Q}$ into a proportional color palette after the steps of background
removal and color extraction. Then the proportional color palette is
transformed into a 128-bin color histogram in all RGB channels and separately
stored in three vectors. We then concatenate these vectors to form the
ultimate color vector $V$. Denote $V_{\mathcal{Q}}$,
$V_{C_{i}},i=1,2,\ldots,n$ as the color vectors of $\mathcal{Q}$,
$V_{C_{i}},i=1,2,\ldots,n$, respectively, we then estimate the cosine
similarity between them.

For $S_{\mathcal{T}_{e}}$, since it is hard to quantify the similarity of
intents, we leverage the power of the CLIP model again to give an accurate
response. We feed the CLIP model with candidate chart $C_{i},i=1,2,\ldots,n$
and text labels in user intent $t_{i},i=1,2,\ldots,m$. For each chart, we
would obtain $m$ outputs $\\{y_{1},y_{2},\ldots,y_{m}\\}$. Prior to the
computation, the user would select one text label indexed $s$ as the attribute
that best represented their intent. Then, we denote $S_{\mathcal{T}_{e}}$ for
a candidate chart as:
$S_{\mathcal{T}_{e}}=\text{e}^{y_{s}}/{\sum_{m}\text{e}^{y_{m}}}.$

Feature Matching Score. Similarly, we match the closeness between the multi-
modal feature and the candidate feature by the following equation:
$S_{\mathcal{M}}=\frac{f_{\mathcal{M}}\cdot{f_{C_{i}}}}{{\|{f_{\mathcal{M}}\|\|{f_{C_{i}}}\|}}}.$

###  4.3 System Interface

We design an interface for the prototype system that allows users to retrieve
a chart according to their intents.

Figure 8: Interface of our prototype system. A) the _Annotation_ view
decouples the properties of the query chart; B) the _Retrieval_ view allows
users to select the primary and extended attributes, and C) the _Result_ view
shows the top 5 retrieved charts.

Annotation View.  In this view, users can upload a query chart and get
disentangled primary attributes, including Type, Color, Trend, and Layout. As
shown in Figure 8 (A), a bar chart is uploaded and the automatically annotated
attributes, “ _Barchart, Categorical Colormap, Increasing Trend, Horizontal
Layout_ ” are presented.

Retrieval View. In this view (Figure 8 (B)), users can view and choose both
primary and extended attributes. The first tag in green corresponds to the
annotated attributes from the query chart, while other options are also shown,
allowing users to select the desired attributes. If users feel that the
default primary attributes are insufficient or have other attributes of
interest, they can add a new classifier by clicking on the chart in the upper
right corner, which reflects the philosophy of the proposed intent-aware
design. In the input box below, users can enter their intentions as intent
prompts for more customized queries. As shown in Figure 8 (B1), the user
selects “Bar chart” for the type attribute and enters “Fancy style with icon”
in the Retrieval view.

Result View. This view (Figure 8 (C)) shows top five query results, arranged
in decreasing order according to their similarity scores.

##  5 Evaluation

Figure 9: Visual comparison of chart retrieval by different methods. First row: Retrieval results by the conventional HOG method. Second row: Retrieval results by the deep-learning-based CNN method. Third to fourth rows: Retrieval results of our WYTIWYR framework. Table 1: Our integrated dataset consists of 18 types of charts from Beagle[BDM∗18] and manual collection. Chart Type | #Count | Chart Type | #Count
---|---|---|---
Bar chart | 7269 | Heatmap | 352
Stacked Bar Chart | 1159 | Line Graph | 11605
Circular Bar Chart | 608 | Star Plot | 491
Donut Chart | 2459 | Choropleth Map | 640
Pie Chart | 2587 | Scatter Plot | 3000
Sankey Diagram | 266 | Word Cloud | 406
Timeline | 324 | Dendrogram | 298
Box Plot | 571 | Network | 395
Histogram | 695 | Circular packing chart | 139

###  5.1 Quantitative Experiment

Dataset. Previous works concentrate on retrieval of limited chart types, e.g.,
single [MTW∗18] or a few type [CCA15, HBL∗19] chart retrieval. Moreover, some
of the literature coarsely considers the categorization of charts, by broadly
grouping several chart categories into one [LWW∗22]. Many datasets are
synthesized with simple composition and monotonous variation, making it
difficult to adapt to real-world scenarios that are much more complex. In this
work, we utilize the Beagle dataset [BDM∗18] _with charts_ in bitmap format,
which offers visualization collections designed by real-world users through
multiple tools, including D3, Plotly, Chartblocks, Fusion Charts, and Graphiq.
To make our framework more robust, we further add more difficult instances by
manually collecting 4k images from Pinterest, which supplements a large number
of stylized and irregular charts. We filter out charts whose type is beyond
our scope (_e.g._ , scientific visualizations) and manually label them with
$\mathcal{T}_{p}$. Finally, our dataset consists of 33,260 images in total.
The detailed distribution is shown in Table 1.

Table 2: Accuracy of Annotation results. Method | Type | Trend | Layout | Color
---|---|---|---|---
ResNet50+MSE Loss | 0.9518 | 0.8290 | 0.9537 | 0.7963
ResNet50+Focal Loss | 0.9601 | 0.8424 | 0.9653 | 0.8019

Annotation Accuracy. We evaluate the performance of
${\color[rgb]{0,0.2,0.572}\definecolor[named]{pgfstrokecolor}{rgb}{0,0.2,0.572}{\mathcal{T}_{p}}}$
annotation using ResNet50 architectures with different losses. As Table 2
lists, the best result is achieved with the designated Focal Loss.

Table 3: F1-scores of Retrieval results. |  | F1-Score
---|---|---
Top-K | Method | Type | Trend | Layout | Color
3 | HOG | 0.6199 | 0.6140 | 0.5449 | 0.6800
CNN | 0.9154 | 0.8043 | 0.7095 | 0.7717
Ours | 0.9549 | 0.8360 | 0.9280 | 0.8260
5 | HOG | 0.5067 | 0.4853 | 0.3857 | 0.6045
CNN | 0.8872 | 0.7550 | 0.6324 | 0.7076
Ours | 0.9364 | 0.7944 | 0.9118 | 0.7730
10 | HOG | 0.4124 | 0.4261 | 0.2741 | 0.5388
CNN | 0.8607 | 0.7085 | 0.5527 | 0.6520
Ours | 0.9091 | 0.7546 | 0.8812 | 0.7223

Retrieval Results. To evaluate the retrieval performance comprehensively, we
examine both precisions and recall via the F1-score. Denoting the
classification result with true positive, true negative, false positive, and
false negative conditions as $TP,TN,FP,FN$, F1-score can be formulated as:

$F1\text{-}score=\frac{2TP}{2TP+FP+FN}.$ (6)

The retrieval performance with $\mathcal{T}_{p}$ is individually computed
using F1-score in the top-$K$ fashion, where $K\in\\{3,5,10\\}$. We devise two
settings of with and without user intent to fully evaluate our approach. For
the setting of the absence of user intent, similarity estimation depends on
the global perception $S_{Global}$.

To demonstrate the effectiveness of our approach, we take two other approaches
as the counterpart: a conventional method (Histogram of Oriented Gradients
[DT05], denoted as HOG from hereon), and a deep-learning-based method
(ResNet50, denoted as CNN from hereon). The quantitative results are displayed
in Table 3. The results reveal that our method significantly outperforms the
others. Furthermore, as $K$ increases, our method keeps the performance
superiority while other methods degrade dramatically. For the evaluation of
user intent retrieval, as there are no ground-truth labels to compute
quantitative results, we visually compare results generated by different
methods for several typical retrieval results. As shown in Figure 9, CNN
outputs visually more similar results than HOG. Our method also produces
similar results without user intent. Nevertheless, our method allows users to
add their intent by selecting the disentangled primary attributes
${\color[rgb]{0,0.41,0.09}\definecolor[named]{pgfstrokecolor}{rgb}{0,0.41,0.09}{\mathcal{T}_{p}}}$
or adding extra intent, which contributes to retrieving results of interest as
shown in the last two rows.

###  5.2 Case Study

During the process of visualization design, many inspirations may arise, but
bringing those inspirations to fruition can be time-consuming. Retrieval is a
fast way to validate those inspirations. Inspired by [BLBL22], two usage
scenarios of chart retrieval are illustrated as a proof-of-concept of the
WYTIWYR framework: 1) extending the design space by explicit goals and 2)
fuzzy retrieval based on implicit user intent. For each scenario, we compare
the results generated by our intent-aware chart retrieval technique with those
generated by intent-free chart retrieval technique.

####  5.2.1 Design Space Extension by Explicit Goals

With the prototype system, users can customize retrieval inputs based on
disentangled attributes of the query chart and the user intent prompt.
Moreover, beyond the four primary attribute classifiers, users can further add
customized classifiers to extract extended attributes. In this scenario, we
divide the disentangled attribute operation into three categories: origin
attribute change, new attribute addition, and existing attribute deletion. In
the end, we show an interesting case of attribute transfer that combines these
operations. Figure 10 shows four cases of our study. In general, retrieval
without user intent provides similar results as the input but gives minimum
design insights. Instead, with the guidance of user intent, our framework
retrieves more diverse results that are well-accepted. The following lists
more details for each case.

Figure 10:  Four cases for extending the design space by explicit attributes.
In each case, the first line presents results without user intent, while the
second line presents results with user intent.

Original Attribute Change. The disentangled attributes are independent of each
other, thus, users can replace one of the attributes while keeping others
unchangeable. In Figure 10 (a), the color of the choropleth map is changed to
user prompt “pink”.

New Attribute Addition. Users can add new attributes, together with the
disentangled attributes, to better describe their specific needs. In Figure 10
(b), users add “with a curve to indicate the distribution”, and the results
are changed to combinations of histograms and line chart.

Existing Attribute Deletion. As the attributes can be disentangled as users’
needs, they can remove some attributes of no interest. In Figure 10 (c), the
user dislikes dark background, then can discard the attribute by adding a new
classifier with labels of [“dark background”, “white background”].

Attribute Transfer. As in Figure 10 (d), given a circular packing chart as a
query chart, the user gets the attribute annotation as $\\{$_Type: “Circular
Packing Chart”, Color: “Sequential Colormap”_ $\\}$. The user would like to
get heatmaps with the same form as the query chart but with more contrast
colors. S/he can get the desired results by changing Type and Color attributes
to $\\{$_Type: “Heatmap”, Color: “Diverging Colormap”_ $\\}$ in the retrieval
stage.

####  5.2.2 Fuzzy Retrieval by User Intent

Designing an expressive visualization incurs a steep learning rate for some
novices, who may not have a specific design prior and prefer to perform trial
and error on the search engine to seek a desirable design. Below, we list
three scenarios to illustrate how our method supports such retrieval.
Similarly, we will compare the results of the intent-free and intent-aware
retrieval.

Figure 11: Three cases for showing fuzzy retrieval. Negative examples are
highlighted with a red border.

Text Information Seeking. Keyword-based searching is popular, but existing
works approach this goal by storing text information in advance [HA19, CCA15]
or relying on OCR [SKC∗11], hindering generalizability and robustness of the
works. Instead, our method allows to recognize text information based on
knowledge transferred by the pretrained CLIP model. As shown in Figure 11 (a),
to generate a chart for visualizing “life expectancy”, the user keeps the
origin chart type Scatter plot and data trend Increase trend, and adds the
user intent as a text prompt “life expectancy”. The intent-aware retrieval
results imitate the famous design in Gapminder.

Relevant Topic Finding. Our method also has the capacity to find chart
examples related to a specific topic. Previous work used word2vec [Chu17] to
model the distances between words [HA19], limiting the search scope within
texts. We enhance the generalizability of matching text prompt with both text
information and visual attributes. In Figure 11 (b), given a word cloud with
the theme of nature, the user can retrieve some examples with text prompt as
“about technology topic”. Our system returns examples with content about
technology. Moreover, with the text prompt of “all words form a shape of a
tree”, our system aligns the semantic information of text with visual
attributes, returning examples with tree shapes instead of only containing the
word “tree”.

Abstract Description Searching. In the process of retrieval, users not having
a clear search target tend to use ambiguous and abstract descriptions. In
Figure 11 (c), the user inputs a basic timeline diagram with “fancy style”,
and gets several well-designed examples. To constrain the scope of vertical
layout, the user adds such attribute and performs targeted retrieval.

###  5.3 Qualitative Evaluation

####  5.3.1 Study Design

Our study recruited seven participants (three women, four men) with ages 22 to
26. To demonstrate that our framework is useful for different levels of users,
we enroll four experts, i.e., visualization designers (E1 - E4), and three
novices with only experience in using visualizations in reports or courses (N1
- N3).

Before the experiment, we collected basic information from the participants
through a questionnaire and introduced them to the use scenarios of chart
retrieval. Then we instructed them to familiarize themselves with the system
workflow. We also showed some examples of using prompts to help participants
understand the prompt better and make a better choice of prompts. After
understanding the background, participants were allowed to freely explore and
use our system. They could upload a query chart and try to get a satisfactory
design in multiple iterations. If there was no satisfactory design, they could
also try combinations of attributes and prompts to see if the retrieved chart
has inspiring results. Throughout the process, participants were encouraged to
think aloud and give feedback whenever they wanted. After they felt it was
sufficient, we conducted interviews about the usability of our system. Three
questions were included: 1) Whether our approach is _effective_ in helping
them find the chart that meets their intents; 2) Whether our approach is
_efficient_ enough to avoid long waits; 3) Whether they are _satisfied_ with
our system. The experiment lasted about 45 minutes. The participants were not
compensated with money but beverages worthing about $5 dollars after the
study.

####  5.3.2 Feedback

Effectiveness. In general, all users agreed that our system effectively
supported for retrieving the desired charts based on their intents. Most
participants (6 out of 7) thought that our setting of the primary attribute
was appropriate and comprehensive. E2 suggested to extract the main hue used
in the chart in the annotation stage. Regarding the comprehensiveness and
accuracy of the prompts, all participants felt that their search needs were
largely met and the results were generally consistent with their intents. N1
spent a long time exploring the map as shown in Figure 10 (a). He found it
exciting when typing “India”, an India map was retrieved. Our system also
effectively recognized his intents when he searched for maps of various
colors.

The participants pointed out some limitations as well. When the prompt given
by the user had multiple meanings, sometimes only one or two of the search
results were exactly what the user expects. E4 tried to add a trend arrow on
the bar chart. However, when he only typed “with arrow”, some bar charts using
arrow icons were also retrieved. He said that sometimes it was required to go
and try several prompts to get the desired result. E3 encountered a similar
situation, and it might be due to our dataset only containing limited charts
that matched her specifications. Besides, sometimes participants may want to
adjust attributes in the query chart, such as changing the white background to
dark (N2), and changing the colorful design to black and white (E2). However,
one or two of the returned results still had the features that they would like
to change. “There are times when I want the query to focus more on the prompt
I give and less on the original chart,” said N2.

Efficiency and Satisfaction. All participants agreed that our system was
responsive and user friendly. They were satisfied with our retrieving
framework, and they pointed out some possible improvements. They found it
helpful to decouple a chart into attributes and have a selective query. N2
stated, “It saved me the time of checking a lot of examples and found the
desired design directly”. N3 appreciated the function of adding a new
classifier since it can be used to explore new types of visualizations. “It is
convenient for novices who do not understand the type and content of the chart
at first,” he gave the reason. E1, a participant with a design background,
said our results were inspiring. She was willing to see more results, even if
they did not exactly match the search intents. “This could inspire me to come
up with new ideas,” she added.

##  6 Discussion

WYTIWYR is an intent-aware chart query framework that can address various
issues related to traditional chart retrieval and expedite the design process.
In conventional retrieval, not all chart attributes may be relevant to users’
intents, whilst some preferred attributes may even be absent from the chart.
Instead, our system disentangles and combines attributes to ensure that the
retrieved results encompass all attributes of user’s intents. There are also
several limitations in our current methods and avenues for future work.

###  6.1 Limitations

User Intent or Query Chart. The input of our method consists of a query chart
and user intent. The composition of two factors affects the retrieval results.
As shown in Figure 10 (b), users have a strong intent of charts that have a
curve, and our system would tend to return the results meeting such intents.
However, the perception of global distribution may be lost. The priority
between user intents as text and global distribution in the input chart is
hard to tackle due to the usage scenarios. Furthermore, within the user
intent, the weights of selected attributes and text prompts are hard to set in
the similarity modeling. A dynamic adjustment of the priority between the user
intent and query chart is demanded in this scenario.

Prompt Sensibility. Text prompt is integrated with query chart as joint input
for retrieval in our work. However, designing effective prompts is challenging
due to the significant impact of even slight wording changes on
performance[ZYLL22]. In our framework, prompt design is limited by the CLIP
pretrained model, which is trained mainly on natural images, which makes it
difficult to accurately interpret expert expressions for visualization domain.
For instance, Figure 12 shows that using a more professional prompt in the
third row results in negative results, when compared to the second row with
identical intent. The CLIP model also struggles to identify rare or novel
objects. Approaches for fine-tuning [LLXH22, LSG∗21, ZZF∗22] the CLIP model or
using advanced language-image models are promising. Obtaining enough chart-
description pairs for training is also necessary, but the process is time-
consuming and resource-intensive.

Dataset. Despite collecting a large number of real-world charts, our database
remains limited in meeting vast user needs. In Figure 11 (b), only the first
three charts are tree-shaped since we only have these charts that meet the
user’s intent in our database. The unbalanced dataset may adversely affect
retrieval performance, as the attribute classifier training could become
biased towards classes with larger volumes. We alleviate it by using focal
loss, which assigns higher weights to minority class and misclassified
examples. Due to the lack of a more detailed category in our dataset, some
inaccurate results will occur in the retrieval. In Figure 12, two types of
heatmaps are visible in the retrieval results.

Figure 12: A case for revealing the limitations. Negative examples are
highlighted with a red border.

Chart Attributes. In the preliminary study, we employed several chart types to
determine the chart attributes that users tend to concentrate on. We randomly
chose samples from both synthesized datasets and real-world collections to
diversify the chart attributes. Nevertheless, the attributes may not fully
cover user intents. A more comprehensive survey will be conducted to address
the issue.

###  6.2 Future Work

Trade-off Control. To balance between user intent and query chart, we plan to
add a slider equipped with dynamic weight control to let the user manage this
trade-off by themselves in the near future.

Text Prompt Auto-completion. Previous works [SHKC20, WHS∗22] utilize auto-
completion as a hint to assist users in the application process, which is also
helpful for our system to alleviate the interpretability shift between the
user side and the model side. Specifically, we aim to build a mapping table to
store the relationship between appropriate prompts and common user text
expressions. Then, when a user completes the input, the keywords of it will be
extracted by the named entity recognition technique [NS07]. Finally, the
corresponding optimal prompt would be automatically completed by the search in
the mapping table.

Text-only retrieval. Our system currently supports chart-only retrieval and
chart-text retrieval. To offer a more generalized retrieval framework, we plan
to add a text-only query, allowing users to obtain their retrieval by only
providing text input. The text-only retrieval is beneficial to users who do
not have any reference chart. To this end, we aim to prepare several basic
charts with primary attributes in advance to serve as temporary query charts,
which can be replaced by a more reliable chart from returned results by
retrieval. The process can be regarded as an iteration of desirable results,
with a clearer search goal and narrowed search scope.

##  7 Conclusion

In this paper, we propose a user intent-aware chart retrieval framework, which
leverages multi-modal input to fuse explicit visual attributes and implicit
user intent into the retrieval process. This pipeline consists of two core
stages, namely Annotation, and Retrieval. The Annotation stage disentangles
visual attributes in the query chart to ensure a flexible combination of user
intent attributes used in retrieval. The Retrieval stage allows users to
integrate the text prompt with the query chart to achieve more customized
retrieval. Quantitative experiments prove the superior performance of our
method compared with previous methods. Furthermore, we conduct two case
studies containing two common retrieval strategies and interviews to
demonstrate the effectiveness and usability. As an initial step to fuse user
prompt in chart retrieval, we hope to enhance the prompt capacity to better
meets users’ growing needs and various usage scenarios. Dataset, code,
pretrained model have been released to promote future research in this
direction.

##  8 Acknowledgments

The authors wish to thank anonymous reviewers for their constructive comments.
The work was supported in part by National Natural Science Foundation of China
(62172398), and the Red Bird Program at the Hong Kong University of Science
and Technology (Guanghzou).

## References

  * [BBK∗18] Behrisch M., Blumenschein M., Kim N. W., Shao L., El-Assady M., Fuchs J., Seebacher D., Diehl A., Brandes U., Pfister H., Schreck T., Weiskopf D., Keim D. A.:  Quality metrics for information visualization.  _Comput. Graph. Forum 37_ , 3 (2018), 625–662.
  * [BDM∗18] Battle L., Duan P., Miranda Z., Mukusheva D., Chang R., Stonebraker M.:  Beagle: Automated extraction and interpretation of visualizations from the web.  In _Proc. ACM CHI_ (2018), pp. 594:1–8.
  * [BFW21] Battle L., Feng D., Webber K.:  Exploring visualization implementation challenges faced by D3 users online.  _arXiv preprint arXiv:2108.02299_ (2021).
  * [BLBL22] Bako H. K., Liu X., Battle L., Liu Z.:  Understanding how designers find and use data visualization examples.  _IEEE Trans. Vis. Comput. Graph._ (2022).
  * [BMR∗20] Brown T., Mann B., Ryder N., Subbiah M., Kaplan J. D., Dhariwal P., Neelakantan A., Shyam P., Sastry G., Askell A., et al.:  Language models are few-shot learners.  In _Proc. NIPS_ (2020), pp. 1877–1901.
  * [BVB∗13] Borkin M. A., Vo A. A., Bylinskii Z., Isola P., Sunkavalli S., Oliva A., Pfister H.:  What makes a visualization memorable?  _IEEE Trans. Vis. Comput. Graph. 19_ , 12 (2013), 2306–2315.
  * [CCA15] Chen Z., Cafarella M., Adar E.:  DiagramFlyer: A search engine for data-driven diagrams.  In _Proc. WWW_ (2015), pp. 183–186.
  * [Chu17] Church K. W.:  Word2vec.  _Natural Language Engineering 23_ , 1 (2017), 155–162.
  * [DCLT18] Devlin J., Chang M.-W., Lee K., Toutanova K.:  Bert: Pre-training of deep bidirectional transformers for language understanding.  In _Proc. NAACL_ (2018), pp. 4171–4186.
  * [DT05] Dalal N., Triggs B.:  Histograms of oriented gradients for human detection.  In _Proc. CVPR_ (2005), pp. 886–893.
  * [DW14] Dang T. N., Wilkinson L.:  Scagexplorer: Exploring scatterplots by their scagnostics.  In _Proc. IEEE PacificVis_ (2014), IEEE, pp. 73–80.
  * [GKSS∗19] Goyal Y., Khot T., Summers-Stay D., Batra D., Parikh D.:  Making the V in VQA matter: Elevating the role of image understanding in visual question answering.  _Int. J. Comput. Vis._ (2019), 398–414.
  * [GLKC22] Gu X., Lin T.-Y., Kuo W., Cui Y.:  Open-vocabulary object detection via vision and language knowledge distillation.  In _Proc. ICLR_ (2022).
  * [GPM∗21] Gal R., Patashnik O., Maron H., Chechik G., Cohen-Or D.:  StyleGAN-NADA: CLIP-guided domain adaptation of image generators.  _ACM Trans. Graph. 41_ (2021), 141:1–13.
  * [HA19] Hoque E., Agrawala M.:  Searching the visual style and structure of D3 visualizations.  _IEEE Trans. Vis. Comput. Graph. 26_ , 1 (2019), 1236–1245.
  * [HBL∗19] Hu K., Bakker M. A., Li S., Kraska T., Hidalgo C.:  VizML: A machine learning approach to visualization recommendation.  In _Proc. ACM CHI_ (2019), pp. 128: 1–12.
  * [HMSA08] Heer J., Mackinlay J., Stolte C., Agrawala M.:  Graphical histories for visualization: Supporting analysis, communication, and evaluation.  _IEEE Trans. Vis. Comput. Graph. 14_ , 6 (2008), 1189–1196.
  * [HZRS16] He K., Zhang X., Ren S., Sun J.:  Deep residual learning for image recognition.  In _Proc. CVPR_ (2016), pp. 770–778.
  * [JKS∗17] Jung D., Kim W., Song H., Hwang J.-i., Lee B., Kim B., Seo J.:  ChartSense: Interactive data extraction from chart images.  In _Proc. ACM CHI_ (2017), pp. 6706–6717.
  * [KRS∗21] Kim H., Rossi R., Sarma A., Moritz D., Hullman J.:  An automated approach to reasoning about task-oriented insights in responsive visualization.  _IEEE Trans. Vis. Comput. Graph. 28_ , 1 (2021), 129–139.
  * [LCF∗15] Li Z., Carberry S., Fang H., McCoy K. F., Peterson K., Stagitis M.:  A novel methodology for retrieving infographics utilizing structure and message content.  _DKE 100_ (2015), 191–210.
  * [LGG∗17] Lin T.-Y., Goyal P., Girshick R., He K., Dollár P.:  Focal loss for dense object detection.  In _Proc. ICCV_ (2017), pp. 2980–2988.
  * [LGR∗20] Lu J., Goswami V., Rohrbach M., Parikh D., Lee S.:  12-in-1: Multi-task vision and language representation learning.  In _Proc. ICML_ (2020), pp. 10437–10446.
  * [LLXH22] Li J., Li D., Xiong C., Hoi S.:  Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation.  In _Proc. ICML_ (2022), pp. 12888–12900.
  * [LOG∗19] Liu Y., Ott M., Goyal N., Du J., Joshi M., Chen D., Levy O., Lewis M., Zettlemoyer L., Stoyanov V.:  RoBERTa: A robustly optimized BERT pretraining approach.  In _Proc. ICLR_ (2019).
  * [LSG∗21] Li J., Selvaraju R., Gotmare A., Joty S., Xiong C., Hoi S. C. H.:  Align before fuse: Vision and language representation learning with momentum distillation.  In _Proc. NIPS_ (2021), pp. 9694–9705.
  * [LWW∗22] Li H., Wang Y., Wu A., Wei H., Qu H.:  Structure-aware visualization retrieval.  In _Proc. ACM CHI_ (2022), pp. 409: 1–14.
  * [MMA18] Majooni A., Masood M., Akhavan A.:  An eye-tracking study on the effect of infographic structures on viewer’s comprehension and cognitive load.  _Inf. Vis. 17_ , 3 (2018), 257–266.
  * [MTW∗12] Moere A. V., Tomitsch M., Wimmer C., Christoph B., Grechenig T.:  Evaluating the effect of style in information visualization.  _IEEE Trans. Vis. Comput. Graph. 18_ , 12 (2012), 2739–2748.
  * [MTW∗18] Ma Y., Tung A. K., Wang W., Gao X., Pan Z., Chen W.:  ScatterNet: A deep subjective similarity model for visual analysis of scatterplots.  _IEEE Trans. Vis. Comput. Graph. 26_ , 3 (2018), 1562–1576.
  * [NS07] Nadeau D., Sekine S.:  A survey of named entity recognition and classification.  _Lingvisticae Investigationes 30_ , 1 (2007), 3–26.
  * [PMH17] Poco J., Mayhua A., Heer J.:  Extracting and retargeting color mappings from bitmap images of visualizations.  _IEEE Trans. Vis. Comput. Graph. 24_ , 1 (2017), 637–646.
  * [PSP21] Parsons P., Shukla P., Park C.:  Fixation and creativity in data visualization design: Experiences and perspectives of practitioners.  In _Proc. IEEE VIS_ (2021), pp. 76–80.
  * [PWS∗21] Patashnik O., Wu Z., Shechtman E., Cohen-Or D., Lischinski D.:  StyleCLIP: Text-driven manipulation of styleGAN imagery.  In _Proc. ICCV_ (2021), pp. 2085–2094.
  * [QDH∗22] Qin X., Dai H., Hu X., Fan D.-P., Shao L., Van Gool L.:  Highly accurate dichotomous image segmentation.  In _Proc. ECCV_ (2022), pp. 38–56.
  * [RKH∗21] Radford A., Kim J. W., Hallacy C., Ramesh A., Goh G., Agarwal S., Sastry G., Askell A., Mishkin P., Clark J., et al.:  Learning transferable visual models from natural language supervision.  In _Proc. ICML_ (2021), pp. 8748–8763.
  * [RNS∗18] Radford A., Narasimhan K., Salimans T., Sutskever I., et al.:  _Improving language understanding by generative pre-training_.  Tech. rep., 2018.
  * [RTOT06] Rodrigues J. F., Traina A. J. M., Oliveira M. C. F. d., Traina C.:  Reviewing data visualization: an analytical taxonomical study.  In _Proc. ICCV_ (2006), pp. 713–720.
  * [RWC∗19] Radford A., Wu J., Child R., Luan D., Amodei D., Sutskever I., et al.:  Language models are unsupervised multitask learners.  _OpenAI blog 1_ , 8 (2019), 9.
  * [SGP∗18] Stitz H., Gratzl S., Piringer H., Zichner T., Streit M.:  Knowledgepearls: Provenance-based visualization retrieval.  _IEEE Trans. Vis. Comput. Graph. 25_ , 1 (2018), 120–130.
  * [SHKC20] Setlur V., Hoque E., Kim D. H., Chang A. X.:  Sneak pique: Exploring autocompletion as a data discovery scaffold for supporting visual analysis.  In _Proc. ACM UIST_ (2020), pp. 966–978.
  * [SHL∗16] Siegel N., Horvitz Z., Levin R., Divvala S., Farhadi A.:  Figureseer: Parsing result-figures in research papers.  In _Proc. ECCV_ (2016), Springer, pp. 664–680.
  * [Shn96] Shneiderman B.:  The eyes have it: a task by data type taxonomy for information visualizations.  In _Proc. IEEE Symp. Vis. Lang._ (1996), pp. 336 – 343.
  * [SKC∗11] Savva M., Kong N., Chhajta A., Fei-Fei L., Agrawala M., Heer J.:  ReVision: Automated classification, analysis and redesign of chart images.  In _Proc. ACM UIST_ (2011), pp. 393–402.
  * [SLC∗23] Shi Y., Liu P., Chen S., Sun M., Cao N.:  Supporting expressive and faithful pictorial visualization design with visual style transfer.  _IEEE Trans. Vis. Comput. Graph. 29_ (2023), 236 – 246.
  * [SSK06] Schneidewind J., Sips M., Keim D. A.:  Pixnostics: Towards measuring the value of visualization.  In _Proc. IEEE VAST_ (2006), pp. 199–206.
  * [SWS∗22] Strobelt H., Webson A., Sanh V., Hoover B., Beyer J., Pfister H., Rush A. M.:  Interactive and visual prompt engineering for ad-hoc task adaptation with large language models.  _IEEE Trans. Vis. Comput. Graph._ (2022).
  * [TM04] Tory M., Möller T.:  Rethinking visualization: A high-level taxonomy.  In _Proc. IEEE InfoVis_ (2004), pp. 151–158.
  * [WHS∗22] Wang Y., Hou Z., Shen L., Wu T., Wang J., Huang H., Zhang H., Zhang D.:  Towards natural language-based visualization authoring.  _IEEE Trans. Vis. Comput. Graph._ (2022).
  * [XBK∗15] Xu K., Ba J., Kiros R., Cho K., Courville A., Salakhudinov R., Zemel R., Bengio Y.:  Show, attend and tell: Neural image caption generation with visual attention.  In _Proc. ICML_ (2015), pp. 2048–2057.
  * [YHZ23] Ye Y., Huang R., Zeng W.:  VISAtlas: An image-based exploration and query system for large visualization collections via neural image embedding.  _IEEE Trans. Vis. Comput. Graph._ (2023), 1–15.
  * [YZF∗22] Yuan L., Zeng W., Fu S., Zeng Z., Li H., Fu C. W., Qu H.:  Deep colormap extraction from visualizations.  _IEEE Trans. Vis. Comput. Graph. 28_ , 12 (2022), 4048 – 4060.
  * [ZDCC21] Zeng W., Dong A., Chen X., Cheng Z.-l.:  VIStory: interactive storyboard for exploring visual information in scientific publications.  _J. Vis. 24_ , 1 (2021), 69–84.
  * [ZFC∗23] Zhang T., Feng H., Chen W., Chen Z., Zheng W., Luo X.-N., Huang W., Tung A. K.:  ChartNavigator: an interactive pattern identification and annotation framework for charts.  _IEEE Trans. Knowl. Data Eng. 35_ , 2 (2023), 1258 – 1269.
  * [ZFF22] Zhao J., Fan M., Feng M.:  ChartSeer: Interactive steering exploratory visual analysis with machine intelligence.  _IEEE Trans. Vis. Comput. Graph. 28_ , 3 (2022), 1500–1513.
  * [ZYLL22] Zhou K., Yang J., Loy C. C., Liu Z.:  Learning to prompt for vision-language models.  _Int. J. Comput. Vis. 130_ , 9 (2022), 2337–2348.
  * [ZZF∗22] Zhang R., Zhang W., Fang R., Gao P., Li K., Dai J., Qiao Y., Li H.:  Tip-adapter: Training-free adaption of CLIP for few-shot classification.  In _Proc. ECCV_ (2022), pp. 493–510.