# Erdős–Lax estimates for rational powers of polynomials

Alex Bergman & Olof Rubin  Centre for Mathematical Sciences, Lund University
Box 118, SE-22100, Lund, Sweden
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract.

The classical Erdős–Lax inequality states that if $P$ is a polynomial of
degree $n$ which is zero free on $\\{|z|<1\\}$ then
$\|P^{\prime}\|\leq\frac{n}{2}\|P\|$ where the norm is taken as supremum norm
on the unit circle. Equality holds if and only if all zeros of $P$ are
situated on the unit circle. In the present article we generalise the equality
to functions of the form $f=P^{1/m}$ where $P$ has all its roots on the unit
circle obtaining $\|f^{\prime}\|=\frac{n/m}{2}\|f\|$. We further prove an
inequality of the form $\|f^{\prime}\|\leq\frac{n/m}{2}\|f\|$ where $P$ is
zero free on $\\{|z|<1\\}$.

###### Key words and phrases:

Polynomial inequalities, Erdős–Lax inequality, Fractional powers of
polynomials

###### 1991 Mathematics Subject Classification:

30A10. 26D05. 41A17. 30C10.

##  1\. Introduction

The purpose of this article is to establish estimates in the uniform norm of
derivatives of fractional powers of polynomials. In particular, we consider
functions $f=P^{1/m}$ where $P$ is a polynomial and $m\in\mathbb{N}$ with the
goal of relating the supremum norms of $f^{\prime}$ and $f$ relative to the
unit circle. Throughout the text we adopt the notation that $\|\cdot\|$
denotes the supremum norm on the unit circle, $\mathbb{T}=\\{z:|z|=1\\}$. We
further adopt the notation $\mathbb{D}=\\{z:|z|<1\\}$ for the unit disk.

For $m>1$ the function $f$ will be multi-valued. However, $f$ and its
derivatives will have single-valued absolute values.

A classical example of a relation of the aforementioned type comes from
Bernstein’s inequality. If $P$ is a polynomial of degree $n$ then

(1) $\|P^{\prime}\|\leq n\|P\|$

with equality if and only if $P(z)=cz^{n}$ for some constant $c\in\mathbb{C}$.

Erdős suggested that this inequality can be sharpened in the case when $P$ is
zero free on $\mathbb{D}$. He suggested that a sharp inequality in this case
is given by

(2) $\|P^{\prime}\|\leq\frac{n}{2}\|P\|.$

This was later proven to be correct by Lax [6] using techniques developed by
Szegő and Pólya. Szegő and Pólya verified the special case of equality in (2)
when the zeros of $P$ are situated on $\mathbb{T}$.

Lax was able to lift this result using so-called reciprocal polynomials to
prove (2). We should mention that preceding Lax’s proof of (2), Turán [10]
showed that if $P$ is a polynomial which is zero free on
$\mathbb{C}\setminus\mathbb{D}$ then

(3) $\|P^{\prime}\|<\frac{n}{2}\|P\|.$

It is interesting to ask if the case of equality in (2) that was proven by
Szegő and Pólya can be generalised to the setting where we instead of the
polynomial $P$ consider the function $f=P^{1/m}$. In particular, can we then
replace the degree $n$ in the equation with the number $n/m$? Here we provide
the affirmative answer to this question. We also consider the case where the
zeros are situated exterior to $\mathbb{D}$ and provide a case for which the
inequality is valid. Several generalisations of the Erdős–Lax inequality
exists for the case of polynomials. See, for example, [7, 8, 2]. In
particular, we mention the $L^{p}$ versions of the Erdős–Lax inequality proved
in [9] ($0<p<\infty$) which were obtained as consequences of $L^{p}$ Bernstein
estimates established by Arestov in [1]. We remark that the results were
obtained earlier in the case $1\leq p<\infty$. For such $p$ the Berstein
estimate was given by Zygmund [11] and the Erdős–Lax estimate was given by de-
Bruijn [4]. However, the direction toward similar inequalities for fractional
powers of polynomials appear to be unexplored.

As $k$ ranges between $1,\dotsc,N$, let $a_{k}$ be complex numbers, $\lvert
a_{k}\rvert\geq 1$, $s_{k}$ positive rational numbers, and $m\geq 1$ an
integer such that $ms_{k}\in\mathbb{N}$ for all $s_{k}$. Denote by $P$ the
polynomial

$P(z)=c^{m}\prod_{k=1}^{N}(z-a_{k})^{s_{k}m}.$

The degree of $P$ will be denoted by $n=m\sum_{k=1}^{N}s_{k}$. Let

(4) $f(z)=P(z)^{1/m}=c\prod_{k=1}^{N}(z-a_{k})^{s_{k}},\quad z\in\mathbb{D}$

where the branches are chosen as to make $f$ analytic in $\mathbb{D}$. The
function $f$ extends continuously to the unit circle and the derivative of $f$
is given by

(5)
$f^{\prime}(z)=\frac{1}{m}f(z)\frac{P^{\prime}(z)}{P(z)}=\frac{f(z)}{m}\sum_{k=1}^{n}\frac{1}{z-c_{k}},$

where the $c_{k}$’s are the zeros of $P$ counting multiplicities. If
$s_{k}\geq 1$ for all $k$, the derivative is also continuous in
$\overline{\mathbb{D}}$. Our goal is to prove the following Erdős–Lax type
equality.

######  Theorem 1.

Let $|a_{k}|=1$ and let $s_{k}\geq 1$ denote rational numbers. If

$f(z)=c\prod_{k}(z-a_{k})^{s_{k}},$

then

$\lVert f^{\prime}\rVert=\frac{\sum s_{k}}{2}\lVert f\rVert.$

Using this equality we are able to prove an Erdős–Lax type inequality with
certain restrictions on the zeros of $f$ in $\left\\{z:\lvert
z\rvert>1\right\\}$.

######  Theorem 2.

Let $|a_{k}|\geq 1$ and let $s_{k}\geq 1$ denote rational numbers. If

$f(z)=c\prod_{k}(z-a_{k})^{s_{k}},$

and $s_{k}\in\mathbb{N}$ whenever $|a_{k}|>1$, then

$\lVert f^{\prime}\rVert\leq\frac{\sum s_{k}}{2}\lVert f\rVert$

with equality if and only if all $a_{k}$’s are unimodular.

This result allows us to have zeros of $f$ away from the unit circle. However,
the restriction is that the zeros which are located away from the circle must
have integer exponents.

It should be noted that all our results extend to the case where $s_{k}$ is
real-valued with the same bound as specified above by a density argument.

The generalisation of the Erdős–Lax inequality presented here serves the
purpose of allowing for fractional powers of roots on the unit circle. The
motivation for this comes from applications to weighted Chebyshev problems on
the unit circle. Given a weight function $w:\mathbb{T}\rightarrow[0,\infty)$
this amounts to finding the unique monic minimiser
$T_{n}^{w}(z)=z^{n}+\sum_{k=0}^{n-1}a_{k}z^{k}$ such that

$\|wT_{n}^{w}\|=\min_{b_{k}}\left\|w(z)\left(z^{n}+\sum_{k=0}^{n-1}b_{k}z^{k}\right)\right\|.$

With Theorem 1 at hand it is possible to generalise work of Lachance, Saff and
Varga [5] characterising such polynomials in the case where $w(z)=|z+1|^{s}$
for $s>0$. The reader is referred to [3] for more details.

##  2\. Proofs of the main Theorems

Assume, as before, that $f$ is of the form

$f(z)=c\prod_{k}(z-a_{k})^{s_{k}}$

where $|a_{k}|,s_{k}\geq 1$. First of all, note that the restriction
$s_{k}\geq 1$ is necessary for otherwise may $f^{\prime}$ not be bounded on
$\mathbb{T}$ if $|a_{k}|=1$ for some $k$. Secondly, note that if
$f(z)=(z-a)^{s}$ with $|a|\geq 1$ and $s\geq 1$ then

$f^{\prime}(z)=s(z-a)^{s-1},$

and thus $\|f^{\prime}\|=s(2+(|a|-1))^{s-1}$. Since $\|f\|=(2+(|a|-1))^{s}$
this implies that

$\frac{\|f^{\prime}\|}{\|f\|}=\frac{s}{(2+|a|-1)}\leq\frac{s}{2}$

with equality if and only if $|a|=1$. Also note that in this case $f^{\prime}$
achieves its maximum modulus at the same point on the unit circle as $f$ does.
This is something we will investigate further. The argument we now give will
be an adaptation of Szegő’s argument for polynomials suitably modified to our
situation. The key step is to prove that any point of maximality for
$f^{\prime}$ is a point of maximality for $f$ (with one exception).

###### Proof of Theorem 1.

Recall that we are assuming $s_{k}\geq 1$, for all $k$ and that
$\sum_{k=1}^{N}s_{k}=n/m$ where $n$ is the degree of the polynomial $P$
satisfying $f(z)=P(z)^{1/m}$. With these restrictions on $s_{k}$, we clearly
see that $f^{\prime}$ is continuous up to the circle. The second derivative of
$f$ is given by

$\begin{split}f^{\prime\prime}(z)&=\frac{1}{m}\left(f^{\prime}(z)\frac{P^{\prime}(z)}{P(z)}+f(z)\frac{P^{\prime\prime}(z)P(z)-(P^{\prime}(z))^{2}}{P(z)^{2}}\right)\\\
&=\frac{1}{m}\left(f^{\prime}(z)\frac{P^{\prime}(z)}{P(z)}+mf^{\prime}(z)\frac{P(z)}{P^{\prime}(z)}\frac{P^{\prime\prime}(z)P(z)-(P^{\prime}(z))^{2}}{P(z)^{2}}\right)\\\
&=\frac{f^{\prime}(z)}{m}\left((1-m)\frac{P^{\prime}(z)}{P(z)}+m\frac{P^{\prime\prime}(z)}{P^{\prime}(z)}\right).\end{split}$

With the intent of initially proving that
$\|f^{\prime}\|\leq\frac{n/m}{2}\|f\|$, we let $z_{0}\in\mathbb{T}$ be a
maximum of $f^{\prime}$. Consider first the case where $P(z_{0})\neq 0$ and
hence also $P^{\prime}(z_{0})\neq 0$ (otherwise $f^{\prime}(z_{0})=0$). The
derivative of the map $\theta\mapsto\lvert
f^{\prime}(z_{0}e^{i\theta})\rvert^{2}$ attains the value $0$ at $\theta=0$.
Thus

$z_{0}f^{\prime\prime}(z_{0})\overline{f^{\prime}(z_{0})}=\overline{z_{0}}f^{\prime}(z_{0})\overline{f^{\prime\prime}(z_{0})},$

meaning that $z_{0}f^{\prime\prime}(z_{0})/f^{\prime}(z_{0})\in\mathbb{R}$.
Combining this with the relation for $f^{\prime\prime}$, we obtain

(6) $\mathbb{R}\ni
z_{0}\cdot\frac{f^{\prime\prime}(z_{0})}{f^{\prime}(z_{0})}=\frac{1}{m}\left((1-m)\text{Re}\left(z_{0}\frac{P^{\prime}(z_{0})}{P(z_{0})}\right)+m\text{Re}\left(z_{0}\frac{P^{\prime\prime}(z_{0})}{P^{\prime}(z_{0})}\right)\right).$

For $z\in\mathbb{T}$,

$z\cdot\frac{P^{\prime}(z)}{P(z)}=\sum_{k=1}^{n}\frac{z}{z-a_{k}}=\frac{n}{2}+i\sum_{k=1}^{n}t_{k}(z),$

where $t_{k}(z)=\text{Im}\Big{(}z(z-a_{k})^{-1}\Big{)}$. If
$\sum_{k=1}^{n}t_{k}(z_{0})=0$ then we attain from equation (5) the inequality

$\|f^{\prime}\|=|f^{\prime}(z_{0})|=\frac{n/m}{2}|f(z_{0})|\leq\frac{n/m}{2}\|f\|.$

We want to show that this is the only possible case (with one exception) and
hence we assume that $\sum_{k=1}^{n}t_{k}(z_{0})\neq 0$. We consider the
quotients

$z^{2}\cdot\frac{P^{\prime\prime}(z)}{P(z)}=\left(\sum_{k=1}^{n}\frac{z}{z-a_{k}}\right)^{2}-\sum_{k=1}^{n}\left(\frac{z}{z-a_{k}}\right)^{2},$
$z\cdot\frac{P^{\prime\prime}(z)}{P^{\prime}(z)}=z^{2}\frac{P^{\prime\prime}(z)}{P(z)}/z\frac{P^{\prime}(z)}{P(z)}=\sum_{k=1}^{n}\frac{z}{z-a_{k}}-\frac{\sum_{k=1}^{n}\left(\frac{z}{z-a_{k}}\right)^{2}}{\sum_{k=1}^{n}\frac{z}{z-a_{k}}}.$

From (6) we see

$\displaystyle(1-m)\sum_{k=1}^{n}t_{k}(z_{0})$ $\displaystyle+$ $\displaystyle
m\left(\sum_{k=1}^{n}t_{k}(z_{0})-\frac{n/2\sum_{k=1}^{n}t_{k}(z_{0})-\sum_{k=1}^{n}t_{k}(z_{0})\sum_{k=1}^{n}(1/4-t_{k}(z_{0})^{2})}{n^{2}/4+\left(\sum_{k=1}^{n}t_{k}(z_{0})\right)^{2}}\right)=0.$

Thus we obtain

$\frac{n^{2}}{4}+\left(\sum_{k=1}^{n}t_{k}(z_{0})\right)^{2}=m\left(n/2-\sum_{k=1}^{n}(1/4-t_{k}(z_{0})^{2})\right).$

Substituting this into the equation for
$\text{Re}(z_{0}P^{\prime\prime}(z_{0})/P^{\prime}(z_{0}))$, we obtain

$\begin{split}\text{Re}\left(z_{0}\frac{P^{\prime\prime}(z_{0})}{P^{\prime}(z_{0})}\right)=\frac{n}{2}-\frac{n/2\sum_{k=1}^{n}(1/4-t_{k}(z_{0})^{2})+\left(\sum_{k=1}^{n}t_{k}(z_{0})\right)^{2}}{n^{2}/4+\left(\sum_{k=1}^{n}t_{k}(z_{0})\right)^{2}}\\\
=\frac{n}{2}-\frac{n/2\left(n/2-n^{2}/4m-\left(\sum_{k=1}^{n}t_{k}(z_{0})\right)^{2}/m\right)+\left(\sum_{k=1}^{n}t_{k}(z_{0})\right)^{2}}{n^{2}/4+\left(\sum_{k=1}^{n}t_{k}(z_{0})\right)^{2}}\\\
=\frac{n}{2}-\frac{(1-n/2m)n^{2}/4+(1-n/2m)\left(\sum_{k=1}^{n}t_{k}(z_{0})\right)^{2}}{n^{2}/4+\left(\sum_{k=1}^{n}t_{k}(z_{0})\right)^{2}}=\frac{n}{2}\left(1+\frac{1}{m}\right)-1.\end{split}$

Finally, we obtain that

$z_{0}\frac{f^{\prime\prime}(z_{0})}{f^{\prime}(z_{0})}=m^{-1}\left((1-m)\frac{n}{2}+m\left(\frac{n}{2}\left(1+\frac{1}{m}\right)-1\right)\right)=\frac{n}{m}-1,$

whenever $z_{0}$ is a maximum of $f^{\prime}$ such that $\sum t_{k}(z_{0})\neq
0$. It is easily seen that $(f^{\prime})^{m}=Q$ is a polynomial of degree
$n-m$. Thus

$\frac{z_{0}}{m}\frac{Q^{\prime}(z_{0})}{Q(z_{0})}=z_{0}\frac{f^{\prime\prime}(z_{0})}{f^{\prime}(z_{0})}=\frac{n}{m}-1.$

Since $Q$ attains its maximum modulus at the same point, $z_{0}$, as
$f^{\prime}$, it follows from the condition for equality in Bernstein’s
inequality that $f^{\prime}=cz^{n/m-1}$ (also, $n/m$ must be an integer). In
this case one verifies by direct computation that the inequality holds.
Otherwise,

$\sum_{k=1}^{n}t_{k}(z_{0})=0.$

In conclusion we obtain the desired inequality

$\lVert f^{\prime}\rVert=\lvert f^{\prime}(z_{0})\rvert=\frac{n/m}{2}\lvert
f(z_{0})\rvert\leq\frac{n/m}{2}\lVert f\rVert.$

It remains to consider the case when $P(z_{0})=0$. Then $z_{0}=a_{k}$ for some
$k$ and $s_{k}=1$, since otherwise $f^{\prime}(z_{0})=0$. Then the right-hand
side of

$\frac{f^{\prime\prime}(z)}{f^{\prime}(z)}=\frac{1}{m}\left((1-m)\frac{P^{\prime}(z)}{P(z)}+m\frac{P^{\prime\prime}(z)}{P^{\prime}(z)}\right),$

has a singularity at $a_{k}$ (unless $m=1$, in which case it is nothing but
the classical version of the Theorem), but the left-hand side does not since
$f^{\prime\prime}$ is continuous at $a_{k}$ and $f^{\prime}$ has a maximum.
Hence this cannot happen (unless $f^{\prime}\equiv 0$).

The lower bound $\|f^{\prime}\|\geq\frac{n/m}{2}\|f\|$ follows from Lemma 1
which generalises Turán’s inequality. ∎

######  Lemma 1.

Let $f(z)=c\prod_{k=1}^{N}(z-a_{k})^{s_{k}}$ with $s_{k}\geq 1$ and
$|a_{k}|\leq 1$. Then

$\|f^{\prime}\|\geq\frac{\sum s_{k}}{2}\|f\|.$

###### Proof.

By differentiating, we obtain

$f^{\prime}(z)=f(z)\sum_{k=1}^{N}\frac{s_{k}}{z-a_{k}}$

and the result follows if we can show that

$\left|\sum_{k=1}^{N}\frac{s_{k}}{z-a_{k}}\right|\geq\frac{\sum
s_{k}}{2}\quad\text{for }z\in\mathbb{T}.$

To see this, note that, with $|z|=1$,

$\displaystyle\text{Re}\sum_{k=1}^{N}s_{k}\frac{z}{z-a_{k}}$
$\displaystyle=\frac{1}{2}\sum_{k=1}^{N}s_{k}\left(\frac{z}{z-a_{k}}+\frac{\overline{z}}{\overline{z-a_{k}}}\right)$
$\displaystyle=\frac{1}{2}\sum_{k=1}^{N}s_{k}\frac{z(\overline{z}-\overline{a_{k}})+\overline{z}(z-a_{k})}{|z-a_{k}|^{2}}
$
$\displaystyle=\frac{1}{2}\sum_{k=1}^{N}s_{k}\frac{2-2\text{Re}(z\overline{a_{k}})}{1-2\text{Re}(z\overline{a_{k}})+|a_{k}|^{2}}\geq\frac{1}{2}\sum_{k=1}^{N}s_{k}.$

∎

From the proof of Theorem 1 we gather the following corollary.

######  Corollary 1.

Let

$f(z)=c\prod_{k=1}^{N}(z-a_{k})^{s_{k}}$

with $|a_{k}|=1$ and $s_{k}\geq 1$. Then $f^{\prime}$ is maximal on the unit
circle at any point $z$ for which $|f(z)|=\|f\|$ and in that case

$\left|\sum_{k=1}^{N}\frac{s_{k}}{z-a_{k}}\right|=\mathrm{Re}\sum_{k=1}^{N}s_{k}\frac{z}{z-a_{k}}=\frac{\sum_{k=1}^{N}s_{k}}{2}.$

Even though it is not difficult to conclude this result for polynomials using
Lax’s argument, it appears to the best of our knowledge that this observation
is new, even for the case where $f$ is a polynomial.

We now proceed to consider the case where $|a_{k}|\geq 1$.

###### Proof of Theorem 2.

Let $f^{\ast}(z)$ denote the reciprocal of $f$ defined via

$f^{\ast}(z)=\overline{c}\prod_{k}(1-\overline{a_{k}}z)^{s_{k}}.$

The reciprocal can equivalently be defined as an $m$th root of the reciprocal
of $P$. Indeed, in this case

$P(z)=f(z)^{m}=c^{m}\prod_{k=1}^{N}(z-a_{k})^{ms_{k}}$

whence

$P^{\ast}(z)=\overline{c}^{m}\prod_{k=1}^{N}(1-\overline{a_{k}}z)^{ms_{k}}.$

We see that $f^{\ast}(z)=P^{\ast}(z)^{1/m}$. Now it is easily verified that
for $z\in\mathbb{T}$, $|P(z)|=|P^{\ast}(z)|$ and
$|P^{\prime}(z)|\leq|(P^{\ast})^{\prime}(z)|$. The last inequality holds since
we are assuming that all the zeros of $P$ are exterior to $\mathbb{D}$. But
this implies that for $z\in\mathbb{T}$, we have $|f(z)|=|f^{\ast}(z)|$ and

$|(f^{\ast})^{\prime}(z)|=\left|\frac{1}{m}P^{\ast}(z)^{1/m-1}(P^{\ast})^{\prime}(z)\right|\geq\left|\frac{1}{m}P^{1/m-1}(z)P^{\prime}(z)\right|=|f^{\prime}(z)|.$

Note further that under the above assumptions,

$f^{\ast}(z)=\prod_{j}(z-a_{k_{j}})^{s_{k_{j}}}Q(z)$

with $|a_{k_{j}}|=1$ where the indices $k_{j}$ correspond to those $s_{k_{j}}$
which are non-integer and $Q$ is a polynomial all of whose zeros are inside
$\mathbb{D}$. We conclude that for any $|\zeta|=1$, it holds true that

$f(z)+\zeta f^{\ast}(z)=\prod_{j}(z-a_{k_{j}})^{s_{k_{j}}}\tilde{Q}(z),$

where $\tilde{Q}$ is some polynomial which has all its zeros on the unit
circle. To see this, note that

$\log\left|\frac{f^{\ast}}{f}\right|$

is subharmonic on $\mathbb{D}$ and superharmonic on
$\mathbb{C}\setminus\mathbb{D}$ while constantly equal to $0$ on the unit
circle. Hence $f+\zeta f^{\ast}$ has all its zeros on $\mathbb{T}$, the same
also being true for $\tilde{Q}$. Therefore we can apply Theorem 1 to the
function $g=f+\zeta f^{\ast}$ where $\zeta$ is chosen such that $\arg
f^{\prime}(z_{0})=\arg\zeta(f^{\ast})^{\prime}(z_{0})$ for a point $z_{0}$
satisfying $|f^{\prime}(z_{0})|=\|f^{\prime}\|$. But then

$\|g^{\prime}\|\geq|g^{\prime}(z_{0})|=|f^{\prime}(z_{0})|+|(f^{\ast})^{\prime}(z_{0})|\geq
2|f^{\prime}(z_{0})|=2\|f^{\prime}\|.$

Since we further have that

$\|g^{\prime}\|=\frac{\sum s_{k}}{2}\|g\|\leq\left(\sum s_{k}\right)\|f\|$

the result follows.

∎

##  3\. Concluding remarks

Theorem 1 is a direct generalisation of a result due to Szegő and Pólya that
first appeared in print in [6]. Theorem 2 generalises the Erdős–Lax
inequality. However, we believe that this result should be valid without
restrictions on the zeros in the exterior of the closed unit disc. In fact, we
conjecture the following.

######  Conjecture 1.

Let $P(z)=c\prod_{k=1}^{N}(z-a_{k})^{ms_{k}}$ where $c\in\mathbb{C}$,
$m\in\mathbb{N}$, $s_{k}\geq 1$, and $|a_{k}|\geq 1$. Then for any branch
$f(z)=P(z)^{1/m}$, we have

$\|f^{\prime}\|\leq\frac{\sum_{k=1}^{N}s_{k}}{2}\|f\|$

with equality if and only if $|a_{k}|=1$ for all $k$.

The reason that the techniques developed in this article does not seem to work
to prove this case is that for such functions $f$ it is not guaranteed that
$f+\zeta f^{\ast}$ from the proof of Theorem 2 is a fractional power of a
polynomial. Instead, we believe that new ideas are needed to prove this
conjecture.

## References

  * [1] Vitalii Vladimirovich Arestov.  On integral inequalities for trigonometric polynomials and their derivatives.  Izvestiya Rossiiskoi Akademii Nauk. Seriya Matematicheskaya, 45(1):3–22, 1981.
  * [2] A. Aziz and Q. G. Mohammad.  Simple proof of a theorem of Erdős and Lax.  Proceedings of the American Mathematical Society, 80(1):119–122, 1980.
  * [3] J. S. Christiansen and O. Rubin.  Chebyshev polynomials on lemniscates.  to appear, 2023.
  * [4] N. G. de Bruijn.  Inequalities concerning polynomials in the complex domain.  Nederl. Akad. Wetensch., Proc., 50:1265–1272 = Indagationes Math. 9, 591–598, 1947.
  * [5] M. Lachance, E. B. Saff, and R. S. Varga.  Inequalities for polynomials with a prescribed zero.  Mathematische Zeitschrift, 168(2):105–116, Jun 1979.
  * [6] P. Lax.  Proof of a conjecture of P. Erdős on the derivative of a polynomial.  Bulletin of the American Mathematical Society, 50(8):509 – 513, 1944.
  * [7] M. A. Malik.  On the Derivative of a Polynomial.  Journal of the London Mathematical Society, s2-1(1):57–60, 01 1969\.
  * [8] A. Mir, A. Hussain, and F. A. Wagay.  On the erdös-lax inequality and its various generalizations concerning polynomials.  Rendiconti del Circolo Matematico di Palermo Series 2, 72(3):2025–2037, Apr 2023.
  * [9] Q.I Rahman and G Schmeisser.  Lp inequalities for polynomials.  Journal of Approximation Theory, 53(1):26–32, 1988.
  * [10] P. Turán’.  über die ableitung von polynomen.  Compositio Math, 7:89–95, 1939.
  * [11] A. Zygmund.  A Remark on Conjugate Series.  Proc. London Math. Soc. (2), 34(5):392–400, 1932.


1055 Empirical study

# Beyond Visuals : Examining the Experiences of Geoscience Professionals With
Vision Disabilities in Accessing Data Visualizations

Nihanth W. Cherukuru e-mail<EMAIL_ADDRESS>David A. Bailey
<EMAIL_ADDRESS>Tiffany Fourment<EMAIL_ADDRESS>Becca
Hatheway<EMAIL_ADDRESS>Marika M. Holland<EMAIL_ADDRESS>Matt Rehme<EMAIL_ADDRESS>National Center for Atmospheric Research
(NCAR)
University Corporation for Atmospheric Research (UCAR)

###### Abstract

Data visualizations are ubiquitous in all disciplines and have become the
primary means of analysing data and communicating insights. However, the
predominant reliance on visual encoding of data continues to create
accessibility barriers for people who are blind/vision impaired resulting in
their under representation in Science, Technology, Engineering and Mathematics
(STEM) disciplines. This research study seeks to understand the experiences of
professionals who are blind/vision impaired in one such STEM discipline
(geosciences) in accessing data visualizations. In-depth, semi-structured
interviews with seven professionals were conducted to examine the
accessibility barriers and areas for improvement to inform accessibility
research pertaining to data visualizations through a socio-technical lens. A
reflexive thematic analysis revealed the negative impact of visualizations in
influencing their career path, lack of data exploration tools for research,
barriers in accessing works of peers and mismatched pace of visualization and
accessibility research. The article also includes recommendations from the
participants to address some of these accessibility barriers.

Human-centered computingVisualizationVisualization Design and evaluation
methods; Human-centered computingAccessibilityAccessibility technologies

Introduction The immense popularity and pervasiveness of data visualizations
have made them the defacto tools for data driven communication. However, this
sole reliance on visual encoding for data interpretation creates significant
accessibility barriers for people who are blind/vision impaired [19, 13]. The
impacts of inequities in information access are particularly amplified during
times of crisis, as evidenced during COVID-19 [25] emphasizing the urgent need
to improve the accessibility of data visualizations. Overall, accessibility
considerations in data visualizations are an under-explored area in
visualization research [19]. Prior work has emphasized the need for
visualization practitioners, accessibility researchers and people from the
blind/low vision community to work together to examine the experiences of
people with vision impairments as well as establish evidence-based guidelines
to tackle the accessibility barriers [19].

The current research seeks to understand the experiences of professionals who
are blind/vision impaired with accessing information presented through data
visualizations in geosciences, one of the least diverse STEM disciplines[9].
We conducted semi-structured, in-depth interviews with professionals who are
blind/vision impaired, exploring the visualization accessibility barriers at
various stages of their career.

The following are the main contributions of this work:

  * •

Providing a descriptive account of the lived experiences of geoscience
professionals who are blind/vision impaired

  * •

Identifying social and technical barriers to visualization accessibility in
geosciences

  * •

Providing recommendations for addressing some of the accessibility barriers
identified in the study and opportunities for accessible technology.

This study was designed to focus on the experiences of professionals from a
single field of study (geosciences) in order to isolate any domain specific
visualization considerations. However, this article covers the findings which
could be broadly applied to other STEM disciplines.

##  1 Related Work and Rationale

The predominant focus of the research community on effective visual encoding
of information inadvertently disenfranchises people with vision disabilities
[15, 14]. This limited access to graphical material can have a snowball effect
of exacerbating the overall under-representation of people with disabilities
in broad disciplines. For instance, a previous study by Butler et al. [7] has
identified that people with vision impairments deliberately chose fields of
study with fewer graphics, driven largely by the accessibility limitations
with graphical material. These accessibility barriers in turn significantly
reduced the career options and led to the disengagement of this population
with their field of study, contributing to the negative perceptions of people
in the field about the capacity of people with vision impairments [3, 7].
Geosciences is one such STEM discipline which has been identified as one of
the least diverse, with 75% fewer individuals with disabilities represented in
the workforce than in the general population [9].

A study by Atchinson and Libarkin [3] investigating the perceptions of
geoscience professionals about accessibility found that practitioners
perceived geosciences through a vision-centric lens, thereby assuming people
with visual impairments to be least viable for geoscience careers among other
types of disabilities. These perceptions of ability are reinforced through
promotional materials and messaging (visualizations being one of them) that
may be actively discouraging individuals with disabilities from considering
geoscience as a viable career option [3]. To this end, providing access to
information and programs was quoted to be one of the imperatives for
successful inclusion of individuals with visual impairments in atmospheric
science careers [10]. Lastly, although this discussion has largely focused on
the importance of accessibility for people with vision impairments, non-visual
techniques of data representation have been known to be beneficial for people
with intellectual and developmental disabilities [26] as well as students
without disabilities [2] highlighting the broader impacts of inclusive design.

Disability studies is an interdisciplinary field focused on the study of
disability through social, cultural and political perspectives. The field has
played a significant role in defining the rhetoric, language and methods
through which to understand the experiences of people with disabilities [18].
A large portion of accessibility research articles in the visualization
domain, published over the last two decades, focused on the evaluation of
technologies and approaches to address specific accessibility barriers of data
visualizations through technological interventions [14]. This approach to
accessibility, with a focus on actionable challenges and opportunities to
address disability through technological interventions, draws its insights
from the medical modal of disability [18]. In contrast, the social model
defines disability as a consequence of exclusion brought forth by structural
and environmental factors [17]. Both these perspectives could help inform
research and new areas of inquiry.

Drawing from these concepts, the participatory design process for the creation
of accessible content is an approach to design which engages people with
disabilities (stakeholders) as equal participants throughout the design
process from the outset (i.e, need-finding and problem definition stage) [18,
16]. This approach emphasises the lived experiences of stakeholders and
reduces the likelihood of developing ”disability dongles”, a term coined by
Liz Jackson to describe a ”well intended elegant yet useless solution to a
problem we never knew we had” [16]. In their call to action for inclusive data
visualizations, Marriot et al. [19] identify the topic of understanding how
people with disabilities currently access data and use data visualizations to
be essential and largely missing from literature. They also identify that
evidence based guidelines to facilitate the creation of accessible content is
lacking, even among the data visualization practitioners who recognize the
importance of accessibility.

The current research study seeks to address the aforementioned shortcoming by
conducting interviews with professionals who are blind/vision impaired, to
understand their experiences accessing data visualizations in geosciences.
While there have been a handful of studies exploring this topic [7, 24, 22],
the current work significantly differs from previous works in that we
investigate this topic with a socio-technical lens, focus on a specific STEM
discipline (geosciences) and do not limit the scope to any one particular
visualization technique.

##  2 Study Design

We conducted in-depth, semi-structured interviews with professionals in
geosciences who are blind/vision impaired exploring the research question:
What are the experiences of students and professionals who are blind/vision
impaired in geosciences in accessing information presented through data
visualizations?

###  2.1 Participants

The seven interviewees were individuals in geosciences who have experience
with data visualizations either in their current work or through their
education. They were recruited through professional contacts, snowball
sampling and advertisements through geoscience listservs. The participants
were offered a $50 honorarium for their participation in a 50-60 minute
interview. All the participants are legally blind and four of them were blind
since birth. Four of the participants identified themselves as female and
three as male. The participants’ ages ranged between 25 and 65 years. All the
participants assessed themselves to be highly proficient with computers. Five
of the participants had a graduate degree and two of them had a highest level
of bachelor’s degree. Six of the participants currently worked in geosciences.

###  2.2 Data Collection

The interviews were conducted using Zoom video conferencing between June 2021
- March 2022. The interview questions were shared with the participants prior
to the interview. The interview started with basic questions followed by open-
ended questions regarding participants’ current occupation and technologies
used to access visual graphics. This was followed by open-ended questions
investigating three stages of their career: (i) K-12, (ii) Higher Education,
and (iii) Professional. The questions explored the importance of data
visualizations at each stage of their career, the impact of visual material on
their career path, and the techniques used to address any barriers that they
have encountered. They were also asked about their favorite visualization
techniques and the reasons for that, as well as areas in visualization
accessibility research that they felt needed to be addressed (Note: The
interview guide is provided as a supplemental material accompanying this
article.) The interviews were audio recorded and transcribed for data
analysis. After each interview, preliminary analysis was conducted to gather
insights and check the relevance of the data in addressing the research
questions. Thematic saturation [23] was observed by the sixth interview.

###  2.3 Data Analysis

####  2.3.1 Theoretical Assumptions

The primary objective of the study was to draw insights on data visualization
accessibility from the participants’ experiences. This requires both the
faithful reflection on the participant’s narrative as well as the researcher’s
interpretive analysis to identify themes relevant to the research question.
Consequently, reflexive thematic analysis (RTA) [4, 5, 6] was determined to be
the best approach for data analysis. This study applied a constructionist
epistemology with an experiential orientation. This approach acknowledges the
importance of both recurrence and meaningfulness of a communicated experience
while enabling the researcher to preserve the subjectivity in a participant’s
experience[8]. Given the data-driven, exploratory nature of this study, a
predominantly inductive approach was adopted for the analysis. Lastly, in
order to focus on the experiences and the reality of the participants,
semantic coding (descriptive analysis) was prioritized over latent coding
(interpretive analysis) whenever possible acknowledging that a certain degree
of latent coding was necessary in order to identify themes relevant to the
research questions.

####  2.3.2 Analytical Process

The six phase, non-sequential, recursive process for RTA proposed by Braun and
Clarke [4, 8] was followed to analyse the transcribed interviews. Following
the familiarization phase, descriptive and interpretive open-codes were
generated for each piece of information that were relevant to the research
questions. Afterwards, these initial codes were reviewed and patterns of codes
were examined for prospective themes using affinity diagramming. This
recursive process was applied for the first two interviews and a list of
preliminary candidate themes was produced. The remaining interviews were then
coded by comparing with the existing candidate themes, reviewing both the
relevance of the new codes and the themes in answering the research questions.
The candidate themes thus produced were then reviewed following Patton’s dual
criteria for judging categories [21, 8] and a final set of themes and sub-
themes were generated. Following the recommendations of Nowell et al. [20]
validity of this approach was determined by maintaining a clear audit trail,
peer debriefing, transferability documented through thick descriptions
(although only a summary is provided in this article).

##  3 Results and Discussion

The first iteration resulted in around 75 codes which were later distilled
down to 23 unique codes by merging similar codes and removing duplicates.
These codes were organized into 5 key findings and 3 recommendations.

###  3.1 Key Findings

####  3.1.1 A deterrent

A common observation that came up repeatedly in all the interviews was that
visualization accessibility continues to be a significant hindrance. While it
did not stop them from continuing in the field, it posed a significant
challenge in their careers. Participants made statements such as:

> ”I don’t want to say it’s career changing, because I didn’t change my
> career, but it’s been career challenging.” - P1

> ”I do feel like I’m missing out on a lot. In not like looking at, like, the
> data visualizations, or understanding them.” - P6

This theme revealed that accessibility limitations of visualizations
inadvertently served as a deterrent for people with vision impairment from
pursuing a career in geosciences, potentially contributing to the training
barriers to inclusion identified in a previous study [3].

####  3.1.2 Limited tools for research

Some of common visualization techniques used in geosciences are line charts
for time series data, contour plots and geographic heat maps for 2D scalar
data (E.g. Temperature, Pressure) and, quiver plots for vector fields (E.g.
wind, vorticity). Participants also mentioned coming across occasional 3D
visualizations such as iso-surfaces. These visualizations are used both in
research as well as education and outreach. While there have been significant
technological advancements enabling the use of alternate modalities for data
exploration such as tactile graphics, participants found that these approaches
had limitations which prevented their application in research. The common
limitations mentioned were: inability of the existing techniques to accurately
encode the information at a resolution needed for research, lack of
refreshability, slow production time, high cost and lack of on-demand
autonomous techniques for independent exploration. Consequently, participants
stated that seeking assistance from a sighted person with subject matter
knowledge was currently the most effective means to access data
visualizations. This reliance on sighted assistance limited their ability to
participate in a scientific discourse as two participants stated:

> ”One of the workarounds for this was collaboration and that worked well. But
> nonetheless, I was constrained in what I could independently do to duplicate
> data.” - P3

> ”I was completely reliant on them, to tell me what their data said, Now,
> that’s actually useful. It’s good to have it if you want the student to tell
> you what the data said. But you also want to be able to check that
> independently.” - P2

One approach to independent data exploration that participants found to be
useful and promising was sonification, where data is encoded in audible
frequencies [11]. However, their application was limited to time series plots
and access to multi-dimensional data, vector fields and comparative
visualizations remain an ongoing challenge.

####  3.1.3 Accessibility barriers in published work

This theme captures the barriers identified by the participants in accessing
the visualizations published by peers in presentations, posters and journal
articles. Although there exist guidelines [1] for including alternate textual
descriptions (alt text) in images, participants mentioned that a majority of
visualizations in academic works either lack alt text or have inadequate
description for them to comprehend the visuals being presented. This puts the
onus on the person with the disability to reach out to the presenter/author,
which might not always be feasible and creates a barrier for their
participation in the scientific discourse. Participants made statements such
as:

> ”Don’t even get me started on conferences … conferences are the places where
> I have the most difficulty with data visualizations.” - P4

> ”I’m on a crusade right now here to help this institution make its all its
> documents more accessible, including Excel, mostly in the office suite,
> Excel, PowerPoint, images and PowerPoint with alt text.” - P1

> ”Some of those constraints about looking at other people’s data were also
> quite serious. So things like seminars, and so on were and are still
> problematic.” -P2

Lundgard et al. [16] mention the awareness of accessibility guidelines as one
of the considerations for accessible visualizations design. This theme
revealed the lack of awareness in academia not only regarding the guidelines
but also the commonly adopted best practices for graphics accessibility
adopted by industry such as alt text.

####  3.1.4 Degree of impact vs career stage

Participants who were in their late career mentioned that they received
sighted assistance through staff employed by their institution or funded
through research grants both of which are not readily available for graduate
students and early career professionals. The same participants also mentioned
that data analysis was a significant part of job duties for people in their
early career stages. Although the concept of data visualizations was
introduced in K-12 education, participants were expected to produce
visualizations in their graduate school. While they managed to fulfil course
requirements through alternate accommodations and creative ways to demonstrate
their understanding of core concepts, the lack of tools for independent data
exploration implicitly influenced their subsequent career choices.
Participants made statements such as:

> ”I was being nudged into areas where this [visualization] was less of a
> limitation. So the more theoretical, and the more computational … It
> [visualization] was a constraint. And I would have had a wider choice of
> topics and styles of accessing data and working with data had it been
> available. ” - P2

> ”The lack of data visualization tools definitely made it harder. It
> influenced the field I chose in the end.” - P5

Butler et al. [7] have investigated the state of accessible graphics in higher
education and have identified significant issues impacting the enrollment and
retention of students with vision impairments in STEM fields. The current
theme has identified similar experiences faced by students in geosciences and
highlights the impact of the higher education experience in determining the
retention of students in the field.

####  3.1.5 Technological advancements: a blessing or a barrier?

Reflecting on the evolution of tools and techniques to access visualizations,
the participants ascertained the overall positive impact of technological
advancements in improving the accessibility barriers with statements such as:

> ”As far as data visualization goes, they’ve [technological advancements]
> helped immeasurably. This interview was actually quite useful to make me
> think about data access now versus say, my early graduate school, and the
> case has immeasurably improved.” -P2

However, the broader technological advancements have also contributed to the
increasing complexity of visualization tools and techniques, which exacerbated
accessibility problems as one participant stated:

> ”Technology has enabled more flashy data visualizations, which the
> scientific community has increasingly relied on. This has caused a migration
> of papers and presentations to become more visual, which leaves a lot of
> people out. -P6”

This theme revealed that accessibility considerations have not received
equivalent attention compared to mainstream visualization techniques. This
mismatched pace could further widen the accessibility gap and highlights the
importance and urgency of accessibility research in data visualizations.

###  3.2 Recommendations

In addition to the above themes, three recommendations emerged from the
interviews regarding strategies to address some of the barriers identified by
the participants:

##### Raise accessibility awareness.

The first recommendation addresses the absence of basic accessibility features
such as alternate textual descriptions in most of the works published in
academia which was mentioned by almost all the participants who used screen-
readers. It also emphasises the importance of understanding the accessibility
guidelines and adopting the current best practices to improve the
accessibility of graphics in academic works. Further, given that most of the
recommendations for alternate textual descriptions do not necessarily
translate to scientific visualizations, it is important to determine the most
effective way to describe data visualizations by consolidating the existing
guidelines from different sources and/or building on them.

##### Provide access to the underlying data.

Many participants in this study devised their own software and techniques to
interpret data. Some of the most commonly used approaches, such as the use of
a braille refreshable display to scan tabular data and code snippets to sonify
time-series data, are only usable if they are able to access the underlying
data used to generate the visualizations. To this end, participants identified
that providing access to the underlying data would be a significant step
towards improving the accessibility of visualization requiring minimal effort
from the presenter.

##### Design inclusive technologies with a multi-sensory approach.

Visualizations are without doubt one of the most effective means of
communicating information. The main limitation of visualizations is their
predominant reliance on a single sensory modality. This last recommendation
acknowledges human diversity and encourages the consideration of more than one
sensory modality in representing data and information. Even among people with
disabilities, the wide spectrum of vision disabilities necessitates solutions
that do not constrain individuals to use a single approach to interaction.

##  4 Summary

We examined the experiences of geoscience professionals who are blind/vision
impaired with accessing data visualizations, using reflexive thematic
analysis. The study revealed that visualization access continues to be a
significant barrier in their careers. The impact was more pronounced for
students and early career professionals and these barriers often influenced
their career decisions and limited their options. The study also revealed
social barriers such as difficulties in accessing visualizations appearing in
academic works due to the lack of awareness among peers regarding the common
best practices for making graphics accessible; and technical barriers such as
lack of accessible data exploration tools for research and underscored the
mismatched pace of visualization and accessibility research. Lastly,
recommendations were provided by the participants highlighting the importance
of both technological development and attitudinal shifts needed to address the
accessibility barriers. Although the focus of this study was on professionals
from geosciences, the findings reported in this article are more generally
applicable to any other field and reinforce some of the observations reported
in previous studies [13, 16, 19].

Additional areas that are worth exploring in future studies based on the data
and insights gathered are listed below.

  * •

Visualizations have both exploratory (visualizations used in research) and
explanatory (visualizations used for communication and story telling)
applications [12]. Some of the technological limitations mentioned by the
participants seemed to depend heavily on the purpose of visualization. Any
study investigating accessibility considerations with data visualizations need
to factor in this distinction.

  * •

There also seems to be a difference between techniques used to absorb
information from data visualizations and, techniques used to create
visualizations to communicate with others. There are very few content creation
tools that are accessible for people who are blind/vision impaired and this
needs to be investigated further for identifying areas of improvement.

  * •

3D data exploration continues to be a major limitation for people who are
blind/vision impaired. Multi-modal visualization techniques using haptics in
immersive environments such as augmented, virtual and mixed reality were
mentioned by one of the participant to be promising for exploring 3D data,
based on their experience with a research prototype. User-interface design in
immersive environments is a rapidly growing area of research. Future studies
exploring the accessibility applications of these emerging technologies in
data visualizations could help address the gaps identified in this and other
related works.

We hope that this work helps motivate both researchers and professionals in
wider STEM disciplines to develop more inclusive data visualizations.

###### Acknowledgements.

We would like to thank the participants for their time in sharing their
experiences and recommendations. We would also like to thank Dr. Bhoomika
Bhagchandani for her valuable guidance on the study design and thematic
analysis. We sincerely appreciate the feedback received from the anonymous
reviewers who helped us improve this article. This work was supported in part
by the 2020 University Corporation For Atmospheric Research President’s
Strategic Initiative Fund and the National Center for Atmospheric Research,
which is a major facility sponsored by the National Science Foundation under
Cooperative Agreement 1852977.

## References

  * [1] Web Content Accessibility Guidelines (WCAG) 2.1, June 2018.
  * [2] P. Asher.  Teaching An Introductory Physical Geology Course To A Student With Visual Impairment.  Journal of Geoscience Education, 49(2):166–169, Mar. 2001. doi: 10 . 5408/1089-9995-49 . 2 . 166
  * [3] C. L. Atchison and J. C. Libarkin.  Professionally held perceptions about the accessibility of the geosciences.  Geosphere, 12(4):1154–1165, Aug. 2016. doi: 10 . 1130/GES01264 . 1
  * [4] V. Braun and V. Clarke.  Successful qualitative research: a practical guide for beginners.  SAGE, Los Angeles, 2013.  OCLC: ocn811733656.
  * [5] V. Braun and V. Clarke.  Reflecting on reflexive thematic analysis.  Qualitative Research in Sport, Exercise and Health, 11(4):589–597, Aug. 2019. doi: 10 . 1080/2159676X . 2019 . 1628806
  * [6] V. Braun and V. Clarke.  One size fits all? What counts as quality practice in (reflexive) thematic analysis?  Qualitative Research in Psychology, 18(3):328–352, July 2021. doi: 10 . 1080/14780887 . 2020 . 1769238
  * [7] M. Butler, L. Holloway, K. Marriott, and C. Goncu.  Understanding the graphical challenges faced by vision-impaired students in Australian universities.  Higher Education Research & Development, 36(1):59–72, Jan. 2017\. doi: 10 . 1080/07294360 . 2016 . 1177001
  * [8] D. Byrne.  A worked example of Braun and Clarke’s approach to reflexive thematic analysis.  Quality & Quantity, June 2021. doi: 10 . 1007/s11135-021-01182-y
  * [9] I. G. Carabajal, A. M. Marshall, and C. L. Atchison.  A Synthesis of Instructional Strategies in Geoscience Education Literature That Address Barriers to Inclusion for Students With Disabilities.  Journal of Geoscience Education, 65(4):531–541, Nov. 2017. doi: 10 . 5408/16-211 . 1
  * [10] I. Durre.  Untapped Career Opportunities for Persons With Visual Impairments.  Bulletin of the American Meteorological Society, 89(7):987–996, July 2008. doi: 10 . 1175/2008BAMS2447 . 1
  * [11] T. Hermann, A. Hunt, and J. G. Neuhoff, eds.  The sonification handbook.  Logos Verlag, Berlin, 2011.  OCLC: ocn771999159.
  * [12] N. Iliinsky and J. Steele.  Designing data visualizations: intentional communication from data to display.  O’Reilly, Beijing Köln, 2011.
  * [13] S. Joyner, A. Riegelhuth, K. Garrity, Y.-S. Kim, and N. W. Kim.  Visualization Accessibility in the Wild: Challenges Faced by Visualization Designers.  p. 29. ACM, New Orleans, LA, USA, May 2022. doi: 10 . 1145/3491102 . 3517630
  * [14] N. W. Kim, S. C. Joyner, A. Riegelhuth, and Y. Kim.  Accessible Visualization: Design Space, Opportunities, and Challenges.  Computer Graphics Forum, 40(3):173–188, June 2021. doi: 10 . 1111/cgf . 14298
  * [15] B. Lee, E. K. Choe, P. Isenberg, K. Marriott, J. Stasko, and T.-M. Rhyne.  Reaching Broader Audiences With Data Visualization.  IEEE Computer Graphics and Applications, 40(2):82–90, Mar. 2020\. doi: 10 . 1109/MCG . 2020 . 2968244
  * [16] A. Lundgard, C. Lee, and A. Satyanarayan.  Sociotechnical Considerations for Accessible Visualization Design.  2019\.  Publisher: arXiv Version Number: 1. doi: 10 . 48550/ARXIV . 1909 . 05118
  * [17] A. Lundgard and A. Satyanarayan.  Accessible Visualization via Natural Language Descriptions: A Four-Level Model of Semantic Content.  IEEE Transactions on Visualization and Computer Graphics, 28(1):1073–1083, Jan. 2022. doi: 10 . 1109/TVCG . 2021 . 3114770
  * [18] J. Mankoff, G. R. Hayes, and D. Kasnitz.  Disability studies as a source of critical inquiry for the field of assistive technology.  In Proceedings of the 12th international ACM SIGACCESS conference on Computers and accessibility - ASSETS ’10, p. 3. ACM Press, Orlando, Florida, USA, 2010. doi: 10 . 1145/1878803 . 1878807
  * [19] K. Marriott, B. Lee, M. Butler, E. Cutrell, K. Ellis, C. Goncu, M. Hearst, K. McCoy, and D. A. Szafir.  Inclusive data visualization for people with disabilities: a call to action.  Interactions, 28(3):47–51, Apr. 2021. doi: 10 . 1145/3457875
  * [20] L. S. Nowell, J. M. Norris, D. E. White, and N. J. Moules.  Thematic Analysis: Striving to Meet the Trustworthiness Criteria.  International Journal of Qualitative Methods, 16(1):160940691773384, Dec. 2017. doi: 10 . 1177/1609406917733847
  * [21] M. Q. Patton and M. Q. Patton.  Qualitative evaluation and research methods.  Sage Publications, Newbury Park, Calif, 2nd ed ed., 1990.
  * [22] V. Potluri, T. E. Grindeland, J. E. Froehlich, and J. Mankoff.  Examining Visual Semantic Understanding in Blind and Low-Vision Technology Users.  In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, pp. 1–14. ACM, Yokohama Japan, May 2021\. doi: 10 . 1145/3411764 . 3445040
  * [23] B. Saunders, J. Sim, T. Kingstone, S. Baker, J. Waterfield, B. Bartlam, H. Burroughs, and C. Jinks.  Saturation in qualitative research: exploring its conceptualization and operationalization.  Quality & Quantity, 52(4):1893–1907, July 2018. doi: 10 . 1007/s11135-017-0574-8
  * [24] A. Sharif, S. S. Chintalapati, J. O. Wobbrock, and K. Reinecke.  Understanding Screen-Reader Users’ Experiences with Online Data Visualizations.  In The 23rd International ACM SIGACCESS Conference on Computers and Accessibility, pp. 1–16. ACM, Virtual Event USA, Oct. 2021\. doi: 10 . 1145/3441852 . 3471202
  * [25] A. F. Siu, D. Fan, G. S.-H. Kim, H. V. Rao, X. Vazquez, S. O’Modhrain, and S. Follmer.  COVID-19 highlights the issues facing blind and visually impaired people in accessing data on the web.  In Proceedings of the 18th International Web for All Conference, pp. 1–15. ACM, Ljubljana Slovenia, Apr. 2021. doi: 10 . 1145/3430263 . 3452432
  * [26] K. Wu, E. Petersen, T. Ahmad, D. Burlinson, S. Tanis, and D. A. Szafir.  Understanding Data Accessibility for People with Intellectual and Developmental Disabilities.  In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, pp. 1–16. ACM, Yokohama Japan, May 2021\. doi: 10 . 1145/3411764 . 3445743


# Extending Contrastive Learning to Unsupervised Coreset Selection

Jeongwoo Ju  KAIST  Heechul Jung  Kyungpook National University  Yoonju Oh
Kyungpook National University  Junmo Kim  KAIST

###### Abstract

Self-supervised contrastive learning offers a means of learning informative
features from a pool of unlabeled data. In this paper, we delve into another
useful approach—providing a way of selecting a core-set that is entirely
unlabeled. In this regard, contrastive learning, one of a large number of
self-supervised methods, was recently proposed and has consistently delivered
the highest performance. This prompted us to choose two leading methods for
contrastive learning: the simple framework for contrastive learning of visual
representations (SimCLR) and the momentum contrastive (MoCo) learning
framework. We calculated the cosine similarities for each example of an epoch
for the entire duration of the contrastive learning process and subsequently
accumulated the cosine-similarity values to obtain the coreset score. Our
assumption was that an sample with low similarity would likely behave as a
coreset. Compared with existing coreset selection methods with labels, our
approach reduced the cost associated with human annotation. The unsupervised
method implemented in this study for coreset selection obtained improved
results over a randomly chosen subset, and were comparable to existing
supervised coreset selection on various classification datasets (e.g., CIFAR,
SVHN, and QMNIST).

##  1 Introduction

Figure 1: Our coreset based on SimCLR performance on three datasets (CIFAR10,
SVHN, QMNIST). The horizontal line depicts the stride as a percentage of the
training set size. The dotted and solid lines represent the test accuracy of
the classification network trained with a randomly chosen subset and our
coreset, respectively. The size of both subsets is 30% of each dataset, i.e.,
15,000, 22,500, and 18,000 samples for CIFAR10, SVHN, and QMNIST,
respectively. At a stride of 0%, our coreset achieves high performance over a
random subset. As the stride moves to the right, the resulting accuracy
decreases. This signifies that the corresponding examples exhibit a low
coreset score. We use ResNet18 architecture for both the coreset selection and
classification.

Deep learning-based methods have been highly effective in performing computer
vision tasks such as image classification[19], object detection[16], and
semantic segmentation[8]. However, these methods generally require large
amounts of data to produce accurate results; in particular, human annotation,
an essential part of supervised learning, can be considerably time consuming
and costly to implement. To address this problem, we selected a subset of the
entire training set: given a training set ${D}$, we selected a subset
${D}^{{}^{\prime}}\subset{D}$. This process requires identifying useful
samples that can collectively represent the entire dataset. In this case, the
underlying assumption is that not all samples contribute equally to a given
task. In other words, each sample has a different level of contribution to the
task—mostly referred to in the literature as high-contribution examples of the
coreset. This assumption is a necessary improvement over a randomly selected
subset (This topic has been thoroughly reviewed [27], [4].). These results can
be divided into two groups: supervised and unsupervised coresets (described in
detail in Section 2).

One of the limitations of employing supervised coresets is that they do not
reduce the annotation cost because they require a fully labeled dataset to
execute successfully. Nonetheless, supervised coresets have been extensively
studied, whereas unsupervised coresets have not received sufficient attention.
Therefore, we aimed to select a coreset without human intervention, as is
often the case when labels are not available. To the best of our knowledge,
only Valvano et al. [30] have addressed the problem of unsupervised coreset
selection. However, their method does not function desirably with large
datasets. Moreover, their results were declared not robust; further research
would be required to prove the usability of this method. Through this study,
we aimed to perform unsupervised coreset selection using a deep learning
approach. It must be noted that a coreset provides more information about the
parent dataset than a non-coreset does to complete the target task (e.g.,
classification). Therefore, it is reasonable to suppose that information-
related metrics and target tasks are required to enable coreset selection.
Because it is impractical to build a target task in the absence of a label, we
adopted an alternative method of building a task, namely, self-supervised
learning.

Figure 2: Overall flowchart of our coreset selection approach. Example of our
coreset selection: During contrastive learning (top left), the cossim value is
calculated (top middle), and its negative value is accumulated (top right)
simultaneously for each example. $N$ and $M$ represent the size of the
training set and the number of epochs, respectively. After the learning is
completed, the training data is sorted in increasing order of the coreset
score (bottom). Lastly, the coreset with a specified size (e.g., eight) is
selected.

Self-supervised learning has arisen for processing unlabeled data. With the
objective of learning useful features from a pool of unlabeled data, self-
supervised learning generally demands the description of a pretext task within
the loss function; good performance depends on the definition of the pretext
task. Hence, several researchers have attempted to devise a pretext task to
enhance downstream task performance. However, it was recently reported that
contrastive learning offers improved performance over existing pretext tasks.
As its name implies, intermediate representations of the deep model are
created to maximize the cossim(cosine similarity) between different views of
the same example. As it is a state-of-the-art method, we decided to focus on
contrastive learning in our study. Moreover, with respect to information-
related metrics, we conjectured that the cossim would be a valid metric for
coreset selection. This is because we assumed that the coreset would exhibit a
low cossim value (in Section 3). As depicted in Fig. 2, we stored the negative
accumulated cossim values for each example at the end of the epoch. We
assigned the saved cossim value as the coreset score and sorted the examples
according to their scores. Finally, we chose a final coreset with a specified
size. Subsequently, we compared the classification accuracy of the model
trained from our coreset examples and from the randomly chosen examples
separately by assigning equal sizes to the subsets. The empirical results
showed that the accuracy of our coreset was more accurate than that of the
random subsets (Section 4). Notably, a low cossim(=high core-set score) value
means that its examples is not easy for contrastive learning to predict
compared with other examples with large cossim values. Therefore, we inferred
that the particular coreset has a characteristic that exhibits a low cossim
value. This empirical evidence is in line with forgetting events [27]—an event
where the sample is wrongly classified during training. In a previous study,
it was discovered that an example with a large number of forgetting events had
a high level of contribution in supervised learning. The opposite is also
true, i.e., the lower the number of forgetting events, the smaller the
contribution of the corresponding example). Our work bridges the gap between
contrastive learning and coreset selection and provides a new direction for
further research. In this paper, we demonstrate the possibility of
transferring an abstract space built from contrastive learning to identify
coreset examples. Our contributions to the literature are summarized as
follows:

  * •

To the best of our knowledge, our study is the first of its kind to
substantiate that self-supervised learning is a highly suitable method for
selecting a subset to generalize deep neural networks.

  * •

To measure the information of an example, we establish a coreset score as a
function of cossim.

Our work is expected to provide guidance on the use of unsupervised coresets.
Our code can be accessed from the following website:
111https://github.com/heechul-knu/self_supervised_learning_without_data.

##  2 Related Work

In our study, we bridged the gap between contrastive learning and unsupervised
coreset selection. To establish context, we provide a brief history of self-
supervision and revisit previous studies on coresets.

Self-supervised learning Self-supervised learning aims to define a pretext
task to enable a model to learn useful features without the presence of
labeled data. In this case, downstream tasks (classification, detection, etc.)
serve as a tool to demonstrate the significance of the pretext task. In the
last decade, numerous researchers have reported that their pretext tasks were
more effective in downstream tasks. The major works are as follows: Context
prediction [5] assigns a task to a model to predict the position of a patch
relative to that of the center-cropped patch. Solving a jigsaw puzzle [24]
recasts a self-supervised problem by learning the permutation index of cropped
patches. The counting task [25] is based on the concept that the sum of visual
primitives from each patch must be equal to that of an entire image.

The above-mentioned studies were mainly based on the use of image patches.
Unlike this stream, rotation [7], PIRL [21], MoCo v1 [12], SimCLR [2], and
MoCo v2 [3] used an entire image as input. The prediction angle at which an
image is rotated is technically sounds and straightforward [7]. Defining
positive and negative pairs and maximizing the cossim value of positive pairs
are key factors in contrastive learning, as mentioned in the MoCo v1-2 [12][3]
and SimCLR [2] papers. Both SimCLR and MoCo describe a positive pair as two
augmented versions of the same example. SimCLR realizes contrastive learning
via a multilayer perceptron (MLP) projection network, heavy data augmentation,
and layer-wise adaptive rate scaling (LARS) optimizer [35]. On the other hand,
MoCo mainly relies on a building dictionary equipped with a momentum encoder.

Coreset Selection A study on coresets was conducted with the goal of achieving
data-processing efficiency with a given label (i.e., achieving faster learning
and consuming lesser storage sapce) Here, the term coreset refers to the most
informative set of examples identified. To aid the reader’s understanding, we
categorized existing studies into four groups depending on the feature type
and the presence or absence of a label: 1) hand-crafted feature + label:
Bayesian inference [1], SVM [28], Bayesian logistic regression [15],
submodular functions [29][32][31], k-means and k-median clustering [11][10],
Gaussian mixture models [20], optimization framework [6], 2) hand-crafted
feature + no label: Submodular optimization for speech data[23], 3) learnable
feature + label: forgetting events [27], selection via proxy(SVP) [4]. 4)
learnable feature + no label: Unsupervised data selection [30]—our work falls
into this category.

In particular, a few attempts focusing on the application of deep neural
networks (DNNs) were made until Toneva et al.[27] discovered a correlation
between coresets and forgetting events. Furthermore, Coleman et al. [4]
extended this work by employing a proxy model for selection. Here, the target
model was not necessarily identical to the model used for selecting the
coreset, but a smaller network could achieve this selection. Valvano et al.
[30] considered an unsupervised coreset selection; their work was based on a
convolutional variational autoencoder (CVAE) to develop an embedding space in
which the distance between successive features was calculated. Their work was
limited to a small amount of data; furthermore, the computation was
intractable, as a large amount of data is presented.

##  3 Unsupervised Coreset Selection

The objective of unsupervised coreset selection is to identify coreset
examples in the absence of labels. Here, we recalled that a coreset provides
more information than a non-coreset. In other words, an information-related
metric must be established to measure the coreset score. Because contrastive
learning is the fundamental idea of our work, the cossim can be readily
obtained. Therefore, we were able to prove that cossim is a valid metric to
represent the coreset. Our hypothesis is that cossim plays a major role in
measuring the coreset score. This hypothesis was made based on visual
inspection, which is presented in Section 5.3. First, to confirm our
hypothesis, we present our observations.

Figure 3: Cossim values for each epoch. Cossim of two examples (CIFAR10) with
the highest and lowest coreset scores based on SimCLR: P1(blue line) and
P2(red line). Note that P1 barely approaches 1 for the entire duration of the
learning process.

Observations. During contrastive learning, we calculated the coreset score as
described in Alg. 1. The resulting array $A$ corresponds to a list of index
examples in the training set. Provided that our score represents the coreset
adequately, the performance increases according to the coreset score. To
provide a more concrete proof, we built a coreset score based on SimCLR and
trained a classification network with cross-entropy loss and a subset of the
training set that is described in Eq. 1. The stride $s$ ranges from 0% to 70%
of the size of the training set and the subset size $L$ is set to 30% of the
size of the training set. $\bm{x}_{k},\bm{y}_{k}$ represents a pair of $k$-th
examples and targets, and $\ell(\cdot)$ is the cross-entropy loss. For
example, in the case of CIFAR10, the sizes of the training and testing set are
50,000 and 10,000, respectively. We chose 30% of the training set (15,000
examples) according to the coreset score and stride. In addition, we set the
stride $s$ as follows: $s\in\\{0,5000,10000,15000,20000,25000,30000,35000\\}$.
Following this, we trained the classification network with these subset
samples and evaluated the performance with the testing set.

$Loss=\frac{1}{L}\sum_{k}\ell(\bm{x}_{k},\bm{y}_{k}),\;\;\;k\in A[s:s+L].$ (1)

Fig. 1 illustrates the performance trend for the three datasets (CIFAR10,
SVHN, and QMNIST). A clear correlation can be observed between the test
accuracy and stride. As the stride increases (and the coreset score
decreases), we observed that the performance of our coreset tends to degrade
almost linearly across the datasets, and at approximately stride=20%, it
descends below that of the randomly generated subset. This observation
emphasized the fact that the examples that provided more information compared
with the others exhibited high coreset scores, and vice versa (the examples
that were not too informative had a low coreset score). Additionally, we
observed that cossim adequately represents the information-related metric
required to measure the score. Hence, it was inferred that our score
effectively sorted the examples according to the coreset. On the contrary, our
coreset performance would exhibit a uniform trend across strides in a failure
case.

.

initialize metric $M[k]=0,k\in[n]$

input: subset size $L$

# calculate the coreset score

while not constrastive learning done do

for all $k\in[n]$ do

$M[k]=M[k]-cossim(z_{i}(k),z_{j}(k))$

# $z_{i},z_{j}$ are a positive pair of latent variables for the k-th example

$A=$argsort($M$, ascending order)

return $A[0:L]$ # resulting coreset

Algorithm 1 Identify coreset

Tracking cossim. It is worth noting that averaging cossim (cossim
accumulation) rather than using a single cossim is an efficient strategy.
After SimCLR learning on CIFAR10 was completed, we selected two examples whose
mean cossim were the highest and lowest, respectively, i.e.,
$x_{k^{\prime}},x_{l^{\prime}},$ where
$k^{\prime}=\operatorname*{arg\,max}_{k}M[k]=A[0],\;l^{\prime}=\operatorname*{arg\,min}_{k}M[k]=A[N]$.
Here, we tracked their cossim values for each epoch during learning, as shown
in Fig. 3, where P1 and P2 correspond to $x_{l^{\prime}},x_{k^{\prime}}$,
respectively. Unexpectedly, the P2 example nearly maintained its value close
to 1 from the beginning of the learning. However, the value of P1 hardly
approaches 1 even at the end of learning. In addition, the cossim values of
both the examples abruptly changed over the entire learning process. Thus,
considering noise-like cossim, averaging rather than choosing a single cossim
value at a certain epoch in the middle of learning was a reasonable approach.

(a) normal t-shirt

(b) abnormal t-shirt

Figure 4: Example of a normal and abnormal T-shirt. The dashes represent two
randomly cropped regions. In normal T-shirts, the contents of the two cropped
regions are not significantly different. On the other hand, in the abnormal
T-shirt, the content of the two cropped regions is significantly different.

(a) CIFAR10

(b) SVHN

(c) QMNIST

Figure 5: Coreset selection performance on three classification datasets. The
vertical bars indicate the standard deviation of the accuracy. In most cases,
our coreset was more accurate than the random subset. Despite the absence of
labels, our results were comparable to those of supervised coreset selection
(SCS). Notably, SCS (with forgetting events) was less accurate than other
methods in the case of SVHN. For QMNIST, we additionally examined the
performance at 20% subset size to verify the saturation point; above 20%
subset size, our SimCLR model and SCS had become saturated at top performance,
whereas our MoCo model and random subset were still undergoing improvement.

(a) CIFAR10

(b) SVHN

(c) QMNIST

Figure 6: Model-agnostic performance on three classification datasets. Coreset
selection model: ResNet18; target model: ResNet101. The vertical bars indicate
the standard deviation of the accuracy. For all three datasets, our coreset
was more accurate than the random subset.

Identifying the coreset. As mentioned above, we calculated the coreset score
by averaging the cossim values. This allowed us to select the coreset based on
the score. Specifically, our coreset score was calculated as follows:
throughout the contrastive-learning process, the negative cossim value is
accumulated for each example. Following this, we sorted the scores in
ascending order. The final coreset was then chosen with the size of the value
specified according to the score, as displayed in Alg.1, where
$[n]=\\{1,...,N\\}$ and $N$ is the size of training set. This approach,
although simple, has proven effective in identifying the coreset, as
demonstrated in Section 4.3. It owes to the fact that in contrastive learning,
strong augmentation plays a crucial role in realizing representation learning.
For instance, considering two cases in which random cropping—a form of
augmentation—is applied to two different types of T-shirts (Fig. 4), we can
assume that the two randomly cropped regions include dissimilar contents,
although they are in an identical class. Consequently, the network experiences
difficulties in minimizing the value of cossim for abnormal T-shirts,
resulting in lower coreset scores compared with those of normal T-shirts.
Thus, we concluded that the abnormal T-shirt possibly provides more
information about the concept of the T-shirt than the normal one did. We
verify this claim in Section 4.3. Intuitively, the most informative example
would include the maximum amount of content possible. The bottom line of this
claim is that an example belonging to a coreset would exhibit an abnormal or
peculiar appearance.

Assessing the coreset. Any approach to obtain a reliable coreset must meet the
following requirements: 1) consistency: across random seeds for the selection
model, the ranking of the resulting coreset score must be as stable as
possible. 2) model-agnostic: once the coreset is chosen, it should outperform
a randomly chosen set regardless of the model of the target task. We set up an
evaluation protocol and describe the results in Sections 5.1 and 5.2.
Furthermore, the absence of labels connotes that the data imbalance among the
interclasses is an inherent problem for unsupervised coreset selection. We
also examine the class imbalance of our coreset compared to a randomly chosen
coreset in Section 5.4.

##  4 Experiments

In this section, we prove that coreset selection is a valid approach for
performing classification tasks. Our coreset is an improvement over the random
performance of three different datasets (CIFAR10, SVHN, and QMNIST). For
comparison with supervised coreset selection (SCS), we conducted SCS
experiments, i.e., using greedy k-centers [26] and forgetting events [27],
which are also referred to in the literature [4]. In most cases, the
performance of SCS, which is label-hungry by nature, manifestly surpasses that
of our method. However, SCS is arguably an invalid approach for cost reduction
because it requires fully annotated data. The importance of our study is thus
highlighted: our proposed method is designed to directly process unlabeled
data although it is outperformed by SCS. In addition, as mentioned in Section
3, along with Fig. 4, we presented another experiment, the coreset cross test,
designed to provide evidence that a set of examples chosen by our coreset
score offer highly informative content.

###  4.1 Implementation Details

Datasets. To vary the example type to gray-scale, RGB, digits, and objects, we
chose three different datasets (CIFAR10[17], SVHN[22], and QMNIST [34]), which
consisted of 10 classes each. CIFAR10 is intended to support object
classification and consists of 50,000 training and 10,000 testing examples.
Each example has a size of 32$\times$32; the examples include center-cropped
objects and RGB channels. QMNIST datasets are generated digits based on NIST
Special Database 19 [9] with the purpose of maintaining MNIST [18]
preprocessing. QMNIST is divided into a training dataset of 60,000 images and
a testing dataset of 10,000 images. These are gray-scale images with a
resolution of 28$\times$28\. In contrast to QMNIST, SVHN comprises RGB
channels. We used 73,257 examples for training and 26,032 for testing. These
are cropped images of digits with a size of 32$\times$32\. We omitted the
extra images from this dataset (531,131 examples). For both SVHN and QMNIST,
the annotated labels directly correspond to digits, i.e., label ‘1’ is the
digit 1, etc.

Models. We deployed ResNet18 [13] for the contrastive and classification
tasks. In SimCLR learning, our coreset was constructed on the basis of the
code available on the website222https://github.com/Spijkervet/SimCLR. We set
the batch size to 1024, number of epochs to 1,000, dimension of projection to
128, and employed the LARS optimizer [35] across the datasets. In MoCo
learning, we shared all the hyperparameters, such as the epochs (600) and
batch size (512), across the datasets, and used cosine learning–rate
scheduling based on the code available on the
website333https://github.com/facebookresearch/moco For both types of
contrastive learning, augmentation techniques including randomly resized
crops, random horizontal flip, color jittering, and random gray-scale were
implemented. In particular, because QMNIST consists of gray-scale images, we
modified the first layer of ResNet18 to contain only a single channel.
Additionally, for both SimCLR and MoCo, we deployed the following augmentation
techniques across datasets: randomly resized crop, color jittering, horizontal
flip, and applying gray-scale.

###  4.2 Coreset Selection Task

For training SCS with forgetting events and greedy k-centers, we utilized an
entirely labeled training set and selected a subset according to their coreset
scores. For SVHN, whose training set size is 73257, we set subset sizes (30%
$\sim$ 70%) as follows: $\\{22500,~{}30000,~{}37500,~{}45000,~{}52500\\}$. The
results of our classification task are shown in Fig. 5. We calculated the mean
and standard deviation of the test accuracy over five runs. Our coreset proved
to exhibit a higher accuracy than the randomly chosen coreset did. This
implies that in contrastive learning, cossim serves as a useful metric for
selecting the coreset. In most cases, SCS outperforms our coreset, whereas for
the SVHN dataset, our results are comparable despite being unlabeled. As
QMNIST contained gray-scale images, and the dataset is relatively easy to
manipulate compared with the others, our improvement was marginal; however, it
outperformed the random coreset at all subset sizes. Across datasets, our
performance lay between SCS and the random subset, highlighting the importance
of our study; despite having no labels, we were able to increase performance
close to that of SCS.

###  4.3 Coreset Cross Test

To verify whether the examples chosen by our coreset score are highly
informative, we designed a cross test as follows: once the coreset score based
on SimCLR was established, we split the original training set into two
subsets; the top 30% $\sim$ 70% as the training set and the bottom 30% as the
testing set (referred to as coreset $\rightarrow$ non-coreset or C
$\rightarrow$ N). Similarly, we set the bottom 30% $\sim$ 70% as the training
set and the top 30% as the testing set (referred to as the non-coreset
$\rightarrow$ coreset or N $\rightarrow$ C), and repeated each test for five
runs. As shown in Table. 1, in all the cases, the C $\rightarrow$ N cross test
was superior to N $\rightarrow$ C. This result supports our claim that samples
with high coreset scores provide more information than those with low scores.
Furthermore, their appearances are significantly different (elaborated in
Section 5.3).

Table 1: Cross-test results. Average ($\pm$ std) obtained from the five runs
on the three datasets. C and N represent the coreset and non-coreset,
respectively. Across the dataset and fraction of the subset, the C
$\rightarrow$ N test achieved significantly higher accuracy than that of the N
$\rightarrow$ C test. Cross tests were conducted using the coreset score based
on SimCLR.

|  | Fraction of subset
---|---|---
Dataset | cross | 30% | 40% | 50% | 60% | 70%
| C $\rightarrow$ N | 94.27 | 95.44 | 96.14 | 96.68 | 96.99
|  | (0.14) | (0.25) | (0.08) | (0.16) | (0.04)
CIFAR10 | N $\rightarrow$ C | 67.39 | 73.95 | 80.18 | 83.54 | 86.68
|  | (0.80) | (0.35) | (0.49) | (0.24) | (0.14)
| C $\rightarrow$ N | 98.24 | 98.38 | 98.50 | 98.50 | 98.80
|  | (0.09) | (0.04) | (0.04) | (0.06) | (0.06)
SVHN | N $\rightarrow$ C | 83.30 | 85.48 | 86.90 | 88.31 | 89.76
|  | (0.25) | (0.41) | (0.13) | (0.15) | (0.24)
| C $\rightarrow$ N | 99.88 | 99.91 | 99.92 | 99.91 | 99.92
|  | (0.02) | (0.01) | (0.01) | (0.02) | (0.01)
QMNIST | N $\rightarrow$ C | 93.07 | 94.07 | 94.84 | 95.44 | 96.03
|  | (0.44) | (0.12) | (0.11) | (0.11) | (0.19)

---

CIFAR10

SVHN

QMNIST

subset size : 30%

(a)

(b)

(c)

subset size : 50%

(d)

(e)

(f)

subset size : 70%

(g)

(h)

(i)

Figure 7: Distribution of the fraction of examples for each class. Our coreset
exhibits a class imbalance for all the datasets. However, a performance
degradation was not observed for subsets above 30% in size. In addition, for
large subsets, the imbalance seems to diminish.

##  5 Ablation Study

In this section, we provide a detailed description of the experiment we
conducted to describe our coreset. First, we attempted to verify that our
coreset produces consistent examples that are agnostic to random seeds. In
addition, by varying the model for contrastive learning and target-task
learning, we confirmed that our coreset delivers consistent performance
agnostic to the model. Second, we visually inspected each example to examine
the appearance of an image belonging to the coreset and non-coreset. Finally,
because class imbalance inevitably occurs owing to the absence of labels, we
plotted the fraction of examples for each class according to the size of the
subset.

coreset
---
deer horse cat cat truck bird bird truck airplane airplane bird
non-coreset
ship airplane bird ship bird airplane airplane bird airplane bird airplane
coreset
nine two six eight six nine one seven nine three three
non-coreset
one one six eight one one one one one one one
coreset
four zero seven nine six five seven seven zero three two
non-coreset
one one one one one one one one one one one
Figure 8: Visual inspection for CIFAR10, SVHN, and QMNIST. Core-set and non-
coreset examples for CIFAR10 (top two rows), SVHN (middle two rows) and QMNIST
(bottom two rows). Examples belonging to the coreset display a less simple
structure than that of the non-coreset. In addition, the majority of non-
coreset examples are concentrated in a few classes that could lead to class
imbalance. Table 2: intersection ratio of five seed runs. For all datasets,
our coreset can be observed to produce consistent examples regardless of the
random seed.

| Contrastive | Fraction of subset
---|---|---
Dataset | Learning | 30% | 40% | 50% | 60% | 70%
| Random | 0.01 | 0.02 | 0.06 | 0.13 | 0.24
CIFAR10 | SimCLR | 0.90 | 0.92 | 0.93 | 0.95 | 0.96
| MoCo | 0.76 | 0.81 | 0.84 | 0.87 | 0.90
| Random | 0.01 | 0.03 | 0.07 | 0.14 | 0.26
SVHN | SimCLR | 0.91 | 0.92 | 0.93 | 0.94 | 0.96
| MoCo | 0.79 | 0.82 | 0.86 | 0.88 | 0.91
| Random | 0.01 | 0.03 | 0.06 | 0.13 | 0.24
QMNIST | SimCLR | 0.95 | 0.95 | 0.96 | 0.97 | 0.98
| MoCo | 0.62 | 0.66 | 0.70 | 0.74 | 0.78

###  5.1 Consistent Coreset

As mentioned earlier, an effective coreset algorithm can produce consistent
coreset elements across random seeds. This property can be easily verified by
calculating the intersection ratio between multiple coresets resulting from
multiple executions. These results are presented in Table 2, where each
intersection ratio is calculated by dividing the number of intersection
elements by the size of the subset. Notably, the intersection ratio of our
coreset was significantly greater than that of the random subset across the
datasets. Hence, our coreset selects examples that are highly similar despite
using multiple random seeds.

###  5.2 Model-Agnostic Coreset Selection

It is apparent that coreset consistency is beneficial in avoiding rerunning
coreset selection when the target of the coreset selection model is altered.
Hence, we claim that model-agnostic coreset selection is one of the essential
properties that any coreset approach has to provide. For our coreset
experiment, all the experimental settings were identical to those described in
Section 4, with the target model, i.e., ResNet101, being the only difference.
As depicted in Fig. 6, our coreset achieved greater performance than that of
the random subset. For extensive proof, we deployed different architectures,
such as wide ResNet (WRN)[36], ResNeXt[33], and DenseNet[14], using a single
coreset with SimCLR on CIFAR10. As listed in Fig. 3, across subset sizes and
target model architectures, our coreset consistently yields high performance
compared with the random.

Table 3: Model-agnostic coreset selection task on CIFAR10. In all the cases,
our coreset exhibited high accuracy compared with that of the random subset,
regardless of the target model architecture. WRN-n-k ,DenseNet-n, and
ResNeXt-n denote WRN with n convolution layers and k widening factor, DenseNet
with n bottleneck layers, and ResNeXt with n layers, respectively.

| Contrastive | Fraction of subset
---|---|---
Model | learning | 30% | 40% | 50% | 60% | 70%
WRN-40-4 | Random | 90.94 | 92.14 | 93.17 | 94.04 | 94.30
| SimCLR | 91.82 | 93.23 | 93.61 | 94.46 | 94.94
DenseNet-121 | Random | 89.52 | 90.85 | 92.06 | 92.69 | 93.10
| SimCLR | 90.39 | 91.59 | 92.81 | 92.80 | 93.39
ResNeXt-29 | Random | 90.68 | 91.99 | 93.68 | 94.30 | 94.61
| SimCLR | 90.71 | 93.13 | 94.01 | 94.54 | 95.05

###  5.3 Visual Inspection

To enable a thorough visual inspection, we identified 11 top/bottom examples
(due to page limitations) using a single coreset score with SimCLR,
represented in Fig. 8. For simplicity, we referred to the top 11 and bottom 11
as the coreset and non-coreset, respectively. As shown in the figure, the
coreset examples exhibit bizarre and peculiar patterns; in contrast, the non-
coreset examples primarily comprised simple structures and backgrounds.
Furthermore, with respect to class imbalance, the coreset of CIFAR10
predominantly consisted of images of airplanes and birds. Similarly, for both
SVHN and QMNIST, most of the top 11 examples consisted of images with a label
of 1. It should be noted that our coreset may have been adversely affected by
class imbalance. This matter is discussed in the subsequent section.

###  5.4 Data Imbalance

Unless annotation is provided, it is difficult to prevent interclass
imbalance. To examine the class imbalance of our coreset, we displayed the
fraction of resulting examples for each class according to the size of the
subset, 30%, 50%, and 70%, as depicted in Fig. 7. A clear imbalance can be
observed in our coreset. However, for a subset size above 30%, the imbalance
problem did not seem to affect the classification accuracy. Moreover, as the
size of the subset increased, the imbalance tended to decrease. Notably, SVHN
training data are inherently unbalanced; hence, even a random subset did not
have a uniform distribution. Moreover, although our coreset presents a data
imbalance, we were able to achieve high accuracy. These facts imply that our
coreset search examples contain highly informative content.

##  6 Discussion

Extending the contrastive learning–based score for the core-dataset. We
claimed that the coreset score adequately represents examples that are more
informative. To extend this result, we attempted to answer the question of
whether it would be possible to distinguish more informative datasets from the
others In other words, our claim encouraged us to explore the possibility of
employing the contrastive learning–based score to represent the core-dataset.
To validate our claim, we plotted the distribution of average cossim for each
dataset, as can be seen in Fig.9, It is evident that relatively more
informative datasets are in the order of CIFAR10, SVHN, and QMNIST because
these datasets are composed of RGB + objects, RGB + digits, and gray scale +
digits, respectively. Accordingly, the mean and median values for the average
cossim values were in the same order. This could imply that the cossim
distribution represents the level of difficulty for a dataset; for example,
QMNIST is a fairly easy dataset for performing classification compared with
CIFAR10 and SVHN; hence, the mean and median of average cossim distribution
yielded a large value. We conjecture that our coreset score can possibly be
used to identify the core-datasets, and could be the subject of further study
in the future.

---

CIFAR10

SVHN

QMNIST

(a) SimCLR : (0.7586, 0.7643)
MoCo : (0.6679, 0.6677)

(b) SimCLR : (0.8204, 0.8311)
MoCo : (0.7145, 0.7175)

(c) SimCLR : (0.8466, 0.8517)
MoCo : (0.7288, 0.7311)

Figure 9: Distribution of average cossim. These figures depict the
distribution of the average cossim resulting from SimCLR and MoCo for three
datasets. $(\cdot,\cdot)$ are a pair of mean and median values. The modes and
means of distribution differ depending on the dataset. The mode and mean of
QMNIST (the easiest case) are higher than those of CIFAR10 and SVHN.

##  7 Conclusion

In this study, we demonstrated that self-supervised contrastive learning can
create a coreset in the absence of a label. Because cossim is readily
available from contrastive learning, we identified a metric that effectively
measures the coreset score. Construction of the coreset score based on the
average cossim value enabled us to successfully select the coreset from a pool
of unlabeled data. This increased the classification accuracy over that of a
randomly chosen subset. In addition, our approach yielded results that were
comparable to those of methods using supervised learning for coreset
selection. Although cossim is appropriate for identifying the coreset per se,
the problem of establishing a new metric still remains. Therefore, in future
research, we hope to establish a metric better tailored to coresets.
Furthermore, when a pool of unlabeled data is gathered from the website and
annotation cost should be reduced, our study possibly provides an guidance
which example to annotate first without sacrificing performance.

## References

  * [1] Trevor Campbell and Tamara Broderick.  Bayesian coreset construction via greedy iterative geodesic ascent.  arXiv preprint arXiv:1802.01737, 2018.
  * [2] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.  A simple framework for contrastive learning of visual representations.  arXiv preprint arXiv:2002.05709, 2020.
  * [3] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He.  Improved baselines with momentum contrastive learning.  arXiv preprint arXiv:2003.04297, 2020.
  * [4] Cody Coleman, Christopher Yeh, Stephen Mussmann, Baharan Mirzasoleiman, Peter Bailis, Percy Liang, Jure Leskovec, and Matei Zaharia.  Selection via proxy: Efficient data selection for deep learning.  arXiv preprint arXiv:1906.11829, 2019.
  * [5] Carl Doersch, Abhinav Gupta, and Alexei A Efros.  Unsupervised visual representation learning by context prediction.  In Proceedings of the IEEE International Conference on Computer Vision, pages 1422–1430, 2015.
  * [6] Hossein Shokri Ghadikolaei, Hadi Ghauch, Carlo Fischione, and Mikael Skoglund.  Learning and data selection in big datasets.  In International Conference on Machine Learning, pages 2191–2200, 2019.
  * [7] Spyros Gidaris, Praveer Singh, and Nikos Komodakis.  Unsupervised representation learning by predicting image rotations.  arXiv preprint arXiv:1803.07728, 2018.
  * [8] S Girisha, Manohara MM Pai, Ujjwal Verma, and Radhika M Pai.  Performance analysis of semantic segmentation algorithms for finely annotated new uav aerial video dataset (manipaluavid).  IEEE Access, 7:136239–136253, 2019.
  * [9] Patrick J Grother.  Nist special database 19.  Handprinted forms and characters database, National Institute of Standards and Technology, page 10, 1995.
  * [10] Sariel Har-Peled and Akash Kushal.  Smaller coresets for k-median and k-means clustering.  Discrete & Computational Geometry, 37(1):3–19, 2007.
  * [11] Sariel Har-Peled and Soham Mazumdar.  On coresets for k-means and k-median clustering.  In Proceedings of the thirty-sixth annual ACM symposium on Theory of computing, pages 291–300, 2004.
  * [12] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick.  Momentum contrast for unsupervised visual representation learning.  arXiv preprint arXiv:1911.05722, 2019.
  * [13] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [14] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger.  Densely connected convolutional networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4700–4708, 2017.
  * [15] Jonathan Huggins, Trevor Campbell, and Tamara Broderick.  Coresets for scalable bayesian logistic regression.  In Advances in Neural Information Processing Systems, pages 4080–4088, 2016.
  * [16] Licheng Jiao, Fan Zhang, Fang Liu, Shuyuan Yang, Lingling Li, Zhixi Feng, and Rong Qu.  A survey of deep learning-based object detection.  IEEE Access, 7:128837–128868, 2019.
  * [17] Alex Krizhevsky, Geoffrey Hinton, et al.  Learning multiple layers of features from tiny images.  2009\.
  * [18] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner.  Gradient-based learning applied to document recognition.  Proceedings of the IEEE, 86(11):2278–2324, 1998.
  * [19] Shin-Jye Lee, Tonglin Chen, Lun Yu, and Chin-Hui Lai.  Image classification based on the boost convolutional neural network.  IEEE Access, 6:12755–12768, 2018.
  * [20] Mario Lucic, Matthew Faulkner, Andreas Krause, and Dan Feldman.  Training gaussian mixture models at scale via coresets.  The Journal of Machine Learning Research, 18(1):5885–5909, 2017\.
  * [21] Ishan Misra and Laurens van der Maaten.  Self-supervised learning of pretext-invariant representations.  arXiv preprint arXiv:1912.01991, 2019.
  * [22] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng.  Reading digits in natural images with unsupervised feature learning.  2011\.
  * [23] Chongjia Ni, Cheung-Chi Leung, Lei Wang, Nancy F Chen, and Bin Ma.  Unsupervised data selection and word-morph mixed language model for tamil low-resource keyword search.  In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 4714–4718. IEEE, 2015.
  * [24] Mehdi Noroozi and Paolo Favaro.  Unsupervised learning of visual representations by solving jigsaw puzzles.  In European Conference on Computer Vision, pages 69–84. Springer, 2016.
  * [25] Mehdi Noroozi, Hamed Pirsiavash, and Paolo Favaro.  Representation learning by learning to count.  In Proceedings of the IEEE International Conference on Computer Vision, pages 5898–5906, 2017.
  * [26] Ozan Sener and Silvio Savarese.  Active learning for convolutional neural networks: a core-set approach.  stat, 1050:27, 2017.
  * [27] Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geoffrey J Gordon.  An empirical study of example forgetting during deep neural network learning.  arXiv preprint arXiv:1812.05159, 2018.
  * [28] Ivor W Tsang, James T Kwok, and Pak-Ming Cheung.  Core vector machines: Fast svm training on very large data sets.  Journal of Machine Learning Research, 6(Apr):363–392, 2005.
  * [29] Sebastian Tschiatschek, Rishabh K Iyer, Haochen Wei, and Jeff A Bilmes.  Learning mixtures of submodular functions for image collection summarization.  In Advances in neural information processing systems, pages 1413–1421, 2014.
  * [30] Gabriele Valvano, Andrea Leo, Daniele Della Latta, Nicola Martini, Gianmarco Santini, Dante Chiappino, and Emiliano Ricciardi.  Unsupervised data selection for supervised learning.  arXiv, pages arXiv–1810, 2018.
  * [31] Kai Wei, Rishabh Iyer, and Jeff Bilmes.  Submodularity in data subset selection and active learning.  In International Conference on Machine Learning, pages 1954–1963, 2015.
  * [32] Kai Wei, Yuzong Liu, Katrin Kirchhoff, Chris Bartels, and Jeff Bilmes.  Submodular subset selection for large-scale speech training data.  In 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 3311–3315. IEEE, 2014.
  * [33] Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He.  Aggregated residual transformations for deep neural networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1492–1500, 2017.
  * [34] Chhavi Yadav and Léon Bottou.  Cold case: The lost mnist digits.  In Advances in Neural Information Processing Systems 32. Curran Associates, Inc., 2019.
  * [35] Yang You, Igor Gitman, and Boris Ginsburg.  Large batch training of convolutional networks.  arXiv preprint arXiv:1708.03888, 2017.
  * [36] Sergey Zagoruyko and Nikos Komodakis.  Wide residual networks.  arXiv preprint arXiv:1605.07146, 2016.