m

Abelian $p$-groups
with a $p$-bounded factor or a $p$-bounded subgroup

Justyna Kosakowska, Markus Schmidmeier, Martin Schreiner

Abstract: In his 1934 paper, G. Birkhoff poses the problem of classifying
pairs $(G,U)$ where $G$ is an abelian group and $U\subset G$ a subgroup, up to
automorphisms of $G$. In general, Birkhoff’s Problem is not considered
feasible. In this note, we fix a prime number $p$ and assume that $G$ is a
direct sum of cyclic $p$-groups and $U\subset G$ is a subgroup. Under the
assumption that the factor group $G/U$ is $p$-bounded, we show that each such
pair $(G,U)$ has a unique direct sum decomposition into pairs of type
$(\mathbb{Z}/(p^{n}),\mathbb{Z}/(p^{n}))$ or $(\mathbb{Z}/(p^{n}),(p))$. On
the other hand, if we assume $U$ is a $p$-bounded subgroup of $G$ and in
addition that $G/U$ is a direct sum of cyclic $p$-groups, we show that each
such pair $(G,U)$ has a unique direct sum decomposition into pairs of type
$(\mathbb{Z}/(p^{n}),0)$ or $(\mathbb{Z}/(p^{n}),(p^{n-1}))$. We generalize
the above results to modules over commutative discrete valuation rings.

MSC 2020: 20K27 (subgroups of abelian groups), 13C05 (Structure,
classification theorems for modules and ideals in commutative rings), 47A15
(invariant subspaces of linear operators)

Key words: Birkhoff Problem, abelian $p$-groups, subgroups, invariant
subspaces

##  1\. Introduction

For $\mathcal{A}$ a full subcategory of the category of abelian groups, the
subgroup category $\mathcal{S}(\mathcal{A})$ consists of all pairs $(G,U)$
where $G\in\mathcal{A}$ and $U$ is a subgroup of $G$. A morphism in
$\mathcal{S}(\mathcal{A})$ from $(G,U)$ to $(G^{\prime},U^{\prime})$ is a
group homomorphism $f:G\to G^{\prime}$ such that the image $f(U)$ is contained
in $U^{\prime}$. There has been a lot of interest in categories of type
$\mathcal{S}(\mathcal{A})$, for example in the case where $\mathcal{A}$
consists of all $p^{n}$-bounded abelian groups.

Recall that an abelian group is $\Sigma$-cyclic, if it is a direct sum of
cyclic groups. In this note, we fix a prime number $p$ and consider the
category $\Sigma$-cyc of all abelian groups which are direct sums of cyclic
$p$-groups. Note that the summands in each direct sum can be indexed by a
(possibly infinite) cardinal number. We will work within ZFC.

By $\mathcal{F}_{1}(\mbox{$\Sigma$-cyc})$ we denote the full subcategory of
$\mathcal{S}(\mbox{$\Sigma$-cyc})$ of all pairs $(G,U)$ where
$G\in\mbox{$\Sigma$-cyc}$ and $U$ is a subgroup of $G$ with $p$-bounded factor
$G/U$, (i.e. $p(G/U)=0$).

Let $\mathcal{S}_{1}(\mbox{$\Sigma$-cyc})$ be the full subcategory of
$\mathcal{S}(\mbox{$\Sigma$-cyc})$ of all pairs $(G,U)$, where
$G\in\mbox{$\Sigma$-cyc}$, $U$ is a $p$-bounded subgroup of $G$.

A picket is a pair $(G,U)$ in $\mathcal{S}(\mbox{$\Sigma$-cyc})$ where the
group $G$ is a cyclic $p$-group.

For example, for $n\in\mathbb{N}$, every picket in
$\mathcal{F}_{1}(\mbox{$\Sigma$-cyc})$ is isomorphic to

$P^{n}_{n}=(\mathbb{Z}/(p^{n}),\mathbb{Z}/(p^{n}))\qquad or\qquad
P^{n}_{n-1}=(\mathbb{Z}/(p^{n}),(p)),$

whereas the pickets in $\mathcal{S}_{1}(\mbox{$\Sigma$-cyc})$ are isomorphic
to

$P_{0}^{n}=(\mathbb{Z}/(p^{n}),0)\qquad or\qquad
P^{n}_{1}=(\mathbb{Z}/(p^{n}),(p^{n-1})).$

We first state the result for $\mathcal{F}_{1}$.

######  Theorem 1.

Every object in $\mathcal{F}_{1}(\mbox{$\Sigma$-cyc})$ is isomorphic to a
direct sum of pickets of type $P^{n}_{n}$ or $P^{n}_{n-1}$. The direct sum
decomposition is unique up to reordering and isomorphy of the summands.

###### Remark:

The corresponding result for pairs $(G,U)$ where the subgroup $U$ is
$p$-bounded (and not the factor $G/U$) does not hold: Let
$G=\bigoplus_{n\in\mathbb{N}}y_{n}\,\mathbb{Z}/(p^{n})$ be the direct sum of
cyclic $p$-groups where $y_{n}$ denotes the generator of the $n$-th direct
summand, and take for $U$ the $p$-bounded subgroup
$U=\bigoplus_{n>1}(y_{1}-p^{n-1}y_{n})\mathbb{Z}$. Then the coset of $y_{1}$
in the factor group $G/U$ is divisible by all powers of $p$, so $G/U$ cannot
be a direct sum of cyclic $p$-groups. A different way to look at it is that
the quotient $G/U$ is a countable reduced primary group which has an Ulm
invariant at $\omega$ [8, Remark (b) on p. 31]. In particular, the pair
$(G,U)$ is not a direct sum of pickets of type $P_{0}^{n}$ or $P_{1}^{n}$.

The corresponding result for $\mathcal{S}_{1}$ requires an additional
condition.

######  Theorem 2.

An object $(G,U)$ in $\mathcal{S}_{1}(\mbox{$\Sigma$-cyc})$ is a direct sum of
pickets of type $P_{0}^{n}$ or $P_{1}^{n}$ if and only if
$G/U\in\mbox{$\Sigma$-cyc}$. In this case, the direct sum decomposition is
unique up to reordering and isomorphy of the summands.

Some related results. (1) The nonsplit short exact sequence

$0\longrightarrow\mathbb{Z}/(p^{2})\stackrel{{\scriptstyle{\iota\choose\pi}}}{{\longrightarrow}}\mathbb{Z}/(p^{3})\oplus\mathbb{Z}/(p)\longrightarrow\mathbb{Z}/(p^{2})\longrightarrow
0$

shows that the condition that the subgroup or the factor group be $p$-bounded
is needed to decompose the embedding (or the short exact sequence) as a direct
sum of pickets.

(2) If we assume that the group $G$ is $p^{m}$-bounded for some
$m\in\mathbb{N}$, and $U$ is a subgroup such that either $U$ or $G/U$ is
$p$-bounded, then our results are well-known: There are only finitely many
finitely generated indecomposable objects, up to isomorphy, (namely the
pickets, see for example [12]), and hence by the Auslander and Ringel-
Tachikawa Theorem any (not necessarily finitely generated) object is a direct
sum of those (see [11] for the Auslander and Ringel-Tachikawa Theorem for
embeddings). Note that under the assumption that $G$ is $p^{m}$-bounded, the
condition that $U$ or $G/U$ be $p$-bounded can even be relaxed to $U$ or $G/U$
being $p^{2}$-bounded; in this case any object is a direct sum of certain bi-
pickets (see [12]).

(3) We will see in Chapter 4 that our results can be generalized from abelian
groups to modules over discrete valuation domains. The recent paper [6]
exhibits categorical equivalences relating embeddings of subgroups in bounded
abelian groups and embeddings of submodules of bounded modules over certain
discrete valuation domains.

##  2\. Proof of Theorem 1

###  2.1. Nice filtrations

Suppose that $G$ is a direct sum of cyclic $p$-groups and $U\leq G$ a
subgroup. In a decomposition of $G$, combine summands of the same order to
obtain the isotypic decomposition

$G=\bigoplus_{n\in\mathbb{N}}T_{n}$

where $T_{n}$ is the direct sum of all summands of order $p^{n}$ in the given
direct sum decomposition of $G$ and $T_{0}=0$.

For each $n\in\mathbb{N}$, write
$T_{n}=\bigoplus_{\kappa_{n}}\mathbb{Z}/(p^{n})$ as a direct sum of
$\kappa_{n}$ copies of $\mathbb{Z}/(p^{n})$ where the index set $\kappa_{n}$
is an arbitrary cardinal number.

We will use the associated filtration,

$G=\bigcup_{n\in\mathbb{N}}G_{n},\qquad\text{where}\qquad
G_{n}=\bigoplus_{\ell\leq n}T_{\ell}.$

Since $G$ has a filtration, so does the subgroup $U$:

$U=\bigcup_{n\in\mathbb{N}}U_{n},\qquad\text{where}\qquad U_{n}=U\cap G_{n}.$

For the proof of Theorem 1, we will assume that the pair $(G,U)$ is in
$\mathcal{F}_{1}(\mbox{$\Sigma$-cyc})$. Then each pair $(G_{n},U_{n})$ is also
in $\mathcal{F}_{1}(\mbox{$\Sigma$-cyc})$. In the following sections we show
that the filtration is nice in the sense that

$(*)\qquad(G_{n},U_{n})=(G_{n-1},U_{n-1})\oplus D\quad\text{where}\quad
D\cong(P_{n}^{n})^{\mu_{n}^{n}}\oplus(P_{n-1}^{n})^{\mu_{n-1}^{n}}$

holds. From formula $(*)$ we can derive the multiplicities of the pickets in
the direct sum decomposition:

Since $G_{n}=G_{n-1}\oplus T_{n}$ and $G_{n-1}+pG_{n}=G_{n-1}\oplus pT_{n}$,
the vector space $\frac{G_{n}}{G_{n-1}+pG_{n}}$ has dimension $\kappa_{n}$.
Considering the chain

$G_{n-1}+pG_{n}\;\subseteq\;G_{n-1}+U_{n}\;\subseteq\;G_{n},$

we obtain $\kappa_{n}=\mu_{n}^{n}+\mu_{n-1}^{n}$ where

$\mu_{n}^{n}=\dim\frac{G_{n-1}+U_{n}}{G_{n-1}+pG_{n}}\quad\text{and}\quad\mu_{n-1}^{n}=\dim\frac{G_{n}}{G_{n-1}+U_{n}}.$

Thus, the embedding $(G,U)$ is the union, and hence the direct sum, as
follows,

$(G,U)\;=\;\bigcup_{n\in\mathbb{N}}(G_{n},U_{n})\;\cong\;\bigoplus_{n\in\mathbb{N}}\left[(P_{n}^{n})^{\mu_{n}^{n}}\oplus(P_{n-1}^{n})^{\mu_{n-1}^{n}}\right],$

proving the existence of the decomposition in the theorem. Since each picket
has a local endomorphism ring, uniqueness follows from the Krull-Remak-
Schmidt-Azumaya theorem, see [1, 3].

###### Remark:

Note that the assumption $(G,U)\in\mathcal{F}_{1}(\Sigma{\rm-cyc})$ is
necessary for $(*)$: For example consider
$G=\mathbb{Z}/(p)\oplus\mathbb{Z}/(p^{2})$ and $U$ generated by $(1,p)\in G$.
Then $(G,U)\cong P^{1}_{1}\oplus P^{2}_{0}$. If $G_{1}$ is the first summand
of $G$ then $U_{1}=G_{1}\cap U=0$, so $(G_{1},U_{1})\cong P^{1}_{0}$. Hence
$(G_{1},U_{1})$ is not isomorphic to a direct summand of
$(G_{2},U_{2})=(G,U)$.

###  2.2. The setup for the induction

We assume the notation from the previous section. It remains to show that
property $(*)$ holds. For this, we construct successively minimal generating
sets for the groups $G_{n}$ in the filtration. Such a generating set has the
form $C_{n}\cup C_{n}^{\prime}$ where

$G_{n}=\bigoplus_{g\in
C_{n}\operatorname{\dot{\cup}}C_{n}^{\prime}}g\mathbb{Z},\qquad
U_{n}=\bigoplus_{g\in C_{n}}g\mathbb{Z}\oplus\bigoplus_{g\in
C_{n}^{\prime}}pg\mathbb{Z}.$

In particular,

$(G_{n},U_{n})\quad\cong\quad\bigoplus_{g\in
C_{n}}(g\mathbb{Z},g\mathbb{Z})\quad\oplus\quad\bigoplus_{g\in
C_{n}^{\prime}}(g\mathbb{Z},p\,g\mathbb{Z}).$

As a byproduct, we obtain sets

$B_{n}=\\{g+pG_{n}:g\in C_{n}\\},\qquad B_{n}^{\prime}=\\{g+pG_{n}:g\in
C_{n}^{\prime}\\}$

such that $B_{n}$ is a basis for $\frac{U_{n}}{pG_{n}}$ and
$B_{n}\operatorname{\dot{\cup}}B_{n}^{\prime}$ a basis for
$\frac{G_{n}}{pG_{n}}$.

In the case where $n=1$, the group $G_{1}$ is a vector space over
$\mathbb{Z}/(p)$ so we can take for $C_{1}=B_{1}$ a basis for the subspace
$U_{1}=G_{1}\cap U$ and extend it to a basis
$C_{1}\operatorname{\dot{\cup}}C_{1}^{\prime}=B_{1}\operatorname{\dot{\cup}}B_{1}^{\prime}$
for $G_{1}$.

###  2.3. Constructing bases

Assume now that minimal generating sets $C_{n-1}$, $C_{n-1}^{\prime}$ and
bases $B_{n-1}$, $B_{n-1}^{\prime}$ have been constructed. Put

$\tilde{B}_{n-1}=\\{g+pG_{n}:g\in
C_{n-1}\\},\qquad\tilde{B}_{n-1}^{\prime}=\\{g+pG_{n}:g\in
C_{n-1}^{\prime}\\},$

then $\tilde{B}_{n-1}$ is in bijection to $B_{n-1}$ under the isomorphism
$\frac{U_{n-1}+pG_{n}}{pG_{n}}\cong\frac{U_{n-1}}{U_{n-1}\cap
pG_{n}}=\frac{U_{n-1}}{pG_{n-1}}$ and hence a basis, and
$\tilde{B}_{n-1}\cup\tilde{B}_{n-1}^{\prime}$ is in bijection to $B_{n-1}\cup
B_{n-1}^{\prime}$ under the isomorphism
$\frac{G_{n-1}+pG_{n}}{pG_{n}}\cong\frac{G_{n-1}}{G_{n-1}\cap
pG_{n}}=\frac{G_{n-1}}{pG_{n-1}}$ and hence a basis. Consider the diagram

$\textstyle{\frac{G_{n}}{pG_{n}}\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\textstyle{\frac{U_{n}+G_{n-1}}{pG_{n}}\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\textstyle{\frac{U_{n}}{pG_{n}}\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\textstyle{\frac{G_{n-1}+pG_{n}}{pG_{n}}\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\textstyle{\frac{U_{n-1}+pG_{n}}{pG_{n}}\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\textstyle{0}$

Note that

$\displaystyle U_{n}\cap(G_{n-1}+pG_{n})$ $\displaystyle=$
$\displaystyle(U_{n}\cap G_{n-1})+pG_{n}$ $\displaystyle=$
$\displaystyle(U\cap G_{n}\cap G_{n-1})+pG_{n}$ $\displaystyle=$
$\displaystyle U_{n-1}+pG_{n}.$

For the first equation, recall $pG_{n}\leq U_{n}$ since we are dealing with
objects in $\mathcal{F}_{1}$.

So, the space in the fourth row in the diagram is the intersection of the
spaces on the left and on the right, while the space in the second row near
the top is their sum. Extend the basis $\tilde{B}_{n-1}$ for
$\frac{U_{n-1}+pG_{n}}{pG_{n}}$ $($in the fourth row) to a basis $B_{n}$ for
$\frac{U_{n}}{pG_{n}}$ $($on the left). Recall that
$\tilde{B}_{n-1}\operatorname{\dot{\cup}}\tilde{B}_{n-1}^{\prime}$ is a basis
for $\frac{G_{n-1}+pG_{n}}{pG_{n}}$. Then
$B_{n}\operatorname{\dot{\cup}}\tilde{B}_{n-1}^{\prime}$ is a linearly
independent set, hence a basis for $\frac{U_{n}+G_{n-1}}{pG_{n}}$. Extend
$\tilde{B}_{n-1}^{\prime}$ to a set $B_{n}^{\prime}$ so that
$B_{n}\operatorname{\dot{\cup}}B_{n}^{\prime}$ is a basis for
$\frac{G_{n}}{pG_{n}}$ $($at the top).

###  2.4. A lifting lemma

We will use the following result, which may be well known, to lift the basis
$B_{n}\operatorname{\dot{\cup}}B_{n}^{\prime}$ to a set of generators for
$G_{n}$.

######  Lemma 3.

Suppose $T$ is a direct sum of copies of $\mathbb{Z}/(p^{n})$. Let
$B=\\{b_{i}:i\in I\\}$ be a linearly independent set in $T/pT$ and
$C=\\{c_{i}:i\in I\\}$ any set of liftings of $B$ to $T$ satisfying
$b_{i}=c_{i}+pT$. Then the subgroup $G$ of $T$ generated by the elements
$c_{i}$ is the direct sum

$G=\bigoplus_{i\in I}c_{i}\mathbb{Z}.$

Moreover, if $B$ is a basis for $T/pT$, then $T=\bigoplus_{i\in
I}c_{i}\mathbb{Z}$.

###### Proof.

We show by induction that for each $m=n-1,\ldots,0$ the following statement
holds:

$(P_{m})\quad p^{m}G=\bigoplus_{i\in I}p^{m}c_{i}\mathbb{Z};\quad\text{if $B$
is a basis then}\quad p^{m}T=p^{m}G.$

We will use that the map

$\varphi_{m}:\frac{T}{pT}\to\frac{p^{m}T}{p^{m+1}T},\;x+pT\mapsto
p^{m}x+p^{m+1}T$

is an isomorphism of vector spaces. In particular, the statement $(P_{m})$
holds for $m=n-1$.

We deal with the induction step $m+1\to m$. Let
$B^{\prime}=\\{b_{i}^{\prime}=\varphi_{m}(b_{i}):i\in I\\}$. Then $B^{\prime}$
is a linearly independent set and, if $B$ is a basis, then so is $B^{\prime}$.
The elements $b_{i}^{\prime}$ satisfy $b_{i}^{\prime}=p^{m}c_{i}+p^{m+1}T$ for
$i\in I$.

The elements $p^{m}c_{i}$ form a generating set for $p^{m}G=\sum_{i\in
I}p^{m}c_{i}\mathbb{Z}$.

To show that the sum is a direct sum, suppose $J\subseteq I$ is a finite
subset and $0=\sum_{i\in J}a_{i}p^{m}c_{i}$ where $a_{i}\in\mathbb{Z}$. Then
in $p^{m}T/p^{m+1}T$, we have $0=\sum a_{i}b_{i}^{\prime}$. Since $B^{\prime}$
is a linearly independent set, for all $i\in J$ there exist
$a_{i}^{\prime}\in\mathbb{Z}$ such that $a_{i}=a_{i}^{\prime}p$. Since $\sum
a_{i}^{\prime}p^{m+1}c_{i}=0$ we can use the induction hypothesis $(P_{m+1})$
to conclude that $a_{i}^{\prime}p^{m+1}c_{i}=0$ for all $i\in J$, hence each
$a_{i}p^{m}c_{i}=0$, so the sum is a direct sum.

It remains to show that if $B$ is a basis then $p^{m}T=p^{m}G$. Let $x\in
p^{m}T$, then $x+p^{m+1}T=\sum a_{i}b_{i}^{\prime}$ with $a_{i}\in\mathbb{Z}$.
Hence $x-\sum a_{i}p^{m}c_{i}$ is in $p^{m+1}T=p^{m+1}G$, by induction
hypothesis $(P_{m+1})$, so there are $a_{i}^{\prime}\in\mathbb{Z}$ with
$x-\sum a_{i}p^{m}c_{i}=\sum a_{i}^{\prime}p^{m+1}c_{i}$. This shows that
$x\in p^{m}G$. $\square$

######  Proposition 4.

Suppose we have the following setup:

  * •

$F=F^{\prime}\oplus T$ where $F^{\prime}$ is $p^{n-1}$-bounded and $T$ is a
direct sum of copies of $\mathbb{Z}/(p^{n})$.

  * •

$B^{\prime}$ is a basis for $F^{\prime}/pF^{\prime}$ and
$B=B^{\prime}\operatorname{\dot{\cup}}B^{\prime\prime}$ a linearly independent
subset in $F/pF$ extending $B^{\prime}$.

  * •

$C^{\prime}$ is a set of elements in $F^{\prime}$ corresponding to the
elements in $B^{\prime}$, and $C^{\prime\prime}$ a set of liftings for the
elements in $B^{\prime\prime}$ to $F$.

Then the sum $G=\sum_{c\in
C^{\prime}\operatorname{\dot{\cup}}C^{\prime\prime}}c\mathbb{Z}$ is a direct
sum. Moreover, if $B$ is a basis for $F/pF$ then $F=G$.

###### Proof.

We show the result by induction on $n$. The case $n=1$ is clear. Suppose
$n>1$. By induction hypothesis, we have that the sum $\sum_{c\in
C^{\prime}}c\mathbb{Z}$ is the direct sum $F^{\prime}=\bigoplus_{c\in
C^{\prime}}c\mathbb{Z}$. Write each $c\in C^{\prime\prime}$ as a pair
$c=(c_{1},c_{2})$ with $c_{1}\in F^{\prime}$ and $c_{2}\in T$, and similarly,
each $b\in B^{\prime\prime}$ as a pair $b=(b_{1},b_{2})$ with $b_{1}\in
F^{\prime}/pF^{\prime}$ and $b_{2}\in T/pT$. If $b=c+pF$ then
$(b_{1},b_{2})=(c_{1}+pF^{\prime},c_{2}+pT^{\prime})$.

By Lemma 3, the sum $\sum_{(c_{1},c_{2})\in C^{\prime\prime}}c_{2}\mathbb{Z}$
is a direct sum, hence so is $G=\sum_{c\in
C^{\prime}\operatorname{\dot{\cup}}C^{\prime\prime}}c\mathbb{Z}$. Moreover, if
$B$ is a basis for $F/pF$ then $B^{\prime\prime}$ is a basis for $T/pT$, hence
we obtain from Lemma 3 that $T=\bigoplus_{(c_{1},c_{2})\in
C^{\prime\prime}}c_{2}\mathbb{Z}$, and it follows that $F=F^{\prime}\oplus
T=\bigoplus_{c\in
C^{\prime}\operatorname{\dot{\cup}}C^{\prime\prime}}c\mathbb{Z}$. $\square$

###  2.5. Constructing generators

Recall that $B_{n}$ is a basis for $U_{n}/pG_{n}$ and that the set $C_{n-1}$
consists of generators in $G_{n-1}$, corresponding to basis elements in
$\tilde{B}_{n-1}=\\{g+pG_{n}:g\in C_{n-1}\\}$. Extend $C_{n-1}$ by choosing
for each basis element $b\in B_{n}\setminus\tilde{B}_{n-1}$ an element $g\in
G_{n}$ which satisfies $b=g+pG_{n}$, this is the set $C_{n}$. Similarly,
extend $C_{n-1}^{\prime}$ by choosing elements $g\in G_{n}$ for the basis
elements $b\in B_{n}^{\prime}\setminus\tilde{B}_{n-1}^{\prime}$ to obtain the
set $C_{n}^{\prime}$.

We obtain from Proposition 4 that

$(**)\qquad G_{n}=\bigoplus_{g\in
C_{n}\operatorname{\dot{\cup}}C_{n}^{\prime}}g\mathbb{Z}.$

It remains to verify $U_{n}=\bigoplus_{g\in
C_{n}}g\mathbb{Z}\oplus\bigoplus_{g\in C_{n}^{\prime}}pg\mathbb{Z}$. The right
hand side is contained in the left hand side: Since $U_{n}\supset pG_{n}$, we
have $pg\mathbb{Z}\subset U_{n}$ for $g\in C_{n}^{\prime}$. Let $g\in C_{n}$.
For $\bar{g}\in U_{n}/pG_{n}$ there is $h\in U_{n}$ such that
$\bar{h}=\bar{g}$, hence $g-h\in p{G_{n}}\subset U_{n}$. Hence $g\in U_{n}$.

The left hand side is contained in the right hand side: Let $x\in U_{n}$, then
$\bar{x}\in U_{n}/pG_{n}$ is a linear combination $\bar{x}=\sum a_{i}b_{i}$
where $a_{i}\in\mathbb{Z}$, $b_{i}\in B_{n}$. Suppose $g_{i}\in C_{n}$,
$\bar{g}_{i}=b_{i}$, so $x-\sum a_{i}g_{i}\in pG_{n}$, hence by $(**)$ there
are $k_{i}\in C_{n}$, $\ell_{i}\in C_{n}^{\prime}$, $d_{i},e_{i}\in\mathbb{Z}$
such that $x-\sum a_{i}g_{i}=p(\sum d_{i}k_{i}+\sum e_{i}\ell_{i})$, hence

$x=[\sum a_{i}g_{i}+p\sum d_{i}k_{i}]+[p\sum e_{i}\ell_{i}]\in\bigoplus_{g\in
C_{n}}g\mathbb{Z}\oplus\bigoplus_{g\in C_{n}^{\prime}}pg\mathbb{Z}.$

To finish the proof of the theorem, note that

$(G_{n},U_{n})\;=\;\bigoplus_{g\in
C_{n}}(g\mathbb{Z},g\mathbb{Z})\oplus\bigoplus_{g\in
C_{n}^{\prime}}(g\mathbb{Z},pg\mathbb{Z})$

extends the previous decomposition for $(G_{n-1},U_{n-1})$.

##  3\. Proof of Theorem 2

We start from notation and definitions.

A subgroup $H$ of an abelian group $G$ is pure, if for all $n\in\mathbb{Z}$,
$h\in H$, $g\in G$ such that $h=ng$, there exists $h^{\prime}\in H$ such that
$h=nh^{\prime}$.

Following [8] (see also [4]) we call a subset $X$ of $G$:

  * •

independent, if for every finite sum $n_{1}x_{1}+\ldots+n_{m}x_{m}=0$, were
$x_{1},\ldots,x_{m}\in X$ and $n_{1},\ldots,n_{m}\in\mathbb{Z}$, we have
$n_{1}x_{1}=\ldots=n_{m}x_{m}=0$,

  * •

pure, if the subgroup $(X)$ of $G$ generated by $X$ is pure,

  * •

pure independent if it is pure and independent.

Let $G$ be a $p$-primary group (i.e. all its elements have orders that are
powers of $p$). An element $x\in G$ has height $h_{G}(x)$ equal $r$, if $x$ is
divisible by $p^{r}$ but not by $p^{r+1}$; $x$ has infinite height if it is
divisible by all powers of $p$.

We call a subgroup $H$ of $G$ height-bounded, if $\mbox{\rm
sup}\\{h_{G}(x)|x\in H\\}$ is finite.

We denote

$G[p]=\\{x\in G\;;\;px=0\\}.$

Let as recall the following two facts [8, Lemma 11, Theorem 12].

######  Lemma 5.

$G$ be a primary group and $P=G[p]$. Let $Q,R$ be subgroups of $P$, with
$Q\subseteq R\subseteq P$, and suppose $R$ is of bounded height. Let $X$ be a
pure independent set satisfying $(X)\cap P=Q$. Then $X$ can be enlarged to a
pure independent set $Y$ satisfying $(Y)\cap P=R$. $\square$

######  Theorem 6.

Let $G$ be a primary group and $P=G[p]$. $G\in\mbox{$\Sigma$-cyc}$ if and only
if $P$ is the union of an ascending sequence of subgroups of bounded height.
$\square$

We split proof of Theorem 2 into Lemmata.

The following lemma seems to be well-known, see [5, Exercise 3 on page 8]

######  Lemma 7.

Let $(G,U)\in\mathcal{S}_{1}(\mbox{$\Sigma$-cyc})$. There exists a pure
subgroup $H$ of $G$ such that $H[p]=U$. Moreover $(H,U)$ is isomorphic to a
direct sum of pickets of type $P_{1}^{n}$.

###### Proof.

Let $P=G[p]$. Since $G\in\mbox{$\Sigma$-cyc}$, we have $P=\bigcup_{i\geq
1}P_{i}$, where $P_{1}\subseteq P_{2}\subseteq P_{3}\subseteq\ldots$, and each
subgroup $P_{i}$ is of bounded height, see Theorem 6. Let $U_{i}=U\cap P_{i}$.
Then $U=\bigcup_{i\geq 1}U_{i}$, where $U_{1}\subseteq U_{2}\subseteq
U_{3}\subseteq\ldots$, and each subgroup $U_{i}$ is of bounded height.

We are going to construct pure independent subsets $X_{1}\subseteq
X_{2}\subseteq X_{3}\subseteq\ldots$ such that $(X_{i})\cap P=U_{i}$. Suppose
that $X_{1},\ldots,X_{n}$ are constructed. We apply Lemma 5 to $Q=(X_{n})\cap
P=U_{n}$, $R=U_{n+1}$ and get a pure independent subset $X_{n+1}$ such that
$(X_{n+1})\cap P=U_{n+1}$. Let $X=\bigcup_{i\geq 1}X_{i}$.Then $H=(X)$ is a
pure subgroup of $G$ with $H[p]=U$.

Since $G\in\mbox{$\Sigma$-cyc}$, we have $H\in\mbox{$\Sigma$-cyc}$, see [8,
Theorem 13]. It follows that $(H,U)$ is a direct sum of pickets of the form
$P_{1}^{n}$, because $H[p]=U$. $\square$

######  Lemma 8.

Let $(G,U)\in\mathcal{S}_{1}(\mbox{$\Sigma$-cyc})$ such that
$G/U\in\mbox{$\Sigma$-cyc}$. Let $H$ be a pure subgroup of $G$ such that
$H[p]=U$. Then $H$ is a direct summand of $G$.

###### Proof.

It follows from [7, Theorem 1]: if $G/U$ is a direct sum of cyclic groups, $U$
is a subgroup of $G[p]$ and $H$ is a pure subgroup of $G$ with $H[p]=U$, then
$G$ is a direct sum of cyclic groups and $H$ is a direct summand of $G$.
$\square$

We are ready to prove Theorem 2.

###### Proof.

Let $(G,U)\in\mathcal{S}_{1}(\mbox{$\Sigma$-cyc})$ such that
$G/U\in\mbox{$\Sigma$-cyc}$. Let $H$ be a pure subgroup of $G$ such that
$H[p]=U$. Moreover let $(H,U)$ be isomorphic to a direct sum of pickets of
type $P_{1}^{n}$. Such a subgroup exists by Lemma 7.

By Lemma 8, the group $H$ is a direct summand of $G$, hence $G=H\oplus
H^{\prime}$ for some subgroup $H^{\prime}$ of $G$. The group $H^{\prime}$ is a
direct sum of cyclic groups, because $G$ is a direct sum of cyclic groups, see
[8, Theorem 13].

Therefore $(G,U)=(H,U)\oplus(H^{\prime},0)$. Since $(H^{\prime},0)$ is
isomorphic to a direct sum of pickets of type $P_{0}^{n}$ we are done.

As each picket has a local endomorphism ring, uniqueness follows from the
Krull-Remak-Schmidt-Azumaya theorem, see [1, 3].

Regarding the converse, whenever $(G,U)$ is isomorphic to a direct sum of
pickets, then the factor $G/U$ is in $\Sigma$-cyc. $\square$

##  4\. Generalizations

Let $R$ be any commutative discrete valuation domain with unique maximal ideal
$(p)$. We refer to [9] for corresponding definitions of: order, height, pure
submodule, pure independent subset etc.

By $\mbox{$\Sigma$-cyc}(R)$ we denote the full subcategory of $R$-Mod
consisting of $R$-modules that are direct sums of cyclic primary (i.e.
torsion) $R$-modules. Then $\mathcal{F}_{1}(\mbox{$\Sigma$-cyc}(R))$ and
$\mathcal{S}_{1}(\mbox{$\Sigma$-cyc}(R))$ consist of all pairs $(M,U)$ with
$M\in\mbox{$\Sigma$-cyc}(R)$ and $U$ an $R$-submodule of $M$ such that
$p(M/U)=0$ and $pU=0$ hold, respectively.

For $n\in\mathbb{N}$, we define the pickets $P_{0}^{n}=(R/(p^{n}),0)$,
$P^{n}_{1}=(R/(p^{n}),(p^{n-1}))$, $P_{n-1}^{n}=(R/(p^{n}),(p))$ and
$P_{n}^{n}=(R,R)$ as usual.

Important examples of commutative discrete valuation domains are the
localizations $R_{(p)}$ of a principal ideal domain $R$ at a prime ideal
$(p)$. In this case we can identify the primary $R_{(p)}$-modules and the
$p$-primary $R$-modules, see [9, page 5020]. Two special cases are of
interest:

Abelian $p$-groups are just the primary modules over the localization
$\mathbb{Z}_{(p)}$ of the integers at the prime ideal $(p)$. Hence the result
below covers Theorems 1 and 2.

Let $k$ be a field. For the second case recall that a $k$-linear operator
$T:V\to V$ is called locally nilpotent if for every $v\in V$ there is
$n\in\mathbb{N}$ with $T^{n}(v)=0$. Such operators form the primary modules
over the localization $R=k[T]_{(T)}$ of the polynomial ring at the ideal given
by the variable $T$. Moreover, if an operator $M=(T:V\to V)$ is in
$\mbox{$\Sigma$-cyc}(R)$ then the vector space $V$ decomposes as a direct sum
of finite Jordan-blocks for $T$.

Suppose that $M=(T:V\to T)$ is a locally nilpotent operator and $U$ a
$T$-invariant subspace of $V$ (or $M$). For a pair $(M,U)$ in
$\mathcal{S}_{1}(\mbox{$\Sigma$-cyc}(R))$, the property $TU=0$ holds, and for
$(M,U)\in\mathcal{F}_{1}(\mbox{$\Sigma$-cyc}(R))$, we have $T(V/U)=0$. If the
pair $(M,U)$ a direct sum of pickets, then there are decompositions of $V$ and
$U$ into Jordan-blocks for $T$ such that each Jordan-block for $V$ (as a
vector space) contains the corresponding Jordan-block for $U$.

Here is the generalization of Theorem 1 and Theorem 2.

######  Theorem 9.

Let $R$ be a commutative discrete valuation domain with maximal ideal $(p)$.

  1. (1)

Every object in $\mathcal{F}_{1}(\mbox{$\Sigma$-cyc}(R))$ is isomorphic to a
direct sum of pickets of type $P_{n}^{n}$ or $P_{n-1}^{n}$.

  2. (2)

An object $(M,U)$ in $\mathcal{S}_{1}(\mbox{$\Sigma$-cyc}(R))$ is isomorphic
to a direct sum of pickets of type $P_{0}^{n}$ or $P_{1}^{n}$ if and only if
$M/U\in\mbox{$\Sigma$-cyc}(R)$.

  3. (3)

Every decomposition of an object as a direct sum of pickets is unique up to
reordering and isomorphy of the summands. $\square$

To prove this result we can follow arguments of previous section with the
following changes.

  * •

To prove Lemma 5 for $R$ we can follow proof of [8, Lemma 11]. The arguments
given can be replaced by corresponding argumenst in [9] for pure $R$-modules.

  * •

Instead of Theorem 6 we apply [9, Theorem 8.4].

  * •

We also can follow arguments in the proof of Lemma 7, but instead of Theorem
13 in [8] we apply Theorem 8.5 in [9].

  * •

Also proof Theorem 1 in [7] can be repeated for modules over $R$.

Acknowledgement. The first named author want to thank Stanisław Kasjan for
fruitful discussions.

## References

  * [1] G. Azumaya, Corrections and Supplementaries to My Paper concerning Krull-Remak-Schmidt’s Theorem, Nagoya Math. J. 1 (1950), 117–124.
  * [2] G. Birkhoff, Subgroups of abelian groups, Proc. Lond. Math. Soc., II. Ser. 38 (1934), 385–401.
  * [3] A. Facchini Module Theory. Endomorphism rings and direct sum decompositions in some classes of modules, MBC Series, 1998.
  * [4] L. Fuchs, Infinite Abelian Groups, Volume 1, Academic Press, 1970.
  * [5] L. Fuchs, Infinite Abelian Groups, Volume 2, Academic Press, 1973.
  * [6] N. Gao, J. Külshammer, S. Kvamme, Ch. Psaroudakis, A functorial approach to monomorphism categories II: Indecomposables, (2023), 46pp., arXiv:2303.07753
  * [7] J. Irwing and J. Swanek, On purifiable subsocles of a primary abelian group, Can. J. Math., Vol. XXIII, (1971), 48–57.
  * [8] I. Kaplansky, Infinite Abelian Groups, Revised edition, Dover edition, Dover Publications, 2018.
  * [9] P. A. Krylov and A. A. Tuganbaev, Modules ocer Discrete Valuation Domains. I, Journal of Mathematical Sciences, Vol. 145, No. 4, (2007), 4497–5117.
  * [10] T-Y. Lam, Lectures on modules and rings, Graduate Texts in Mathematics No. 189, Berlin, New York, 1999.
  * [11] A. Moore, The Auslander and Ringel-Tachikawa Theorem for submodule embeddings, Communications in Algebra 38, (2010), 3805–3820.
  * [12] M. Schmidmeier, Bounded submodules of modules, J. Pure Appl. Algebra 203, (2005), 45–82.


# Increasing the Speed and Accuracy of Data Labeling
Through an AI Assisted Interface

Michael Desmond<EMAIL_ADDRESS>,  Zahra Ashktorab
<EMAIL_ADDRESS>,  Michelle Brachman<EMAIL_ADDRESS>,
Kristina Brimijoin<EMAIL_ADDRESS>,  Evelyn Duesterwald
<EMAIL_ADDRESS>,  Casey Dugan<EMAIL_ADDRESS>,  Catherine Finegan-
Dollak<EMAIL_ADDRESS>,  Michael Muller<EMAIL_ADDRESS>,  Narendra
Nath Joshi<EMAIL_ADDRESS>,  Qian Pan<EMAIL_ADDRESS>and
Aabhas Sharma<EMAIL_ADDRESS>IBM ResearchUSA

(2021)

###### Abstract.

Labeling data is an important step in the supervised machine learning
lifecycle. It is a laborious human activity comprised of repeated decision
making: the human labeler decides which of several potential labels to apply
to each example. Prior work has shown that providing AI assistance can improve
the accuracy of binary decision tasks. However, the role of AI assistance in
more complex data-labeling scenarios with a larger set of labels has not yet
been explored. We designed an AI labeling assistant that uses a semi-
supervised learning algorithm to predict the most probable labels for each
example. We leverage these predictions to provide assistance in two ways: (i)
providing a label recommendation and (ii) reducing the labeler’s decision
space by focusing their attention on only the most probable labels. We
conducted a user study (n=54) to evaluate an AI-assisted interface for data
labeling in this context. Our results highlight that the AI assistance
improves both labeler accuracy and speed, especially when the labeler finds
the correct label in the reduced label space. We discuss findings related to
the presentation of AI assistance and design implications for intelligent
labeling interfaces.

AI Assistance, Data Labeling, Human Computer Interaction.

††journalyear: 2021††copyright: acmcopyright††conference: 26th International
Conference on Intelligent User Interfaces; April 14–17, 2021; College Station,
TX, USA††booktitle: 26th International Conference on Intelligent User
Interfaces (IUI ’21), April 14–17, 2021, College Station, TX, USA††price:
15.00††doi: 10.1145/3397481.3450698††isbn: 978-1-4503-8017-1/21/04††ccs:
Human-centered computing Empirical studies in HCI

##  1\. Introduction

As AI becomes ever more ubiquitous, so too does the need to label and annotate
training data. Despite significant algorithmic advances (Vinyals et al., 2016;
Arazo et al., 2020) and recent labeling paradigm shifts (Ratner et al., 2017),
human-in-the-loop data labeling still remains the most reliable method of
acquiring labeled data. But humans make mistakes, and human time is costly.
Many real-world AI systems, such as conversational agents, require labelers to
work with large label sets containing tens, hundreds, or even thousands of
labels. As the number of labels increases, the labeler’s decision process
becomes more complex, and the mechanics of labeling become more laborious and
time consuming.

A common paradigm for supporting and optimizing human decision processes is to
leverage human-AI collaboration (Cai et al., 2019b; Wang et al., 2019a; Wolf
and Blomberg, 2019). Lai and Tan (Lai and Tan, 2019) have shown that AI
assistance—in the form of exposing a human to the predictions of an AI
model—can significantly improve the accuracy of human decisions on a
dichotomous task. This finding is of particular interest for data labeling, a
process of repeatedly capturing and encoding human decisions. We show that the
benefits of AI assistance for decision making can also be leveraged in more
complex data labeling scenarios with large label sets.

We designed and evaluated an AI assisted data-labeling interface wherein a
predictive algorithm assists the human labeler by (a) identifying the most
probable labels that apply to the unlabeled example and (b) reducing the
labeling decision space to help the labeler quickly choose the correct label.
We conducted a user study, recruiting 54 participants from Amazon Mechanical
Turk to perform a series of labeling tasks with a label set of 21 labels. One
group of participants, the baseline, were unassisted, while the other two
groups were assisted by two variants of an AI model, a weak minimally trained
model and a strong model. The strong model had access to more labeled data and
thus exhibited better predictive performance (accuracy) than the weak model.
Both AI models were trained using a transductive semi-supervised learning
algorithm (Zhou et al., 2004). Labeling assistance was provided by using the
model’s calculated label probability distribution to rank, annotate and reduce
the set of labels presented to the labeler. The result is a set of recommended
labels to focus the labeler’s attention on the most likely label choices. Our
experiment set out to test the following hypotheses:

  * •

H1: Providing AI assistance increases human accuracy during data-labeling
tasks.

  * •

H2: Providing AI assistance decreases the time required to perform data-
labeling tasks.

  * •

H3: Improving the accuracy of AI assistance improves the accuracy of the human
labeler.

Our results indicate that human labelers are significantly more accurate when
assisted by a predictive model during labeling activities. Overall our AI
assisted participants were 6% more accurate than the unassisted participants.
We found no significant difference in accuracy between weaker (less accurate)
AI assistance and stronger (more accurate) AI assistance. This was somewhat
surprising and suggests that labelers are unlikely to overly rely on
assistance; instead, they can discern when the predictions are inaccurate and
make their own labeling decisions. We also found significant improvement in
the amount of time required to perform labeling tasks in the presence of AI
assistance. Labeling speed improvements were especially pronounced when
labelers found the desired label in the reduced label space provided by the AI
assistant. This paper makes the following contributions:

  * •

We developed an AI-assisted data-labeling approach where, for each example
presented to the labeler, a predictive model ranks, annotates, and reduces the
label set, focusing the labeler’s attention on the most probable labels.

  * •

We conducted a user study with 54 participants from Mechanical Turk to
evaluate the impact of our AI assistance on human labeler performance
demonstrating that AI assistance significantly improves labeler accuracy.

  * •

The user study also shows that AI assistance reduces the time taken to
complete a labeling task, especially when labelers find a label among the
reduced set of labels recommended.

  * •

We discuss the implications of our findings around the design of AI assistance
for labeling tasks.

##  2\. Related Work

Human-AI Interaction: Human-computer interaction is increasingly concerned
with the collaboration between AI and human (Stinson, 2018; Cai et al.,
2019a). Users are optimistic about the incorporation of AI-assisted decision
making in data science (Wang et al., 2019a) and pedagogical tools (Abu Ghali
et al., 2018; Stinson, 2018) and researchers are interested in the outcome of
these interactions (Oh et al., 2018; Wang et al., 2019a). Joint human-AI
decision making outcomes can be improved if user trust towards the AI is
calibrated appropriately (Zhang et al., 2020) and when there is an appropriate
level of contribution from each party in a human-AI collaborative context (Lai
and Tan, 2019; Mackeprang et al., 2019).

An implicit part of AI-assisted decision making is how the AI system’s results
are presented to the user. Factors like explainability and transparency can
influence humans’ impressions and trust in an AI system. Studies investigating
explainability have explored the role of transparency in AI systems. The
Expectation Confirmation Model (ECM) states that user expectations are
directly related to the user satisfaction of a system (Bhattacherjee, 2001).
One way to set user expectation is by making the system more transparent by
showing accuracy or confidence of a model. Researchers have investigated
making the AI system more transparent to users by displaying model predictions
and confidence values (Rosenthal and Dey, 2010; Lai and Tan, 2019). Rosenthal
and Dey (Rosenthal and Dey, 2010) explored the types of information that are
most useful to present to labelers to maximize labeling accuracy. They found
that showing model predictions and uncertainty had a positive effect on
accuracy. Lai and Tan. (Lai and Tan, 2019) demonstrated that providing humans
with machine predictions significantly improved human decision-making
performance in a deception-detection task. They found that showing predictions
of an AI model resulted in a 21% accuracy improvement and showing the
predictions along with confidence scores resulted in a 46% relative
improvement. In a similar study, Zhang et al. (Zhang et al., 2020) measured
improvement in users’ trust when provided with prediction and confidence
scores but found no significant improvement in accuracy. The authors
attributed the insignificant accuracy gain to the human and AI having little
performance divergence on the task. The human and the AI rarely disagreed.

In the aforementioned studies, the human subjects were presented with
dichotomous tasks. Our aim is to evaluate accuracy and efficiency effects of
AI assistance but in a more complex decision making scenarios, such as
choosing amongst a large label set.

Labeling Assistance Algorithms & Tools: Since labeling data is considered
time-consuming, expensive and difficult, a significant body of work has
focused on ways to increase the labeler’s accuracy and efficiency, and reduce
tedium. Many of these are interactive machine learning approaches (Dudley and
Kristensson, 2018) in which human judgement and computational intelligence are
paired in a cooperative setting.

Active Learning (Settles, 2009) is a labeling paradigm in which the learner
(the model being trained or a proxy) assists the human labeler by choosing
which examples to label from a pool of unlabeled data. Active learning is a
corner stone of interactive machine learning and has been shown to
significantly reduce the amount of data that needs to be labeled; however, it
does not make individual labeling decisions easier, and may make labeling more
difficult due to use of uncertainty as a common selection heuristic (Settles
et al., 2008).

Sun et al. (Sun et al., 2017) studied visualizations of the learning process
during interactive labeling, and found that visual representations of model
performance, such as model predictions, improved understanding and motivation
and helped to avoid redundant labeling. Rosenthal and Dey (Rosenthal and Dey,
2010) also found that model predictions and uncertainty scores helped to
improve labeling accuracy. For instance AILA (Choi et al., 2019) incorporates
label predictions in an assisted labeling system.

A particular aspect of labeling accuracy is related to consistency. Kulesza et
al. (Kulesza et al., 2014) introduced concept evolution, a phenomenon where
the labeler’s understanding of a label changes over time due to exposure to
data, resulting in temporally inconsistent labels. It’s reasonable to assume
that the predictions of a model might help to improve accuracy by recommending
labels in a manner consistent with existing labeled examples.

A variety of more elaborate assisted labeling systems have been proposed.
Settles (Settles, 2011) designed a dual interface labeling tool entitled
“Dualist” that uses an information gain algorithm to learn feature-label
associations from labeled data. The associations are presented to the labeler
who can then confirm the association, thus rapidly labeling the data in a very
efficient manner. Studies indicated that Dualist significantly improved
labeling efficiency. In a similar vein Williams et al. (Williams et al., 2015)
developed an interactive approach to intent labeling for text-based virtual
assistants. An initial search of an unlabeled corpora provides training data
for a machine learning model, which is then applied to the unlabeled corpora
in an active learning style refinement process. The labeler’s experience is
simplified to a sequence of binary labeling tasks performed with the aid of
model predictions. Ratner et al. (Ratner et al., 2020) introduced the idea of
weak supervision, an approach to data labeling where human provided heuristics
(labeling functions) are used in conjunction with a generative model to label
data at scale. These projects show that the application of predictive models
has potential to transform the labeling process into a simpler and less
tedious experience.

Overreliance: “The tendency to over-rely” or automation bias (Goddard et al.,
2012) has been investigated in areas such as aviation (Riley, 1994), as well
as medicine(Goddard et al., 2012). In human-AI collaborative environments,
humans make decisions based in part on the AI’s recommendations. When humans
overly rely, they are not using their own cognitive capacities to make a
decision and rather are blindly following the system’s recommendations. Any
“human in the loop” process relies on humans to provide valuable feedback.
Thus, overreliance, i.e., the human not using their judgment to make a
decision, can be detrimental to that process (Shneiderman, 2020). Labelers
overly relying on AI Assistance is a concern. We know that labeling tasks are
tedious, and the temptation to just leave the decision to the predictive model
is common. We study overreliance by considering how humans perform when
working with two different AI systems, one more accurate than the other.

##  3\. AI-Assisted Labeling

A driving goal of our work was to understand how to infuse AI assistance into
the labeling decision process to enhance human performance. In particular, we
wanted to study how the predictions of an AI model can be used to recommend
labels and reduce the decision space that the labeler needs to consider. To
further this work we implemented an AI Assisted Labeling system. The system is
driven by a semi-supervised learning algorithm that uses both labeled and
unlabeled data to generate label probability distributions for unlabeled data.
The label distributions are used to both rank and annotate the labels that are
presented to the labeler. The system interface presents only the five most
probable labels (referred to as the ”top 5”), but the user has the option to
view the rest of the labels, if necessary.

###  3.1. AI Assistance Algorithm

At the core of our AI Assisted Labeling system is label spreading (Zhou et
al., 2004), a transductive semi-supervised learning algorithm that predicts
label distributions by propagating label information probabilistically via a
graph representation of both labeled and unlabeled data. The algorithm is
transductive, meaning that it predicts labels for the unlabeled examples that
have already been seen by the algorithm, as opposed to a generalizing model
that predicts labels for unseen examples. In general, semi-supervised learning
(Van Engelen and Hoos, 2020) is an good choice for implementing labeling
assistance due to the coexistence of labeled and unlabeled data. Moreover,
transductive graph-based approaches are suitable as they can still operate
reliably even if the amount of labeled data is limited. Our labeling system
invokes label spreading after a small number of seed labels have been obtained
in order to create recommendations for the remaining labeling tasks.

###  3.2. User Experience

AI assistance is presented to the labeler via a visual system of ordering,
annotating, and reducing the visible label set as shown in Figure 1. The list
of labels is ordered based on the AI assistance prediction probabilities. The
ordering is explained by overlaying probability information (blue bars) which
serves as an indication of the assistant’s confidence.

Figure 1. The assisted labeling interface.

The assisted labeling interface.

Figure 2. The assisted labeling interface after the user clicked “View More.”

The assisted labeling interface.

The list of labels is initially limited to the top five with the highest
confidence scores. Limiting the label display to the top five reduces the
complexity of the label decision; however, the user is free to expand the full
list of labels using the “View More” option, as illustrated in Figure 2. We
chose five based on an inspection of the probability and accuracy values of
the AI models. As shown in Figure 2, in most cases, probability values drop
below one percent past the top five, rendering additional labels improbable
from the perspective of the AI models. Also, in most cases, the true label of
the example is among the top five. The label distribution is always shown
rounded to whole percentages.A score less than one but greater than zero is
presented as “¡1%” so that all categories are possible answers; none has zero
probability.

##  4\. Study Design

We ran a between-subjects experiment to investigate whether our AI labeling
assistance system improves labeling accuracy and speed, particularly in the
context of a large label set. The study was designed to compare how labelers
perform with and without AI assistance. Furthermore, the study considers two
variants of AI assistance, weak and strong, to understand if improving the
accuracy of the assistance also improves human performance. We next describe
the details of our study: the labeling task, conditions, methodology, our
participants, materials, and the data collected.

###  4.1. Labeling Task & Dataset

Participants were asked to examine and label a total of 100 (21 training + 79
experimental) short-text customer inquiries, within 30 minutes. The topic of
the inquiries was consumer telecommunication service, and each inquiry had one
‘ground truth’ label. There was a choice of 21 unique labels, each
corresponding to a customer intent. For example, the inquiry “Why does my call
drop every time I drive to work?” is correctly labeled with the intent
“Network Complaints,” meaning that the customer is unhappy about some aspect
of the network and their intent is to complain. Table 1 shows the 21 labels.

We chose the dataset due to its general domain accessibility for non-
specialists, its realistic use case, and because of the number of labels
included. Our goal was to focus on larger label spaces to create a more
challenging decision space. Considering work related to working memory
capacity (Miller, 1956; Cowan, 2010), feasibility of visual presentation, and
availability of appropriate data, we deemed 21 labels to be sufficiently
‘large’ for our purposes.

Table 1. The 21 Study labels Label
---
Activate Device
Activate Prepaid Plan
Activate Roaming Add Insurance
Add Service Features
Change Price Plan
Coverage Area Inquiry
Deactivate Prepaid Plan
Deactivate Roaming
Device Upgrade Eligibility
International Rate Plan Inquiry
Network Complaints
Network Unlock
Port In
Price Plan Inquiry
Recharge SIM
Remove Service Features
Return Device
Roaming Inquiry
Swap Device
Troubleshooting

###  4.2. Conditions

The study was organized into three distinct labeling conditions, spanning two
interfaces. Subjects in the baseline unassisted condition performed the
labeling task with no AI assistance using the labeling interface in Figure 3.
The baseline interface presented all available labels in alphabetical order.

Subjects in the weak assistance condition were assisted by a predictive model
that was minimally trained to predict labels, while those in the strong
assistance condition were assisted by a model with more training that provided
more accurate predictions. Including both weak and strong assistance enabled
us to analyze how performance of the assistant affects performance of the
human labeler. Participants in both assisted conditions performed the study
task using the assisted interface shown in Figure 1 and 2, and the predictive
performance of the assistant differentiated the conditions.

To implement the weak and strong assistance conditions with differing degrees
of predictive performance, we varied the amount of labeled data input to the
label-prediction algorithm and predicted two snapshot label distributions over
the study dataset. Table 2 describes the amount of labeled data, the model’s
accuracy on the study dataset, and top five accuracy—the percentage of
examples for which the true label is included in the top five—for the two
conditions.

Table 2. AI-assistance performance details on the set of 79 study examples. Assistance Condition | # Labeled | Acc. | Top 5 Acc.
---|---|---|---
Weak | 21 | 0.44 | 0.86
Strong | 221 | 0.59 | 0.94

Figure 3. The unassisted labeling interface used by the baseline group.

The unassisted labeling interface used in the baseline condition.

###  4.3. Methodology

We prepared the participants for the study using an interactive training
session, which was completed first and presented using the appropriate
interface for the condition (baseline or assisted). During training each
participant labeled 21 examples, one from each of the possible labels in the
dataset. The examples used for training were chosen to be the most typical
examples for their class/label. We selected these examples algorithmically by
calculating the set of examples closest to the centroid (class mean vector) of
each class. Participants were actively reinforced during the training phase.
Upon selecting the correct label the participant was presented with a message
indicating that the selection was correct, along with the corresponding label
guideline. An incorrect selection resulted in a message indicating such, along
with the correct label and its corresponding guideline. Participants were also
reminded that label guidelines were available via contextual tooltips
throughout the study.

After completing the training phase, participants moved on to the experimental
phase. They were asked to label the remaining 79 examples, one at a time.
Unlike training, the participants did not receive feedback on whether the
selected label was correct; however, the tooltips with label guidelines were
available. The order of the 79 examples (which constituted the experimental
portion of the study data) was random, and the order was reversed for one half
of the participants exposed to each condition, to counteract any potential
order effects.

###  4.4. Participants

A particular concern of the study was to create a realistic data-labeling
scenario wherein the labelers were sufficiently motivated to perform the
labeling task to the best of their abilities. We recruited our study
participants via the Amazon Mechanical Turk platform and specifically
requested ‘Master Workers’ who have a track record of sincere effort. To
motivate workers we advertised a monetary bonus linked specifically to
labeling accuracy. Each participant earned a 2¢ bonus per example correctly
labeled in addition to the flat payment of $4.25 (chosen to be above the
current US hourly minimum wage) for completing the 30-minute study. A maximum
bonus of $1.58 could be earned on the task.

Most of our participants had significant experience on Mechanical Turk: 43 had
more than 3 years of experience, 7 had 2-3 years, and 4 had 1-2 years. Of our
participants, 50 primarily spoke English, while 2 primarily spoke Hindi and 1
primarily spoke Tamil. All participants had at least a high school education,
and 41 participants had formal education beyond high school. While most
participants (30) had heard about AI, 13 closely followed AI-related news, 8
had AI related work experience, and 3 had extensive experience in AI research
or development.

###  4.5. Materials

The dataset used in the study contained a total of 420 examples, labeled by
experts with experience designing customer assistance chatbots. The data was
balanced among the 21 classes (labels) with some minor variance. We encoded
each example in the dataset to a 512-dimensional vector using the Universal
Sentence Encoder (Cer et al., 2018). To create a high-quality and consistent
training experience for the participants to get familiar with the labeling
problem, we calculated the ‘centroid’ or mean vector for each class, and
selected a set of 21 examples closest to the centroids as training data. We
then randomly selected and removed a set of 79 of the remaining examples to be
labeled by the participants during the study, ensuring the set was balanced
across the 21 labels. This left us with 320 examples to implement the
assistance conditions.

####  4.5.1. Implementing Weak Assistance

For the weak assistance condition, we selected 21 labeled examples from the
320 remaining examples using the same nearest-to-centroid approach. We then
applied label spreading 111We used Scikit Learn (Pedregosa et al., 2011)’s
implementation of label spreading (kernel=“knn”, n_neighbors=10) for both weak
and strong assistance. to propagate labels from the 21 examples into the
remaining 299 examples and the study data (79 examples) combined. This models
a realistic scenario where label information is diffused from an initial
labeled dataset into a larger unlabeled dataset. We used the predicted label
distributions on the study data examples to create the weak assistance
condition.

####  4.5.2. Implementing Strong Assistance

For the strong assistance condition we simply expanded the labeled data
iteratively using a low-margin active learning selector. After 20 active
learning iterations (adding 10 labeled examples per iteration) we again
applied the label spreading algorithm to generate label distributions over the
study dataset. These distributions were used in the strong assistance
condition.

###  4.6. Data Collected

The following information was collected from each participant during the
study.

####  4.6.1. Selected Labels

For each example presented we stored the label selected by the participants.
We were then able to compare the participant’s selection to the ground truth
label to determine individual participant labeling accuracy.

####  4.6.2. Labeling Duration

The amount of time spent labeling was calculated as the duration between the
labeler being presented with the example and when the participant clicked the
button to submit the label and transition to the next example. This approach
removed any variation due to differences in page loading times. The labeling
durations were also winsorized with a 5% lower limit and a 95% upper limit in
order to reduce the the effect of spurious outliers, such as when a labeler
would take a break or need to attend to some unrelated matter while performing
their task. Winsorizaton was applied per participant.

####  4.6.3. “View More” events

We recorded all use of the “View More” function, which was available to the
assisted conditions only. This event indicates that the participant expanded
the visible subset of labels to see the full list. This would provide a
valuable insight into how users interacted with the decision space reduction
feature.

####  4.6.4. Questionnaire

Participants also completed a post-experiment questionnaire, which requested
feedback about the labeling experience in general, and perceptions of the weak
and strong AI if conditions allowed.

##  5\. Results

A total of 54 participants successfully completed the experiment and an
average bonus of $1.20 was earned. Of the 54, 19 participants completed the
baseline condition, 18 completed the weak assistance condition, and 17
completed the strong assistance condition.

###  5.1. Labeling Accuracy

We performed a one-way ANOVA to evaluate the accuracy differences between the
three study conditions: baseline, weak assistance, and strong assistance. Data
is presented as mean $\pm$ standard deviation. Accuracy was significantly
different between the three conditions, F2, 51 = 4.18, p ¡ 0.02. Accuracy
decreased in the baseline condition (0.72 $\pm$ 0.10) compared to the weak
assistance condition (0.79 $\pm$ 0.05) and the strong assistance condition
(0.78 $\pm$ 0.05), as shown in Figure 4. Tukey post-hoc analyses revealed that
the increase from baseline to weak assistance was statistically significant (p
¡ 0.03). Participants were more accurate in the weak assistance condition than
in the baseline condition. No other condition differences were statistically
significant. The accuracy difference between the baseline condition and strong
assistance condition was not statistically significant (p ¡ 0.07), but survey
results suggest that strong assistance positively affected labeler accuracy.

Figure 4. Mean labeling accuracy across the three study conditions. The mean
value is represented by the white circle.

Mean labeling accuracy across the three study conditions.

In the post-study survey participants rated the helpfulness and accuracy of
the label recommendations highly in both the weak and strong assistance
conditions. Across assisted conditions, 91% of participants rated the
assistance as helpful. Of the three participants who did not find the
assistance helpful, two were in the weak assistance condition. Fewer
participants rated the suggested labels as accurate than helpful, with 80% of
participants agreeing that the assistance was accurate, 9% neutral, and 11% in
disagreement. Using the Fisher test, we did not find significant differences
between the strong and weak assistance conditions in how participants rated
the helpfulness or accuracy of the assistance. Overall, participants often
commented that the suggested labels were helpful ”when they were right”.

> ”The ones where the recommendation was correct were exciting because it
> saved me time.” \- P9-WA (weak assistance)

These results support our hypothesis H1 since our findings show that AI
assistance increases accuracy in labeling tasks. A further breakdown of
assisted labeling accuracy, in particular considering the use of the “View
More” function, revealed the following. In general participants in the weak
assistant condition used the “View More” feature in 29% (28.8 $\pm$ 0.8) of
labeling tasks, and participants in the strong assistant condition used the
feature in 17% (16.6 $\pm$ 0.7) of their tasks. This is consistent with the
performance differences between the AI conditions, namely that better
predictive performance would account for reduced necessity of the “View More”
feature. In the cases where the view more option was used participants in the
weak assistance condition achieved 66% accuracy (65.9 $\pm$ 0.1), while those
in the strong assistance condition achieved 60% accuracy (60.1 $\pm$ 1.2).
Alternatively when the view more option was not used participants in the weak
assistance condition achieved 84% accuracy (83.4 $\pm$ 0.4), while those in
the strong assistance condition achieved 81% accuracy (81.2 $\pm$ 0.4).

###  5.2. Labeling Speed

A general linear mixed-effects model (a.k.a. mixed model) analysis of variance
was used to analyze labeling duration. The assistance condition and whether
the “View More” option was selected were modeled as fixed effects, while
trials were nested within each subject and modeled as a random effect. Mixed
models are more robust since they consider the output of each individual trial
and also are more robust to missing data and unbalanced designs (Krueger and
Tian, 2004).

There was a significant effect of assistance condition (baseline, weak
assistance, strong assistance) on overall labeling duration (measured in
seconds) F2,58.2=5.71, p ¡ .001. Tukey post hoc tests showed that users spent
less time on each trial in the weak assistance condition (8.81 $\pm$ 14.6)
than baseline condition (9.36 $\pm$ 9.58), p ¡ 0.004, and spent less time on
each trial in the strong assistance condition (9.07 $\pm$ 16.5) than the
baseline condition (9.36 $\pm$ 9.58), p ¡ 0.04, as shown in Figure 5. There
was also a significant effect of “View More” selection on duration (“View
More” selected, “View More” not selected) F1,3730.3 = 202.59, p ¡ 0.001; the
trials in which participants selected the “View More” button (15.97 $\pm$
15.17) were longer in duration than those who did not (7.89$\pm$ 13.12),
regardless of the AI assistance condition. Additionally, when participants
used the “View More” feature in the assistance conditions (strong (18.38 $\pm$
22.50), and weak (14.65 $\pm$8.69)), they took longer to complete the labeling
task than participants in the baseline condition (9.36 $\pm$ 9.58). We
attribute this result to the fixed alphabetical order of labels in the
unassisted condition, which was more efficient to use compared to the
confidence ordered label list used in the assistance conditions, for more
discussion on this see 6.1

Figure 5. A plot of average duration of labeling time per trial for various
assistance conditions.

graph

To better understand user behavior and the outcome of the labeling task, we
must consider the user interface interaction effects of the use of “View more”
and the condition. We found a significant interaction of View More Selection x
AI Assistance Condition on duration F1,4253.5=6.5. p ¡ 0.05. Tukey post-hoc
analysis revealed significant differences when “View More” was not selected
between the baseline condition (9.36 $\pm$ 9.58) and the weak assistance
condition (6.45 $\pm$ 15.79) p<0.05, but not between the baseline condition
and the strong assistance condition, (7.21 $\pm$ 14.33) p=0.15. When users did
not select the “View More” button, they completed the labeling task more
quickly in the weak assistance condition. When the “View More” button was
selected, there were significant differences between the strong assistance
condition (18.4 $\pm$ 22.5) and the weak assistance condition (14.65
$\pm$8.69) p<0.05. This result is likely related to the reduced usage of the
“View More” feature (learning effects) and the more conservative confidence
distributions (See section 6.2) exhibited by the strong assistance condition.

Figure 6 shows the durations for each of the conditions. Our results support
H2, showing that AI assistance makes labeling tasks more efficient, by
decreasing the amount of time users take per task, especially when the labeler
finds the label in the reduced set of recommendations provided by the
assistant and does not need to use the “View More” button.

Figure 6. A plot of average duration of labeling time per trial for various
assistance conditions. Hues represent whether “View More” button was selected.

graph

Our survey results provide further insight into the approach of reducing the
initial label set to the top 5 most probable labels, and reordering of the
label list by most probable. All participants in the baseline condition agreed
that it was easy to find the label they were looking for. Most participants
with labeling assistance (86%) also found it easy to find the label they
needed in their original five recommended labels. However, when participants
with AI assistance found it necessary to “View More” to see the full set of
labels, only 49% of participants found it easy to find the label that they
were looking for. When asked to describe their process looking through the
long list of labels, participants wrote:

> ”If it wasn’t in the top five, it was a bit annoying. I just used the search
> feature in my browser (ctrl+f) to find what I wanted in the long list.”
> P23-WA

> ”I had to look through them [the labels] more closely/taking more time to
> find one that I thought fit.” P31-SA (strong assistance)

Participants cited the “View More,” the long length, and lack of organization
of labels as features they liked the least.

###  5.3. Variation in Predictive Accuracy

Surprisingly, a comparison of labeler accuracy between the two assistance
groups shows that improving the predictive performance of the AI did not lead
to improved labeling accuracy. The accuracy difference between the weak
assistance condition (0.79 $\pm$ 0.05) and the strong assistance condition
(0.78 $\pm$ 0.05) was not statistically significant. Furthermore, the weak
assistance group used the “View More” feature significantly more often than
the strong assistance group F(12.3) = 202.59, p <0.001. Recall that weak
assistance had inferior model accuracy when compared to strong assistance, and
labelers need to search for labels outside of the recommended set more often.
In light of these results we reject H3. Participants were not more accurate
when their labeling tasks were assisted by a more accurate predictive model.

With any system that is automated, there is a risk that the user will overly
rely on the system and relax their own perceptual abilities when making a
decision (Heer, 2019). This was a concern for us when designing the study. An
overreliant labeler might simply accept the top recommended label without
questioning the correctness. However, our results suggest that labelers were
liable to make their own determination of correctness regardless of the
quality of the assistant’s recommendation. We analyzed the level of agreement
between our participants and the AI predictions, and found that in the weak
assistance condition participants agreed with the top predicted label in 45%
of tasks (45.0 $\pm$ 0.4) and in the strong assistance condition participants
agreed with the top predicted label in 64% (64.0 $\pm$ 0.5) of the tasks. This
finding is consistent with prior work by Yu, Kun, et al. (Yu et al., 2016)
which established that AI-assisted users were able to perceive the accuracy of
an AI decision support system and adjust their trust accordingly. Sun et al.
(Sun et al., 2017) observed a similar pattern when studying the effectiveness
of visual representations of learning during interactive labeling tasks.

Participants’ survey responses often reflected their efforts to find the
correct response rather than merely relying on the recommended labels. Most
participants seemed to consider the recommendations critically, rather than
assuming that they were necessarily correct, such as P13:

> The recommendations by the AI assistant gave a good ‘base’ to go off of but
> was not full accurate enough. P13-WA

##  6\. Discussion and Design Recommendations

###  6.1. Ordering of Labels

The inclusion of predictive recommendations should compliment the human
tendency to learn and exploit spatial knowledge in a consistent information
space. In some of our open-ended responses in the survey following the
labeling task, users complained about the ordering of labels changing. In the
assisted conditions the labels are ordered based on the confidence of the
label predictions (both before and after clicking “View More”), and a
consistent spatial/alphabetical label order was never materialized. Some
participants noted this in the survey, wishing it was alphabetical or ordered
in another useful way. When asked about finding labels using “View More”,
participants wrote:

> ”They didn’t seem to be ordered very well. It should have been
> alphabetical.” P23-WA

> ”They weren’t grouped in a way that made sense to me. They seemed to be
> randomly listed.” P26-WA

Prior work has shown that users quickly form and rely on spatial knowledge in
menu systems (Cockburn et al., 2007), that brief menus can lead to faster
performance and learning or over-learning a menu-structure over time can
provide performance enhancements (Ahlström et al., 2010). Watts-Perotti and
Woods (Watts-Perotti and Woods, 1999) explain that when information is
spatially dedicated (kept in a consistent place within a display system) the
location itself serves as a memory aid for users who become familiar with the
structure. In our assisted labeling scenario, when a user is able to find the
desired label in the reduced decision space they gain a significant efficiency
advantage by working with a smaller set of relevant options. However when the
user cannot find the correct label in the reduced set they need to expand the
full set of labels and search for the label they want. We suspect that
searching an inconsistently ordered set of items is far less efficient than
relying on spatial knowledge learned from a consistent display space. As such
further labeling efficiency might be achievable by providing an alphabetically
ordered list of labels, annotated with model predictions, alongside the subset
of recommended labels (Top 5 ordered by prediction confidence). Alternatively
the labeler might be allowed to switch between probabilistic or alphabetical
orderings of the label set on demand. These solutions would eliminate the
mechanical task of opening the full set of labels (“View More” button) and
also allow the labeler to fall back on their spatial knowledge in the event
that a satisfactory label prediction was not found.

###  6.2. Confidence of Label Predictions

Future designs should carefully consider the presentation of model confidence
and set user expectations accordingly. Prior work shows that when user
expectations are high or do not match the level that the system is capable of
delivering, users are less satisfied with their experience (Cheng et al.,
2019; Wang et al., 2019b; Wiegand et al., 2019). A participant noted in the
survey that the system presented recommendations with high confidence that
seemed incorrect:

> ”Sometimes it was clearly way off the mark and had very high confidence.”
> P10-SA

Surprisingly, labelers assisted by the weak AI performed slightly better than
those assisted by the strong AI, even through the strong AI demonstrated
considerably better predictive performance on the labeling task. An analysis
of the predictions provided by both AI assistants indicated that the weak AI
was more certain in its predictions than the strong AI. We define certainty as
the inverse scaled entropy of the predictions provided by the assistant over
the course of the labeling task. In particular the weak AI was prone to highly
confident correct and incorrect predictions, while the strong AI exhibited a
more conservative distribution of confidence scores across its predictions.

Figure 7. Top confidence of correct label predictions. The X axis represents
confidence values on the unit interval. The Y axis represents the number of
predictions. The weak assistant tended to produce more highly confident
predictions.

graph

Figure 8. Top confidence of incorrect label predictions. The X axis represents
confidence values on the unit interval. The Y axis represents the number of
predictions. The weak assistant tended to produce more highly confident
predictions.

graph

A deeper comparison of labeling trials showed that in the cases where the
assistant made an incorrect prediction, labelers agreed with the weak AI in
11% (11.1 $\pm$ 5.6) of those trials, but labelers agreed with the strong AI
in 26% (26.4 $\pm$ 11.3) of the trials, an unexpected difference between the
two conditions. It’s possible that the confident but often incorrect weak AI
fostered a greater degree of distrust on behalf of the labelers, who were then
more likely to scrutinize its predictions. On the other hand, the strong AI,
being more often correct and exhibiting a more appropriate confidence range,
may have caused labelers to more often trust incorrect predictions. Overall
the labelers using the weak assistant performed slightly better, perhaps due
to having a little less trust in the predictions of the assistant. See figures
7 and 8 which show the distributions of top confidence scores for correct and
incorrect prediction across the two assistance conditions. Note the skew
towards highly confident predictions for the weak assistance condition.

The effect of confidence scores on labeler performance needs further study.
Characteristics like the certainty of predictions and confidence presentation
may have unexpected consequences on the decision making process.

###  6.3. AI Assistance in low information environments

AI assistance appears to have value in the early stages of labeling
activities, but having a representative set of training examples is vital. AI
assistance can improve labeling performance even with very little labeled
data. Our weak assistant only had 21 labeled examples in order to make
predictions. However provision of representative examples to train the
labeling assistance model is a fundamental concern. A skewed or incomplete set
of training examples may cause labeling errors which compound over time, as
mislabeled examples from previous labeling iterations are rolled into the
assistants training data. It would be pragmatic to provide assistance only
when a representative set of labeled examples is available. In our experiment
we selected one labeled example per label as training for the weak assistant,
but this requirement may change depending on the particulars of the dataset
and the labeling task under consideration.

##  7\. Limitations

This paper reflects one particular context. We studied a single task in this
paper; future work should extend to additional types of tasks with varying
degrees of difficulty. Also, we recruited participants from Mechanical Turk,
which is known to have intersectional gender imbalances in different countries
and cultures (Difallah et al., 2018). Mechanical Turk workers are not subject
matter experts and results may differ for labeling tasks that require domain
expertise. We do believe the results of this study can be leveraged in future
work exploring AI assistance for datasets that require subject matter
expertise.

##  8\. Conclusion

In this paper we studied the impact of AI assistance on human data labeling
performance. In particular we focused on a labeling task with a larger label
set than had previously been studied.

Our results indicate that using a predictive model as an ‘AI assistant’ can
improve labeling accuracy. We found that varying the predictive performance of
the AI did not significantly impact the accuracy of human labelers.
Participants in the study did not over-rely on the predictions of the labeling
assistant, and it seems that the accuracy of human labeling can be improved
with relatively weak AI support.

We also studied the effects of AI assistance on labeling speed. We found that
reducing the labeler’s “decision space” by showing only the top 5 most
confidently predicted labels, improves labeling speed, particularly when the
labeler finds the desired label in the filtered subset. When labelers need to
expand the full label set using the “View More” function, the speed
improvements are reversed. Future work will involve understanding how to
optimally present label predictions without negatively affecting the labelers
ability to efficienty navigate and select from the full set of labels.

In the course of the study, and during subsequent analysis of the results, we
noticed subtle effects of prediction confidence on labeling behavior. Further
study is required to understand such phenomena. We also acknowledge that to
understand the full potential of AI assistance for data labeling, a broader
range of studies with varying degrees of task difficulty and label set size
would be worthwhile.

## References

  * (1)
  * Abu Ghali et al. (2018) Mahmoud J Abu Ghali, Abdullah Abu Ayyad, Samy S Abu-Naser, and Mousa Abu Laban. 2018.  An intelligent tutoring system for teaching english grammar.  (2018).
  * Ahlström et al. (2010) David Ahlström, Andy Cockburn, Carl Gutwin, and Pourang Irani. 2010.  Why it’s quick to be square: modelling new and existing hierarchical menu designs. In _Proceedings of the SIGCHI Conference on Human Factors in Computing Systems_. 1371–1380.
  * Arazo et al. (2020) Eric Arazo, Diego Ortego, Paul Albert, Noel E O’Connor, and Kevin McGuinness. 2020.  Pseudo-labeling and confirmation bias in deep semi-supervised learning. In _2020 International Joint Conference on Neural Networks (IJCNN)_. IEEE, 1–8.
  * Bhattacherjee (2001) Anol Bhattacherjee. 2001\.  Understanding information systems continuance: an expectation-confirmation model.  _MIS quarterly_ (2001), 351–370.
  * Cai et al. (2019a) Carrie J Cai, Emily Reif, Narayan Hegde, Jason Hipp, Been Kim, Daniel Smilkov, Martin Wattenberg, Fernanda Viegas, Greg S Corrado, Martin C Stumpe, et al. 2019a.  Human-centered tools for coping with imperfect algorithms during medical decision-making. In _Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems_. ACM, 4.
  * Cai et al. (2019b) Carrie J Cai, Samantha Winter, David Steiner, Lauren Wilcox, and Michael Terry. 2019b.  ” Hello AI”: Uncovering the Onboarding Needs of Medical Practitioners for Human-AI Collaborative Decision-Making.  _Proceedings of the ACM on Human-Computer Interaction_ 3, CSCW (2019), 1–24.
  * Cer et al. (2018) Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St. John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, Yun-Hsuan Sung, Brian Strope, and Ray Kurzweil. 2018.  Universal Sentence Encoder.  _arXiv preprint_ abs/1803.11175 (2018).  arXiv:1803.11175 http://arxiv.org/abs/1803.11175
  * Cheng et al. (2019) Hao-Fei Cheng, Ruotong Wang, Zheng Zhang, Fiona O’Connell, Terrance Gray, F Maxwell Harper, and Haiyi Zhu. 2019.  Explaining Decision-Making Algorithms through UI: Strategies to Help Non-Expert Stakeholders. In _Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems_. ACM, 559.
  * Choi et al. (2019) Minsuk Choi, Cheonbok Park, Soyoung Yang, Yonggyu Kim, Jaegul Choo, and Sungsoo Ray Hong. 2019\.  Aila: Attentive interactive labeling assistant for document classification through attention-based deep neural networks. In _Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems_. 1–12.
  * Cockburn et al. (2007) Andy Cockburn, Carl Gutwin, and Saul Greenberg. 2007\.  A predictive model of menu performance. In _Proceedings of the SIGCHI conference on Human factors in computing systems_. 627–636.
  * Cowan (2010) N. Cowan. 2010.  The Magical Mystery Four: How is Working Memory Capacity Limited, and Why?  _Current Directions in Psychological Science_ 19 (2010), 51–57.
  * Difallah et al. (2018) Djellel Difallah, Elena Filatova, and Panos Ipeirotis. 2018\.  Demographics and dynamics of mechanical Turk workers. In _Proceedings of the eleventh ACM international conference on web search and data mining_. 135–143.
  * Dudley and Kristensson (2018) John J Dudley and Per Ola Kristensson. 2018.  A review of user interface design for interactive machine learning.  _ACM Transactions on Interactive Intelligent Systems (TiiS)_ 8, 2 (2018), 1–37.
  * Goddard et al. (2012) Kate Goddard, Abdul Roudsari, and Jeremy C Wyatt. 2012\.  Automation bias: a systematic review of frequency, effect mediators, and mitigators.  _Journal of the American Medical Informatics Association_ 19, 1 (2012), 121–127.
  * Heer (2019) Jeffrey Heer. 2019\.  Agency plus automation: Designing artificial intelligence into interactive systems.  _Proceedings of the National Academy of Sciences_ 116, 6 (2019), 1844–1850.
  * Krueger and Tian (2004) Charlene Krueger and Lili Tian. 2004.  A comparison of the general linear mixed model and repeated measures ANOVA using a dataset with multiple missing data points.  _Biological research for nursing_ 6, 2 (2004), 151–157.
  * Kulesza et al. (2014) Todd Kulesza, Saleema Amershi, Rich Caruana, Danyel Fisher, and Denis Charles. 2014.  Structured labeling for facilitating concept evolution in machine learning. In _Proceedings of the SIGCHI Conference on Human Factors in Computing Systems_. 3075–3084.
  * Lai and Tan (2019) Vivian Lai and Chenhao Tan. 2019.  On human predictions with explanations and predictions of machine learning models: A case study on deception detection. In _Proceedings of the Conference on Fairness, Accountability, and Transparency_. 29–38.
  * Mackeprang et al. (2019) Maximilian Mackeprang, Claudia Müller-Birn, and Maximilian Timo Stauss. 2019.  Discovering the Sweet Spot of Human-Computer Configurations: A Case Study in Information Extraction.  _Proceedings of the ACM on Human-Computer Interaction_ 3, CSCW (2019), 1–30.
  * Miller (1956) G. A. Miller. 1956\.  The Magical Number Seven, Plus or Minus Two: Some Limits on our Capacity for Processing Information.  _Psychological Review_ 63 (1956), 81–97.
  * Oh et al. (2018) Changhoon Oh, Jungwoo Song, Jinhan Choi, Seonghyeon Kim, Sungwoo Lee, and Bongwon Suh. 2018\.  I lead, you help but only with enough details: Understanding user experience of co-creation with artificial intelligence. In _Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems_. 1–13.
  * Pedregosa et al. (2011) F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011.  Scikit-learn: Machine Learning in Python.  _Journal of Machine Learning Research_ 12 (2011), 2825–2830.
  * Ratner et al. (2017) Alexander Ratner, Stephen H Bach, Henry Ehrenberg, Jason Fries, Sen Wu, and Christopher Ré. 2017.  Snorkel: Rapid training data creation with weak supervision. In _Proceedings of the VLDB Endowment. International Conference on Very Large Data Bases_ , Vol. 11. NIH Public Access, 269.
  * Ratner et al. (2020) Alexander Ratner, Stephen H. Bach, Henry Ehrenberg, Jason Fries, Sen Wu, and Christopher Ré. 2020.  Snorkel: Rapid training data creation with weak supervision.  _The VLDB Journal_ 29, 2 (2020), 709–730.
  * Riley (1994) Victor Andrew Riley. 1994\.  _Human use of automation._ Ph.D. Dissertation. ProQuest Information & Learning.
  * Rosenthal and Dey (2010) Stephanie L Rosenthal and Anind K Dey. 2010.  Towards maximizing the accuracy of human-labeled sensor data. In _Proceedings of the 15th international conference on Intelligent user interfaces_. 259–268.
  * Settles (2009) Burr Settles. 2009\.  _Active learning literature survey_.  Technical Report. University of Wisconsin-Madison Department of Computer Sciences.
  * Settles (2011) Burr Settles. 2011\.  Closing the loop: Fast, interactive semi-supervised annotation with queries on features and instances. In _Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing_. 1467–1478.
  * Settles et al. (2008) Burr Settles, Mark Craven, and Lewis Friedland. 2008\.  Active Learning with Real Annotation Costs. In _Proceedings of the NIPS Workshop on Cost-Sensitive Learning._
  * Shneiderman (2020) Ben Shneiderman. 2020\.  Human-Centered Artificial Intelligence: Reliable, Safe & Trustworthy.  _International Journal of Human–Computer Interaction_ 36, 6 (2020), 495–504.
  * Stinson (2018) Catherine Stinson. 2018\.  _Healthy Data: Policy solutions for big data and AI innovation in health_.  Mowat Centre for Policy Innovation.
  * Sun et al. (2017) Yunjia Sun, Edward Lank, and Michael Terry. 2017.  Label-and-Learn: Visualizing the Likelihood of Machine Learning Classifier’s Success During Data Labeling. In _Proceedings of the 22nd International Conference on Intelligent User Interfaces_. 523–534.
  * Van Engelen and Hoos (2020) Jesper E Van Engelen and Holger H Hoos. 2020.  A survey on semi-supervised learning.  _Machine Learning_ 109, 2 (2020), 373–440.
  * Vinyals et al. (2016) Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. 2016\.  Matching networks for one shot learning. In _Advances in neural information processing systems_. 3630–3638.
  * Wang et al. (2019a) Dakuo Wang, Justin D. Weisz, Michael Muller, Parikshit Ram, Werner Geyer, Casey Dugan, Yla Tausczik, Horst Samulowitz, and Alexander Gray. 2019a.  Human-AI Collaboration in Data Science: Exploring Data Scientists’ Perceptions of Automated AI.  _Proceedings of the ACM on Human-Computer Interaction_ 3, CSCW (2019), 1–24.
  * Wang et al. (2019b) Danding Wang, Qian Yang, Ashraf Abdul, and Brian Y Lim. 2019b.  Designing Theory-Driven User-Centric Explainable AI. In _Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems_. ACM, 601.
  * Watts-Perotti and Woods (1999) Jennifer Watts-Perotti and David D Woods. 1999.  How experienced users avoid getting lost in large display networks.  _International Journal of Human–Computer Interaction_ 11, 4 (1999), 269–299.
  * Wiegand et al. (2019) Gesa Wiegand, Matthias Schmidmaier, Thomas Weber, Yuanting Liu, and Heinrich Hussmann. 2019.  I Drive-You Trust: Explaining Driving Behavior Of Autonomous Cars. In _Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems_. ACM, LBW0163.
  * Williams et al. (2015) Jason D. Williams, Nobal B. Niraula, Pradeep Dasigi, Aparna Lakshmiratan, Carlos Garcia Jurado Suarez, Mouni Reddy, and Geoff Zweig. 2015\.  Rapidly scaling dialog systems with interactive learning.  In _Natural Language Dialog Systems and Intelligent Assistants_. Springer, 1–13.
  * Wolf and Blomberg (2019) Christine Wolf and Jeanette Blomberg. 2019.  Evaluating the Promise of Human-Algorithm Collaborations in Everyday Work Practices.  _Proc. ACM Hum.-Comput. Interact._ 3, CSCW, Article 143 (Nov. 2019), 23 pages.  https://doi.org/10.1145/3359245
  * Yu et al. (2016) Kun Yu, Shlomo Berkovsky, Dan Conway, Ronnie Taib, Jianlong Zhou, and Fang Chen. 2016\.  Trust and reliance based on system accuracy. In _Proceedings of the 2016 Conference on User Modeling Adaptation and Personalization_. 223–227.
  * Zhang et al. (2020) Yunfeng Zhang, Q. Vera Liao, and Rachel K. E. Bellamy. 2020\.  Effect of Confidence and Explanation on Accuracy and Trust Calibration in AI-Assisted Decision Making. In _Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency_ _(FAT* ’20)_. Association for Computing Machinery, New York, NY, USA, 295–305.  https://doi.org/10.1145/3351095.3372852
  * Zhou et al. (2004) Dengyong Zhou, Olivier Bousquet, Thomas N Lal, Jason Weston, and Bernhard Schölkopf. 2004\.  Learning with local and global consistency. In _Advances in neural information processing systems_. 321–328.