<EMAIL_ADDRESS>
# Anisotropic magnon damping by zero-temperature quantum fluctuations in
ferromagnetic CrGeTe3

Lebing Chen  Department of Physics and Astronomy, Rice University, Houston,
Texas 77005, USA  Chengjie Mao  Department of Mechanical Engineering and
Materials Science, Duke University, Durham, NC 27708, USA  Jae-Ho Chung  –
Department of Physics, Korea University, Seoul 02841, Korea  Matthew B. Stone
Alexander I. Kolesnikov  Xiaoping Wang  Neutron Scattering Division, Oak Ridge
National Laboratory, Oak Ridge, Tennessee 37831, USA  Naoki Murai  J-PARC
Center, Japan Atomic Energy Agency, Tokai, Ibaraki 319-1195, Japan  Bin Gao
Department of Physics and Astronomy, Rice University, Houston, Texas 77005,
USA  Olivier Delaire<EMAIL_ADDRESS>Department of Mechanical
Engineering and Materials Science, Duke University, Durham, NC 27708, USA
Pengcheng Dai<EMAIL_ADDRESS>Department of Physics and Astronomy, Rice
University, Houston, Texas 77005, USA

###### Abstract

Spin and lattice are two fundamental degrees of freedom in a solid, and their
fluctuations about the equilibrium values in a magnetic ordered crystalline
lattice form quasiparticles termed magnons (spin waves) and phonons (lattice
waves), respectively. In most materials with strong spin-lattice coupling
(SLC), the interaction of spin and lattice induces energy gaps in the spin
wave dispersion at the nominal intersections of magnon and phonon modes. Here
we use neutron scattering to show that in the two-dimensional (2D) van der
Waals honeycomb lattice ferromagnetic CrGeTe3, spin waves propagating within
the 2D plane exhibit an anomalous dispersion, damping, and breakdown of
quasiparticle conservation, while magnons along the $c$ axis behave as
expected for a local moment ferromagnet. These results indicate the presence
of dynamical SLC arising from the zero-temperature quantum fluctuations in
CrGeTe3, suggesting that the observed in-plane spin waves are mixed spin and
lattice quasiparticles fundamentally different from pure magnons and phonons.

††preprint: APS/123-QED

In a magnetic ordered crystalline lattice, spin and lattice vibrations about
their equilibrium positions form quasiparticles termed magnons (spin waves)
and phonons (lattice waves), respectively boothroyd . Since these
quasiparticles emerge from linearized theories that ignore all terms of order
higher than quadratic and neglect interactions among the quasiparticles
themselves, they are extremely stable against decay Holstein ; Oguchi .
Moreover, because of the invariance of the ferromagnetic (FM) ground state
under a spin rotation about the magnetization direction, the number of magnons
is conserved colloquium  and spin waves have infinite lifetime throughout the
Brillouin zone Dietrich . In a conventional local moment ferromagnet with a
spin-rotational invariant Heisenberg Hamiltonian, spin waves are characterized
by definite values of the $z$ projection of the total spin $S^{z}$, meaning
every magnon has an intrinsic quantum number $\left|\Delta S^{z}\right|=1$ and
is conserved in magnon scattering processes colloquium . In this case, magnon-
magnon interactions play a minimum role in the modification of spin waves, as
its Hamiltonian only contains the renormalization and the two-particle
scattering term colloquium . As a consequence, the intensity of spin waves
measured by inelastic neutron scattering (INS) as a function of temperature in
a local moment ferromagnet should only be renormalized by the Bose population
factor boothroyd . Therefore, the energy width of peaks in spectra of spin
waves with infinite lifetime should only be limited by the instrumental
resolution, as seen in FM ordered EuO2 Dietrich . In systems where the spin
and lattice degrees of freedom are coupled, the spin-lattice coupling (SLC)
can modify spin waves in several ways. First, a static lattice distortion
induced by SLC may affect the anisotropy of magnon exchange couplings, as seen
in the spin waves of iron pnictides jzhao09 . Second, time-dependent lattice
vibrations interacting with spin waves may give rise to significant SLC. One
possible consequence of such SLC is the formation of energy gaps in the spin
wave dispersion at the nominal intersections of magnon and phonon modes
Anda1976 ; Guerreiro2015 ; Oh2016 . Alternatively, dynamic lattice deformation
may change the spin coupling coefficient that may then decrease the lifetime
of spin waves Silberclitt ; Jensen . Nevertheless, the experimental
observation of spin wave damping in ferromagnets is rare Hwang1998 ; Dai2000 ;
Ye2007 , and may not arise from SLC Helton2017 . Recently, SLC was suggested
to be critical in understanding the ground state properties of two-dimensional
(2D) van der Waals (vdW) FM CrGeTe3 and CrI3 Tian2016 ; YSun2018 ; XLi2014 ;
Webster2018 ; BHZhang2019 ; JLi2021 . In these honeycomb ferromagnets, the
superexchange coupling between nearest neighbor (NN) Cr-Cr bonds mediated with
ligand Te/I atoms is FM, which competes with the antiferromagnetic (AF) Cr-Cr
direct exchange, yielding a net FM interaction between NNs [Figs. 1(a,b)]
CST_INS ; LBChen ; LBChen2020 ; LBChen2021 ; FengfengZhang . A consequence of
this competition is the strong coupling between the inter-atomic distance and
magnetic exchange couplings. For example, in CrGeTe3, the AF Cr-Cr direct
exchange decreases much faster with increasing Cr-Cr distance, compared with
the Cr-Te-Cr superexchange [Fig. 1(b)]. According to $ab$ $initio$
calculations BHZhang2019 ; JLi2021 , the FM exchange coupling has a slope of
$\sim$10 meV/Å as the inter-atomic distance between NN Cr-Cr pairs increases.
Therefore, small dynamic lattice vibrations of the Cr atoms can directly
affect spin waves in CrGeTe3. Experimentally, strong SLC has been suggested
from Raman scattering, where optical phonon modes in CrGeTe3 undergo narrowing
in width and hardening in energy as the system is cooled below $T_{C}$
Tian2016 . However, Raman measurements can only probe a few optical phonon
modes and certain magnon at the zone center $\Gamma$ point, and are unable to
study spin waves throughout the Brillouin zone and their directional lifetime
anisotropy. In addition, the in-plane lattice parameter $a$ of CrGeTe3
displays negative thermal expansion around $T_{C}$=65K Carteaux1995 ,
consistent with the calculations showing enhanced FM interaction with the
expansion of the lattice. Finally, INS experiments found broadened spin waves
in CrGeTe3 throughout the Brillouin zone, suggestive of a strong SLC
FengfengZhang . Although these results provided circumstantial evidence for
SLC in CrGeTe3, there is currently no direct experimental proof and a
microscopic understanding of the SLC in CrGeTe3 is lacking. Since CrGeTe3 can
be cleaved to a monolayer with long-range FM order CGT_nature  and potential
for 2D spintronic devices due to possible dissipationless topological spin
excitations FengfengZhang , it is important to understand interactions of
magnetic excitations with lattice vibrations because such SLC may
fundamentally modify the topological nature of spin excitations. In this work,
we use INS to measure the spin and lattice dynamics in bulk single crystal
CrGeTe3, complemented by density functional theory (DFT) calculations of
phonon spectra and neutron diffraction measurements to determine temperature
dependence of the atomic Debye Waller factor. While spin waves along the $c$
axis are resolution limited with well-defined dispersion following the
expected behavior for a local moment ferromagnet, spin waves within the
honeycomb lattice plane show broadening and damping throughout the Brillouin
zone. Furthermore, the number of magnons is not conserved with increasing
temperature. By comparing these results with DFT calculations of phonon
spectra and neutron diffraction measurements of directional dependent atomic
Debye Waller factor, we conclude that the observed in-plane spin wave
anomalies arise from the large in-plane magnetic exchange coupling variations
induced by anisotropic zero-temperature motion of Cr atoms. These results
unveil the quantum zero-point motion induced SLC, suggesting that the observed
in-plane spin waves are mixed spin and lattice quasiparticles fundamentally
different from pure magnons and phonons.

##  I Results

Real/Reciprocal space and in-plane/$c$-axis spin waves. Figures 1(a,b) show
real space images of CrGeTe3, where the in-plane NN magnetic exchange $J_{1}$,
second NN exchange $J_{2}$, third NN exchange $J_{3}$ and $c$-axis exchange
$J_{c1}$ are marked. The corresponding reciprocal space with high symmetry
points is shown in Fig. 1(c). Figure 1(d) summarizes a schematic picture of
the temperature dependent root mean square displacement of different atoms in
CrGeTe3 within the $ab$ plane and along the $c$-axis direction determined from
our single crystal neutron diffraction analysis SI . The left and right panels
of Figures 1(e) and 1(f) show calculated and measured spin wave spectra along
the $c$-axis and in-plane, respectively, where the calculation is obtained
using a local moment Heisenberg Hamiltonian and linear spin wave theory (LSWT)
boothroyd . While the calculation and data agree rather well and are
resolution limited throughout the Brillouin zone for spin waves along the
$c$-axis [Fig. 1(e)], the measured spin waves within the $ab$-plane are weaker
and broader than the calculated spectra [Figs. 1(f), 2(a), 2(b)]. To test if
population of magnons is conserved and follows the Bose population factor as
expected for a local moment ferromagnet, we plot in Figs. 1(g) and 1(h)
temperature dependence of the imaginary part of the dynamic susceptibility
obtained using fluctuation-dissipation theorem, $\chi^{\prime\prime}({\bf
Q},E)=(1-e^{-E/k_{B}T})S({\bf Q},E)$ where $S({\bf Q},E)$ is the spin wave
intensity at energy $E$ and wave vector ${\bf Q}$ and $k_{B}$ is Boltzmann
constant, at $E=1.5$ meV in-plane and along the $c$-axis, respectively. Upon
increasing temperature from 7 K to 50 K in the FM ordered state,
$\chi^{\prime\prime}({\bf Q},E)$ decreases for in-plane spin waves while it
remains the same for magnons along the $c$-axis. This suggests a violation of
magnon conservation for in-plane spin waves but not along the $c$-axis. The
right panels in Figures 2(a) and 2(b) show the overall spin-wave spectra in
the $[H,K]$ plane, where we used incident neutron energies $E_{i}=37$ and 50
meV, integrated over $L=[-5,5]$, and $L=[-6,6]$ to obtain the $[H,H]$ and
$[H,0]$ dispersion, respectively. Figure 2(c) shows the experimental geometry
to probe phonons around wave vector $(0,0,12)$ where magnetic contributions
can be safely ignored due to small magnetic form factor of Cr3+ at this large
Q. Figure 2(d) compares the dispersions of the calculated magnons and phonons
along the high smmetry directions within the $ab$ plane. The $[0,0,L]$ spin-
wave spectrum along the $c$-axis in Fig. 1(e) is obtained using $E_{i}=4.7$
meV. Inspection of Figs. 2(a) and 2(b) reveals clear acoustic and optical spin
waves separated by a spin gap at the Dirac point, consistent with previous
work FengfengZhang  and very similar to spin waves in honeycomb lattice
ferromagnets CrI3 LBChen ; LBChen2020 ; LBChen2021  and CrBr3 CrBr3 ; Cai2021
. Consistent with earlier work LBChen ; LBChen2020 ; LBChen2021 , the magnon
dispersion can be calculated with the Heisenberg Hamiltonian

$H_{0}=J_{ij}\textbf{S}_{i}\cdot\textbf{S}_{j}+\textbf{A}_{ij}\cdot(\textbf{S}_{i}\times\textbf{S}_{j})+D_{z}(S_{i}^{z})^{2}$
(1)

where $J_{ij}$ includes the in-plane NN $J_{1}$, second NN $J_{2}$, third NN
$J_{3}$ and $c$-axis exchange $J_{c1}$; $\textbf{A}_{ij}$ is the the
Dzyaloshinskii-Moriya (DM) term that exists only between second NN in
honeycomb lattice according to Moriya’s rule [Figs. 1(a) and 1(b)] DM1 ; DM2 .
The single-ion anisotropy term $D_{z}=0.033$ meV opens a 0.1 meV spin gap at
the $\Gamma$ point [Figs. 3(a,b)]. By fitting the overall spin-wave
dispersion, we find $J_{1}=-2.76$ meV, $J_{2}=-0.11$ meV, $J_{3}=-0.33$ meV,
$J_{c1}=-0.86$ meV, and $|\textbf{A}|=0.20$ meV. However, the calculated spin-
wave spectra based on the Heisenberg model and instrumental resolution cannot
fully explain the observed scattering intensity and broadening of the in-plane
dispersion, especially for the optical magnons located at the $\Gamma$ point
[Figs. 1(f) and 2(a,b)]. In contrast, the $c$-axis $[0,0,L]$ dispersion agrees
well with the Heisenberg Hamiltonian and spin waves are instrumental
resolution limited [Fig. 1(e)]. Furthermore, we rule out the trivial
broadening due to finite mosaic spreads of the co-aligned crystals SI . As a
FM insulator, CrGeTe3 has an electronic band gap of $\sim$380 meV YFLi2018 ,
which is orders of magnitude higher than the thermal and magnon energy. The
observed magnon broadening and damping thus cannot be a result of magnon-
magnon couplings from itinerant electrons colloquium . Even considering
magnon-magnon interaction, quasiparticle numbers in a ferromagnet should still
be conserved, meaning that $\chi^{\prime\prime}(E,{\bf Q})$ should be
independent of temperature in the FM state. Figures 1(g) and 1(h) show the
$\chi^{\prime\prime}({\bf Q},E)$ calculated from experimentally measured spin
waves. While $\chi^{\prime\prime}({\bf Q},E)$ is highly temperature dependent
along the in-plane direction, it is temperature independent along the
$c$-axis. These results indicate the breaking of in-plane magnon number
conservation, and can only be explained by anisotropic SLC. Low-energy
acoustic phonons and calculated phonon dispersions. To understand how phonons
are coupled with spin waves, we use INS to measure acoustic phonon modes with
polarization along the $c$-axis. Using Bragg peak position $(0,0,12)$ as the
zone center $\Gamma$ point, we avoid strong magnetic scattering and can
therefore directly probe lattice vibrations [Figs. 2(c)]. Experimentally, the
probed phonon vibration direction is parallel to the momentum transfer Q, and
the phonon energy is a function of its reduced momentum vector q in the first
Brillouin zone, where $\textbf{q}=\textbf{Q}-\textbf{G}$ with G being a
reciprocal lattice vector. Since any q in the first Brillouin zone will be an
order of magnitude smaller than $\textbf{G}=(0,0,12)$, we can safely assume
the probed phonon spectrum around $(0,0,12)$ has a vibration direction along
the $c$($z$)-axis. To compare with INS experiments, we also use DFT to
calculate the phonon dispersions and intensities in bulk CrGeTe3 [Fig. 2(d)].
Although phonon and magnon spectra have similar bandwidth as well as many
mutual crossover points as seen in Fig. 2(d), the optical spin waves from $M$
to $K$ point have no overlap with any phonon modes. Therefore, the optical
magnon broadening observed in Figs. 2(a) and 2(b) cannot be a result of
magnon-phonon coupling induced by mode crossovers Oh2016 ; Dai2000 . The left
and right panels of Figure 2(e) show dispersions of the measured and
calculated acoustic phonon along the $c$-axis, respectively. Overall, the
measured phonon spectra is compatible to our $ab$-initio calculations and the
reported phonon structure in 2D CrGeTe3 BHZhang2019 . As discussed earlier,
SLC can have several effects on the magnon and phonon spectra. First, a
lattice distortion induced by the magnetic order can affect the exchange
coupling as well as phonon energy, consistent with the observed phonon modes
hardening below $T_{C}$ in CrGeTe3 Tian2016 ; Second, if the phonon and magnon
modes intersect with each other, it will either open up a gap or broaden the
magnon signal due to SLC. However, we find no sudden magnon broadening or
energy gap at possible magnon-phonon cross points [Figs. 1(f), 2(a,b,d)].
Instead, we find two spin wave anomalies: 1\. The width of acoustic spin waves
increases quadratically with increasing ${\bf q}$ [Figs. 3(a) and 3(c)]; 2.
The low-energy spin wave dispersion deviates from the calculated dispersion
using Heisenberg Hamiltonian fits from the overall spin-wave spectra [Figs.
3(b) and 3(d)]. Figure 3(a) shows the low-energy spin-wave spectrum along the
in-plane $[H,H]$ direction SI . By carrying constant-$E$ and ${\bf Q}$ cuts
from the spin-wave spectrum, we obtain $H$ dependence of the full width of
half maximum (FWHM) of the magnon energy (life time). We subtracted the fitted
FWHM by the calculated instrumental resolution and find $E_{FWHM}=\gamma
H^{2}+E_{0}$, where $E_{0}=0.2$ meV accounts for the additional broadening
effect other than instrumental resolution (See Sec. 1.3 in the supplemental
information for details). [Fig. 3(c)]. From the Heisenberg Hamiltonian fits to
the overall spin wave spectra in Figs. 2(a) and 2(b), we expect the spin-wave
stiffness in the long-wavelength limit (small ${\bf q}$) to be
$D=4\pi^{2}S(J_{1}+6J_{2}+4J_{3})=280.4$ meV/(r.l.u)2, clearly different from
the observation of $D\approx 140$ meV/(r.l.u)2 [Fig. 3(d)]. For comparison,
spin waves along the $c$-axis are resolution limited and can entirely be
described by a Heisenberg Hamiltonian [Figs. 1(e)]. Therefore, our results
suggest the presence of significant in-plane magnetic exchange coupling
variations that cannot be accounted for by the average exchange couplings
obtained from a Heisenberg Hamiltonian fit to the overall spin-wave dispersion
[Figs. 2(a) and 2(b)].

##  II Discussion

To quantitatively understand these observations, we consider the effect of SLC
in a Heisenberg Hamiltonian

$H_{SLC}=\sum_{ij}(\textbf{S}_{i}\cdot\textbf{S}_{j})\frac{dJ_{ij}}{d\textbf{u}}\textbf{u},$
(2)

where ${\bf u}$ is the dynamical lattice displacement of Cr atoms away from
their equilibrium positions. Therefore, the total spin Hamiltonian will be
modified as $H=H_{0}+H_{SLC}$. Various DFT calculations estimated the SLC
coefficient ${dJ_{ij}}/{d\textbf{u}}$ in CrGeTe3 BHZhang2019 ; JLi2021 . For
example, the calculated NN exchange $J_{1}$ has an unidirectional relationship
with atomic displacement $dJ_{1}/du_{x}^{Cr}=-8.48$ meV/Å, with $x$ along the
nearest Cr-Cr bond [Fig. 1(b)] JLi2021 . Fig. 1 (d) visualizes the temperature
dependence of the in-plane ($U_{11}$) and $c$-axis ($U_{33}$) Debye-Waller
factor in root-mean-square displacement for Cr, Ge, and Te obtained by crystal
structure analysis of neutron single crystal diffraction data of CrGeTe3 SI .
By multiplying the calculated SLC coefficient and refined Debye-Waller factor,
we can quantitatively estimate the fluctuation of the exchange interactions at
different temperatures. At 5 K, the Cr in-plane $u_{x}^{Cr}$, as estimated by
$u_{x}^{Cr}=\sqrt{U^{Cr}_{11}}$, equals to 0.036 Å and therefore can modulate
the NN exchange $J_{1}$ by $\sim$0.72 meV (FWHM$=2.36\times 0.036\times
8.48$). Including atomic displacements of Ge and Te atoms will have additional
effect on $J_{1}$. On the other hand, the fluctuation of $J_{1}$ due to SLC
directly results in the spin-wave broadening of CrGeTe3. Therefore, we
utilized Monte-Carlo simulations to reproduce the observed magnon spectra
(Fig. 4). In the simulation, we calculate several magnon spectra with
different $J_{1}$, and sum them up after multiplying a Gaussian coefficient
centered at $J_{1}=-2.76$ meV. In this case, the width of the Gaussian
coefficient is directly related to the extent of the $J_{1}$ variations. In
order to reproduce the magnon broadening of the optical modes at 3.5 K, a
$15\%$ and $25\%$ (in FWHM) variation of $J_{1}$ is needed, respectively
[Figs. 4(a,b,e,f)]. The difference of the broadening effect between Figs. 4(a)
and 4(b) may be attributed to SLCs with different atomic vibrational modes and
will have different effect on $J$ fluctuation. Nevertheless, the simulated
$J_{1}$ variation of 0.42 $\sim$ 0.71 meV is close to the estimated value of
0.72 meV, suggesting that the observed magnon broadening can be understood as
SLC induced by the lattice vibrations. Upon increasing temperature to
$T\approx 0.85T_{C}=55$ K, spin waves are softened by $\sim$5% but a 35%
variation of $J_{1}$ is needed to explain the magnon broadening [Fig. 4(c)].
For comparison, the energy width of spin waves along the $L$ direction is
resolution limited at 3.5 K [Fig. 1(e)], and only broadens by $\sim$5% at 50 K
confirming the anisotropic nature of the SLC [Figs. 4(d)]. Since the magnon
broadening is observed at 3.5 K, atomic displacement (lattice vibrations) due
to thermal fluctuations can be safely ignored and the SLC must be induced by
the zero temperature quantum motion of Cr atoms. Since neutron is a weakly
interacting probe, we do not expect neutron scattering itself to affect the
populations of magnons and phonons at these temperatures. In a local moment
system with spin $S$, the total moment sum rule requires
$M_{0}^{2}=M^{2}+\left\langle{\bf m}^{2}\right\rangle=g^{2}S(S+1)$
$\mu_{B}^{2}$, where $M$ is the static ordered moment, $\left\langle{\bf
m}^{2}\right\rangle$ is the local fluctuating moment, and $g\approx 2$ is the
Land$\rm\acute{e}$ electron spin $g$ factor, to be a temperature independent
constant Lorenzana2005 ; Dai2015 . While the ordered moment in a magnetic
ordered system can be directly measured via temperature dependence of the
magnetic Bragg peak intensity, the local fluctuating moment $\left\langle{\bf
m}^{2}\right\rangle$ can be estimated through integrating the magnetic
excitations within the first Brillouin zone over all energies. To test if the
total sum rule is satisfied in FM CrGeTe3, we compare the changes in magnetic
Bragg peak intensity together with spin waves integrated within the first
Brillouin zone. Figure 4(g) shows the temperature evolution of the $(1,1,0)$
Bragg peak intensity. The 150 K data (black) has nuclear and incoherent
background scattering but without static magnetic scattering. The additional
intensity in the low-temperature data comes solely from static ordered moment
$M$, and is proportional to $M^{2}$. The moment $M$ is expected to reach $gS$
$\mu_{B}$ at zero temperature. By integrating the elastic magnetic intensity
at 3.5 K, we find $S=1.45$ consistent with the $S=3/2$ picture. At this
temperature, the local fluctuating moment $\left\langle{\bf
m}^{2}\right\rangle$ should integrate up to $S\sim$ 1.5 to complete the sum
rule of $S(S+1)$. Figure 4(h) shows the calculated INS intensity using LSWT
with the integration of $(S_{xx}+S_{yy}+S_{zz})$ yielding $S=1.5$ (blue dashed
line). The experimental INS intensity, although preserving the overall shape,
is over 50% less than the expectation values of a $S=3/2$ ferromagnet,
suggesting that spin waves in CrGeTe3 do not follow the local moment LSWT.
Upon increasing temperature to 55 K, the ordered moment decreases and yields
an effective $S=1.03$ (fig. 4(g)). To maintain the sum rule, the local
fluctuating moment $\left\langle{\bf m}^{2}\right\rangle$ should integrate up
to an effective $S=2.7$. This intensity increase is mostly due to the Bose
factor as well as the additional intensity at the neutron energy gain side (in
which spin excitations transfer energy to the incident neutron). The red
dashed line in Fig. 4(h) shows the LSWT calculated intensity at 55 K, with the
integration of $(S_{xx}+S_{yy}+S_{zz})=2.7$. Comparing with experimental data,
the observed spin waves show larger intensity reduction at 55 K, thus further
suggesting violation of the total moment sum rule. In summary, we
systematically investigate the magnon and phonon spectrum as well as the SLC
in CrGeTe3 using INS and DFT calculations. Our results reveal the existence of
a strong, highly anisotropic SLC mainly affecting magnons along the in-plane
directions. The strong SLC severely shortens the in-plane magnon lifetime even
with minimal thermal fluctuation at low temperatures, making CrGeTe3 the first
example to have visible effect of SLC caused by quantum zero-point motion of
the lattice. This means the spin excitations in CrGeTe3 will dissipate
regardless of their topological nature FengfengZhang . Therefore, CrGeTe3 will
not be suitable for dissipationless spintronics, but can be an ideal candidate
for pressure sensitive spintronic devices due to its exceptionally large and
anisotropic SLC.

##  III Methods

Single crystal growth and reciprocal space. High-quality single crystals of
CrGeTe3 was made by Ge-Te flux method CGT_growth . The crystals are typically
$5\times 5$ mm2 within the $ab$ plane and a few micron thick along the
$c$-axis. Although CrGeTe3 belongs to the rhombohedral $R$-3 space group, we
use a hexagonal lattice with $a=b=6.86$ Å, $c=20.42$ Å as shown in Figs.
1(a,b) to describe its crystal structure. In this notation, the momentum
transfer $\textbf{Q}=H\textbf{a}^{\ast}+K\textbf{b}^{\ast}+L\textbf{c}^{\ast}$
is denoted as $(H,K,L)$ in reciprocal lattice units (r.l.u.) [Fig. 1(c)]. The
high symmetry points $\Gamma$, $M$, $K$, $A$ in reciprocal space and their
respectively equivalent points are specified [Fig. 1(c)]. Neutron scattering.
Neutron diffraction experiments were performed at HB-3 triple-axis
spectrometer at High Flux Isotope Reactor, Oak Ridge National Laboratory
(ORNL) on a piece of single crystal sample aligned in the $[H,H,L]$ scattering
plane. Temperature dependence of the $(1,1,0)$ Bragg peak shows $T_{C}=65$ K
and negative in-plane thermal expansion below $T_{C}$, consistent with
previous reports SI ; CGT_growth . To further determine the lattice structural
distortion and atomic Debye Waller factors of Cr, Ge, and Te atoms at
different temperatures across $T_{C}$, we performed time-of-flight neutron
diffraction experiments on a piece of single crystal sample at TOPAZ single
crystal diffraction at Spallation Neutron Source (SNS), ORNL TOPAZ . We
measured many diffraction peaks at 150 K, 70 K and 5 K, and used the JANA 2006
software jana  to refine the crystallographic parameters. Table 1 shows the
diagonal elements of the atomic displacement matrix ($U_{11}=U_{22}$,
$U_{33}$) as well as the isotropic displacement $U_{iso}$. With decreasing
temperature, the lattice displacements of the atoms are highly anisotropic
favoring vibration along the $c$-axis. Upon cooling from 150 K to 5 K, The
$U_{33}$ of the magnetic Cr atom only decreases by $\sim 50\%$ while values of
$U_{11}$ and $U_{22}$ are suppressed by an order of magnitude. The same
anisotropic suppression of the atomic displacement is also observed in the
non-magnetic Ge and Te atoms, but not as strong as that in Cr. This suggests
the presence of a strong anisotropic magnetoelastic coupling. We also
confirmed the negative thermal expansion of the lattice parameter $a$ SI  as
reported previously Carteaux1995 . In order to map out the spin waves, time-
of-flight INS experiments were performed using the SEQUOIA spectrometer at
SNS, ORNL SEQUOIA  and the AMATERAS spectrometer at MLF, J-PARC, Japan
AMATERAS . Single crystals of total mass 0.42 gram were co-aligned on an
aluminum plate with the help of an X-ray Laue machine. Five different incident
energies of $E_{i}=50,37$ meV (SEQUOIA) and $E_{i}=15,4.7,1.0$ meV (AMATERAS)
were used to carry out INS measurements in the Horace mode horace . The magnon
band top and bottom appears at $\Gamma$, while the Dirac cone appears at the
$K$ point, splitting the higher energy optical and lower energy acoustic
magnon bands. We utilized the MATLAB Horace and SpinW package horace ; spinw
to reconstruct the neutron structure factor $S(E,\textbf{Q})$, convolve with
instrumental resolution, and integrate the neutron intensity in the momentum
space. Harmonic Density Functional Theory Simulations Phonon simulations were
performed in the framework of DFT as implemented in the Vienna $\it{Ab\text{
}initio}$ Simulation Package (VASP 5.4.1) Kresse1993 ; Kresse1996 ;
Kresse1996b . We used $6\times 6\times 6$ gamma-centered Monkhorst-Pack
electronic _k_ -point mesh to integrate over the Brillouin zone for the
rhombohedral unit cell with 10 atoms. The plane-wave cut-off energy of 450 eV
provided satisfactory degree of convergence (energy difference of $\sim$0.1
meV/atom). The convergence criteria for the self-consistent electronic loop
was set to 10-8 eV. The projector-augmented-wave potentials explicitly
included six valence electrons for Cr ($4s^{2}3d^{4}$), four for Ge
($3s^{2}3p^{2}$) and six for Te ($4s^{2}4p^{4}$). We performed spin-polarized
calculations without spin-orbit coupling, with a magnetic momentum of 2.91
$\mu_{B}$ on each Cr atom. We used the generalized gradient approximation
(GGA) in the Perdew-Burke-Ernzerhof (PBE) parametrization refPBE ; refPBE2 .
During the relaxation of the unit cell, the lattice parameters and atomic
positions were optimized until forces on all atoms were smaller than 1 meV
Å-1. The resulting lattice parameters were $a=b=c=7.8364$ Å, and
$\alpha=\beta=\gamma=51.9824^{\circ}$. Phonon dispersions were calculated in
the harmonic approximation, using the finite displacement approach as
implemented in Phonopy Phonopy_2015 . The phonon calculations used a $2\times
2\times 2$ supercell of the rhombohedral cell containing 80 atoms. The atomic
displacement amplitude was 0.01 Å. Phonon Intensity Simulations The simulated
wave vector-resolved phonon intensity was calculated using the following
expression:

$\displaystyle S({\bf Q},E)\propto$
$\displaystyle\sum_{s}\sum_{\mathbf{\tau}}\frac{1}{\omega_{s}}$ (3)
$\displaystyle\times\left|\sum_{d}\frac{{b^{\rm
coh}_{d}}}{\sqrt{M_{d}}}\rm{exp}(-W_{d})\rm{exp}(i\mathbf{Q}\cdot\mathbf{d})(\mathbf{Q}\cdot\mathbf{e}_{ds})\right|^{2}$
$\displaystyle\times\langle
n_{s}+\frac{1}{2}\pm\frac{1}{2}\rangle\delta(\omega\mp\omega_{s})\delta(\mathbf{Q}-\mathbf{q}-\mathbf{\tau})$

where $b^{\rm coh}_{d}$ is the coherent neutron scattering length for atom
$d$, Q = k - k’ is the wave vector transfer, $\mathbf{d}$ the equilibrium
position of atom $d$, $\mathbf{e}_{ds}$ the eigenvector of phonon mode s for
atom $d$, and k’ and k are the final and incident wave vector of the scattered
particle, q the phonon wave vector, $\omega_{s}$ the eigenvalue of the phonon
corresponding to the branch index $s$, $\bm{\tau}$ is the reciprocal lattice
vector, $d$ the atom index in the unit cell, $\exp(-2W_{d})$ the corresponding
DW factor, and $n_{s}=\left[\exp\left(\frac{\hbar\omega_{s}}{k_{\rm
B}T}\right)-1\right]^{-1}$ is the Bose-Einstein occupation factor
($E=\hbar\omega_{s}$). The $+$ and $-$ sign in Eq. (3) correspond to phonon
creation and phonon annihilation, respectively. The phonon eigenvalues and
eigenvectors in Eq. (3) were obtained by solving dynamical matrix using
Phonopy Phonopy_2015 .
Data availability The data that support the plots within this paper and other
findings of this study are available from the corresponding authors upon
reasonable request. Code availability The codes used for the DFT calculations
of phonon spectra in this study are available from the corresponding authors
upon reasonable request.

## References

  * (1) Boothroyd, A. T. Principles of neutron scattering from condensed matter. Chapter 8 (Oxford Press, 2020).
  * (2) Holstein, T. & Primakoff, H. Field dependence of the intrinsic domain magnetization of a ferromagnet. Phys. Rev. 58, 1098 (1940).
  * (3) Oguchi, T. Theory of spin-Wave interactions in ferro- and antiferromagnetism. Phys. Rev. 117, 117 (1959).
  * (4) Zhitomirsky, M. E. & Chernyshev, A. L. Colloquium: Spontaneous magnon decays. Rev. Mod. Phys. 85, 219 (2013).
  * (5) Dietrich, O. W., Als-Nielsen, J. & Passell, L. Neutron scattering from the Heisenberg ferromagnets EuO and EuS. III. Spin dynamics of EuO. Phys. Rev. B 14, 4923, (1976).
  * (6) Zhao, J. et al. Spin waves and magnetic exchange interactions in CaFe2As2. Nat. Phys. 5, 555 (2009).
  * (7) Anda, E. Effect of phonon-magnon interaction on the Green functions of crystals and their light-scattering spectra. J. Phys. C 9, 1075 (1976).
  * (8) Guerreiro, S. C. & Rezende, S. M. Magnon-phonon interconversion in a dynamically reconfigurable magnetic material. Phys. Rev. B 92, 214437 (2015).
  * (9) Oh, J. et al. Spontaneous decays of magneto-elastic excitations in non-collinear antiferromagnet (Y,Lu)MnO3. Nat. Commun. 7, 13146 (2016).
  * (10) Silberclitt, R. Effect of spin waves on the phonon energy spectrum of a Heisenberg ferromagnet. Phys. Rev. 188, 786 (1969).
  * (11) Jensen J. & Houmann, J. G. Spin waves in terbium. II. Magnon-phonon interaction. Phys. Rev. B 12, 320 (1975).
  * (12) Hwang, H. Y. et al. Softening and broadening of the zone boundary magnons in Pr0.63Sr0.37MnO3. Phys. Rev. Lett. 80, 1316 (1998).
  * (13) Dai, P. et al. Magnon damping by magnon-phonon coupling in manganese perovskites. Phys. Rev. B 61, 9553 (2000).
  * (14) Ye, F. et al. Spin waves throughout the Brillouin zone and magnetic exchange coupling in the ferromagnetic metallic manganites La1-xCaxMnO3 ($x=0.25,0.30$). Phys. Rev. B 75, 144408 (2007).
  * (15) Helton, J. S. et al. Spin wave damping arising from phase coexistence below $T_{C}$ in colossal magnetoresistive La0.7Ca0.3MnO3. Phys. Rev. B 96, 104417 (2017).
  * (16) Tian, Y., Gray, M. J., Ji, H., Cava, R. J. & Burch, K. S. Magneto-elastic coupling in a potential ferromagnetic 2D atomic crystal. 2D Mater. 3, 025035 (2016).
  * (17) Sun, Y. et al. Effects of hydrostatic pressure on spin-lattice coupling in two-dimensional ferromagnetic Cr2Ge2Te6. Appl. Phys. Lett. 112, 072409 (2018).
  * (18) Li, X. & Yang, J. CrXTe3 (X = Si, Ge) nanosheets: two dimensional intrinsic ferromagnetic semiconductors. J. Mater. Chem. C 2, 7071 (2014).
  * (19) Webster, L. & Yan, J.-A. Strain-tunable magnetic anisotropy in monolayer CrCl3, CrBr3, and CrI3. Phys. Rev. B 98, 144411 (2018).
  * (20) Zhang, B. H., Hou, Y. S., Wang, Z., & Wu, R. Q., First-principles studies of spin-phonon coupling in monolayer Cr2Ge2Te6. Phys. Rev. B 100, 224427 (2019).
  * (21) Li, J., Feng, J., Wang, P., Kan, E. & Xiang, H. Nature of spin-lattice coupling in two-dimensional CrI3 and CrGeTe3 Sci. China-Phys. Mech. Astron. 64, 286811 (2021).
  * (22) Williams, T. J. et al. Magnetic correlations in the quasi-two-dimensional semiconducting ferromagnet CrSiTe3. Phys. Rev. B 92, 144404 (2015).
  * (23) Chen, L. et al. Topological spin excitations in honeycomb ferromagnet CrI3. Phys. Rev. X 8, 041028 (2018).
  * (24) Chen, L. et al. Magnetic anisotropy in ferromagnetic ${\mathrm{CrI}}_{3}$. Phys. Rev. B 101, 134418 (2020).
  * (25) Chen, L. et al. Magnetic field effect on topological spin excitations in CrI3. Phys. Rev. X 11, 031047 (2021).
  * (26) Carteaux, V., Brunet, D., Ouvrard, G. & Andr$\rm\acute{e}$, G. Crystallographic, magnetic and electronic structures of a new layered ferromagnetic compound Cr2Ge2Te6. J. Phys. Cond. Matt. 7, 69 (1995).
  * (27) Zhu, F. F. et al., Topological magnon insulators in two-dimensional van der Waals ferromagnets CrSiTe3 and CrGeTe3: towards intrinsic gap-tunability. Sci. Adv. 7, eabi7532 (2021).
  * (28) Gong, C. et al. Discovery of intrinsic ferromagnetism in two-dimensional van der Waals crystals. Nature 546, 265 (2017).
  * (29) Samuelsen, E. J., Silberglitt, R., Shirane, G., & Remeika, J. P. Spin waves in ferromagnetic CrBr3 studied by inelastic neutron scattering. Phys. Rev. B 3, 157 (1971).
  * (30) Cai, Z. W. et al. Topological magnon insulator spin excitations in the two-dimensional ferromagnet CrBr3. Phys. Rev. B 104, L020402 (2021).
  * (31) Dzyaloshinskii, I. A thermodynamic theory of weak ferromagnetism of antiferromagnetics. J. Phys. Chem. Solids 4, 241 (1958).
  * (32) Moriya, T. Anisotropic superexchange interaction and weak ferromagnetism. Phys. Rev. 120, 91 (1960).
  * (33) Li, Y. F. et al. Electronic structure of ferromagnetic semiconductor CrGeTe3 by angle-resolved photoemission spectroscopy, Phys. Rev. B 98, 125127 (2018).
  * (34) Lorenzana, J., Seibold, G. & R. Coldea, Sum rules and missing spectral weight in magnetic neutron scattering in the cuprates, Phys. Rev. B 72, 224511 (2005).
  * (35) Dai, P. C. Antiferromagnetic order and spin dynamics in iron-based superconductors. Rev. Mod. Phys. 87, 855 (2015).
  * (36) Yao, X. et al. Record high-proximity-induced anomalous Hall effect in (BixSb1-x)2Te3 thin film grown on CrGeTe3 substrate. Nano Lett. 19, 4567 (2019).
  * (37) Coates L. et al. A suite-level review of the neutron single-crystal diffraction instruments at Oak Ridge National Laboratory, Review of Scientific Instruments, 89, 092802 (2018).
  * (38) Petricek, V., Dusek, M. & Palatinus, L. Crystallographic Computing System JANA2006: General features. Z. Kristallogr. 229 5 (2014)
  * (39) Granroth, G. E. et al. SEQUOIA: A newly operating chopper spectrometer at the SNS. Journal of Physics: Conference Series 251, 12058 (2010).
  * (40) Nakajima, K. et al. AMATERAS: A cold-neutron disk chopper spectrometer. J. Phys. Soc. Jpn. 80, SB028 (2011).
  * (41) Ewings, R. A. et al. Horace: Software for the analysis of data from single crystal spectroscopy experiments at time-of-flight neutron instruments.  Nuclear Instruments and Methods in Physics Research A 834, 132 (2016).
  * (42) Toth, S. & Lake, B. Linear spin wave theory for single-Q incommensurate magnetic structures. J. Phys.: Condens. Matter 27, 166002 (2015).
  * (43) Kresse G. & Hafner J., $Ab$ $initio$ molecular dynamics for liquid metals. Phys. Rev. B 47, 558-561, (1993)
  * (44) Kresse, G. & Furthm${\rm\ddot{u}}$ller, J., Efficient iterative schemes for ab initio total-energy calculations using a plane-wave basis set. Phys. Rev. B 54, 11169 (1996).
  * (45) Kresse, G. & Furthm${\rm\ddot{u}}$ller, J., Efficiency of ab-initio total energy calculations for metals and semiconductors using a plane-wave basis set. Computational Materials Science 6, 15 (1996).
  * (46) Perdew, J. P. & Zunger, A., Self-interaction correction to density-functional approximations for many-electron systems. Phys. Rev. B 23, 5048 (1981).
  * (47) Perdew, J. P., Burke, K. & Ernzerhof, M., Generalized gradient approximation made simple. Phys. Rev. Lett. 77, 3865 (1996).
  * (48) Togo, A. & Tanaka, I. First principles phonon calculations in materials science. Scr. Mater. 108, 1 (2015).
  * (49) See supplementary information for additional data and analysis.

Acknowledgement We thank Dr. Seiko Ohira-Kawamura for her help in experiments
on AMATERAS and Prof. R. Q. Wu for discussions. The neutron scattering and
sample growth work at Rice is supported by U.S. NSF-DMR-2100741 and by the
Robert A. Welch Foundation under Grant No. C-1839, respectively (P.D.). The
works of JHC was supported by the National Research Foundation (NRF) of Korea
(Grant nos. 2020R1A5A1016518 and 2020K1A3A7A09077712). Work at Duke (first-
principles modeling) was supported by the U.S. Department of Energy, Office of
Science, Basic Energy Sciences, Materials Sciences and Engineering Division,
under Award No. DE-SC0019978. A portion of this research used resources at the
Spallation Neutron Source, a DOE Office of Science User Facilities operated by
the Oak Ridge National Laboratory. The neutron experiment at the Materials and
Life Science Experimental Facility of the J-PARC was performed under a user
program proposal number 2020A0099. Author contributions P.D., L.C., and J.H.C.
conceived the project. L.C. and B.G. grew the single crystals and aligned them
using a Laue x-ray diffractometer. The neutron scattering experiments were
carried out by L.C., J.H.C., M.B.S., A.I.K., X.P.W., N.M., and P.D., Phonon
calculations are carried out by C.M. and O.D. The paper was written by P.D.,
L.C., C.M., and O.D. and all coauthors made comments on the paper. Competing
interests The authors declare no competing interests. Additional information
Correspondence and requests for materials should be addressed to J.C., O.D. or
P.D.

Figure 1:  Real/reciprocal space of CrGeTe3 and spin waves along the in-plane
and $c$-axis directions. (a) The hexagonal lattice of CrGeTe3, showing only
Cr3+ ions. The magnetic exchange couplings discussed in the main text are
shown. (b) A closer view of the Cr (blue) hexagon together with the ligand Ge
(gray) and Te (yellow) atoms. The bold (dashed) lines indicate bonds
above(below) the Cr plane, and the Te atoms above (below) the Cr plane are
drawn in heavy (light) yellow. The bode length of Cr-Te and Cr-Cr are
specified. (c) The first Brillouin zone of the hexagonal lattice, high
symmetry points $\Gamma$ (black), $M$ (blue), $K$ (red), and $A$ (green) are
indicated. (d) A schematic picture of the temperature dependence of the root
mean square displacement of atoms in CrGeTe3. (e) Spin waves along the
$[0,0,L]$ direction. The color scale is applicable to all color plots in this
paper. (f) The low-energy magnons along the $[H,H,3]$ direction. The left
panels in (e,f) are the LSWT calculations convoluted with instrumental
resolution using parameters given in the main text, and the right panels are
experimental results. (g,h) Imaginary part of the dynamic susceptibility along
the (g) $[H,H]$ and (h) $[0,0,L]$ directions at different temperature. Error
bars in all the figures represent statistical errors of 1 standard deviation.
Figure 2:  Magnon and phonon spectra of CrGeTe3. (a, b) INS spectra in the
$[H,H]$ and $[H,0]$ directions, respectively. (c) Experimental geometry for
phonon modes probed in the experiment. (d) The calculated bulk phonon spectrum
and fitted magnon spectrum. (e) The experimental (left) and DFT calculated
(right) spectrum of $L(c)$ direction longitudinal acoustic (L-LA) phonon,
respectively; (f) The experimental (left) and calculated (right) spectrum of
in-plane direction transverse acoustic phonon, respectively. Figure 3:
Details of in-plane magnon broadening. (a) INS data along the $[H,H]$
direction with $E_{i}=4.7$ meV and $L=3$. (b) INS data along the $[0,0,L]$
direction showing a $\sim$0.1 meV spin gap at the $\Gamma$ point. (c) Magnon
line-width (FWHM) as a function of reciprocal lattice vector $[H,H]$, and its
comparison with the $L$-dispersion FWHM (black dots) at the same momentum
transfer. (d) Long-wavelength magnon energy as a function of $[H,H]$ with
quadratic fits (dashed lines). The solid line indicates the expected magnon
dispersion calculated using LSWT with fitted parameters given in the text. The
vertical and horizontal error bars in (c) and (d) represent uncertainty in
fitted energy in meV and $[H,H]$ in r.l.u., respectively. Figure 4:
Simulations on magnon spectrum broadening and the total moment sum rule. (a)
LSWT calculations along the $[H,H]$ direction with instrumental resolution,
with 15$\%$ in-plane $J$ fluctuation compared with experimental results from
SEQUOIA at 3.5 K; (b) LSWT calculation along the $[H,0]$ direction with
instrumental resolution, with 25$\%$ in-plane $J$ fluctuation, compared with
experimental results; (c) same calculation as in (a) with 35$\%$ in-plane $J$
fluctuation and experimental results at 55 K; (d) The $[0,0,L]$ dispersion at
50 K, with no fluctuation of $J$ in the calculation. (e,f) Respective $E$-cuts
at the $M$ point and $\Gamma$ point for the experimental data (blue dots),
calculation without $J$ fluctuation (black lines) and with $J$ fluctuation
(red lines). (g) The $(1,1,0)$ elastic peak at 3.5 K, 55 K, and 150 K,
normalized to absolute units; (h) The INS intensity $S(E)$ at 3.5 K and 55 K,
averaged over the first Brillouin zone. The dashed lines are calculations of
spin wave intensities from the LSWT. The vertical error bars in (e) and (f)
represent statistical errors of 1 standard deviation.


# Surprisingly Popular Voting Recovers Rankings, Surprisingly!

Hadi Hosseini
Pennsylvania State University
<EMAIL_ADDRESS>Debmalya Mandal
Columbia University
<EMAIL_ADDRESS>Nisarg Shah
University of Toronto
<EMAIL_ADDRESS>Kevin Shi
University of Toronto
<EMAIL_ADDRESS>
###### Abstract

The wisdom of the crowd has long become the de facto approach for eliciting
information from individuals or experts in order to predict the ground truth.
However, classical democratic approaches for aggregating individual _votes_
only work when the opinion of the majority of the crowd is relatively
accurate. A clever recent approach, _surprisingly popular voting_ , elicits
additional information from the individuals, namely their _prediction_ of
other individuals’ votes, and provably recovers the ground truth even when
experts are in minority. This approach works well when the goal is to pick the
correct option from a small list, but when the goal is to recover a true
ranking of the alternatives, a direct application of the approach requires
eliciting too much information. We explore practical techniques for extending
the surprisingly popular algorithm to ranked voting by partial votes and
predictions and designing robust aggregation rules. We experimentally
demonstrate that even a little prediction information helps surprisingly
popular voting outperform classical approaches.

##  1 Introduction

The wisdom of the crowd has been the default choice for uncovering the ground
truth. Suppose we wish to determine the true answer to the question: “Is
Philadelphia the capital of Pennsylvania?” Condorcet’s Jury Theorem suggests
that if we elicit votes from a large crowd, the majority answer will be
correct with high probability even if, on average, the crowd is only slightly
more accurate than a random selection. However, in some domains the crowd can
be highly inaccurate and experts may be in minority. For example, when the
very question listed above is posed to real crowds, the majority answer is
often (the incorrect) ‘yes’ [8].

To circumvent this difficulty and uncover the ground truth even when the
majority is wrong, Prelec et al. [21] introduce the _surprisingly popular_
(SP) algorithm. This algorithm asks each individual not only what she thinks
the answer is (the _vote_), but also what fraction of the other participants
she thinks will say yes/no (the _prediction_). Then, instead of simply
selecting the majority (i.e. popular) answer, the algorithm selects the answer
that is _surprisingly popular_ , i.e., whose actual frequency in the votes is
greater than its average predicted frequency. They show that as the crowd gets
larger in the limit, this approach will provably recover the correct answer
with probability $1$, even if the crowd is less accurate than a random
selection on average.

The intuition behind their algorithm, borrowed from their work, is as follows.
Suppose there are two hypothetical worlds, one where Philadelphia is the
capital and one where it is not. In the former world, a greater fraction (say
$90\%$) would say ‘yes’ than the fraction (say $60\%$) that would say ‘yes’ in
the latter. However, the $60\%$ of the people who believe the correct world is
the former would predict the frequency of ‘yes’ to be $90\%$, whereas the
remaining $40\%$ would predict it to be $60\%$. This would make the average
predicted frequency of ‘yes’ to be somewhere between $60\%$ and $90\%$, higher
than its actual frequency of $60\%$. In other words, the majority but
incorrect answer ‘yes’ would be surprisingly _unpopular_ while ‘no’ would be
surprisingly popular and correct.

Several works have demonstrated the effectiveness of this approach in a wide
range of domains [21, 14, 25, 19, 24, 16]. Prediction questions have also been
used to boost the accuracy of surveys on social networks [10]. Prelec et al.
[21] show how to apply their approach to questions with non-binary votes and
non-binary ground truth. When the true answer lurks among $r$ options, their
approach requires each individual to predict the exact frequency of each of
$r$ options among other individuals’ votes. We are interested in ranked
voting, i.e., when the ground truth is a ranking of $m$ alternatives. Note
that in this case, the approach of Prelec et al. [21], which we refer to as
_surprisingly popular (SP) voting_ , would require eliciting predictions in
the form of a distribution over $r=m!$ options, which is clearly infeasible
for even moderate values of $m$. Thus, the main research questions we address
are:

> _How do we extend surprisingly popular voting to effectively recover a
> ground truth ranking of alternatives? If we elicit partial vote and
> prediction, how do we aggregate them and what information-accuracy tradeoff
> does this offer?_

#### Our contributions.

We focus on eliciting only ordinal vote and prediction information. For the
_vote_ , we ask individuals to provide their opinion of either just the top
alternative of the ground truth ranking (_Top_) or the full ground truth
ranking (_Rank_). For the _prediction_ , informally, we ask individuals to
predict either just a single alternative (_Top_) or a ranking of alternatives
(_Rank_) based on the other individuals’ votes. The exact prediction elicited
under various conditions is described in Section 3. In addition to these four
elicitation formats, we use as benchmark two classical elicitation formats in
which Top and Rank votes are elicited but no prediction is elicited. Because
the SP algorithm of Prelec et al. [21] does not work on partial votes and
predictions, we first design a novel aggregation method for such partial
information.

Next, we conduct an empirical study with $720$ participants from Amazon’s
Mechanical Turk platform. We ask the participants questions on geography,
movies, and artwork which admit a ground truth ranking of four alternatives
and elicit their responses in the aforementioned six elicitation formats. We
compare the different elicitation formats using four metrics: difficulty
(measured through response time as well as perceived difficulty),
expressiveness, error in recovering the ground truth top alternative, and
error in recovering the ground truth ranking.

Our results show that even when the vote and prediction information are
individually no better than random guesses, by combining the two pieces of
information SP voting performs significantly better. Further, it outperforms a
whole slew of conventional voting rules which ignore prediction information
and only aggregate the votes. We also observe that when it is necessary to
choose between eliciting more complex vote information and eliciting more
complex prediction information, the latter may be the right choice.

###  1.1 Related Work

Our work builds on the SP voting approach of Prelec et al. [21]. This approach
in turn builds on its precursor, the Bayesian truth serum (BTS) [22], which
also uses participants’ predictions, but for a different objective: to decide
payoffs to the participants which incentivize them to honestly report their
votes and predictions.

Prediction markets [3, 7], quadratic voting [13], and peer prediction [18] are
alternative approaches to recovering the ground truth, which, like SP voting,
allow a minority of experts to override the majority opinion. Instead of
eliciting participants’ predictions of other participants’ votes, prediction
markets and quadratic voting ask participants to place a bet on their vote
while peer prediction methods require them to participate in multiple tasks.

These recent approaches stand in contrast to a large body of work on epistemic
social choice [20] and noisy voting [6], which build on the seminal work of de
Condorcet [9], Galton [11], and Young [26]. Some of this literature focuses on
statistical models of errors in participants’ votes such as the Mallows model,
the Bradley-Terry model, the Thurstone-Mosteller model, and the Plackett-Luce
model. However, all these models assume that a participant is ever-so-slightly
more likely to report the correct option than an incorrect option. Hence,
approaches based on these models can fail to recover the ground truth when the
majority of the crowd is misinformed.

Finally, our work is reminiscent of a recent flurry of work on the
elicitation-distortion tradeoff in computational social choice [15, 1, 17, 12,
2]. In this line of work, there is no ground truth; instead, participants have
subjective preferences and the goal is to identify the decision that maximizes
the social welfare. Rather than directly eliciting participants’ utility
functions, various elicitation formats are used to elicit partial preferences
to analyze the tradeoff between the amount of information elicited and the
approximation to social welfare (called distortion). Our work replaces the
distortion with its counterpart, that is, the accuracy of recovering an
underlying ground truth.

##  2 Model

Let $A$ be a set of $m$ alternatives and $\mathcal{L}(A)$ be the set of
rankings over $A$. For a ranking $\sigma\in\mathcal{L}(A)$ and
$x\in\left\\{1,\ldots,m\right\\}$, let $\sigma(x)$ be the alternative in the
$x^{\text{th}}$ highest position in $\sigma$.

SP voting uses a Bayesian model; in the following, we present a special case
of the model for ranked voting. There exists a ground truth ranking
$\pi^{*}\in\mathcal{L}(A)$ drawn from a _prior_ $\mathcal{P}$. There are $n$
voters; each voter $i$ observes a noisy ranking $\sigma_{i}\in\mathcal{L}(A)$
drawn from a _signal distribution_ $\textstyle\Pr_{s}(\cdot|\pi^{*})$. The
voters know both the prior $\mathcal{P}$ and the signal distribution
$\textstyle\Pr_{s}(\cdot|\pi^{*})$; however, the principal is unaware of both.
Following Prelec et al. [21], we assume that
$\mathcal{P}(\pi),\textstyle\Pr_{s}(\sigma|\pi)>0$ for all rankings
$\sigma,\pi\in\mathcal{L}(A)$ to avoid degeneracy.

Conventional voting would ask each voter $i$ to simply report her observed
noisy ranking $\sigma_{i}$ and use a voting rule such as the Kemeny rule or
Borda count to aggregate the reported rankings. SP voting additionally asks
each voter $i$ to make inferences about the reports of other voters. Given her
observed noisy ranking $\sigma_{i}$ and the prior $\mathcal{P}$, voter $i$ can
compute a posterior distribution over the ground truth, given by

$\textstyle\Pr_{g}(\pi^{*}|\sigma_{i})=\frac{\textstyle\Pr_{s}(\sigma_{i}|\pi^{*})\cdot\mathcal{P}(\pi^{*})}{\sum_{\pi^{\prime}\in\mathcal{L}(A)}\textstyle\Pr_{s}(\sigma_{i}|\pi^{\prime})\cdot\mathcal{P}(\pi^{\prime})}.$

In turn, the voter can also infer a distribution over the noisy ranking
$\sigma_{j}$ observed by another voter $j$:

$\textstyle\Pr_{o}(\sigma_{j}|\sigma_{i})=\textstyle\sum_{\pi^{*}\in\mathcal{L}(A)}\textstyle\Pr_{s}(\sigma_{j}|\pi^{*})\cdot\textstyle\Pr_{g}(\pi^{*}|\sigma_{i}).$

SP voting asks each voter $i$ to report not only her observed noisy ranking
$\sigma_{i}$ (the _vote_), but also her inferred distribution
$\textstyle\Pr_{o}(\cdot|\sigma_{i})$ over other voters’ noisy rankings (the
_prediction_). Given these reports, for a ranking $\pi\in\mathcal{L}(A)$, let
$f(\pi)=\sum_{i=1}^{n}\mathbbm{1}[\sigma_{i}=\pi]$ denote the number of voters
who vote $\pi$ and $g(\cdot|\pi)$ denote the average of reported predictions
$\textstyle\Pr_{o}(\cdot|\sigma_{i})$ across all voters $i$ with
$\sigma_{i}=\pi$. Then, the SP algorithm of Prelec et al. [21] computes the
prediction-normalized vote count for each possible ground truth $\pi$ as

$\overline{V}(\pi)=f(\pi)\cdot{\textstyle\sum_{\pi^{\prime}\in\mathcal{L}(A)}}\frac{g(\pi^{\prime}|\pi)}{g(\pi|\pi^{\prime})}.$
(1)

The following result due to Prelec et al. [21], rephrased in our context,
guarantees that the ground truth ranking will have the highest prediction-
normalized vote count under the assumption that the highest posterior
probability for ground truth ranking $\pi$ will be assigned by a voter who
observes noisy ranking $\pi$.

######  Theorem 1 ([21]).

Suppose the prior $\mathcal{P}$ and the signal distribution
$\textstyle\Pr_{s}$ are such that
$\textstyle\Pr_{g}(\pi|\pi)>\textstyle\Pr_{g}(\pi|\pi^{\prime})$ for all
distinct rankings $\pi,\pi^{\prime}\in\mathcal{L}(A)$. Then, we have that
$\Pr[\pi^{*}\in\mathrm{argmax}_{\pi\in\mathcal{L}(A)}\overline{V}(\pi)]\to 1$
as $n\to\infty$.

##  3 Elicitation Formats & Aggregation Rules

Note that the prediction requested from voter $i$,
$\textstyle\Pr_{o}(\cdot|\sigma_{i})$, is a distribution over $m!$ rankings.
Eliciting this would undoubtedly place significant cognitive burden on the
voter. Thus, our goal is to elicit partial vote and prediction information
from the voters. Since eliciting numerical information is known to be
difficult [5], we focus on eliciting ordinal information for prediction. We
develop aggregation rules for recovering the ground truth from ordinal
information and empirically evaluate the effectiveness of SP voting.

#### Elicitation formats:

We focus on two types of vote reports, and for each of them, two types of
prediction reports. Below we provide formal explanations of these formats in
the context of our model. In the next section, we provide example phrasings
that were used to pose the various questions to the participants in our
empirical study. Let $r_{i}$ and $q_{i}$ respectively denote the vote and
prediction reports submitted by voter $i$.

  * •

_Top vote:_ Voter $i$ reports the top alternative in her observed noisy
ranking, i.e., $r_{i}=\sigma_{i}(1)$.

    * –

_Top prediction:_ Voter $i$ estimates the most frequent alternative among the
other votes, i.e. $q_{i}=\mathrm{argmax}_{a\in
A}\sum_{\sigma\in\mathcal{L}(A):\sigma(1)=a}\Pr_{o}(\sigma|\sigma_{i})$.

    * –

_Rank prediction:_ Voter $i$ estimates the ranking of the alternatives by
their frequency among the other votes, i.e., $q_{i}\in\mathcal{L}(A)$ such
that
$\sum_{\sigma\in\mathcal{L}(A):\sigma(1)=q_{i}(x)}\Pr_{o}(\sigma|\sigma_{i})\geq\sum_{\sigma\in\mathcal{L}(A):\sigma(1)=q_{i}(y)}\Pr_{o}(\sigma|\sigma_{i})$
for all $x>y$.

  * •

_Rank vote:_ Voter $i$ reports her entire observed noisy ranking, i.e.,
$r_{i}=\sigma_{i}$.

    * –

_Top prediction:_ Voter $i$ estimates the alternative that appears most
frequently in the top position of the other votes. Formally, this is
equivalent to the top prediction in case of a top vote:
$q_{i}=\mathrm{argmax}_{a\in
A}\sum_{\sigma\in\mathcal{L}(A):\sigma(1)=a}\Pr_{o}(\sigma|\sigma_{i})$.

    * –

_Rank prediction:_ Voter $i$ estimates the most frequent ranking among the
other votes, i.e.,
$q_{i}\in\mathrm{argmax}_{\sigma\in\mathcal{L}(A)}\Pr_{o}(\sigma|\sigma_{i})$.
Note that this is different from the rank prediction in case of a top vote.

This gives rise to four elicitation formats, which we refer to as Top-Top,
Top-Rank, Rank-Top, and Rank-Rank with the first component denoting the vote
format and the second denoting the prediction format. As a benchmark, we use
Top-None and Rank-None, where top and rank votes are elicited, respectively,
but no prediction information is elicited.

#### Aggregation rules:

There are two difficulties in applying the SP algorithm of Prelec et al. [21]
— maximizing $\overline{V}(\pi)$ given in Equation 1 — in our setting.

First, the effectiveness of the approach depends on how accurately functions
$f$ and $g$ from Equation 1 match their expected values, which in turn depends
on how large the number of voters is compared to the number of options among
which the ground truth lurks. In our case, since the ground truth is one of
$m!$ rankings, the approach would be ineffective unless each question is
answered by a number of voters much larger than $m!$. Instead, we determine
the ground truth comparison of each of $\binom{m}{2}$ pairs of alternatives
independently by applying the algorithm from Equation 1 on the relevant
pairwise comparison data extracted from the reports of the voters.

Second, even for comparing a pair of alternatives, Equation 1 requires
cardinal prediction information whereas our input is ordinal. We propose a
simple parametric model in which, for each elicitation format, we use two
parameters, $\alpha\in(0.5,1)$ and $\beta\in(0,0.5)$, to convert ordinal
pairwise predictions into cardinal pairwise predictions to be utilized by the
SP algorithm. In Section 4, we describe how we train these parameter values.
The formal algorithm and its detailed description are provided in the
appendix.

Note that applying our algorithm for comparing each pair of alternatives
independently results in a tournament, which we use for two prediction tasks:
predicting the top alternative in the ground truth ranking and predicting the
entire ground truth ranking. For the former task, we select the alternative
that defeats the maximum number of other alternatives in the resulting
tournament, breaking ties uniformly at random, and consider the frequency of
predicting the correct top alternative. For the latter task, we compute the
Kendall Tau distance of the tournament from the ground truth ranking.

Finally, note that there are no prediction reports for Top-None and Rank-None
and we consider a natural extension of SP voting. In particular, for Top-None,
SP voting returns an acyclic tournament comparing alternatives by their
plurality scores, and for Rank-None, it returns the (potentially cyclic)
majority preference tournament. We then select an alternative/ranking as
described earlier.

##  4 Experiment Design

To test the effectiveness of SP voting for recovering ranked ground truth with
only ordinal elicitation, we conducted an empirical study by recruiting $720$
participants (turkers) from Amazon Mechanical Turk (MTurk), a popular
crowdsourcing marketplace. An average turker spent about 15 minutes to
complete the survey. The survey was designed as follows.

#### Datasets.

To generate questions with an underlying ground truth comparison of
alternatives, we used three datasets from three distinct _domains_ :

  1. 1.

The _geography_ dataset111Retrieved from worldpopulationreview.com contains
230 countries with their 2019 population estimates according to the United
Nations.

  2. 2.

The _movies_ dataset222Retrived from
boxofficemojo.com/chart/top_lifetime_gross contains 15,743 movies with their
lifetime box-office gross earnings.

  3. 3.

The _paintings_ dataset333Generously provided by the authors of Prelec et al.
[21]. contains 80 paintings with their latest auction prices.

#### Questions.

In each domain, the numerical values associated with the alternatives allow a
ground truth comparison among the alternatives. For each domain, we considered
the top $50$ alternatives with the highest values. From these, we generated
$20$ questions, each comparing four alternatives selected such that two
consecutive alternatives in the ground truth ranking were exactly $6$ ranks
apart in the global ranking of all $50$ alternatives. Collectively, we had
$60$ questions across all three domains. For each of the $60$ questions and
each of the $6$ elicitation formats described in Section 3, we elicited $20$
responses, generating a total of $7,200$ responses.

#### Turker assignment.

Figure 1 shows the workflow faced by a turker. Each of the $720$ turkers
responded to $10$ questions split evenly among two randomly assigned
elicitation formats. , The turkers were divided roughly equally between the
$30$ ordered pairs of elicitation formats called _treatments_. Further, as
mentioned above, each question under each elicitation format was assigned to
the same number of turkers.

Preview & Consent Tutorial (Elicitation Format 1) 5 Questions (Elicitation
Format 1) Review (Elicitation Format 1) Tutorial (Elicitation Format 2) 5
Questions (Elicitation Format 2) Review (Elicitation Format 2) Quiz Submit

Figure 1: The workflow of a turker.

#### Tutorials.

As shown in Figure 1, each set of five questions in a fixed elicitation format
was preceded by a tutorial. The tutorial was designed specifically for the
elicitation format and tested turkers’ understanding of the vote and
prediction formats. It contained a sample question along with pre-specified
beliefs over the correct answer as well as over the other responses. Turkers
had to successfully pass the tutorial by converting the given beliefs into the
requested vote and prediction format in order to proceed to the questions.

#### Reviews.

Each set of five questions was also succeeded by a review, which asked the
turkers to rate the _difficulty_ (from Very Easy to Very Difficult) and
expressiveness (Very Little to Very Significant) of the elicitation format of
the preceding questions. While we controlled the difficulty level of various
questions from a given domain, as we show in Section 5 the three domains
themselves differed significantly in their difficulty. In anticipation of this
and to ensure that the turkers’ implicit comparison between their two assigned
elicitation formats is not influenced by the domains, the study was designed
such that the sequence of domains encountered by a turker in the first five
questions precisely matched that in the next five questions.

See the appendix for details such as the consent form, the tutorial for each
domain, the review, and other details.

Figure 2: Average time spent.

Figure 3: Perceived difficulty.

Figure 4: Perceived expressiveness.

#### Response qualifications & payment.

To ensure high-quality responses, in addition to providing training in the
form of tutorials, we restricted participation in our study to turkers who had
(a) at least $90\%$ approval rate on previous tasks, (b) at least $100$
completed tasks, and (c) the region set to US East (us-east-1) on MTurk.

Finally, at the end of the survey, the turkers were required to answer a quiz,
which repeated the four alternatives from the last question they answered and
asked them to identify the alternative they chose or ranked first in their
vote. The turkers were incentivized to answer the quiz correctly (see below).
In our case, over $82\%$ of turkers passed the quiz.

The payment was divided into two parts. A _base_ payment of 50¢ was provided
conditioned on completing the entire survey including all tutorials,
questions, and reviews. A _bonus_ payment of 50¢ was provided conditioned on
correctly answering the quiz question.

#### Elicitation formats.

In Section 3, we discussed six elicitation formats and described what vote and
prediction a given voter $i$ should submit as a function of her observed noisy
ranking $\sigma_{i}$, the prior $\mathcal{P}$, and the signal distribution
$\textstyle\Pr_{s}$. In our empirical study, we design natural and intuitive
phrasing to elicit the corresponding responses from the turkers.

For example, consider a question which asks to compare four countries (United
Kingdom, Vietnam, Russia, and Kenya) by their population. Under the Top-Rank
elicitation format, the vote and prediction questions would be as follows:

  * •

Part A (vote): _Which country do you think is the most populated among the
following?_

  * •

Part B (prediction): _Imagine that other participants will also answer Part A.
How do you think the following countries will be ordered from the most common
response (top) to the least common (bottom)?_

See the appendix for sample phrasings for all six elicitation formats and
screenshots from our user interface.

#### Training.

Recall that in our aggregation method, for each elicitation format, we use two
parameters, $\alpha\in(0.5,1)$ and $\beta\in(0,0.5)$, to convert ordinal
predictions into cardinal predictions that can be then used in the SP
algorithm. To learn effective values of these parameters, we split the dataset
into a training and a test set. For each elicitation format, we selected $5$
questions from each of three domains, reserving the remaining $15$ questions
from each domain for the test set. Using these $15$ questions, we performed a
grid search over $\alpha$ ranging from $0.55$ to $0.95$ in increments of
$0.025$ and $\beta$ ranging from $0.05$ to $0.45$ in increments of $0.025$ and
selected the values with the lowest mean squared error.

##  5 Results

In this section, we present our results averaged across all three domains. In
the appendix, we present more detailed results averaged across each domain
separately. All confidence intervals shown are $95\%$ intervals. We compare
the elicitation formats using four key metrics: difficulty (i.e. cognitive
burden), expressiveness, error in predicting the ground truth top alternative,
and error in predicting the ground truth ranking.

###  5.1 Difficulty & Expressiveness

We measure the following three metrics.

  * •

_Response time:_ Response time is known to be a good objective proxy for the
cognitive load associated with a task [23]. We measure the amount of time
spent by the turkers on the tutorials and questions of the elicitation format.

  * •

_Perceived difficulty:_ As a subjective indicator of difficulty, we consider
the perceived difficulty reported by the turkers (from Very Easy to Very
Difficult) during the review stage of the elicitation format.

  * •

_Perceived expressiveness:_ Expressiveness indicates the amount of information
that the turkers felt they were able to convey through the elicitation format
(from Very Significant to Very Little).

Figure 4 shows the average time spent by the workers on the tutorial and on an
average question under the six elicitation formats along with $95\%$
confidence intervals (lower is better). We observe a statistically significant
trend: when we fix a vote format (say Top or Rank), the average time spent
increases for both tutorials and questions as we make the prediction format
more complex (None $\to$ Top $\to$ Rank). In the appendix, we show the average
time spent for each domain and observe that the choice of the domain does not
significantly affect it regardless of the elicitation format.

Figure 4 and Figure 4 respectively show the reported distributions of
perceived difficulty (easier is better) and perceived expressiveness (higher
is better). Interestingly, the turkers found the six elicitation formats to be
of very similar difficulty and similar expressiveness.

Figure 5: Average error in predicting the top alternative in the ground truth.
By combining both the vote and predictions, SP voting achieves a much lower
error than in either piece of information.

Figure 6: Comparing SP voting with conventional voting for predicting the top
alternative. Incorporating the prediction reports helps SP voting
significantly outperform conventional voting.

Figure 7: Average error in predicting the ground truth ranking. By combining
both the vote and prediction information, SP voting achieves a much lower
error than in either piece of information.

Figure 8: Comparing SP voting with conventional voting for predicting the
ground truth ranking. Incorporating the prediction reports helps SP voting
significantly outperform conventional voting.

###  5.2 Predicting the Ground Truth Top Alternative

We now turn to analyzing how effectively the different elicitation formats
help us predict the ground truth. In addition to measuring the error of the
ground truth estimate returned by our algorithm, we also measure the error in
the input votes and predictions themselves. Note that every vote and
prediction is an estimate of some truth (either the ground truth or a summary
statistic of the other votes); thus, its error can be measured with respect to
the truth it is attempting to uncover.

First, we consider predicting simply the top alternative in the ground truth
ranking. For our algorithm as well as for the input votes and predictions, we
use, as error measure, the frequency of incorrectly guessing the top
alternative of the truth they attempt to estimate. Figure 8 shows the average
prediction errors for various elicitation formats (lower is better).444SP
voting errors are obtained by averaging over $60$ elections associated with
$60$ questions. Vote/Prediction errors are averaged over $1200$ responses and
have narrower confidence intervals. We remind the reader that the
effectiveness of SP voting should be judged based only on elicitation formats
which include some prediction information.

Given four alternatives, selecting an alternative uniformly at random would
result in a prediction error of $0.75$. Interestingly, both the vote and
prediction reports individually have average error around this benchmark. Yet,
by combining these two pieces of individually erroneous information, SP voting
is able to achieve significantly lower error. This is not surprising because
SP voting approach is design precisely to pick out the minority of experts
lurking among a majority of non-experts by combining vote and prediction
information. Moreover, for a fixed type of vote (either Top or Rank), as the
prediction formats become more complex (None $\rightarrow$ Top $\rightarrow$
Rank), the performance of SP voting improves.

Figure 8 compares SP voting to several standard voting rules including
Plurality, Plurality with Runoff, Borda, Copeland, Instant Runoff Voting
(IRV), and Maximin Rule, which ignore the prediction information and simply
aggregate the vote information in a democratic manner.555See [4] for
definitions of these rules. The conventional voting rules run on elections
containing votes from three elicitation formats (Rank-None, Rank-Top, and
Rank-Rank) whereas SP voting runs on each elicitation format individually. We
can see that for Rank-Rank, SP voting (rightmost orange bar) outperforms all
conventional voting rules, despite having access to just a third of the
samples. This indicates that the prediction information helps significantly.

These observations hold even when we consider each domain separately. These
results are provided in the appendix.

###  5.3 Predicting the Ground Truth Ranking

We now consider predicting the full ground truth ranking. For SP voting result
as well as the individual votes and predictions, we use the Kendall-Tau (KT)
distance to measure the error of the SP voting result, votes, and predictions
compared to the true ranking they aim to estimate. Figure 8 shows the average
KT distance for different elicitation formats (lower is better). Given four
alternatives, selecting a uniformly random ranking will have an average KT
distance of $3$. Both the votes and prediction reports have average error
around this benchmark. Similar to predicting the top alternative, SP voting
produces significantly lower average error by combining these two noisy pieces
of information. Moreover, for each vote format (either Top or Rank), as the
prediction report becomes more expressive (None $\to$ Top $\to$ Rank) the
average error of SP voting decreases.

Finally, we compare SP voting with standard voting rules (Figure 8) in terms
of the average KT distance and find that SP voting again outperforms all
voting rules for Rank-Rank.

###  5.4 Prediction vs. Vote

Our results illustrate the importance of prediction in recovering the ground
truth. While eliciting ranked votes and predictions (Rank-Rank) achieves the
lowest error, an intriguing question arises when we seek to choose an
elicitation format that provides a reasonable tradeoff between accuracy and
difficulty/expressiveness. Figures 8 and 8 show that Top-Rank significantly
outperforms Rank-Top while both formats are comparable in terms of response
time, perceived difficulty, and perceived expressiveness. Thus, if we wish to
choose an elicitation format slightly more complex than Top-Top, making the
prediction more expressive is more promising than that of the vote. The same
observation holds when comparing Top-Top versus Rank-None. This shows that
when a tradeoff between more complex vote and more complex prediction is
necessary, eliciting more complex prediction may be better.

##  6 Discussion

We extended surprisingly popular voting to recover a ground truth ranking of
alternatives and, through a crowdsourcing study across different domains,
showed that it outperforms conventional voting approaches without
significantly increasing elicitation. In our study, the ground truth is a
ranking over four alternatives, and a challenging future direction is to
extend this approach to rankings with more than four alternatives. For a large
number of alternatives, any practical elicitation scheme would ask the voters
to report a partial rank over the alternatives, which will make it challenging
to design aggregation rules for such partial ranks.

Another interesting direction would be to derive theoretical performance
guarantees for surprisingly popular voting when the number of participants is
finite (the results of Prelec et al. [21] hold only in the limit) and when
only partial votes and predictions are elicited (this may require assuming a
parametric signal distribution such as the Mallows model).

#### Acknowledgements.

The authors were partly supported by NSF grant #1850076 (HH), a postdoctoral
fellowship from Columbia DSI (DM), and an NSERC Discovery Grant (NS).

## References

  * Abramowitz et al. [2019] Ben Abramowitz, Elliot Anshelevich, and Wennan Zhu.  Awareness of voter passion greatly improves the distortion of metric social choice.  In Proceedings of the 15th International Conference on Web and Internet Economics (WINE), pages 3–16, 2019.
  * Amanatidis et al. [2020] Georgios Amanatidis, Georgios Birmpas, Aris Filos-Ratsikas, and Alexandros A. Voudouris.  Peeking behind the ordinal curtain: Improving distortion via cardinal queries.  In Proceedings of the 34th AAAI Conference on Artificial Intelligence (AAAI), pages 1782–1789, 2020.
  * Arrow et al. [2008] Kenneth J. Arrow, Robert Forsythe, Michael Gorham, Robert Hahn, Robin Hanson, John O. Ledyard, Saul Levmore, Robert Litan, Paul Milgrom, Forrest D. Nelson, George R. Neumann, Marco Ottaviani, Thomas C. Schelling, Robert J. Shiller, Vernon L. Smith, Erik Snowberg, Cass R. Sunstein, Paul C. Tetlock, Philip E. Tetlock, Hal R. Varian, Justin Wolfers, and Eric Zitzewitz.  The promise of prediction markets.  Science, 320(5878):877–878, 2008.
  * Brandt et al. [2016] Felix Brandt, Vincent Conitzer, Ulle Endriss, Jérôme Lang, and Ariel D Procaccia.  Handbook of computational social choice.  Cambridge University Press, 2016.
  * Camerer [2011] Colin F Camerer.  Behavioral game theory: Experiments in strategic interaction.  Princeton University Press, 2011.
  * Caragiannis et al. [2016] Ioannis Caragiannis, Ariel D Procaccia, and Nisarg Shah.  When do noisy votes reveal the truth?  ACM Transactions on Economics and Computation (TEAC), 4(3):1–30, 2016.
  * Chen and Pennock [2010] Yiling Chen and David M Pennock.  Designing markets for prediction.  AI Magazine, 31(4):42–52, 2010.
  * De Boer and Bernstein [2017] Patrick M De Boer and Abraham Bernstein.  Efficiently identifying a well-performing crowd process for a given problem.  In Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing, pages 1688–1699, 2017.
  * de Condorcet [1785] Marquis de Condorcet.  Essai sur l’application de l’analyse à la probabilité de décisions rendues à la pluralité de voix.  Imprimerie Royal, 1785.  Facsimile published in 1972 by Chelsea Publishing Company, New York.
  * Galesic et al. [2018] Mirta Galesic, W Bruine de Bruin, Marion Dumas, A Kapteyn, JE Darling, and E Meijer.  Asking about social circles improves election predictions.  Nature Human Behaviour, 2(3):187–193, 2018.
  * Galton [1907] Francis Galton.  Vox populi.  Nature, 75:450–451, 1907.
  * Kempe [2020] David Kempe.  Communication, distortion, and randomness in metric voting.  In Proceedings of the 34th AAAI Conference on Artificial Intelligence (AAAI), pages 2087–2094, 2020.
  * Lalley and Weyl [2018] Steven P Lalley and E Glen Weyl.  Quadratic voting: How mechanism design can radicalize democracy.  In AEA Papers and Proceedings, volume 108, pages 33–37, 2018.
  * Lee et al. [2018] Michael D Lee, Irina Danileiko, and Julie Vi.  Testing the ability of the surprisingly popular method to predict nfl games.  Judgment and Decision Making, 13(4):322, 2018.
  * Mandal et al. [2019] Debmalya Mandal, Ariel D Procaccia, Nisarg Shah, and David Woodruff.  Efficient and thrifty voting by any means necessary.  In Advances in Neural Information Processing Systems, pages 7180–7191, 2019.
  * Mandal et al. [2020a] Debmalya Mandal, Goran Radanović, and David Parkes.  The effectiveness of peer prediction in long-term forecasting.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 2160–2167, 2020.
  * Mandal et al. [2020b] Debmalya Mandal, Nisarg Shah, and David P Woodruff.  Optimal communication-distortion tradeoff in voting.  In Proceedings of the 21st ACM Conference on Economics and Computation, pages 795–813, 2020.
  * Miller et al. [2005] Nolan Miller, Paul Resnick, and Richard Zeckhauser.  Eliciting informative feedback: The peer-prediction method.  Management Science, 51(9):1359–1373, 2005.
  * Palley and Soll [2019] Asa B Palley and Jack B Soll.  Extracting the wisdom of crowds when information is shared.  Management Science, 65(5):2291–2309, 2019.
  * Pivato [2019] Marcus Pivato.  Realizing epistemic democracy.  In The Future of Economic Design, pages 103–112. 2019.
  * Prelec et al. [2017] Dražen Prelec, H Sebastian Seung, and John McCoy.  A solution to the single-question crowd wisdom problem.  Nature, 541(7638):532, 2017.
  * Prelec [2004] Dražen Prelec.  A bayesian truth serum for subjective data.  science, 306(5695):462–466, 2004.
  * Rauterberg [1992] Matthias Rauterberg.  A method of a quantitative measurement of cognitive complexity.  Human-computer interaction: Tasks and organisation, pages 295–307, 1992.
  * Rutchick et al. [2020] Abraham M Rutchick, Bryan J Ross, Dustin P Calvillo, and Catherine C Mesick.  Does the “surprisingly popular” method yield accurate crowdsourced predictions?  Cognitive research: principles and implications, 5(1):1–10, 2020\.
  * Wang et al. [2019] Juntao Wang, Yang Liu, and Yiling Chen.  Forecast aggregation via peer prediction.  arXiv:1910.03779, 2019.
  * Young [1988] H. P. Young.  Condorcet’s theory of voting.  The American Political Science Review, 82(4):1231–1244, 1988.

## Appendix

##  Appendix A Additional Material for Section 4

###  A.1 Experiment Design

Since we are interested in recovering the rank of four alternatives, any
question where the four alternatives are ranked very close to each other in
the ranked list of $50$ alternatives, would be a hard question for most of the
turkers. For this reason, we decided to formulate each question by selecting
four alternatives where the distance between successive alternatives is
exactly six. For example, some questions had alternatives ranked
$(1,7,13,19),(3,9,15,21),\ldots,(31,37,43,49)$. The consecutive alternatives
were kept exactly $6$ distance apart to control the difficulty level of the
questions. The first $16$ questions have alternatives ranked
$(1,7,13,19),(3,9,15,21),\ldots,(31,37,43,49)$. The last four questions have
alternatives ranked $(2,8,14,20),(12,18,24,30),(22,28,34,40),(32,38,44,50)$.

#### Elicitation Formats.

Below, we provide the phrasings used in our study to elicit the various types
of responses from a turker. As an example question, consider a question which
asks to compare four countries (United Kingdom, Vietnam, Russia, and Kenya) by
their population.

  1. 1.

Top-None: A turker is provided with four choices and is asked to vote the best
option according to her opinion.

     * •

Part A (vote): _Which country do you think is the most populated among the
following?_

The turker is just asked one question and there is no additional prediction
question regarding others’ opinions.

  2. 2.

Top-Top: A turker is asked two questions in this format. She is asked to vote
her top choice as in format Top-None. Moreover, she is asked a prediction
question about the votes of other participants.

     * •

Part A (vote): _Which country do you think is the most populated among the
following?_

     * •

Part B (prediction): _Imagine that other participants will also answer Part A.
Which of the following four countries do you think will be the most common
response?_

  3. 3.

Top-Rank: Like Top-Top, this rule also asks a turker two questions. However,
the prediction question is different, and asks to rank the four choices based
on the votes of other participants.

     * •

Part A (vote): _Which country do you think is the most populated among the
following?_

     * •

Part B (prediction): _Imagine that other participants will also answer Part A.
How do you think the following countries will be ordered from the most common
response (top) to the least common (bottom)?_

  4. 4.

Rank-None: This elicitation format asks the turker to order the four choices
based on her own opinion.

     * •

Part A (vote): _How do you think the following four countries should be
ordered from the most-populated (top) to the least-populated (bottom)?_

  5. 5.

Rank-Top: This format also asks a turker to rank four choices and same as in
Rank-None. Additionally, it asks the turker a prediction question about the
votes of other participants.

     * •

Part A (vote): _How do you think the following four countries should be
ordered from the most-populated (top) to the least-populated (bottom)?_

     * •

Part B (prediction): _Imagine that other participants will also answer Part A.
In your opinion, which country will be the most common top choice?_

  6. 6.

Rank-Rank: In this format, both the vote and prediction questions ask the
turker to rank the four choices.

     * •

Part A (vote): _How do you think the following four countries should be
ordered from the most-populated (top) to the least-populated (bottom)?_

     * •

Part B (prediction): _Imagine that other participants will also answer Part A.
In your opinion, which will be the most common ordering of the following
countries?_

##  Appendix B Details of the Aggregation Method

We now discuss our aggregation method that takes as input pairs of (vote,
prediction) reports from the voters and returns either a rank or a single
alternative. Recall that we considered six different elicitation formats with
different types of votes and prediction reports. Both the vote and the
prediction report can be either the top alternative or a ranking over the
alternatives. Our aggregation method is a general aggregation rule and works
when the votes and prediction reports are either ranked choices or top
alternatives. At a high level, our method (algorithm 1) considers pairs of
alternatives (say $(a,b)$) in turn, extracts the relevant information for that
pair from the input, and determines whether $a\succ b$ or not in the true
ranking $\pi_{\star}$.

Algorithm 2 describes how to extract the relevant information about a pair
$(a,b)$ from the input. We first describe how to extract the relevant votes
about the pair $(a,b)$ from the input. Consider a vote $r_{i}$ from voter $i$.
If $r_{i}$ is a rank over the alternatives, then we set $r_{i}^{(a,b)}$ either
$1$ or $0$ based on whether $a\succ_{r_{i}}b$ or not. On the other hand, if
$r_{i}$ is just an alternative, then set $r_{i}^{(a,b)}$ to either $1$ or $0$
depending on whether the reported alternative equals $a$ or $b$. If the top
alternative $r_{i}$ is neither $a$ nor $b$, we just discard report of voter
$i$ for determining the order for the pair of alternatives $a$, and $b$.

In order to extract the relevant prediction report about the pair $(a,b)$,
note that $q_{i}$ can be either a rank or an alternative, and we want to
convert it to a probability estimate of $P(a\succ b|a\succ b)$ or $P(a\succ
b|b\succ a)$. So algorithm 2 takes as input two additional parameters
$\alpha$, and $\beta$ and they are used to determine the value of the
probability estimates. If $q_{i}$ is a rank over the alternatives, then we
first choose either $\alpha$ or $\beta$ depending on the value of
$r^{(a,b)}_{i}$. Call this choice $p$. Then set $q_{i}^{(a,b)}$ either $p$ or
$1-p$ based on whether $a\succ_{q_{i}}b$ or not. On the other hand, if $q_{i}$
is just an alternative, then set $q_{i}^{(a,b)}$ to either $p$ or $1-p$
depending on whether the prediction equals $a$ or $b$. Finally, in case the
predicted alternative is neither $a$ nor $b$, we set $q_{i}^{(a,b)}$ to $1/2$.

After extracting the relevant information about the pair $(a,b)$, algorithm 1
executes the SP algorithm on two possible comparisons between $a$ and $b$ –
$a\succ b$ and $b\succ a$. In particular, we compute the _prediction-
normalized vote_ for $a\succ b$ and $b\succ a$, denoted as
$\overline{V}(a\succ b)$ and $\overline{V}(b\succ a)$ respectively, and choose
whichever has higher normalized vote. The formula for the prediction-
normalized vote for a comparison (say $a\succ b$) is given as

$\overline{V}(a\succ b)=f(a\succ b)\sum_{i}\frac{g\left(r_{i}^{(a,b)}|a\succ
b\right)}{g\left(a\succ b|r_{i}^{(a,b)}\right)}$

Here $f(a\succ b)$ is the true frequency of the order $a\succ b$ among the
voters, which we approximate from the votes $\\{r^{(a,b)}_{i}\\}_{i\in[n]}$.
Similarly, the conditional probabilities are approximated from the prediction
reports $\\{q^{(a,b)}_{i}\\}_{i\in[n]}$. For more details about the guarantees
of this algorithm, the reader is referred to the supplementary materials of
[21].

Input: Information reports $\\{r_{i}\\}_{i\in[n]}$, prediction reports
$\\{q_{i}\\}_{i\in[n]}$, and probabilities $\alpha>0.5$ and $\beta<0.5$.

for  _each pair of alternatives $(a,b)$_ do

$\left(\\{r^{(a,b)}_{i},q^{(a,b)}_{i}\\}_{i\in[n]}\right)\leftarrow\textrm{Extract-
Reports}(\\{r_{i},q_{i}\\}_{i\in[n]},(a,b),\alpha,\beta)$

/* Signal $1$ (resp. $0$) corresponds to $a\succ b$ (resp. $b\succ a$). */

$N_{a\succ b}=\left\\{j:r_{j}^{(a,b)}=1\right\\}$

$N_{b\succ a}=\left\\{j:r_{j}^{(a,b)}=0\right\\}$

$f(a\succ
b)=\sum_{i}\mathbbm{1}\left\\{r_{i}^{(a,b)}=1\right\\}/(\left|N_{a\succ
b}\right|+\left|N_{b\succ a}\right|)$

$f(b\succ a)=1-p_{\star}(a\succ b)$

$g(1|1)=\frac{1}{\left|N_{a\succ b}\right|}\sum_{i\in N_{a\succ
b}}q_{i}\textrm{ and }P(0|1)=1-P(1|1)$

$g(1|0)=\frac{1}{\left|N_{b\succ a}\right|}\sum_{i\in N_{b\succ
a}}q_{i}\textrm{ and }P(0|0)=1-P(1|0)$

/* Compute prediction-normalized vote */

$\displaystyle\overline{V}(a\succ b)=f(a\succ
b)\sum_{i}\frac{g\left(r_{i}^{(a,b)}|1\right)}{g\left(1|r_{i}^{(a,b)}\right)}$
$\displaystyle\overline{V}(b\succ a)=f(b\succ
a)\sum_{i}\frac{g\left(r_{i}^{(a,b)}|0\right)}{g\left(0|r_{i}^{(a,b)}\right)}$

$T\leftarrow\phi$ /* Create a tournament */

for  _each pair of alternatives $(a,b)$_ do

/* Ties are broken u.a.r. */

if  _$\overline{V}(a\succ b) >\overline{V}(b\succ a)$_ then

$T\leftarrow T\cup a\succ b$

else

$T\leftarrow T\cup b\succ a$

return _$T$_

ALGORITHM 1 SP Voting

Input: Information reports $\\{r_{i}\\}_{i\in[n]}$, prediction reports
$\\{q_{i}\\}_{i\in[n]}$, pair $(a,b)$, and probabilities $\alpha>0.5$, and
$\beta<0.5$.

for  _$i=1,\ldots,n$_ do

/* Extract information report */

if  _$r_{i}$ is a rank_ then

Set $r^{(a,b)}_{i}=\left\\{\begin{array}[]{cc}1&\textrm{ if
}a\succ_{r_{i}}b\\\ 0&\textrm{ o.w. }\end{array}\right.$

else if  _$r_{i}$ is a top alternative and $r_{i}\in\left\\{a,b\right\\}$_
then

Set $r^{(a,b)}_{i}=\left\\{\begin{array}[]{cc}1&\textrm{ if }r_{i}=a\\\
0&\textrm{ if }r_{i}=b\end{array}\right.$

else

Ignore $(r_{i},q_{i})$ for determining $a\succ b$

/* Extract prediction report */

if  _$q_{i}$ is a rank_ then

Set $q^{(a,b)}_{i}=\left\\{\begin{array}[]{cc}\alpha&\textrm{ if
}a\succ_{q_{i}}b\textrm{ and }r_{i}^{(a,b)}=1\\\ 1-\alpha&\textrm{ if
}b\succ_{q_{i}}a\textrm{ and }r_{i}^{(a,b)}=1\\\ 1-\beta&\textrm{ if
}a\succ_{q_{i}}b\textrm{ and }r_{i}^{(a,b)}=0\\\ \beta&\textrm{ o.w.
}\end{array}\right.$

else if  _$q_{i}$ is a top alternative and $q_{i}\in\left\\{a,b\right\\}$_
then

Set $q^{(a,b)}_{i}=\left\\{\begin{array}[]{cc}\alpha&\textrm{ if
}q_{i}=a\textrm{ and }r_{i}^{(a,b)}=1\\\ 1-\alpha&\textrm{ if }q_{i}=b\textrm{
and }r_{i}^{(a,b)}=1\\\ 1-\beta&\textrm{ if }q_{i}=a\textrm{ and
}r_{i}^{(a,b)}=0\\\ \beta&\textrm{ o.w. }\end{array}\right.$

else

Set $q^{(a,b)}_{i}=1/2$.

return _$\left(\\{r^{(a,b)}_{i},q^{(a,b)}_{i}\\}_{i\in[n]}\right)$_

ALGORITHM 2 Extract-Reports

##  Appendix C Missing Figures for Different Domains

###  C.1 Mean Time Spent

Figure 9 shows the average time spent by a turker on a single question, for
three different domains. For each domain, we observe the same pattern. For a
fixed type of vote (either Top or Rank), as we ask more complex prediction
reports ($\text{none}\rightarrow\text{top}\rightarrow\text{rank}$), the
particular elicitation format requires more time to answer one question.

Figure 9: Average time-spent on a question, for different elicitation formats,
grouped by domain.

###  C.2 Predicting Top Alternative for Different Domains

Figure 11 show the average error in predicting the top alternative of the
ground truth ranking for different elicitation formats and different domains.
For each domain, we see that, for a fixed type of vote (Top or Rank) as we
make the prediction reports more complex, the average prediction error
generally goes down. In particular, except for the domain Paintings, the
following orders always hold among the elicitation formats: $\text{Top-
None}>\text{Top-Top}$ and $\text{Rank-None}>\text{Rank-Top}$.

Figure 11 compares our method with six conventional voting rules in terms of
predicting the top alternative of the ground truth. We see the same phenomenon
as we saw when all questions were combined. SP voting trained on just Rank-
Rank elicitation format, outperforms all six voting rules for the domains
Geography and Movies.

Figure 10: Average error in predicting the top alternative of the ground truth
ranking, across different elicitation formats and different domains.

Figure 11: Average error in predicting the top alternative of the ground truth
ranking, for different voting rules, and SP voting on three elicitation
formats (Rank-None, Rank-Top, and Rank-Rank).

###  C.3 Predicting Ground Truth Ranking for Different Domains

Figure 13 shows the average Kendall-Tau distance from the ground truth ranking
for different elicitation formats and different domains. For each domain, we
see that, for a fixed type of vote (Top or Rank) as we make the prediction
reports more complex, the average prediction error generally goes down. In
particular, except for the domain Paintings, the following orders always hold
among the elicitation formats: $\text{Top-None}>\text{Top-Rank}$ and
$\text{Rank-None}>\text{Rank-Rank}$.

Figure 13 compares our method with six conventional voting rules in terms of
the average Kendall-Tau distance from the underlying true ranking. We see the
same phenomenon as we saw when all questions were combined. SP voting trained
on just Rank-Rank elicitation format, outperforms all six voting rules for all
the domains.

Figure 12: Average Kendall-Tau distance from the true rankings, across
different elicitation formats, and different domains.

Figure 13: Average Kendall-Tau distance from the true rankings, for different
voting rules, and SP voting on three elicitation formats (Rank-None, Rank-Top,
and Rank-Rank).

##  Appendix D Screenshots from Our User Interface

In this section, we provide screenshots of different pages of our user
interface.

###  D.1 Preview and Consent Form

Figure 14 shows the Preview and the Consent pages. After accepting our HIT, a
turker first sees the Preview page. The turker needs to accept the consnt for
participation before continuing with the tutorials.

Figure 14: Preview and consent form

###  D.2 Tutorials

Figure 15 shows the tutorials for Rank-Top and Top-Rank elicitation formats.
The tutorial provides the turker with a scenario (a correct answer, and a
belief about other participants’ votes), and asks the turker to complete the
vote and prediction (if required) questions. Each turker must successfully
complete the tutorial to proceed to the actual questions.

Figure 15: Tutorials for Rank-Top and Top-Rank formats.

###  D.3 Sample Questions

Figures 16 and 17 show sample questions for different elicitation formats.
Each turker completes five questions from two elicitation formats.

Figure 16: Sample questions from Top-None, Top-Top, and Top-Rank questions.

Figure 17: Sample questions from Rank-None, Rank-Top, and Rank-Rank questions.

###  D.4 Difficulty/Expressiveness

We assign each turker to two elicitation formats. After answering five
questions from each format, the turker is asked to complete a survey about the
difficulty and expressiveness of that format (shown in figure 18).

Figure 18: Difficulty and expressiveness questions