# Synchrotron radiation from cosmic string wakes.

Dilip Kumar, Soumen Nayak and Soma Sanyal  School of Physics, University of
Hyderabad, Gachibowli, Hyderabad, India 500046

###### Abstract

Magnetic fields can be generated in cosmic string wakes due to the Biermann
mechanism in the presence of neutrino inhomogeneities. As the cosmic string
moves through the plasma the small magnetic field is amplified by the
turbulence in the plasma. Relativistic charged particles which cross the
magnetized wake of a cosmic string will therefore emit synchrotron radiation.
The opening angle of the cosmic string is very small and so the wake appears
like a relativistic jet. Assuming a homogeneous magnetic field in the wake of
the string, we obtain the synchrotron emission from non thermal relativistic
electrons in the wake of the string. The emitted radiation has a broad peak
and is over a wide range of frequency. We show that the spectrum can be mapped
to some of the unknown sources in different ranges of the current available
catalogues.

cosmic string, wakes, synchrotron radiation

##  I Introduction

Synchrotron radiation has been studied for a very long time to identify and
locate various non - luminous astrophysical objects blackhole2021  in our
universe. It is the radiation emitted from relativistic electrons moving in
the magnetic field close to an astrophysical object. One of the most elusive
objects which are currently being sought using various methods are the cosmic
strings. These strings are produced during symmetry breaking phase transitions
in the early universe csreview . They are essentially defects in one dimension
with a conical spacetime. These strings can give rise to interesting
phenomenon in the early universe. The conical space around the string leads to
the formation of wakes behind them deruelle . In relativistic fluid flows,
both strong shocks as well as weak shocks are generated hiscock ; vollick ;
beresnyak  in these wakes. There are many kinds of cosmic strings depending on
the nature of the phase transition. Out of all these, the superconducting
cosmic string carry electric current and can generate magnetic fields in their
wake dimopoulos . So, the spectrum of synchrotron radiation from electrons
moving close to a superconducting string has been studied in detail rogozin .
Recently, it has been shown that magnetic fields can be formed in shocks of
wakes of non-superconducting strings by the Biermann battery mechanism sovan .
The Biermann battery mechanism generates a magnetic field whenever the
temperature gradient and the density gradient in an inhomogeneous plasma are
not aligned to each other schoeffler . Once, the magnetic field is generated,
it gets enhanced by the turbulence in the wake region. As the cosmic string
moves in the plasma, electrons crossing these magnetized wakes will emit
synchrotron radiation. The energy spectrum of these electrons along with other
signatures may help in identifying these cosmic strings in the current
universe. Currently, there are various attempts to predict the signature of
these cosmic string from the 21 cm observational data maibach . Other methods
of identifing cosmic strings are also being pursued. In this article, we are
proposing yet another method of identifying cosmic string wakes by using
electromagnetic radiation.

The study of diffusive synchrotron radiation has already yielded several
interesting results. The spectrum of the radiation depends on the correlation
length of the magnetic field through which the electrons travel. For a uniform
large scale magnetic field the spectrum is referred to as the synchrotron
spectrum while for small scale random magnetic field the radiation is often
refered to as the ”jitter” radiation medvedev ; fleishman . A theory has also
been developed for the small scale jitter radiation. The theory gives rise to
several power law asymptotes which smoothly change as the frequency changes.
This leads to the fact that a wide range of frequencies are now covered by the
synchrotron radiation.

In recent times, numerical simulations have been developed to study the
microphysics of non-relativistic as well as relativistic shocks. For cosmic
strings too, there have been several studies of unmagnetized shocks stebbins ;
sornborger ; ponce . In a previous work, we have studied the evolution of a
magnetic field in the wake of a cosmic string nayak . Cosmic strings can have
magnetic fields in their wakes due to various mechanisms. One mechanism that
has been mentioned before is the Biermann battery mechanism, alternative
mechanisms involve the generation of vorticity in the plasma. Two cosmic
strings moving past one another can generate vortices between them vachaspati
. Since ions and electrons do not have the same mass, a magnetic field can be
generated by the differential angular momentum between the ions and electrons
via the Harrison mechanism harrison . Other ways of generating vorticity have
also been explored. Once the field is generated in the wake, it may decay but
it will not dissipate completely, so the electrons moving across these wakes
will be moving through a magnetic field which means that they would emit
synchrotron radiation. However, only the synchrotron radiation coming from
wakes of superconducting strings have been studied previously in the
literature.

In this article we will focus on Abelian Higgs strings. Unlike superconducting
strings these do not have a current flowing through the length of the string.
The magnetic field in these wakes are generated by the Biermann mechanism
sovan . The Biermann mechanism generates oppositely directed magnetic fields
on the different sides of the wake. The wake has a traingular structure, being
narrow close to the string and becoming wider as the distance from the string
increases. Electrons will move across the wake as the string moves through the
plasma. In this work, we obtain the spectrum of synchrotron radiation emitted
by relativistic electrons in the string wake. We find that the synchrotron
radiation emitted from these electrons are in the range of
$10^{2}Hz-10^{23}Hz$. The overall spectrum has a broad peak. Due to the large
range of frequencies that it spans, the radiation should be detected by the
current all sky surveys. We discuss how this can be another signature for
cosmic string wakes in addition to the ones that are already being pursued.

In section 2, we discuss the wakes of cosmic strings in the intergalactic
medium. In section 3, we discuss the synchrotron radiation emitted by the
electrons moving in the magnetized wake of these cosmic strings. In section 4,
we discuss the spectrum obtained and in section 5, we present the summary and
conclusions of the paper.

##  II Cosmic string wake in the intergalactic medium

Cosmic string wake forms as a long cosmic string moves through the
intergalactic medium. It has been shown that the wake is very wide for a hot
gas tightly coupled to radiation but becomes quite narrow for the case of a
hydrogen gas ponce . Since the intergalactic medium consists of mostly
baryonic and leptonic matter, hence the wake caused by a moving cosmic string
would be closer to the case of the hydrogen gas. Thus the wake would be more
like a narrow jet with a large number of streaming relativistic particles. As
the velocity of the string reaches supersonic velocities, shock waves are
generated behind the cosmic string.

Cosmic string shock waves have been discussed for the high temperature plasma
of the very early universe. The relativistic motion of the string as well as
the finite temperature of the medium has been taken into account in these
studies stebbins ; rees ; deruelle . These shocks occur before the start of
recombination. Realistic studies of shocks also include the the possibility of
ionization and interaction with the background radiation ponce . The velocity
of the shock would depend on the nature of the plasma and it’s Equation of
State (EoS). In the post recombination era, the universe is matter dominated.
Assuming that the interaction between the matter and radiation is minimal, the
sound speed at those temperatures is given by,

$c_{s}^{2}=\frac{\Gamma_{m}p_{1}}{\rho_{1}+p_{1}}$ (1)

Here $\Gamma_{m}$ is the ratio of the specific heats of a radiation gas and
can be considered to equal to $5/3$, while $p_{1}$ and $\rho_{1}$ are the
pressure and density in the pre-shock region. Detailed studies of shocks in
these plasmas have been carried out ponce . The wakes usually have different
values of density, pressure and temperature in the pre-shock and the postshock
regions. The shock which is similar to a front moves with a velocity known as
the shock velocity. Generally this shock velocity depends both on the pre
shock and post shock density and pressure. It has been shown that, in the
case, where the specific heat ratio remains approximately the same in the
preshock and the postshock regime, one can obtain the velocity of the shock
wave as

$v_{sh}=\frac{1}{4}[(\Gamma_{m}+1)^{2}u^{2}+16c_{s}^{2}]^{1/2}-\frac{(3-\Gamma_{m})}{4}u$
(2)

Here $u=v\delta(1-v^{2})^{-1/2}$. $\delta$ is the angle of deflection of the
particles due to the cosmic string metric and $v$ is the velocity of the
string. Therefore the shock velocities can go as high as $0.9c$.

Electron acceleration from various shock waves have been studied in detail
galeev . Both subrelativistic shock waves as well as ultra relativistic shock
waves are known to generate accelerated electrons which subsequently emit
synchrotron radiation. Most of these studies are therefore done with
hydrodynamic shock waves. In the presence of a magnetic field, the wake region
will also have mulitple shock waves nayak . So it is highly likely that the
electrons moving through these magnetized wakes will emit synchrotron
radiation.

The following basic properties of the wake need to be taken into account while
studying this synchrotron radiation. The particle number density in the wake
is inhomogeneous. This is because different particles in the wake move
differently in the metric of the Abelian Higgs string. Depending on their mass
and angular momentum, particles appear to cluster closer to the string due to
the presence of bounded orbits saha . Due to the asymmetry amongst the right
and left handed neutrinos a neutrino current is also generated. This neutrino
current is oscillating in nature and the number density of the neutrinos
depends on their distance from the string sovan . So, the wake structure is
very inhomogeneous in nature. Apart from these inhomogeneities, the
temperature in the preshock and post shock region is also quite different. The
temperature in the post shock regime is given by

$T_{2}=\frac{p_{2}n_{1}}{p_{1}n_{2}}T_{1}$ (3)

Here the suffix $1$ is used for the pre shock quantities, while the suffix $2$
is used for the post shock quantities. In the next section, we present our
model for determining the synchrotron radiation from the wakes of these
Abelian Higgs cosmic strings. Using the basic propoerties of inhomogeneities
in the wake, we obtain the spectrum of the synchrotron radiation emitted by
electrons crossing the magnetized wake of an Abelian Higgs string.

##  III Synchrotron radiation from electrons in the wake

As described in the previous section, the electron distribution in the cosmic
string wake is inhomogeneous. There are regions which have a higher density of
electrons and regions which have a lower density of electrons. Since these
electrons are accelerated by the motion of the shock generated in the string’s
wake, they will lose energy as synchrotron radiation. For electrons, the peak
frequency at which this loss happens is given by,

$\nu_{c}\sim\frac{eB\gamma^{2}}{2\pi mc}$ (4)

Here the magnetic field $B$ is considered to be homogeneous over the mean free
path of the electrons. The average power that is radiated by the electrons is
given by,

$\langle
P^{syn}(\nu)\rangle=\frac{\sqrt{3}e^{3}B}{m_{e}c^{2}}\int_{1}^{\infty}d\gamma
N_{e}(\gamma)R(\alpha)$ (5)

Here $\gamma$ is the relativistic Lorentz factor and
$\alpha=\frac{\nu}{\nu_{c}}$. $N_{e}(\gamma)$ is the distribution of electrons
which are moving across the cosmic string wake, $R(\alpha)$ is a function of a
combination of Bessel functions.

One of the simplest models used for obtaining the spectrum is the ”Blob model”
or the standard one zone model. Here it is assumed that the random magnetic
field fills up a spherical volume of radius $r_{b}$ and comoving volume of
$V_{b}=\frac{4\pi r_{b}^{3}}{3}$. The nonthermal relativistic electrons are
assumed to be uniformly distributed throughout the blob with an isotropic
pitch angle distribution. The electrons have a Lorentz factor distribution
given by $N_{e}(\gamma^{\prime})$ where $\gamma^{\prime}$ is the comoving
Lorentz factor. For this model, the synchrotron radiation is given by,

$\nu F_{\nu}^{syn}=\frac{\delta_{D}^{4}\nu^{\prime}\langle
P^{syn}(\nu^{\prime})\rangle}{4\pi d^{2}_{L}}$ (6)

The Lorentz factor determines the Doppler factor for the relativistic outflow
given by $\delta_{D}$ defined by,

$\delta_{D}=\frac{1}{\gamma_{2}(1-\beta cos\theta)}$ (7)

Here $\beta=\frac{v_{f}}{c}$ and $\theta$ is the viewing angle which is very
small.The other input that is important for an observable is the luminosity
distance. This distance will depend on the redshift of the object. The
luminosity distance $d_{L}$ is given by,

$d_{L}=d_{A}(1+z)^{2}$ (8)

where $z$ is the redshift and $d_{A}$ is the angular size distance. It is
therefore the ratio of the transverse extent of the object and the angle it
subtends in the sky. Generally, when studying synchrotron radiation from jets
or other astrophysical objects, blobs of electrons present in the jets are
considered to emit the synchrotron radiation. Here too, we have assumed that
there are a distribution of electrons which are moving across the cosmic
string wake and these electrons have a power law distribution, given by,

$N_{e}(\gamma)=K_{e}\frac{1}{4\pi}\gamma^{-p}H(\gamma;\gamma_{1},\gamma_{2})$
(9)

Here $K_{e}$ is the normalization constant.The synchrotron radiation emitted
by the electrons for an isotropic pitch angle distribution is then given by,

$\nu
F^{syn}_{\nu}(p)=\frac{3^{(p+2)/2}}{2^{(p+1)/2}}a(p)\frac{4}{3}c\sigma_{T}U_{B}K_{e}\left(\frac{\nu}{\nu_{B}}\right)^{(3-p)/2}$
(10)

Here $\sigma_{T}$ is the Thomson cross section, $U_{B}=\frac{B^{2}}{8\pi}$,
$\nu_{B}=\frac{eB}{2\pi m_{e}c}$. For relativistic shocks $p\approx 2$ and
$a(p)=0.1032$. Details of these calculations are available in ref.dermer .

The python module AGNPY agnpy  has been developed based on the blob model and
has been tested for different cases. As is seen the primary inputs come from
the particular source being studied and the the final output spectrum would
depend upon the electron distribution, the Lorentz $\gamma$ factor, the
Doppler factor etc. We obtain these input parameters for the case of the
cosmic string and use the AGNPY module to obtain the final spectrum.

We will now proceed to briefly mention our inputs to the module and then
present the final spectrum obtained for various different parameters of our
model. In the blob model, the radius of the blob is an input parameter. We
model the source such that the radius of the blob must be smaller than the
width of the shock. Now the width of the wake at $z=30$ is given approximately
by $2.3\times 10^{-4}Mpc$. Since we obtain the synchrotron spectrum for values
of $z<30$, we take the radius of the blob to be $R_{B}<8\times 10^{20}$ cms.
We however find that the spectrum does not depend on the exact value of the
radius (as the volume cancels out in the final expression) so we keep it at
$R_{B}\sim 10^{16}$ cms.

Figure 1: The synchrotron spectrum for different values of the $\gamma$
parameter at redshift $z=1$.

The most important parameter associated with the cosmic string shock is the
Lorentz factor of the accelerated electrons. In the post shock region this is
given by $4\gamma_{2}=\frac{n_{2}}{n_{1}}$. For cosmic strings we have layek ,

$\frac{n_{2}}{n_{1}}=\frac{16\pi G\mu
v_{f}^{2}}{3\sqrt{v_{f}^{2}-v_{s}^{2}}}+1$ (11)

Here $v_{f}$ is the velocity of the string and $v_{s}$ is the velocity of the
shock wave. Therefore we can obtain the Lorentz factor from the epression,

$\gamma_{2}=\frac{1}{4}\left[\frac{16\pi G\mu
v_{f}^{2}}{3\sqrt{v_{f}^{2}-v_{s}^{2}}}+1\right]$ (12)

Since the value of $4\pi G\mu\sim 10^{-5}$, therefore the value of
$\gamma_{2}\sim 0.25$. Since it is assumed that the strings move with
velocities close to that of light, the value of $\delta_{D}$ depends on the
Lorentz factor. So this is an important parameter in the emission spectrum of
the synchrotron radiation. We have used this as a parameter and used different
values of redshift to obtain the final spectrum. Since radiation has been
observed from very low redshift objects only, we keep the maximum of $z$ as
30.

##  IV Results

We now present the synchrotron spectrum for the different parameters discussed
in the previous section. In fig.1, we show the plots corresponding to
different values of the parameter $\gamma_{2}$ and $\delta_{D}$.They are
plotted for redshift value of $z=1$. The redshift values are limited by the
observational component and so as mentioned before, the largest redshift we
have considered is $z=30$ while the smallest is $z=0.069$. We have also
plotted the same graphs for different values of the redshift parameter in fig
2

Figure 2: The synchrotron spectrum for different values of the redshift
parameter at $\gamma=1$

We see that the turnover frequency is about $100$ GHz. The spectral break
comes after $10^{11}$ GHz. As expected we find that, the spectral break shifts
to low frequencies as we move back in time. The synchrotron spectrum obtained
can be fit with three distinct power laws. The initial low frequency self
absorption part has a power law exponent of $\sim 1.24$, this is followed by
the optically thin emission region which has a very low spectral index. This
is the region of the spectrum where the relativistic electrons are scattered
mulitple times. Finally it dips very sharply in the high frequency range. This
is different from the usual spectrum obtained from the Gamma Ray Bursts where
there is a more bell shaped curve.

Galactic radiation has been observed at such frequencies in several cases. Due
to the high sensetivity of the all sky surveys there are many sources of
radiation which have been detected. Not all these sources can be associated
definitely and unambigously with known sources such as an Active Galactic
Nuclei (AGN). There are some unidentified sources which are being studied too
hannes . We proceeded to search the catalogues to determine if there are any
unidentified sources in the frequency range of the synchrotron spectrum that
we have obtained from cosmic string wakes. In the infra red (IR) region, the
Wide-field Infrared Survey Explorer (WISE) data has several point sources
attributed to Blazers which are also in the range of the spectrum that we have
obtained. The similarity to blazars comes from the fact that the the gamma ray
emission in blazars has a very similar geometry to the expected gamma ray
emission from cosmic string wakes. Also the emission in blazers may be due to
shock shock collision, magnetic reconnection or turbulence, all three of which
can occur in the cosmic string wakes. Thus it would be difficult to
distinguish between the two sources; blazer from a cosmic string wake. One
point which distinguishes the two spectra is that the blazer usually has two
broadly peaked components whereas the cosmic string wake will have only one
broad peak.

We now proceed to plot a few sources from the various observations in fig. 3
to show that the current spectra that we have obtained is within the scope of
current observations. As we see the peak value of the flux is between
$10^{-13}-10^{-12}ergcm^{-2}s^{-1}$ and the location is over a broad range of
frequencies. We used the WISE catalogue to search for sources of infrared
radiation in this flux scale. In fig 3, we have plotted a few points from the
WISE catalogue WISE  on the obtained spectrum for $\gamma=1$. Though the
initial rise is not as steep for the synchrotron radiation with a power law
behaviour of about $1.2$ in the optically thick region, some of the data
points are close to the obtained spectrum.

Figure 3: The synchrotron spectrum with data points from the WISE, SWIFT and
GALEX catalogues.

Similarly, we have included observed fluxes from the SWIFT SWIFT  and the
GALEX catalogue GALEX . We have used the data that is available online and
find that different surveys in different frequency bands have observed fluxes
in this range. This seems to indicate that a more detailed study of
synchrotron radiation in cosmic string wakes may lead to results matching with
experimental data.

##  V Summary and Conclusions

In this work we have studied the synchrotron radiation emitted by relativistic
electrons moving in a cosmic string wake. The narrow width of the wake
determines the Lorentz $\gamma$ factor of the accelerating electrons. We
obtain the spectrum for different values of the $\gamma$ factor using a one
blob model at different values of the redshift. Unlike the Gamma Ray Bursts
studied previously the $\gamma$ factor is limited to much lower values in this
case. A cosmic string wake cannot generate $\gamma$ factors ranging from
$10^{4}-10^{11}$ so our results are for values of $\gamma$ ranging from
$0.1-10$. We have assumed that the overall magnetic field is homogeneous over
the width of the wake. Though it is possible that the magnetic field will have
some small scale fluctuations due to turbulence in the plasma, for the current
work we have not looked at such details.

We have found that the frequncy spectrum obtained from the relativistic non-
thermal electrons moving in the wake of a cosmic string will be over a wide
range of frequencies. There is a broad region in the spectrum due to the
multiple scattering of the electrons. The spectrum can be fitted with three
separate power law exponents. There is only one broad peak in the spectrum.
Thus the nature of the spectrum differs from the spectrum usually obtained
from blazers or AGN’s. Since there are still quite a few unidenitfied sources
of radiations in the catalogues for the all sky surveys, we have attempted to
find some of the data for the point sources in the given frequency range. We
do find that several of the surveys (WISE, SWIFT and GALEX) have unknown
sources with similar flux in the range of frequencies that are covered by the
cosmic string wake.

We plan to extend this study further to include more details about the cosmic
string wake in the modelling of the synchrotron radiation. Currently we have
used a rather simple one blob model which simplifies the actual geometry of
the wake structure. We have also used a homogeneous magnetic field in our
calculation which may not be the case for the magnetic field in the cosmic
string case. It is quite possible that after generation of the seed field in
the wake of the cosmic string, the evolution of the field by turbulence leads
to a randomized field over very short lengthscales. This would change the
final spectrum of the synchrotron radiation. We hope to look at all these
details in a future work so that we have a more disctinct picture of the
synchrotron radiation from electrons moving in cosmic string wakes.

Acknowledgments

This research was carried out on the computational facility set up from funds
given by the DST- SERB Power Grant no. SPG/2021/002228 of the Government of
India. S.N acknowledges financial support from CSIR fellowship No.
09/414(2001)/2019-EMR-I given by the Human Resource Development, Government of
India. D.K acknowledges financial support from DST- SERB Power Grant no.
SPG/2021/002228 of the Government of India. The authors would like to thank
Abhisek Saha for help with the numerical codes and Sayantan Bhattacharya and
Susmita Barman for suggestions and discussions regarding the experimental
results.

## References

  * (1) Ramesh Narayan et. al. ApJ 912 35 (2021).
  * (2) Copeland Edmund J. and Kibble T. W. B. Proc. R. Soc. A.466623–657 (2010).
  * (3) N. Deruelle, B. Linet, Class. Quantum Grav. 5, 55 (1988).
  * (4) W.A. Hiscock, J.B. Lail, Phys. Rev. D 37, 869 (1988).
  * (5) D. N. Vollick, Phys. Rev. D 48, 3585, (1993).
  * (6) A. Beresnyak, ApJ 804 121, (2015).
  * (7) K. Dimopoulos, Phys. Rev. D 57, 4629 (1998); E. M. Chudnovsky, G. B. Field, D. N. Spergel and A. Vilenkin, Phys. Rev. D 34, 944 (1986).
  * (8) D. A. Rogozin and L.V. Zadorozhna, Astron. Nachr., 334: 1051-1054 (2013).
  * (9) S. Sau, S. Sanyal, Eur. Phys. J. C 80, 152 (2020).
  * (10) K.M. Schoeffler, N.F. Loureiro, R.A. Fonseca, L.O. Silva. Phys. Plasmas 23(5), 056304 (2016).
  * (11) D. Maibach, R. Brandenberger, D. Crichton, and A. Refregier, Physical Review D, 104(12), 123535 (2021).
  * (12) M. V. Medvedev, ApJ 540 704 (2000).
  * (13) G. D. Fleishman, ApJ 638 348 (2006).
  * (14) A. Stebbins, S. Veeraraghavan, R.H. Brandenberger, J. Silk, N. Turok, Astrophys. J. 322, 1 (1987).
  * (15) Andrew T. Sornborger and Robert H. Brandenberger and Bruce Fryxell and Kevin Olson, The Astrophysical Journal, 482, 22-32 (1996).
  * (16) G. A. Ponce and E. T. Vishniac, The Astrophysical Journal, 332:57-66, (1988).
  * (17) S. Nayak, S. Sau, S, Sanyal, Astroparticle Physics, 146,102805, (2023).
  * (18) T. Vachaspati, Phys. Rev. D 45, 3487 (1992).
  * (19) E. R. Harrison, Phys. Rev. Lett. 30, 188 (1973).
  * (20) M. J. Rees, Mon.Not. R. Astron. Soc.222, 27 (1986); J. Silk and A. Vilenkin, Phys. Rev. Lett.53, 1700 (1984); T. Vachaspati, Phys.Rev. Lett.57, 1655 (1986).
  * (21) A.A. Galeev, Advances in Space Research,Volume 4, Pages 255-263,(1984).
  * (22) A. Saha and S. Sanyal, JCAP03, 022, (2018).
  * (23) High Energy Radiation from Black Holes, C. D. Dermer and G. Menon, Princeton University Press (2009).
  * (24) D. Kumar and S. Sanyal, ApJ 944 183 (2023).
  * (25) C. Nigro, J. Sitarek, P. Gliwny and D. Sanchez, A. Tramacere and M. Craig, Astronomy and Astrophysics, Volume 660, A18, (2022).
  * (26) B. Layek, S. Sanyal and A. M. Srivastava, Phys. Rev. D.63.083512, (2001).
  * (27) Hannes-S. Zechlin and Dieter Horns JCAP11, 050 (2012).
  * (28) WISE data, Wright et. al (2010).https://wise2.ipac.caltech.edu/docs/release/allsky/ (Last accessed 16/09/23).
  * (29) A. Falcone et al., Systematic Search for X-ray Counterparts of Fermi-LAT Unassociated Sources Using Swift Observations, volume. 12 of AAS/High Energy Astrophysics Division, (2011).
  * (30) Leire Beitia-Antero and Ana I. Gómez de Castro, Astronomy and Astrophysics, 596, A49 (2016) ;https://galex.stsci.edu/GR6/ (Last accessed 16/09/23).


# Eliciting New Wikipedia Users’ Interests via Automatically Mined
Questionnaires: For a Warm Welcome, Not a Cold Start

Ramtin Yazdanian,1 Leila Zia,2 Jonathan Morgan,2 Bahodir Mansurov,2 Robert
West1
1EPFL
2Wikimedia Foundation
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Every day, thousands of users sign up as new Wikipedia contributors. Once
joined, these users have to decide which articles to contribute to, which
users to seek out and learn from or collaborate with, etc. Any such task is a
hard and potentially frustrating one given the sheer size of Wikipedia.
Supporting newcomers in their first steps by recommending articles they would
enjoy editing or editors they would enjoy collaborating with is thus a
promising route toward converting them into long-term contributors. Standard
recommender systems, however, rely on users’ histories of previous
interactions with the platform. As such, these systems cannot make high-
quality recommendations to newcomers without any previous interactions—the so-
called cold-start problem. The present paper addresses the cold-start problem
on Wikipedia by developing a method for automatically building short
questionnaires that, when completed by a newly registered Wikipedia user, can
be used for a variety of purposes, including article recommendations that can
help new editors get started. Our questionnaires are constructed based on the
text of Wikipedia articles as well as the history of contributions by the
already onboarded Wikipedia editors. We assess the quality of our
questionnaire-based recommendations in an offline evaluation using historical
data, as well as an online evaluation with hundreds of real Wikipedia
newcomers, concluding that our method provides cohesive, human-readable
questions that perform well against several baselines. By addressing the cold-
start problem, this work can help with the sustainable growth and maintenance
of Wikipedia’s diverse editor community.

##  1 Introduction

Every day, roughly 10,000 users open an account on Wikipedia across its more
than 160 actively edited language editions. The retention and diversification
of these newcomers is crucial to Wikipedia’s expansion and success (?), as the
largest encyclopedia of humankind relies on a large and diverse community of
active editors.

Welcoming the newcomers to Wikipedia, however, comes with its own challenges.
One of the first challenges is to quickly learn about the newcomers’
interests, a critical piece of information for building personalized
recommendation systems to read and edit Wikipedia, for matching newcomers with
other newcomers and with more experienced editors, for assessing the diversity
of the editor pool, etc. But learning about newcomers’ interests and profiles
is hard as they do not have a history of contributions to Wikipedia or
interactions with the platform yet. This so-called _cold-start problem_ (?) is
well known in the context of many recommendation algorithms such as
collaborative filtering (?) or content-based filtering (?), families of
algorithms that rely heavily on user histories. Generally, the cold-start
problem arises when recommendations are to be made for a new user, for whom no
past history exists, thus drastically reducing recommendation quality.

Many of the methods used to solve the cold-start problem are based on
automatically generated questionnaires posed to the user, by means of which
the system creates an initial profile for the user and then recommends items
based on said profile. The main problem to be tackled, then, is how a
questionnaire can be automatically generated from available data, which, in
the case of Wikipedia, consists of the content of articles and the editing
history of existing users, which is a form of collaborative data. In
particular for Wikipedia, the challenge is that there is no explicit feedback
(such as the ratings omnipresent in platforms such as Amazon etc.) available
by editors on the articles they like or dislike, but only implicit feedback
(?): the only data that is publicly available about the interactions of
individual users with articles is logs of the edits users have made to
articles.

In this paper, we propose a set of language-independent methods for the data-
driven generation of questionnaires that allow us to elicit initial profiles
from Wikipedia newcomers, with an application to article recommendations.

The following are our main contributions: (i) We propose a language-
independent question generation method that takes topics (represented as
vectors with a weight for every Wikipedia article, indicating the relevance of
the respective article to the topic) as input and turns them into questions
that ask the user to compare a pair of article lists and choose their
preferred list. (ii) We propose three topic extraction methods, based on the
content of Wikipedia articles, their editing history, or both. This provides
the topics needed for our question generation method. After presenting these
questions to a user and thus eliciting their initial profile, we provide them
with article recommendations by combining their questionnaire answers with the
extracted topics. Other forms of assistance (_e.g._ matching newcomers to
other editors based on shared interests) are also possible (but left for
future work). Fig. 1 depicts our pipeline. (iii) Finally, we evaluate the
article recommendations produced based on our questionnaires using historical
edit-history data, also investigating the cohesion and readability of
questions. We then evaluate our method’s article recommendation performance
online, with actual Wikipedia newcomers as participants. Our results show that
our method beats recommendation baselines based on article popularity, a
commonly used signal. Its performance comes relatively close, especially in
terms of reducing undesirable recommendations, to a collaborative filtering
ceiling—which is in fact unsuitable for a real cold-start scenario since it
requires user histories.

Figure 1: An overview of our whole pipeline for creating the questionnaire and
providing users with recommendations, along with section numbers for each
element of the pipeline.

The remainder of this paper is organized as follows: We first discuss related
work on recommender systems in Sec. 2, particularly recommender systems for
Wikipedia. Afterwards, we discuss our questionnaire generation approach and
article recommendation method, followed by our three approaches to extracting
topics from Wikipedia articles, in Sec. 3. We then present the results of our
offline evaluation in Sec. 4, followed by our online evaluation in Sec. 5, and
we discuss our results from both evaluations. At last, we discuss some of the
nuances of deploying our system in practice, describe other uses of our
questionnaire, and point towards directions for future work in Sec. 6.

##  2 Related work

###  2.1 Types of feedback in recommender systems

Generally speaking, two types of feedback exist in recommender systems:
explicit, and implicit (?). Explicit feedback refers to ratings that users can
provide to indicate that they like or dislike an item. Implicit feedback
refers to interactions that do not involve explicit statement of like or
dislike, _e.g._ , browsing history, purchase history, number of listens for a
particular song, etc. This means that like/dislike information has to be
inferred from the available implicit data, usually less reliably than explicit
ratings, since the line between a user disliking an item, and that user simply
not knowing about the item, becomes blurred. Wikipedia lacks explicit ratings
from editors on the articles they like or dislike, and as such, only has
implicit feedback.

###  2.2 Recommender systems for Wikipedia

There exist several works proposing recommender systems tailored for
Wikipedia, which aim to expand Wikipedia in different ways. SuggestBot (?)
matches people with editing tasks in a personalized fashion based on their
editing history. ? (?) expand Wikipedia by recommending articles that are
present in one language and missing in another, while ? (?) recommend new
sections to be added to articles. However, none of these systems address the
cold-start problem for editors; ? (?) focus on recommending the correct
sections for a given article, while ? (?) and ? (?) require a user’s editing
history to recommend articles to them, hence being unsuitable for a complete
newcomer to Wikipedia.

###  2.3 Questionnaire methods

Questionnaire methods for the user cold-start problem ask the user a few
questions and create an initial profile for them, so that the system can
recommend items to the user right from the start. In addition to the relevance
of those items, it is important for the questionnaire to be short, to avoid
tiring the user (?).

To our knowledge, the existing methods all work on explicit feedback systems -
thus making them much less applicable to implicit feedback settings like
Wikipedia - and each of their questions is concerned with the user’s like or
dislike of (or sometimes, lack of knowledge about) a single item. These
methods can broadly be divided into two categories: non-adaptive, and
adaptive. In adaptive methods as opposed to the non-adaptive ones, the
question posed to the user at each step depends on their answers to the
previous ones.

Non-adaptive methods usually use an information theoretic measure on all the
items to choose the top n items, which will then presented to the users as
questions of the type “Do you know this item and do you like it?”. Some of
these measures (?; ?) are as follows:

  * •

Popularity: Items are chosen based on their popularities (_i.e._ how many
ratings they have), a popular item being preferred over more obscure ones.

  * •

Contention (or Entropy): Items are chosen based on how much users disagree
about them. The use of entropy without popularity may result in esoteric and
obscure items being selected (?); thus, methods such as Entropy0 (that takes
missing values into account as zeros) or HELF (Harmonic mean of Entropy and
Logarithm of Frequency) have superior performance to raw entropy (?).
Unfortunately, the lack of reliable negative ratings makes entropy-based
methods become much less applicable in an implicit feedback setting.

  * •

Coverage: As noted by ? (?), certain items might be controversial, but their
ratings might be weakly correlated with those of others, and therefore the
rating the user gives them will not be indicative of what ratings they would
probably give to other items.

Adaptive methods are usually based on decision trees (?; ?; ?), while some are
based on bootstrapping using item-item similarity (?; ?). Some methods fail to
improve accuracy over the best non-adaptive methods (popularity-times-entropy
in case of (?)), while others (?) beat methods such as HELF and Entropy0 in an
online setting. ? (?) use a decision tree with multiple questions at each
step, combining the user’s answers to those questions for the selection of the
next node. Their multi-question approach is designed to increase the
probability that the user would know at least one of the items in question in
each node.

The performance of these methods can be evaluated both offline and online.
Offline evaluation is the most common type, wherein the model is trained on a
training set, and then the questionnaire-answering process is simulated with
users in the test set. Each data point is a user’s rating for an item and the
training and test sets have no users in common. The test error is the mean of
the prediction errors for individual user-item-rating tuples. In online
testing as used by ? (?), actual users answer the questionnaire, and their
answers are treated as their profile; their subsequent ratings of recommended
movies are treated as test data.

##  3 Proposed method

Figure 2: Overview of the process of a user answering the questionnaire and
the generation of recommendations using those responses. The steps mentioned
in the figure are those explained in Sec. 3.1. The vectors $\hat{T_{i}}$ are
the original topic vectors $T_{i}$ with their entries sorted descending from
top to bottom. Both the topic and interest vectors have one entry per
Wikipedia article, capturing the relevance of each article to the topic or the
user, respectively.

Our method works by first eliciting a new user’s interests through a short
questionnaire and then recommending articles that are well aligned with the
inferred interests.

The questionnaire is generated ahead of time in a preprocessing step. It
consists of $K$ questions, each of which asks for the user’s preference among
two lists of articles, A and B: “Between lists A and B, which one contains
more articles that you would be interested in editing?” Answers are collected
on a 7-item Likert scale, indicating a great, moderate, or slight preference
for one of the lists or a lack of preference.

For generating the questions, we require a set of _topic vectors_ $T_{i}$.
Topic vectors must have an entry for each Wikipedia article, specifying the
relevance of the respective article to the topic. Also, topic vectors must
capture topical _dichotomies,_ such that the articles corresponding to the
largest positive entries form a cluster that is clearly distinct from another
cluster formed by the articles corresponding to the largest negative entries
(e.g., singers vs. beetles, cf. example of Table 3). For instance, the topics
discovered by a singular value decomposition (SVD) on a user–article
interaction matrix meet this requirement, since they define topics as
directions of maximum variance among users (represented as vectors with
entries corresponding to articles). In this setup, each topic vector $T_{i}$
can be turned into a question $Q_{i}$ by simply using $T_{i}$’s highest-
scoring $N$ articles as list A and its lowest-scoring $N$ articles, as list
B.111 We choose $N=20$, which gives a good trade-off between quick readability
and sufficient context (cf. evaluation of Sec. 4.1).

Any method producing such dichotomous topic vectors can be plugged into the
above procedure as a black box. We outline three particular methods later
(Sec. 3.2); for now, we assume the $K$ questions are given and focus on how
the user’s answers to the questions are leveraged for recommending articles to
edit (Sec. 3.1).

###  3.1 Generating recommendations from questionnaire answers

The procedure for generating recommendations from questionnaire answers
consists of four steps (depicted schematically in Fig. 2):

  1. 1.

User $u$ answers the $K$ questions (all of the form, “Between lists A and B,
which one contains more articles that you would be interested in editing?”) on
a 7-item Likert scale.

  2. 2.

The categorical answers are turned into numerical scores between $-1$ and $1$
(great, moderate, and slight preference for A are mapped to $1$,
$\frac{2}{3}$, $\frac{1}{3}$, respectively; negative values are used if the
preference is for B, and $0$ is for no preference).

  3. 3.

The numerical scores serve as weights for combining the topic vectors
$T_{1},\dots,T_{K}$ in a weighted sum, which we call the _interest vector_
$I_{u}$ of user $u$. Formally, the interest vector is calculated as

$I_{u}=\sum_{i=1}^{K}\rho_{ui}T_{i},$ (1)

where $\rho_{ui}$ is the numerical representation (cf. item 2 above) of $u$’s
response to the $i$-th question.

  4. 4.

Recommend to $u$ the articles corresponding to the largest entries of the
interest vector $I_{u}$.

This procedure is very similar to collaborative filtering methods based on
matrix factorization (e.g., SVD-based methods): For a given user $u$, those
methods consider a vector $I_{u}^{*}$ with an entry for each article, storing
how much $u$ has previously interacted with (e.g., edited) the respective
article. This vector is projected into a low-dimensional subspace spanned by a
small number of “topic” vectors (via the dot product of $I_{u}^{*}$ with each
topic vector, capturing how well $u$ is aligned with each topic) and then back
into the original space (via a weighted sum of topic vectors, where weights
correspond to the previously computed dot products). Projecting the vector
$I_{u}^{*}$ back and forth smoothes the vector, making values that were
previously equal to zero, non-zero, and thus giving all articles (even those
the user has not interacted with) a meaningful score.

Our procedure can be seen as an approximation of this approach: Since we deal
with newcomers, $I_{u}^{*}$ is not available, so we cannot compute the dot
product of $I_{u}^{*}$ with the topic vectors. Instead, we ask $u$ explicitly
how well she is aligned with each topic vector and use her responses
(interpreted as scores between $-1$ and $1$) as proxies for the dot products.
From here on, our method is identical to the above-described matrix-
factorization approach, so our resulting user interest vectors $I_{u}$ may be
considered approximations of the smoothed vectors that would be output by that
approach if we had access to historical interaction vectors $I_{u}^{*}$.

###  3.2 Preprocessing: Extracting topic vectors

As stated at the beginning of Sec. 3, our questionnaire generation method
relies on topic vectors containing an entry for each Wikipedia article where
the sets of highest- and lowest-scoring articles form dichotomous clusters. In
this subsection, we will describe three approaches for extracting such topic
vectors, which we term “content-only”, “collaborative-only”, and “joint”.
Preprocessing steps for preparing the data may be found in the appendix.

#### Content-only

Our content-only approach uses the textual content of Wikipedia articles as
data, obtained from the English Wikipedia dump.222https://dumps.wikimedia.org/
We use a bag-of-words representation of each article with TF-IDF weighting to
create a matrix $T$ whose columns are articles and whose rows are
unigrams.333Word embedding approaches such as word2vec (?) may also be used in
lieu of TF-IDF. Based on the singular value decomposition (SVD) $T=USV^{T}$
(where $U$ and $V$ are orthogonal matrices and the diagonal $S$ contains the
singular values in sorted order), we define the topic matrix as
$\bar{T}=VS^{T}$.444 Since $T$ is large and sparse, we perform the SVD without
first mean-centering it. As a consequence, the singular vectors (columns of
$U$ and $V$) corresponding to the largest singular value capture mostly the
mean of the data and we hence discard the first column (i.e., topic vector) of
$\bar{T}=VS^{T}$.  We may now use the first $K$ columns of $\bar{T}$ to create
$K$ questions using the question generation method outlined in the beginning
of Sec. 3. This method essentially captures directions of high variance in
terms of the usage of words across articles, and since it outputs orthogonal
directions in space, the correlation between the topics is minimized.

The disadvantage of only using the content is that many words may have
multiple meanings (_e.g._ proper nouns that refer to different entities, such
as the name “Michael”), which could create similarities between dissimilar
articles. This could result in the reduction of readability and semantic
cohesion in the questions.

#### Collaborative-only

The full editing history of
Wikipedia555https://www.mediawiki.org/wiki/API:Revisions provides rich
insights into how editors contribute to the project. Based on this dataset, we
create a matrix $E$ whose rows are users and whose columns are articles, with
$E_{ij}$ being the number of edits user $i$ has made to article $j$.

Our collaborative-only topic extraction method is similar to our content-only
method, except that the SVD is performed on the editing history matrix $E$.
This approach captures directions of high variance in terms of the editors
contributing to articles. Our hope is that there will exist editor
communities, each of which focuses on a specific set of topics, thus allowing
us to capture those topics by finding these high-variance directions. The
disadvantage of this method, however, is that there will still be many users
whose contributions are not necessarily semantically coherent (since many
edits may be about fixing typos, etc. (?)). This might cause semantically
incoherent articles to be grouped together in one topic. In addition, edit
counts alone may not be sufficiently representative of an article’s
importance, because many important articles will have fewer edits because they
lack controversy, are not targeted by vandalism, or require more domain
expertise to edit.

#### Joint topic extraction method

In order to leverage the advantages of both types of data at our disposal, we
propose a joint method that embeds both users and articles in a latent space
using matrix factorization. We call the matrix containing the article latent
representations (in a row-wise fashion) $Q$ and the matrix with the user
latent representations $P$. Our loss function is as follows:

$\begin{split}\min_{P,Q}\;\sum_{u,i}c_{ui}(r_{ui}-p_{u}^{T}q_{i})^{2}&+\alpha\|P\|^{2}+\lambda\|Q-\bar{Q}\|^{2}\\\
&+\theta\|Q^{T}Q-\operatorname{diag}(Q^{T}Q)\|^{2},\end{split}$ (2)

where $\alpha$, $\lambda$, and $\theta$ are hyperparameters set by a grid
search using a validation set (which will be explained later). The matrix
$\bar{Q}$ is obtained by stacking the first 200 columns of $\bar{T}$ (i.e.,
content-based topic vectors) and row-normalizing the result, and serves as a
regularizer for our $Q$ matrix; it is our way of including the content of
articles in the objective function. The reason for the row-normalization will
be explained in the optimization section. The last term in the function is
designed to enforce an orthogonality constraint on the columns of $Q$ in a
soft manner; this is to minimize correlation between dimensions. The $r_{ui}$
and $c_{ui}$ are derived from the respective entry of the user-article editing
history matrix, $e_{ui}$, following ? (?) as

$\displaystyle r_{ui}$ $\displaystyle=$ $\displaystyle\begin{cases}1&\text{if
}e_{ui}>0,\\\ 0&\text{otherwise,}\end{cases}$ (3) $\displaystyle c_{ui}$
$\displaystyle=$ $\displaystyle
1+\kappa\,\ln\left(1+\frac{e_{ui}}{\epsilon}\right).$ (4)

where $r_{ui}$ is the inferred rating by user $u$ for article $i$, while
$c_{ui}$ indicates how confident we are in the inferred rating. The
hyperparameters $\kappa$ and $\epsilon$ determine how quickly confidence grows
with increasing edit counts; we have set them to 10 and 20 respectively as we
believe these to be reasonable values, reducing the number of hyperparameters
for our grid search.

As mentioned before, the number of latent dimensions for $P$ and $Q$ is 200.
As with other matrix factorization problems, this objective function is non-
convex. It is important to mention that the existence of the $c_{ui}$ and
$r_{ui}$ means that our joint method does not reduce to the collaborative-only
method if $\bar{Q}$ is set to 0.

#### Optimization and evaluation metrics

We optimize the objective function following ? (?) and replacing stochastic
gradient descent with mini-batch gradient descent. Since our primary aim is to
extract topics, our focus is more on our more reliable information, _i.e._ ,
the non-zero entries. Therefore, at each iteration, we take a mini-batch from
our $E$ matrix that is $90\%$ non-zeros and $10\%$ zeros.

Our objective function being non-convex means that we are likely to arrive at
a local - rather than global - minimum using gradient descent, and our
initialization step will be an important one. We set $Q$ to $\bar{Q}$ and
randomly initialize $P$, and then we row-normalize $P$, such that all three
matrices are row-normalized. This is because the $r_{ui}$ in our objective
function are either $0$ or $1$, and if the rows of $P$ and $Q$ are normalized,
the value of the dot product between a row of $P$ and a row of $Q$ ranges
between $-1$ and $1$. This does mean that the columns of $\bar{Q}$ are not
orthogonal anymore, but our results show that the orthogonality constraint on
$Q$ still results in almost-orthogonal columns for $Q$, and our observed
results are superior to the case where we do not use row-normalization.

The values of our hyperparameters are determined by taking a validation set
out of our user-article matrix. The validation set is created by choosing a
fraction of the users and then taking out a fraction of their edits, and then
training the objective on the remaining user-article matrix. In the end, we
calculate two values for the selection of hyperparameters:

  * •

The validation error, which is the mean squared error (_i.e._ the first term
in Eq. 2) on the held-out edits for the validation set users.

  * •

The cohesion score for $Q$, that is calculated as follows: for each topic, we
calculate the cosine between each pair of article latent vectors for the set
of the 20 most positively weighted articles in that topic and similarly for
the set of the 20 most negatively weighted articles, and we then take the
average separately for each of the two sets. The average of these two values
is the cohesion score for that topic, which has a maximum of 1. This score is
an important measure because the questions we generate and the dichotomies
they capture need to be easily intelligible to the questionnaire respondent.
We will also use this cohesion score to compare the quality of the questions
generated by our three methods.

To choose the best hyperparameters, we look at these (error,score) pairs and
identify the point that offers the best trade-off between a low validation
error and a high cohesion score.

Table 1: The mean and 95% confidence interval of cohesion for 20 and 50
questions generated using our three methods.

Method | Avg. cohesion, 20 qs | Avg. cohesion, 50 qs
---|---|---
Joint | $0.672[0.626,0.719]$ | $0.674[0.627,0.719]$
Collab-only | $0.578[0.494,0.665]$ | $0.582[0.533,0.632]$
Content-only | $0.434[0.375,0.493]$ | $0.439[0.388,0.493]$

Table 2: The mean and five precentiles of recall at 300 for the three
baselines and our three methods (content-only, collaborative-only, and joint),
with the addition of a collaborative-only method without the septile-based
stratification (Collab-only, no strat.) in the question answering simulation.
Our methods are in bold. The total number of test users is 15,028.

Method | mean | 50% | 75% | 90% | 95% | 99% | 99.9%
---|---|---|---|---|---|---|---
CF-based | 0.06758 | 0.05 | 0.10 | 0.20 | 0.25 | 0.45 | 0.70
Content-only | 0.02113 | 0.00 | 0.05 | 0.05 | 0.10 | 0.20 | 0.40
Edit-pop | 0.01081 | 0.00 | 0.00 | 0.05 | 0.05 | 0.10 | 0.20
Joint | 0.01077 | 0.00 | 0.00 | 0.05 | 0.05 | 0.15 | 0.30
Collab-only, no strat. | 0.00681 | 0.00 | 0.00 | 0.05 | 0.05 | 0.10 | 0.25
View-pop | 0.00490 | 0.00 | 0.00 | 0.00 | 0.05 | 0.05 | 0.10
Collab-only | 0.00488 | 0.00 | 0.00 | 0.00 | 0.05 | 0.10 | 0.20

Table 3: Top and bottom 5 articles in the first 3 questions, generated by the
joint method.

| Question 1 | Question 2 | Question 3
---|---|---|---
Top 5 |  | Meat Loaf
---
Elvis Costello
Stevie Nicks
Eddie Vedder
Tom Jones (singer)
| Sikeston, Missouri
---
Selma, Alabama
Timeline of Baltimore
St. Augustine, Florida
Clarksville, Tennessee
| Philipp, Prince of Eulenburg
---
Maximilien Robespierre
Charles George Gordon
In Solitary Witness
Mortara case
Bottom 5 |  | Apomecynini
---
Desmiphorini
Agaristinae
Endemic goitre
C21H20O10
| Incomplete Nature
---
Troubleshooting
Affective computing
Isothermal microcalorimetry
Positive feedback
| Tri-Cities, Washington
---
Kent, Ohio
Champaign, Illinois
Limon, Colorado
Carlsbad, New Mexico

##  4 Offline evaluation

###  4.1 Question cohesion

Our first approach to evaluating our questions is calculating the cohesion of
the questions. Table 1 shows the cohesion values for 20 and 50 questions for
the joint method, the content-only SVD, and the collaborative-only SVD. The
cohesion scores here, in each of the three schemes, have been calculated using
the same matrix for both the questions and the latent representations, thus in
some sense simultaneously measuring both the quality of individual questions
and their latent representations. We also manually check the first 20
questions generated by the method. Our manual investigation reveals that the
content-only method has a few relatively outlier articles in each 20-article
list, which could confuse the user regarding the meaning of the question. The
top and bottom 5 articles for the first 3 questions can be seen in Table 3.

###  4.2 Recommendation performance

The second element of our offline evaluation is recommendation performance; we
perform it by simulating users in the test set answering the questionnaire.
This simulation is as follows: for each test user, we hold out 20 of their
distinct edited articles, _i.e._ in their edit vector, we set to zero 20 of
the (non-zero) entries, creating a “modified edit vector”. Then, in order to
simulate the online experiment, we calculate the dot product between each
question vector and the user’s modified edit vector - which results in a
quantity we call the user’s simulated response to that question - and then
calculate the 1st to 6th septiles of the set of all these values across all
users. These septiles will allow us to rank-normalize the users’ simulated
responses to the questions into the 7-level Likert scale that we discussed
earlier. This gives us a response vector for the user, consisting of all their
simulated responses. This method is equivalent to projecting the user’s
article-space edit vector onto the subspace spanned by the topics (_i.e._
questions), and then representing the resulting vector back in the original
basis, with the difference that we also discretize their projection onto each
dimension of the subspace.

We generate the user’s recommendations as described previously. Using this
list, we calculate recall@$k$ (the fraction of the holdout article set
appearing in the top $k$ recommendations) and precision@$k$ (the fraction of
the top $k$ recommendations appearing in the holdout article set) for the 20
holdout articles of the user in question. Note that since the total number of
true positives is known (there are 20 holdout articles), recall@$k$ and
precision@$k$ are trivially related by a known fraction and therefore, we only
report recall@$k$.

Due to the sparsity of the modified edit vector, our simulation of the
question answering is using the full topic vector, rather than a topic vector
with everything except the top and bottom 20 articles masked; such a masking
would have resulted in most responses being 0. This, combined with our method
for evaluation (the 20-article holdout set), means that the offline evaluation
is relatively unrealistic.

We use two baselines and one ceiling to compare the performance of our method
with:

  * •

Recommendation based on editing popularity, where we recommend the most edited
articles. This is a weak baseline since it is not personalized for each user.
We call this baseline “Edit-pop”.

  * •

Recommendation based on viewing popularity, where we recommend the most viewed
articles. This is also a weak baseline. We call this baseline “View-pop”.

  * •

Recommendation using collaborative filtering (with 50 latent dimensions,
selected using a validation set) on the user-article editing matrix, which we
call “CF-based”. This is done using the Python package _implicit_
666https://github.com/benfred/implicit, using an objective function proposed
in (?), which is essentially our own with $\bar{Q}$ set to 0 and the soft
orthogonality constraint removed.

The collaborative filtering (CF) method is a ceiling and not a baseline,
because the primary target audience of a cold-start questionnaire comprises
users who have no editing history since they have just created an account, and
CF is unusable in the absence of existing history for the user. As such, a
comparison of our method’s performance with the performance of CF for users
for whom there is some existing editing history, allows us to gauge how far we
are from a potentially unattainable high level of performance.

The results can be seen in Table 2. We have additionally included the
collaborative-only method without the septile-based stratification (whose
absence would make the offline evaluation less realistic) to show the negative
effect of said stratification on offline recommendation performance.
Interestingly, the content-only method outperforms our joint method in the
offline simulation, and is only outperformed by the CF-based recommendations.
This is especially surprising because the holdout set of users is essentially
part of the greater editing history matrix, which plays no role in the
content-only method, but does play a role in our joint method. It is also
worth noting that the collaborative-only method is the worst-performing
method. This, combined with the cohesion values we see in Table 1, means that
the editing history matrix itself is noisy and reduces our offline performance
in the joint method; however, it also helps denoise the questions generated by
our method, making them much more cohesive. This could be because dissimilar
articles that possess similar terms do not have similar editor communities,
and this allows our joint method to separate these dissimilar articles from
each other (while the content-only approach would fail to do so).

As mentioned before, our offline test is rather unrealistic because of how the
question answering process is simulated and how the results are evaluated,
even though it is a necessary sanity check for the output of our method. An
online evaluation with real users will be a more realistic evaluation of the
system, and we will discuss how we have conducted this online evaluation in
the next section.

##  5 Online evaluation

Our online evaluation will attempt to measure the satisfaction of recently
joined users with the recommendations provided by our method, in comparison
with two baselines and a ceiling. For this evaluation, we will have to choose
one of the three question sets, based on the offline evaluation. Our joint
method provides the best trade-off between question cohesion and offline
recommendation performance, given that it performs acceptably in the latter
while being by far the best in the former. As a result, we will use the
questions generated using topics extracted by our joint method in our online
evaluation.

Compliant with Wikimedia’s “Wikipedia is not a laboratory” policy, our
research project has been fully documented on Wikimedia’s meta-
wiki777https://meta.wikimedia.org/wiki/Research:Voice˙and˙exit˙in˙a˙voluntary˙work˙environment/Elicit˙new˙editor˙interests
as a research project, and we have taken cautionary steps in order not to
disrupt the normal operation of any Wiki in any shape or form.

###  5.1 Design

We took several steps before launching the experiment itself, detailed below:

  * •

We took all the users who had signed up for an account on the English
Wikipedia in the 3 months from September to December 2018,888A link to the
privacy policy for the experiment is found on the meta page, under “Second
Online Experiment”. since we wanted our participants to be relatively new to
the platform, and not seasoned veterans. We divided these users into two sets:
those with more than 3 edits, and those with 3 or fewer edits. For the former
set, we have three baselines: View-pop, Edit-pop, and CF-based, while for the
latter, we only have View-pop and Edit-pop. We will call the former condition
“with CF”, while the latter will be called “no CF”.

  * •

In order to calculate the CF-based recommendations for the first set of users,
we first filtered their edits to only include _articles_ (rather than
talk/user pages). Then, similarly to what we did for our offline evaluation,
we used our user-article editing history matrix with the Python package
“implicit” (again with 50 latent dimensions) to pre-calculate the
recommendations, which we then stored. At this point, the “with CF” condition
numbered about 26,000 users. For each of these users, we calculated a unique
participation token that would allow us to identify them when they took the
questionnaire, so that we could give them their pre-calculated CF
recommendations when they participated.

  * •

We subsampled the set of “no CF” users to also contain 26,000 users, and also
generated participation tokens for them, for the sake of homogeneity.

Table 4: Number of “wins”, “draws” and “losses” for our method versus the
three competitors for the reading preference question in each of the
conditions.

Reading preference | with CF | no CF
---|---|---
losses | draws | wins | losses | draws | wins
Q-based vs CF-based | 172 | 73 | 115 | - | - | -
Q-based vs View-pop | 108 | 70 | 182 | 81 | 53 | 163
Q-based vs Edit-pop | 55 | 94 | 211 | 53 | 65 | 179

Table 5: Number of “wins”, “draws” and “losses” for our method versus the
three competitors for the editing preference question in each of the
conditions.

Editing preference | with CF | no CF
---|---|---
losses | draws | wins | losses | draws | wins
Q-based vs CF-based | 178 | 79 | 103 | - | - | -
Q-based vs View-pop | 103 | 88 | 169 | 68 | 74 | 155
Q-based vs Edit-pop | 46 | 113 | 201 | 50 | 92 | 155

At this point, the experiment was ready to begin, and we set it up as follows:

  * •

We sent emails to users from both sets, containing an invitation message, a
link to the questionnaire, and their participation tokens. We attempted
sending emails to all the users in both sets, but some had not enabled the
receipt of emails, and only 13,794 users in total could be contacted.

  * •

Each user answered the questionnaire, which consisted of the first 20
questions generated by the joint topic extraction method, where each question
was about comparing two lists of 20 articles each and indicating the list that
contained more articles that they would be interested in editing. For each
list of articles, a word cloud of the most important terms in that list of
articles was provided to provide further summary information about that list
of articles, in addition to the articles themselves. Without the word clouds,
some article lists (from different questions) would appear to be about the
exact same subject, whereas the word clouds would reveal nuanced differences
between them. For each question, the user was given the seven options as
described earlier on. Their responses to the questionnaire were used to
construct their article-space profile and thus to generate their
questionnaire-based recommendations.

  * •

For each user, twelve lists of recommendations were generated, each of which
contained 5 articles. For all the users, six of these lists were generated
using their responses to the questionnaire (which we will call “Q-based” from
now on). For “with CF” users, two of the remaining six were View-pop, two
Edit-pop, and two CF-based. For “no CF” users, three of the remaining six were
View-pop and the other three Edit-pop. The six lists of Q-based
recommendations were each paired with one of the baseline lists, resulting in
six pairs of lists, which the user would then provide feedback on.

Since over-specialization is a well-known issue in recommender systems (?) and
in order to increase the diversity of our recommendations, we have used a
simple diversification scheme for each of the four recommendation methods: we
first generated a larger set of recommendations (randomly from the most
popular articles in case of View-pop and Edit-pop), which we then clustered
into several groups based on the latent representations we had generated for
the articles (the rows of the $Q$ matrix), and then we randomly took one
article from each cluster. This scheme was mainly meant for the Q-based and
CF-based recommendations, but for the sake of a fair comparison, we also used
it for View-pop and Edit-pop.

  * •

For each of the six pairs of lists, the users were asked three questions
(where each, again, had 7 options):

    1. 1.

Among the two lists, which list has more articles that you would be interested
in reading?

    2. 2.

Among the two lists, which list has more articles that you would be interested
in editing?

    3. 3.

Among the two lists, which list has more articles that you would not be
interested in at all, neither for reading nor for editing?

  * •

The user’s responses and their username (which was uniquely identifiable based
on their participation token) were recorded.

Table 6: Number of “wins”, “draws” and “losses” for our method versus the
three competitors for the uninterestedness question in each of the conditions.

Uninterestedness | with CF | no CF
---|---|---
losses | draws | wins | losses | draws | wins
Q-based vs CF-based | 137 | 114 | 109 | - | - | -
Q-based vs View-pop | 90 | 103 | 167 | 67 | 83 | 147
Q-based vs Edit-pop | 54 | 111 | 195 | 54 | 89 | 154

(a)

(b)

Figure 3: Violin plots for original responses to the three feedback questions
and response type proportions to questionnaire questions. (a) Violin plots for
original responses to the three feedback questions for users in the “with CF”
condition. (b) Violin plot of questionnaire response type proportions for each
response type, for strongly dissatisfied and satisfied users. The distribution
of “no preference” responses is different between the two sets with $p<0.001$.
(Black bar: inter-quartile range; white dot: median.)

Note that we ask separate questions for editing and reading preferences, since
reading an article and contributing to it are fundamentally different, but a
respondent might mix them up if they are not in separate questions. In
addition, we asked the question about which list had more articles they were
not interested in, since previous work has shown that users may distrust
recommender systems that make recommendations that are obviously bad (?).

###  5.2 Results

We received a total of 279 responses, 180 of which were from the “with CF” set
while the remaining 99 were “no CF”. This imbalance in the response rates is
not surprising, since more active new editors (those who edit more in the
first few days after creating an account) are much more likely to continue
participating in Wikipedia (?).

In order to present the results, we categorize the results for each question
into three types of comparisons (Q-based versus each of the three competitor
methods), mapping the responses to their numerical representations from $-1$
to $1$ in steps of $\frac{1}{3}$, with the sign indicating whether the Q-based
recommendations beat the other method (and $0$ corresponding to a draw). In
the reading and editing preference questions, a positive value indicates that
our method recommended more interesting articles than the other method, while
for the uninterestedness question, a positive value indicates that our method
recommended fewer uninteresting articles than the other method.

The ternarized results (win, draw and loss, corresponding to positive, zero
and negative responses) for all three comparison types in each of the three
questions can be seen in Tables 4, 5 and 6, showing that our method beats the
popularity-based baselines for both conditions and all three questions, while
the CF-based ceiling remains the best-performing method. For the “with CF”
users, violin plots for the original response values (from -1 to 1) can be
seen in Fig. 3a. Interestingly, both Table 6 and Fig. 3a show that the margin
of defeat for our method against CF-based recommendations is considerably
smaller in the uninterestedness question, which points towards the gap between
our method and CF-based recommendations being smaller than the gap between the
popularity-based baselines and our method. In particular, our method seems to
be close to the CF-based ceiling when it comes to not providing undesirable
recommendations, as evidenced by the results of the uninterestedness question.

###  5.3 Analysis of dissatisfied users

An interesting question is the following: how many of our respondents are
dissatisfied with their Q-based recommendations, and if so, are there any
patterns in their recommendations or in the way they answered the
questionnaire? As a response to the first question, we define three sets of
respondents: strongly dissatisfied, mildly dissatisfied, and satisfied. We
define strongly dissatisfied users as those who have indicated the Q-based
recommendations as the less interesting ones compared to popularity-based
baselines in at least half of their uninterestedness questions. These users
number 58 respondents (out of 279), which means about 20% of our respondents,
pointing towards the potential existence of systematic effects. Satisfied
users are those who have not done so in any of the uninterestedness questions,
and they make up 135 respondents, which is about 50% of all the respondents.
Slightly over two-thirds of each of the two sets have the “with CF” condition.
We call the rest of the users “mildly dissatisfied”, and we exclude them from
this analysis. In order to better understand the characteristics of the
strongly dissatisfied users compared to the satisfied users, we analyze three
features in these two sets of users:

  * •

For each user, the proportion of each of the four types of response (no
preference, slight preference, moderate preference, and great preference)
across all of their 20 questionnaire responses.

  * •

The proportion, across each of the two sets, of their preferential and non-
preferential (_i.e._ non-zero and zero) responses for each question in the
questionnaire.

  * •

The lists of recommendations given to them, investigated manually.

Our reason for investigating _how_ the dissatisfied users answered the
questionnaire is that our questions have different numerical latent
representations in the article space, and the effects and interactions of
these latent representations are unclear to us. An analysis of how the
dissatisfied users differed in their questionnaire answering from the
satisfied users can give us useful insights into the improvements we could
make to our method.

The distribution of proportions of each response type for each of the two
groups are shown in Fig. 3b. We conducted a Kruskal-Wallis H-test to see if
the difference between the proportions of the “no preference” option between
the two sets of users was significant, which gave us a p-value of 0.00014,
meaning that there is a significant difference between the distributions of
said proportion among the two sets of users. This tells us that the satisfied
users have had a greater tendency to use the “no preference” option (we do not
have enough evidence to state the inverse, namely that using the “no
preference” option more would lead to greater satisfaction). This could point
to our method performing worse when many non-zero responses are given,
especially since the article-space representations of questions can be
conflicting, _i.e._ a positive response to a question up-weights some articles
while down-weighting others, and these down-weighted articles may themselves
have been up-weighted by another question, which creates a conflict. Thus,
this is a area for future improvement.

We investigated the per-set (the sets being strongly dissatisfied and
satisfied) proportions of non-preferential and preferential responses to each
question, but we did not see any meaningful pattern in the differences, and as
such, we cannot say that the issue comes mainly from a certain part of the
questionnaire (_e.g._ the earlier questions or the later questions). As the
final part of our investigation, we also manually looked at the
recommendations given to the dissatisfied users. However, no distinct patterns
could be found in their recommendations, other than occasional over-
specialization, which indicates that a more advanced diversification scheme is
an important direction for future work.

##  6 Discussion and future work

In this section, we present the implications of this study and future
directions.

#### Tailoring recommendations for contribution

In the evaluation of the questionnaire, we asked editors to assess lists of
recommended articles for reading and editing purposes. Such lists can be
improved in two ways. First, the reading recommendations for editors may be
different than reading recommendations for readers as such reading
recommendations can inform the editing. Such lists are currently being
generated agnostic to whether the user is an editor or a reader. Second, the
editing recommendations do not consider the specific needs of the articles
that are being recommended for editing. For example, articles of lower quality
or stubs may need to be prioritized in such recommendations using new and
existing system (?). One can also provide recommendations based on the type of
edits an article is missing: for example typo fixing, adding citations,
expanding a section, etc. Since our method essentially creates a total
ordering for the article space, it can, given a list of stubs and other
articles needing contribution, limit its recommendations to those.

#### Improving the recommendations

One direction for future work is taking into account the sizes and other
details of individual edits, and the discussions on talk pages of articles,
which would give us richer information on users and potentially improve our
editing history matrix. Another direction would be addressing the issue of
lowered recommendation quality when faced with potentially conflicting
responses to the questionnaire, which could be achieved through a more
advanced diversification scheme that could make sure that all questions get
their fair share of presence in the recommendations while retaining the
ability of our method to recommend articles that bridge different topics.
Also, an interesting improvement would be making the questionnaire adaptive,
thus technically allowing for more topics to be included in the questionnaire,
as not everyone would be getting the same 20 questions in an adaptive
questionnaire. This would allow for more detailed initial profiles for users.
In addition to adaptivity, one question that could be addressed in future work
is finding an optimal number of questions which creates a balance between
eliciting enough information and being short enough for the user to answer
quickly. We estimate (based on small-scale tests) that each question in the
current questionnaire would take about one minute to answer, thus bringing up
the total answering time to about 20 minutes, which could drive some newcomers
away. A simple solution would be to allow the user to answer as many questions
as they would like, rather than a fixed number of questions.

#### User matching

An interesting approach to helping newcomers (and an alternative to article
recommendation) is matching them with other newcomers or with veteran editors
to help get them started. The latter can provide the newcomer with a mentor of
sorts, while the former allows the formation of more flat and non-hierarchical
groups who can collaboratively contribute to Wikipedia. Our questionnaire
allows us to perform such a matching, since each user can be represented as
their vector of responses to the questions, and the similarity between two
users can be defined as the dot product of their response vectors. We
conducted an experiment to assess how such a matching would help newcomers,
but our experiment was unsuccessful due to low participation. We hypothesize
that the low participation was due to the complexity of the experiment design,
some of which is imposed by the way Wikipedia communications between editors
work today. We believe that exploring matching based on the constructed user
interest profiles is a potential direction for the future, and an interesting
use of our questionnaire.

#### Measuring retention

In this work, we focused on measuring the quality of our recommendations and
the immediate satisfaction of the users with them. Another important question
is whether the deployment of this cold-start recommender leads to increased
retention among users who may not have become active editors otherwise. There
will need to be a different and longer experiment to measure retention,
potentially by providing some newcomers with personalized recommendations and
others with random recommendations and comparing their contributions over
time.

##  7 Conclusions

We have proposed a questionnaire creation pipeline—consisting of a novel
automatic question generation method, based on pairwise comparisons, and a set
of three topic extraction methods—that can create questionnaires to help
tackle the editor cold-start problem on Wikipedia. We showed that our proposed
joint topic extraction provides the best trade-off between the semantic
cohesion of questions and offline article recommendation performance. Our
online evaluation shows that our method provides article recommendations that
are superior to non-personalized baselines, while having relatively close
performance to a collaborative filtering ceiling which is inapplicable to a
real cold-start scenario due to its need for the editing history of the user.
Our method’s performance is achieved using a non-adaptive questionnaire, and
we believe that adding adaptiveness can boost its performance even further.

##  8 Acknowledgments

We would like to thank Tiziano Piccardi and Hristo Paskov for their
assistance, and the Wikipedia editor community for participating in our online
evaluation and making this study possible. We would also like to thank the
Wikimedia research community for their valuable feedback.

##  Appendix A Preprocessing the data

To improve the results of our method, we have performed several preprocessing
steps on the set of articles and the set of users before training our model,
including:

  * •

Removing very rare (appearing in less than 50 articles) and very frequent
(appearing in more than 10% of the articles) words.

  * •

Removing users and articles with very low edit counts (the lower bound being
20 in both cases).

  * •

Removing very short stubs (under 100 words), since their content is too sparse
and noisy.

  * •

Removing lists and list-like articles since they a) exist more for reasons of
organization rather than pure content, and b) contain many rare terms (_e.g._
proper nouns) due to their nature, and according to our tests, they end up
disproportionately dominating the topics if not removed. We have not conducted
this filtering in an exhaustive manner since it is sometimes debatable whether
or not an article is a list; a more exhaustive removal of such articles is
left for future work.

Our filtered article space contains approximately 2,200,000 articles, which is
still a very large set. However, if the possibility for the recommendation of
any and all articles is required, then the article recommendations made by our
method may be used as an intermediate step, with the final recommendations
(out of the full article set) being selected based on the intermediate
recommendations. This selection can be based on content similarity to the
intermediate recommendations, the need for contributions and its severity,
various metadata about the article (_e.g._ its length, number of sections,
whether or not it is a list), etc. For the sake of reproducibility and easier
future expansion of our work, we have made both our code and our data publicly
available.999https://github.com/RamtinYazdanian/wikipedia˙coldstart˙questionnaire

## References

  * [Cosley et al. 2007] Cosley, D.; Frankowski, D.; Terveen, L.; and Riedl, J.  2007\.  Suggestbot: Using intelligent task routing to help people find work in Wikipedia.  In Proceedings of the 12th International Conference on Intelligent User Interfaces, 32–41.
  * [Golbandi, Koren, and Lempel 2010] Golbandi, N.; Koren, Y.; and Lempel, R.  2010\.  On bootstrapping recommender systems.  In Proceedings of the 19th ACM International Conference on Information and Knowledge Management, 1805–1808.
  * [Golbandi, Koren, and Lempel 2011] Golbandi, N.; Koren, Y.; and Lempel, R.  2011\.  Adaptive bootstrapping of recommender systems using decision trees.  In Proceedings of the 4th ACM International Conference on Web Search and Data Mining, 595–604.
  * [Goldberg et al. 1992] Goldberg, D.; Nichols, D.; Oki, B. M.; and Terry, D.  1992\.  Using collaborative filtering to weave an information tapestry.  Communications of the ACM 35:61–70.
  * [Halfaker et al. 2013] Halfaker, A.; Geiger, R. S.; Morgan, J. T.; and Riedl, J.  2013\.  The rise and decline of an open collaboration system: How Wikipedia’s reaction to popularity is causing its decline.  American Behavioral Scientist 57(5):664–688.
  * [Halfaker 2017] Halfaker, A.  2017\.  Interpolating quality dynamics in Wikipedia and demonstrating the keilana effect.  In Proceedings of the 13th International Symposium on Open Collaboration, 19:1–19:9.
  * [Hu, Koren, and Volinsky 2008] Hu, Y.; Koren, Y.; and Volinsky, C.  2008\.  Collaborative filtering for implicit feedback datasets.  In 8th IEEE International Conference on Data Mining.
  * [Koren, Bell, and Volinsky 2009] Koren, Y.; Bell, R.; and Volinsky, C.  2009\.  Matrix factorization techniques for recommender systems.  Computer 42(8):30–37.
  * [McNee, Riedl, and Konstan 2006a] McNee, S. M.; Riedl, J.; and Konstan, J. A.  2006a.  Being accurate is not enough: How accuracy metrics have hurt recommender systems.  In Extended Abstracts on Human Factors in Computing Systems, 1097–1101.
  * [McNee, Riedl, and Konstan 2006b] McNee, S. M.; Riedl, J.; and Konstan, J. A.  2006b.  Making recommendations better: An analytic model for human-recommender interaction.  In Proceedings of the 2006 Conference on Human Factors in Computing Systems, 22–27.
  * [Mikolov et al. 2013] Mikolov, T.; Chen, K.; Corrado, G.; and Dean, J.  2013\.  Efficient estimation of word representations in vector space.  CoRR abs/1301.3781.
  * [Mooney and Roy 2000] Mooney, R. J., and Roy, L.  2000\.  Content-based book recommending using learning for text categorization.  In Proceedings of the 5th ACM Conference on Digital Libraries, 195–204.
  * [Panciera, Halfaker, and Terveen 2009] Panciera, K.; Halfaker, A.; and Terveen, L.  2009\.  Wikipedians are born, not made: A study of power editors on Wikipedia.  In Proceedings of the ACM 2009 International Conference on Supporting Group Work, 51–60.
  * [Piccardi et al. 2018] Piccardi, T.; Catasta, M.; Zia, L.; and West, R.  2018\.  Structuring Wikipedia articles with section recommendations.  CoRR abs/1804.05995.
  * [Rashid et al. 2002] Rashid, A. M.; Albert, I.; Cosley, D.; Lam, S. K.; McNee, S. M.; Konstan, J. A.; and Riedl, J.  2002\.  Getting to know you: Learning new user preferences in recommender systems.  In Proceedings of the 7th International Conference on Intelligent User Interfaces, 127–134.
  * [Rashid, Karypis, and Riedl 2008] Rashid, A. M.; Karypis, G.; and Riedl, J.  2008\.  Learning preferences of new users in recommender systems: An information theoretic approach.  ACM SIGKDD Explorations Newsletter archive 10(2):90–100.
  * [Schein et al. 2002] Schein, A. I.; Popescul, A.; Ungar, L. H.; and Pennock, D. M.  2002\.  Methods and metrics for cold-start recommendations.  In Proceedings of the 25th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, 253–260.
  * [Sun et al. 2013] Sun, M.; Li, F.; Lee, J.; Zhou, K.; Lebanon, G.; and Zha, H.  2013\.  Learning multiple-question decision trees for cold-start recommendation.  In Proceedings of the 6th ACM International Conference on Web Search and Data Mining, 445–454.
  * [Wulczyn et al. 2016] Wulczyn, E.; West, R.; Zia, L.; and Leskovec, J.  2016\.  Growing Wikipedia across languages via recommendation.  In Proceedings of the 25th International Conference on World Wide Web, 975–985.
  * [Zhou, Yang, and Zha 2011] Zhou, K.; Yang, S.; and Zha, H.  2011\.  Functional matrix factorizations for cold-start recommendation.  In Proceedings of the 34th International ACM SIGIR Conference on Research and Development in Information Retrieval, 315–324.