# Generative De-Quantization for Neural Speech Codec
via Latent Diffusion

###### Abstract

In low-bitrate speech coding, end-to-end speech coding networks aim to learn
compact yet expressive features and a powerful decoder in a single network. A
challenging problem as such results in unwelcome complexity increase and
inferior speech quality. In this paper, we propose to separate the
representation learning and information reconstruction tasks. We leverage an
end-to-end codec for learning low-dimensional discrete tokens and employ a
latent diffusion model to de-quantize coded features into a high-dimensional
continuous space, relieving the decoder’s burden of de-quantizing and
upsampling. To mitigate the issue of over-smooth generation, we introduce
midway-infilling with less noise reduction and stronger conditioning. In
ablation studies, we investigate the hyperparameters for midway-infilling and
latent diffusion space with different dimensions. Subjective listening tests
show that our model outperforms the state-of-the-art at two low bitrates, 1.5
and 3 kbps. Codes and samples of this work are available on our webpage
111https://saige.sice.indiana.edu/research-projects/LaDiffCodec.

Index Terms—  Speech Codec, Latent Diffusion Model, Speech Synthesis

##  1 Introduction

Neural speech codecs are designed to capture human speech’s intricate patterns
more effectively than traditional methods. Recently, with the successful
attempts of codec-based generation [1, 2, 3], high-bitrate neural codecs [4,
5, 6] gain much attention, for its capability of recovering high-fidelity
audio.

Current high-fidelity codecs mostly perform waveform coding [7, 8]. They
typically learn encoder and decoder models end-to-end, aiming at extracting
expressive latent features and powerful decoders in a single network.
Reconstruction objectives favor preserving essential information. Thus, with
abundant data, end-to-end models are generally good at learning
representations and achieving high-fidelity reconstruction at higher bitrates.
For example, SoundStream achieves reasonable speech quality at 3kbps with its
fully convolutional architecture and residual vector quantization [4]. Pre-
trained transformer and language models have been used to assist low-bitrate
coding in open-sourced EnCodec [6] and an ultra-low-bitrate codec [9]. More
recently, DAC [5] implements RVQGAN and snake activation into the architecture
and improves coding quality, especially on high bandwidth. Many other
improvements involve disentanglement [10, 11, 12] into end-to-end coding to
remove redundancy. However, the above works achieve good quality speech only
at medium or high bitrate ($\geq$ 3 kbps). In the low-bitrate case,
excessively deep and complex networks are necessary to learn low-dimensional
representation, which can impair end-to-end training. Learning expressive
features and powerful decodes at the same time becomes challenging.

Therefore, on coding low bitrate, many works turn to a vocoder-based codec to
leverage generative models, and reconstruction of speech from existing
features, e.g., WaveNet-based codecs [13, 14], LPCNet for real-time coding
[15], GAN-empowered resynthesis [16]. Recent employment of AudioLM [1] in
SoundStream as in LMCodec [17], reduces the bitrate down to $\sim$ 1kbps.
Neural feature predictor [18] applies a generative model on the feature domain
to assist LPCNet with more efficient feature input.

Fig. 1: A diffusion model de-quantizes the low-dimensional, low-bitrate
discrete speech tokens into high-dimensional continuous variables.

We observed from pilot experiments that low-bitrate coded features from
waveform codec can preserve distinguishable speech features and better capture
essential information than the traditional speech features at the same
bitrate. Thus, it is worth utilizing the representation learned from the end-
to-end audio coding schemes while replacing the end-to-end trained decoders
with more powerful generative models. To this end, the proposed system
consists of three modules. First, we employ an autoencoder whose bottleneck is
defined by a continuous and high-dimensional latent space. We assume that its
decoder is responsible for high-fidelity reconstruction as it is exempt from
the de-quantization and dimension expansion tasks. Second, we employ another
end-to-end codec that properly performs the dimension reduction and
quantization task in its bottleneck. This end-to-end codec will perform coarse
reconstruction from its low-bitrate code, but high fidelity is not guaranteed.
Finally, we employ a latent diffusion model that bridges the gap between the
two feature representations. By conditioning the diffusion model with the
lower-dimensional quantized code from the end-to-end codec, we expect this
model to perform generative de-quantization and upsampling tasks.

We opt for the diffusion model among other generative models because 1)
diffusion models have been successful in generating natural-sounding speech
and audio [19], from an extensive range of conditions; 2) Diffusion models
provide a way to generate the entire feature map altogether and refine it
iteratively. In contrast to the autoregressive models, diffusion models look
at a large condition space, which significantly increases the quality upper
bound. Specifically, we demonstrate that a latent diffusion (LD) model [20] is
suitable in this problem setup, because running on the latent space offloads
the diffusion model’s potential burden to reconstruct raw waveforms: Instead,
the LD model focuses on generating latent feature vectors. We explore the
trade-off between quality and efficiency in choosing a diffusion space. As
diffusion models are prone to generate over-smooth speech and hallucinate
content, we introduce a new sampling technique that adds stronger prior to the
conditional generation. We notice a concurrent work that explores high-
fidelity audio generation from the speech codec by multi-band diffusion [21].
In comparison, our model focuses on speech generation and achieves better
quality at both low- and high-bitrate cases with only one latent diffusion
model, thus more efficient with respect to the model design. We call our model
the latent diffusion codec (LaDiffCodec).

##  2 Generative De-quantization with Latent Diffusion

Fig. 1 provides an overview of the proposed LaDiffCodec. It consists of the
latent diffusion process in the middle, which converts the quantized code
generated from the discrete coding module (on the right) into the continuous
code (on the left).

###  2.1 Latent Diffusion

Diffusion models are generative models characterized by two Markov processes:
diffusion and reverse processes. The diffusion process,
$q(\bm{x}_{1:T}|\bm{x}_{0})=\prod^{T}_{t=1}q(\bm{x}_{t}|\bm{x}_{t-1})$,
corrupts clean data point $\bm{x}_{0}$ by gradually adding Gaussian noise,
until it reaches a random variable $\bm{x}_{T}$ that is close to the standard
normal distribution. Hence,
$q(\bm{x}_{t}|\bm{x}_{t-1})\sim\mathcal{N}(\sqrt{1-\beta_{t}}\bm{x}_{t-1},\beta_{t}I)$,
where $\beta_{t}$ is the pre-defined noise schedule
($0<\beta_{0}<...<\beta_{T}<1$). With reparameterization, sampling a step from
the diffusion process can be acquired by,
$\mathcal{F}:\bm{x}_{0}\mapsto\bm{x}_{t}$,

$\mathcal{F}(\bm{x}_{0},t)=\sqrt{\bar{\alpha}_{t}}\bm{x}_{0}+\sqrt{1-\bar{\alpha}_{t}}\bm{\epsilon},$
(1)

where $\bm{\epsilon}\sim\mathcal{N}(0,1)$ and
$\bar{\alpha}_{t}=\prod_{i=0}^{t}(1-\beta_{i})$.

The learned reverse process
$p_{\theta}(\bm{x}_{0:T})=p(\bm{x}_{T})\prod^{T}_{t=1}p_{\theta}(\bm{x}_{t-1}|\bm{x}_{t})$
is commonly represented by a parametric function, e.g., a neural network,
which predicts noise $\bm{\epsilon}_{t}$ at step $t$ to denoise the data point
$\bm{x}_{t}$. This process can be conditioned by various types of auxiliary
information.

While diffusion models succeed on many data generation tasks, the high
computational complexity limits their accessibility. Meanwhile, the models are
prone to spend excessive amounts of resources on modeling imperceptible
details, especially on the multi-modality tasks [20]. As a solution, latent
diffusion models propose to operate in the latent space $\bm{z}$ that is
usually learned by a pre-trained autoencoder

$\hat{\bm{x}}_{0}\leftarrow
f_{\text{dec}}(\bm{z}_{0}),\quad\bm{z}_{0}\leftarrow
f_{\text{enc}}(\bm{x}_{0}),$ (2)

assuming this space to be computationally preferable and perceptually
equivalent to the data domain.

For example, in LD, the diffusion process is defined in the latent space:
$q(\bm{z}_{1:T}|\bm{z}_{0})=\prod^{T}_{t=1}q(\bm{z}_{t}|\bm{z}_{t-1})$.

###  2.2 Diffusion-Based De-Quantization

LaDiffCodec maps the two latent spaces, the low-dimensional discrete code
$\mathbb{H}$ and the high-dimensional continuous feature $\mathbb{Z}$. The
restorative nature of this mapping requires conditional generation. In this
section, we describe the three components of LaDiffCodec: discrete coding,
continuous coding, and conditional diffusion sampling.

Discrete coding: Our discrete coding module $g(\cdot)$ is an autoencoder-type
codec that learns the discrete code space $\mathbb{H}$ using its encoder
component, $g_{\text{enc}}:\mathbb{R}^{N}\rightarrow\mathbb{H}^{D}$. It
provides the discretized features $\bm{h}\in\mathbb{H}^{D}$, which serve as
the transmission bitstream in the ordinary codec usage. Ideally, we want the
quantized speech tokens $\bm{h}$ to contain enough information for faithful
speech reconstruction. Therefore we opt for state-of-the-art autoencoder
codecs, which are designed to better capture coding features. Existing codec
relies solely on the decoder function
$g_{\text{dec}}:\mathbb{H}^{D}\rightarrow\mathbb{R}^{N}$, which could be the
performance bottleneck when $\mathbb{H}^{D}$ is low dimensional and discrete.
In LaDiffCodec, we repurpose this discrete code $\bm{h}$ to condition the
reverse diffusion sampling process. We employ Encodec [6] as the backbone of
this discrete coding module.

Continuous coding: To de-quantize the discrete token $\bm{h}$ into a
continuous feature vector $\bm{z}$, LaDiffCodec utilizes an LD model defined
in the continuous space $\mathbb{Z}$. To this end, we build another EnCodec-
like continuous autoencoder, whose encoder maps the raw signal space
$\mathbb{X}$ into the feature space $\mathbb{Z}$, i.e.,
$f_{\text{enc}}:\mathbb{X}^{N}\rightarrow\mathbb{Z}^{L}$, followed by a
decoder that maps it back to the signal domain:
$f_{\text{dec}}:\mathbb{Z}^{L}\rightarrow\mathbb{X}^{N}$. There is a trade-off
in this continuous latent space. One may want the latent dimension $L$ to be
as large as possible to keep its high expressiveness. However, with a
diffusion model defined in this space, high dimensionality entails a long
sampling time, causing reduced efficiency. Moreover, the gap between the high-
dimensional continuous space and a low-dimensional discrete space has to be
filled with upsampling layers, which brings additional artifacts. In our
experiments, we investigate a few options of $L$ to minimize the tradeoff.

Conditional Latent Diffusion: A diffusion model built on $\mathbb{Z}$
gradually adds noise $\bm{\epsilon}_{t}$ to $\bm{z}_{t}$ in its diffusion
process. It employs a conditioned neural network model, which estimates
$\bm{\epsilon}_{t}$ with the reweighted training objective for training the
denoising (reverse) process [22],

$\mathbb{E}_{\bm{z}_{0},t,\bm{h}}(||\bm{\epsilon}_{t}-\bm{\epsilon}_{\theta}(\bm{z}_{t},t,\bm{h})||)$
(3)

where $\bm{\epsilon}_{\theta}(\bm{z}_{t},t,\bm{h})$ is parameterized by a
neural network with weights $\theta$. It predicts the noise in the generation
process. Hence, the loss is its expected difference from $\bm{\epsilon}_{t}$.
The quantized features $\bm{h}$ condition both training and sampling stages to
steer the generation.

###  2.3 Midway-Infilling

The original sampling algorithm proposed in denoising diffusion probabilistic
models (DDPM) [22] iteratively removes predicted noise
$\bm{\epsilon}_{\theta}$ from the noisy data samples,
$\mathcal{G}\colon\bm{x}_{t}\mapsto\bm{x}_{t-1}$,

$\mathcal{G}(\bm{x}_{t},t,\bm{h})\\!=\\!\frac{1}{\sqrt{1\\!-\\!\beta_{t}}}\\!\left(\bm{x}_{t}\\!-\\!\frac{\beta_{t}}{\sqrt{1-\bar{\alpha}_{t}}}\bm{\epsilon}_{\theta}\big{(}\bm{x}_{t},\sqrt{\bar{\alpha}_{t}},\bm{h}\big{)}\right)+\sqrt{\beta_{t}}\bm{n},$

from $\bm{x}_{T}$ to $\bm{x}_{0}$, where $\bm{n}$ is a Gaussian noise. $T$
usually equals the number of time steps used for training, e.g., 1,000. DDPM
sampling can be tedious, given the large number of sampling steps. Denoising
diffusion implicit models’ (DDIM) sampling method [23] significantly reduces
the sampling steps. However, in our task, both DDPM and DDIM are prone to
generate overly smooth samples at very low bitrates, i.e., 1 and 1.5 kbps,
with some hallucination effects, such as missing or replaced phonemes.

We believe the smoothing effect is caused by excessive noise reduction and
insufficient assistance from the condition. Therefore, we propose a new
sampling method, midway-infilling, which improves the sampling quality and
efficiency in two folds. 1) It starts sampling from a mid-point step $\tau<T$
rather than from $\bm{x}_{T}$, a random noise space, which reduces sampling
steps by 10 to 20 times without sacrificing the sampling quality; 2) it
implements a separate conditioning branch to impose stronger conditioning
during sampling.

Midway-infilling is inspired by the infilling algorithm proposed in [24].
Infilling aims to condition and steer the sampling steps on the unconditional
diffusion model. In the original infilling process, an occluded sample
$\bm{s}_{0}$ is provided. The diffusion process runs on $\bm{s}_{0}$
(infilling branch) to meet the time step of reversed sampling branch
$\bm{x}_{t}$ (sampling branch), i.e., $\bm{s}_{t}=\mathcal{F}(\bm{s}_{0},t)$.
$\bm{x}_{t}$ and $\bm{s}_{t}$ are then interpolated with a certain ratio at
each step. Akin to infilling, the proposed midway-infilling has two branches.
The difference is, instead of providing $\bm{s}_{0}$, we use condition
$\bm{h}$ or its upsampled version to approximate $\bm{s}_{\tau}$, which is a
midway variable of the infilling branch’s Markov chain path. Accordingly, our
infilling branches can run the reverse process from step $\tau$ to 0, as
described in Algorithm 1.

Input: Conditioning vector $\bm{h}$, midway step $\tau$, interpolation ratio
$\gamma$, sampling function $\mathcal{G}(\cdot)$

$\bm{s}_{\tau}\leftarrow\bm{h},\quad\bm{x}_{\tau}\sim\mathcal{N}(0,1),\quad\bm{x}_{\tau}=(1-\gamma)\bm{x}_{\tau}+\gamma\bm{s}_{\tau}$

for  _$t=\tau\ldots 1$_ do

$\bm{s}_{t-1}=\mathcal{G}(\bm{s}_{t},t,\bm{h})$ – Infilling branch

$\bm{x}_{t-1}=\mathcal{G}(\bm{x}_{t},t,\bm{h})$ – Sampling branch

$\bm{x}_{t-1}=(1-\gamma)\bm{x}_{t-1}+\gamma\bm{s}_{t-1}$

end for

return $\bm{x}_{0}$

Algorithm 1 Midway-Infilling

##  3 Experiment Setup

(a) Reference

(b<EMAIL_ADDRESS>
(c) EnCodec@3kbps

(d<EMAIL_ADDRESS>
(e) LaDiffCodec@3kbps

Fig. 2: Spectrograms of the reference speech, and the coded versions by
EnCodec and LaDiffcodec at 1.5kbps and 3kbps. The audio samples of these
spectral are available at the sample page. White blocks point out the example
areas where EnCodec shows aliasing artifacts.

###  3.1 Model Design and Hyperparameter Setup

Fig. 3: Average MUSHRA scores and their confident intervals of LaDiffCodec and
baseline codec systems at 1.5kbps and 3kbps.

(a) 1kbps

(b) 1.5kbps

(c) 3kbps

Fig. 4: PESQ scores of LaDiffCodec using different midway-infilling
hyperparameters. X-axis denotes the mask ratios $\gamma$ from 0 to 1. Y-axis
denotes the midway timestep $\tau$.

In the forward process, we use $T\\!=\\!1000$ steps and set noise schedule
linearly from $\beta_{1}\\!=\\!0.0001$ to $\beta_{T}\\!=\\!0.02$. The reversed
diffusion process is parameterized with a U-Net-based backbone, similar to
[20, 25]. The U-Net is built with ResNet blocks, each containing three
convolution layers and one four-head self-attention layer. The model comprises
five encoder blocks, one middle block, and five decoder blocks. The channel
dimensions of encoder blocks are $[128,256,256,512,512]$. Decoder blocks have
the reversed dimensions of encoder blocks, and the channel dimension of the
middle block is 512. While AudioLDM [25] uses FiLM conditioning, we condition
quantized tokens to the diffusion model by stacking the tokens $\bm{h}$ and
model input $\bm{z}_{t}$ together, as we found it more effective than FiLM in
this task. When the dimensionality of the continuous and discrete code spaces
do not match, i.e., $D<L$, $\bm{h}$ is firstly upsampled with transposed
convolutional layers. We scale each frame of the upsampled tokens to $[-1,1]$
before conditioning them on the diffusion.

We use the proposed midway-infilling method for sampling, where the
hyperparameters are shared in all bitrate cases to keep the usage simple. We
set the midway step $\tau=100$ and $\gamma=0.3$. Once latent diffusion
sampling is finished, the acquired continuous sample $\bm{z}_{0}$ is fed to
the decoder, which maps it to the time-domain signal. Since $\tau=100$ is
small, it takes $\sim$5.65 seconds to generate a 3.2-second sample.

We retrain the non-streamable version of EnCodec with the 16 kHz Librispeech
dataset [26] as the discrete autoencoder. Its encoder and decoder have SEANet
[27] as the backbone. The encoder downsamples input through four convolution
layers with strides of size 2, 4, 5, and 8, respectively. Using transposed
convolution, the decoder upsamples the latent space in the reverse order. In
addition, we build a continuous autoencoder akin to EnCodec. We keep only one
downsampling layer with a stride size of 8 to achieve higher dimension and,
consequently, expressiveness.

###  3.2 Data and training

All experiments are conducted on the Libirspeech dataset, train-
clean-100 fold for training, and dev-clean for testing. Model training runs on
the 3.2s sequences. The three components, i.e., discrete autoencoder (16khz
EnCodec), continuous autoencoder, and the latent diffusion model, are trained
separately. When training the latent diffusion model, both autoencoders are
frozen. Our diffusion models are bitrate-specific. For example, LaDiffCodec at
1.5 kbps uses EnCodec’s 1.5 kbps tokens as its condition. We use Adam
optimizer for all the training tasks, with a batch size of 20 and a learning
rate 5e-5. It takes 6 hours to converge the autoencoder training and three
days for the latent diffusion model on one NVIDIA A100 GPU.

###  3.3 Evaluation and Ablation study

Our MUSHRA-like subjective test [28] compares LaDiffCodec with 16 kHz EnCodec
at two bitrates, 1.5kbps and 3kbps. Sequences from a 16 kHz DAC at 3kbps are
also included for comparison. 13 audio experts participated in and rated ten
gender-balanced samples of 3 seconds.

All the ablation studies are evaluated with PESQ [29] on randomly picked 50
samples from the test set. We notice that PESQ doesn’t reflect the real
perceptual preference for different coding systems. However, as for the
comparison of sequences generated from the same kind of system, they tend to
exhibit a steady trend.

##  4 Experimental Results

###  4.1 Comparison with other codec

Figure 3 shows MUSHRA scores of different codec systems. We see LaDiffCodec
surpasses EnCodec and DAC at both bitrates. Particularly, LaDiffCodec’s 1.5
kbps samples are preferred by the subjects to the 3 kbps samples of the other
codecs.

We believe that LaDiffCodec has two main traits that contribute to its
superior performance. Firstly, it recovers coding artifacts. Lossy compression
can cause various speech alterations and degradation. We observe that at the
1.5kbps, EnCodec starts to lose intelligibility, because some phonemes are not
recovered precisely. DAC exhibits a similar artifact at 1.5 kbps (using three
quantizer cookbooks). At 3kbps, while the intelligibility is better preserved,
the baseline codecs still produce artifacts such as subtle background noise
(DAC) or metallic and hissing sound (EnCodec). LaDiffCodec can fix severe
quantization artifacts by generating variables in the continuous latent space.
The fact that the LD process works in the pre-learned latent distribution
narrows its synthesis process to a more straightforward problem. In addition,
the well-trained continuous latent space leads to higher speech reconstruction
quality, eliminating the non-speech artifacts and distortion.

Secondly, it generates more natural-sounding speech. Fig. 2 presents
spectrograms of the reference and various decoded versions. These low-bitrate
EnCodec results suffer from losing high-frequency information due to the
reduced expression space. Waveform coding, in particular, often experiences
aliasing artifacts, as shown in 2(b) and 2(c). In these spectrograms, the
high-frequency area over $\sim$4 kHz shows a mirrored reflection of the lower
frequency harmonics. These high-frequency aliasing effects can add unnatural
artifacts to the reconstruction. In contrast, LaDiffCodec makes up some high-
frequency energy and eludes the aliasing effect. As a result, it produces a
more natural and pleasant sound. As we use DAC’s public 16khz checkpoint,
which is not re-trained on the Librispeech dataset, it appears less performed
than the re-trained EnCodec.

###  4.2 Ablation Studies

Hyperparameters of Midway-Infilling: This ablation explores different settings
of midway-infilling hyperparameters. Smaller $\gamma$ correlates to less
involvement of the condition branch $[\bm{s}_{\tau},...,\bm{s}_{0}]$ during
sampling, while a large $\tau$ means the sampling process conducts more noise
reduction. When $\gamma=0$ and $\tau=1000$, it is equivalent to DDPM’s
original sampling method. With the correct set of hyperparameters, midway-
infilling gains higher PESQ than the original DDPM sampling. The leftmost
columns of each graph present the degrading performance by reducing DDPM
sampling steps, with no extra infilling branch involved (i.e., $\gamma=0$).
The rightmost columns, on the other hand, present the infilling branch’s sole
contribution to sampling. According to the PESQ score, the best quality is
obtained when sampling step $\tau$ is small and $\gamma$ is close to 0 or 1.
Our perceptual rating aligns with PESQ concerning $\tau$. However, a large
interpolation ratio $\gamma$ leads to a better phoneme-level reconstruction at
the cost of less naturalness. A sequence of speech samples regarding different
interpolation ratios can be found on our webpage.

Latent Dimensionality: Table 1 presents the PESQ scores of LaDiffCodec with
different latent space dimensions. The total downsampling rate is the product
of stride sizes. The first row shows results from the ordinary (non-latent)
diffusion model, which samples in the time domain with no continuous
autoencoder involved. All the experiments are made to run for the same amount
of time. In comparison, the LD models outperform the time-domain diffusion
method (stride$=1$), indicating that a reduced dimension and auxiliary feature
learning can facilitate diffusion modeling, especially in this speech coding
task. However, reduced dimension does not always lead to superior performance.
When more downsampling layers are added to the continuous autoencoder, the
latent space starts losing expression power, or becomes hard to model with the
diffusion process.

Strides | @1kbps | @1.5kbps | @3kbps
---|---|---|---
[1] |  1.18 $\pm$ 0.04 |  1.20 $\pm$ 0.04 |  1.77 $\pm$ 0.19
[8] | 1.81 $\pm$ 0.15 |  1.95 $\pm$ 0.15 | 2.23 $\pm$ 0.17
[4, 8] |  1.71 $\pm$ 0.71 | 2.19 $\pm$ 0.75 |  2.16 $\pm$ 0.69
[4, 5, 8] |  1.66 $\pm$ 0.11 |  1.71 $\pm$ 0.12 |  1.84 $\pm$ 0.10
[2, 4, 5, 8] |  1.49 $\pm$ 0.09 |  1.65 $\pm$ 0.13 |  1.71 $\pm$ 0.12

Table 1: Performance of diffusion models with different latent space
dimension. The arrays in the first column present the stride sizes of each
down-sampling layer in the continuous encoder.

##  5 Conclusion

This work proposed LaDiffCodec and demonstrated the effectiveness of
integrating waveform coding-based feature learning and latent diffusion model
for high-quality, low-bitrate speech coding. By mapping the low-dimensional
discrete speech token into high-dimensional continuous space using latent
diffusion, the codec released the burden of upsampling and de-quantization
from the decoder. It achieved improved speech quality with reduced artifact
and increased naturalness. While mainly focusing on the low-bitrate scenarios,
our work potentially sheds light on the high-fidelity codec-based generation.
Our models provides a solution that enables using fewer codebooks for
categorical generation, which reduces the task’s difficulties without
sacrificing output sound quality.

## References

  * [1] Z. Borsos, R. Marinier, D. Vincent, E. Kharitonov, O. Pietquin, M. Sharifi, D. Roblek, O. Teboul, Da. Grangier, M. Tagliasacchi, et al.,  “AudioLM: a language modeling approach to audio generation,”  IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 31, pp. 2523–2533, 2023.
  * [2] C. Wang, S. Chen, Y. Wu, Z. Zhang, L. Zhou, S. Liu, Z. Chen, Y. Liu, H. Wang, J. Li, et al.,  “Neural codec language models are zero-shot text to speech synthesizers,”  arXiv preprint arXiv:2301.02111, 2023.
  * [3] F. Kreuk, G. Synnaeve, A. Polyak, U. Singer, A. Défossez, J. Copet, D. Parikh, Y. Taigman, and Y. Adi,  “Audiogen: Textually guided audio generation,”  arXiv preprint arXiv:2209.15352, 2022.
  * [4] N. Zeghidour, A. Luebs, A. Omran, J. Skoglund, and M. Tagliasacchi,  “Soundstream: An end-to-end neural audio codec,”  IEEE/ACM Trans. Audio, Speech and Lang. Proc., vol. 30, pp. 495–507, jan 2022.
  * [5] R. Kumar, P. Seetharaman, A. Luebs, I. Kumar, and K. Kumar,  “High-fidelity audio compression with improved RVQGAN,”  arXiv preprint arXiv:2306.06546, 2023.
  * [6] A. Défossez, J. Copet, G. Synnaeve, and Y. Adi,  “High fidelity neural audio compression,”  arXiv preprint arXiv:2210.13438, 2022.
  * [7] K. Zhen, M. S. Lee, J. Sung, S. Beack, and M. Kim,  “Psychoacoustic calibration of loss functions for efficient end-to-end neural audio coding,”  IEEE Signal Processing Letters, vol. 27, pp. 2159–2163, 2020.
  * [8] D. Petermann, S. Beack, and M. Kim,  “HARP-Net: Hyper-autoencoded reconstruction propagation for scalable neural audio coding,”  in Proc. of the IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA), 2021.
  * [9] A. Siahkoohi, M. Chinen, T. Denton, W. Kleijn, and J. Skoglund,  “Ultra-low-bitrate speech coding with pretrained transformers,”  in Proc. Interspeech, 2022.
  * [10] H. Yang, K. Zhen, S. Beack, and M. Kim,  “Source-aware neural speech coding for noisy speech compression,”  in Proc. of the IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2021.
  * [11] X. Jiang, X. Peng, Y. Zhang, and Y. Lu,  “Disentangled feature learning for real-time neural speech coding,”  in ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023, pp. 1–5.
  * [12] A. Omran, N. Zeghidour, Z. Borsos, F. de Chaumont Quitry, M. Slaney, and M. Tagliasacchi,  “Disentangling speech from surroundings with neural embeddings,”  in ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2023, pp. 1–5.
  * [13] W. B. Kleijn, F. S. C. Lim, A. Luebs, J. Skoglund, F. Stimberg, Q. Wang, and T. C. Walters,  “WaveNet based low rate speech coding,”  in Proc. of the IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2018, pp. 676–680.
  * [14] Y. Li C. Garbacea, A. van den Oord,  “Low bit-rate speech coding with VQ-VAE and a WaveNet decoder,”  in Proc. of the IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2019.
  * [15] J.-M. Valin and J. Skoglund,  “A real-time wideband neural vocoder at 1.6 kb/s using LPCNet,”  in Proc. Interspeech, 2019.
  * [16] A. Polyak, Y. Adi, J. Copet, E. Kharitonov, K. Lakhotia, W.-N. Hsu, A. Mohamed, and E. Dupoux,  “Speech resynthesis from discrete disentangled self-supervised representations,”  arXiv preprint arXiv:2104.00355, 2021.
  * [17] T. Jenrungrot, M. Chinen, W. B. Kleijn, J. Skoglund, Z. Borsos, N. Zeghidour, and M. Tagliasacchi,  “LMCodec: A low bitrate speech codec with causal transformer models,”  in Proc. of the IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2023.
  * [18] H. Yang, W. Lim, and M. Kim,  “Neural feature predictor and discriminative residual coding for low-bitrate speech coding,”  in Proc. of the IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2023.
  * [19] S. Pascual, C. Bhattacharya, G.and Yeh, J. Pons, and J. Serrà,  “Full-band general audio synthesis with score-based diffusion,”  in ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023, pp. 1–5.
  * [20] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer,  “High-resolution image synthesis with latent diffusion models,”  in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 10684–10695.
  * [21] R. S. Roman, Y. Adi, A. Deleforge, R. Serizel, G. Synnaeve, and A. Défossez,  “From discrete tokens to high-fidelity audio using multi-band diffusion,” 2023.
  * [22] J. Ho, A. Jain, and P. Abbeel,  “Denoising diffusion probabilistic models,”  Advances in neural information processing systems, vol. 33, pp. 6840–6851, 2020.
  * [23] J. Song, C. Meng, and S. Ermon,  “Denoising diffusion implicit models,”  in International Conference on Learning Representations, 2021.
  * [24] G. Mittal, J. Engel, C. Hawthorne, and I. Simon,  “Symbolic music generation with diffusion models,”  arXiv preprint arXiv:2103.16091, 2021.
  * [25] H. Liu, Z. Chen, Y. Yuan, X. Mei, X. Liu, D. Mandic, W. Wang, and M. D. Plumbley,  “Audioldm: Text-to-audio generation with latent diffusion models,”  arXiv preprint arXiv:2301.12503, 2023.
  * [26] V. Panayotov, G. Chen, D. Povey, and S. Khudanpur,  “Librispeech: An ASR corpus based on public domain audio books,”  in Proc. of the IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2015, pp. 5206–5210.
  * [27] M. Tagliasacchi, Y. Li, K. Misiunas, and D. Roblek,  “Speech enhancement by online non-negative spectrogram decomposition in non-stationary noise environments,”  in Proc. Interspeech, 2012.
  * [28] ITU-R Recommendation BS 1534-3,  “Method for the subjective assessment of intermediate quality levels of coding systems (MUSHRA),” 2015.
  * [29] A. W. Rix, J. G. Beerends, M. P. Hollier, and A. P. Hekstra,  “Perceptual evaluation of speech quality (PESQ)-a new method for speech quality assessment of telephone networks and codecs,”  in Proc. of the IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2001, vol. 2, pp. 749–752.


11institutetext: King’s College London, United Kingdom
11email: {kholoud.alghamdi, miaojing.shi<EMAIL_ADDRESS>

# Learning to Recommend Items to Wikidata Editors

Kholoud AlGhamdi  Miaojing Shi  Elena Simperl

###### Abstract

Wikidata is an open knowledge graph built by a global community of volunteers.
As it advances in scale, it faces substantial challenges around editor
engagement. These challenges are in terms of both attracting new editors to
keep up with the sheer amount of work, and retaining existing editors.
Experience from other online communities and peer-production systems,
including Wikipedia, suggests that personalised recommendations could help,
especially newcomers, who are sometimes unsure about how to contribute best to
an ongoing effort. For this reason, we propose a recommender system
_WikidataRec_ for Wikidata items. The system uses a hybrid of content-based
and collaborative filtering techniques to rank items for editors relying on
both item features and item-editor previous interaction. A neural network,
named neural mixture of representations, is designed to learn fine weights for
the combination of item-based representations and optimize them with editor-
based representation by item-editor interaction. To facilitate further
research in this space, we also create two benchmark datasets, a general-
purpose one with $220,000$ editors responsible for $14$ million interactions
with $4$ million items, and a second one focusing on the contributions of more
than $8,000$ more active editors. We perform an offline evaluation of the
system on both datasets with promising results. Our code and datasets are
available at https://github.com/WikidataRec-developer/Wikidata_Recommender.

##  1 Introduction

Wikidata is an open knowledge graph built by a global community of volunteers
[40]. Since its launch in $2012$ it reached more than $90$ million items and
$24,000$ active editors111www.wikidata.org/wiki/Wikidata:Statistics/en. Manual
contributions are core to Wikidata [40]: editors add new content, keep it up-
to-date, model knowledge as graph items and properties, and decide on all
rules of content creation and management. However, as Wikidata advances in
scale, it faces substantial challenges around editor engagement [34].

Experience from other online communities and peer-production systems,
including Wikipedia, Quora and others, suggests that one way to improve
engagement is through recommendations [7, 9, 44]. This is especially true for
editors who are relatively new to Wikidata, who need to overcome so-called
‘legitimate peripheral participation’ (LPP) effects to continue to engage
[28]. According to LPP, newcomers are more likely to become members of a
community if they are provided with suggestions of (typically lower-risk)
tasks they could carry out to further the goals of the community [19].

At the moment, looking for relevant items to edit in Wikidata remains
challenging because of the sheer number of options available [34]. The
suggested and open tasks page
222www.wikidata.org/wiki/Wikidata:Contribute/Suggested_and_open_tasks gives a
useful but daunting overview of the various ways in which people could
contribute to Wikidata. Many task lists are automatically generated, without
taking into account aspects such as editor tenure, previous editing history or
interests. This lack of focus is also said to prevent editors from developing
reinforced editing habits, which increases the likelihood of dropouts [34].
Seasoned editors use tools such as
QuickStatements333https://quickstatements.toolforge.org/#/ and
Watchlist444www.wikidata.org/wiki/Help:Watchlist to organise their work, but
such tools become relevant only once editors have identified productive ways
to contribute.

For these reasons, we propose WikidataRec, a recommender system for Wikidata
items. The system uses content-based and collaborative filtering techniques to
rank items for editors relying on both item features and item-editor previous
interaction. Collaborative filtering is a representative approach to address
recommendation task with implicit feedback, which learns item-centric and
editor-centric edit representations using matrix factorization [17] from item-
editor interaction data. We adopt it in this work and further facilitate it
with additional information from item content and relations, where we learn
these representations from by sentence embedding model ELMo [26] and graph
embedding model TransR [20], respectively. To combine the multiple
representations, we introduce the neural mixture of representations (NMoR), a
neural network inspired by mixture of experts [13] that produces fine weights
for different item-based representations. It is optimized over the editor-
based representation to rank items to editors. The proposed WikidataRec
demonstrates a large capacity of leveraging items’ contents, relations, and
edit history between editors and items into the recommender.

To facilitate further research in this space, we also create two benchmark
datasets, a general-purpose one with $220,000$ editors responsible for $14$
million interactions with $4$ million items, and a more focused one with
contributions of more than $8,000$ active editors with more than $200$ edits
each. We evaluate the system on these datasets against several baselines and
analyse the impact of different features and levels of data sparsity on
performance. The results are promising, though challenges remain because of
unbalanced participation, sparse interaction data and little explicit
information about editors’ interests and feedback.

##  2 Background and Related Work

###  2.1 Wikidata Data Model

Wikidata consists of structured records stored in form of entities, where each
entity is allocated a separate page and described by a set of terms and
statements [40]. There are two types of entities, items and properties,
situated in different namespaces. The namespaces are identified by URIs using
numbers and letters, with the letter $Q$ relating to items and the letter $P$
to properties. Although there are other namespaces in Wikidata, we focus only
on these two, as they are most relevant for our recommender system. Items and
property pages have a similar interface, starting with labels, descriptions
and aliases, which are language-specific, and then moving on to statements,
which are language-agnostic. Each statement contains a set of triplet edges
(head, relation, tail), capturing the different types of relations between
entities in the world, e.g., London $\overrightarrow{\text{\scriptsize{capital
of}}}$ United Kingdom. Pages also include sitelinks connecting to Wikipedia
articles or other Wikimedia projects [24]. Each page is indexed by a unique
identifier. Figure 1 shows an example of a Wikidata item page.

Figure 1: The structure of an Wikidata item.
Source: [42]

###  2.2 Editors and Editing in Wikidata

Anyone can edit Wikidata. The literature tends to distinguish between two
types of contributions: manual ones, carried out by registered or anonymous
editors, and bots. Bots are used for repetitive editing tasks, but do not
contribute to discussions, modeling decisions, or rules for content creation,
and management [24]. Human editors may get involved in any type of activities,
though some require specific access rights or ontology engineering skills [28,
29]. Furthermore, there are two types of editing tasks which are higher-risk
tasks and lower-risk ones. Edits are higher-risk if they affect a larger share
of the graph, e.g. property editing; hence not all editors may create
properties in Wikidata. Lower-risk edits are, for example adding/changing
labels, descriptions, etc. Formally, the Wikidata editing process starts when
an editor logs into the platform and contributes by inserting data on an item
page using the basic editing interface (See Figure 1). Every action performed
by an editor is recorded on the so-called revision history page. In our
current work, we focus on item edits rather than properties or other types of
contributions. Our aim is to establish whether recommendations are technically
feasible with the available data. Previous studies into the Wikidata community
suggest that item edits are popular with less experienced editors, who are the
main audience for personalised recommendations. Properties or other sorts of
works are normally dealt by seasoned and active editors [28, 29].

###  2.3 Recommending Tasks to Communities and Crowds Online

There are a lot of works proposing the use of personalized recommendations in
online communities and peer-production systems [7, 9, 44]. Recommendations aim
to make work more effective and increase retention, by matching open tasks to
people’s skills and interests [5]. As noted in the introduction, they can also
help new members of a community find their way and contribute [9, 28].

In Wikipedia, a first recommender system, SuggestBot, was proposed in [5]. It
used article titles, links, and co-editing patterns as main features to
recommend articles to editors. A more recent recommender system by [23]
represents Wikipedia articles using Graph Convolutional Networks. Both works
aim to recommend items (Wikipedia articles) to editors based on features of
items and editors. There is no explicit feedback that would confirm an
editor’s interest in an article. Equally, there is very little information
about editors beyond what they have edited so far [23]. Our system is similar
in spirit to [5] and [23], though our neural approach incorporates additional
item-based features and a mix of representations.

In community question-answering (CQA), several papers develop recommender
systems to route users to questions they might be interested in answering,
hence improving their engagement on the platform and reducing question
answering time [7, 21, 37, 38]. [7, 37] model recommendation as a
classification problem, using different machine learning techniques. They
implement a hybrid approach with content and collaborative knowledge to
address the sparsity problem. [21, 38] apply graph embedding techniques to
tackle the same problem. Similarly, we mix item and relational representations
with collaborative filtering to solve in a novel application context.

Another related area is online crowdsourcing, where the aim is to allocate
tasks published by a requester to an online crowd. [18, 33] employ a
probabilistic matrix factorization (PMF) to recommend suitable tasks to
crowdworkers based on their previous activities, performance, and preferences.
They handle the cold start problem by utilizing predefined categories (e.g.,
sentiment analysis, translation, image labeling) as additional features to
improve the recommendation accuracy.

###  2.4 Evaluating Recommender Systems

_Evaluation Methodologies._ To evaluate the performance of a recommender
system, precision@k and recall@k are the most common metrics for top-k
recommendations task with implicit data [36]. Precision@k measures the
relevance of the recommended items list, whereas recall@k gives insight about
how well the recommender is able to recall all the items the editor has rated
positively (or interacted with) in the test set [36]. Moreover, there is a set
of metrics that cares about where the relevant item appears in the recommended
list. For instance, mean average precision(MAP) and mean average recall (MAR)
are popular metric for search engines and are applied to the recommendation
task [2]. In these metrics, the relevant items are required to be placed as
high on the recommendation list as possible [36]. There is also a family of
metrics such as catalog coverage and diversity that pays special attention to
editor experience [2]. The catalog coverage evaluates whether the recommender
algorithm can generate recommendations with a high proportion of items, and
the diversity evaluates how diverse the set of proposed recommendations within
a single recommended list [2].

_Recommender Datasets._ Recommender datasets are available for items such as
movies (using MovieLens)555https://grouplens.org/datasets/movielens/ or
products (using the Amazon dataset)666https://jmcauley.ucsd.edu/data/amazon/.
In other tasks, such benchmarks are not widely available. The systems
discussed in Section 2.3 are mostly evaluated on custom-built datasets derived
from the platforms’ activity logs. This is also the case with our system,
which to the best of our knowledge is the first of its kind for Wikidata. To
encourage further research in this space, we hence provide two new datasets
for the community to reuse, specified in Section 3.

##  3 Wikidata Recommender Datasets

###  3.1 From Wikidata Dumps to Relational Tables

Wikidata dumps are readily available as JSON, RDF and
XML777https://dumps.wikimedia.org/wikidatawiki/. In our work we use the JSON
and XML ones from July 1, 2019. The JSON dump contains all Wikidata pages
without their edit history, which is available as XML. We parse the JSON data
to extract all Wikidata items along with their corresponding identifiers,
labels, descriptions, and statements. For the text data (i.e. label and
description), we fetch only the English language version and discard data for
other languages, because English is the best covered language in Wikidata
[15]. Also, we ignore items’ metadata about aliases and sitelinks as we do not
use them as features in our system. We transform the parsed data into CSV and
import it into a PostgreSQL database (as a _Wikidata-items-content_ table);
further processed the raw data containing the items’ statements (i.e. claims)
in similar way (as a _Wikidata-items-relations_ PostgreSQL table). Next, we
work on the XML dump to extract editing information for each individual
editor. We only focus on edits that are performed on an item’s namespace and
ignore all other non-item-related actions, as we do not model this information
in our system. For each edit, the following information is kept: who carried
out the edit, the item being edited, the timestamps of the edit, and the
comment indicating the specific action executed by the editor on the item.
They are stored in PostgreSQL (as a _Wikidata-editors-edits_ table).

###  3.2 Sampling and Cleaning

We randomly sample $14$ million editing activities performed by $221,353$
editors who are human registered (i.e. non-bots). We sort these activities in
ascending order based on timestamps. We refer to this as the Wikidata-14M
dataset. We also want to test the system on a denser dataset that contains the
editors who interacted with a larger number of items in their editing history
and the items that received edits by a high number of editors. The
Wikidata-14M dataset is filtered by keeping editors who edited at least $200$
unique items during their tenure and items that have been edited by at least
$5$ different editors. We refer to it as the Wikidata-Active-Editors dataset.
We then removed some outliers from both datasets - $2.14\%$ of editors in
Wikidata-14M and $5\%$ in Wikidata-Active-Editors edited a very large number
of items in a very short time relative to the size of their contributions.
This is considered as the case for institutional accounts that publish their
data via Wikidata [39]. We remove those accounts and their edits from the
datasets, as they are not the main target editors for recommendations. We use
both datasets to evaluate our system. The first one Wikidata-14M depicts the
actual diversity of editing activities in Wikidata with a mix of active and
more casual editors. The second one Wikidata-Active-Editors focuses on more
active editors and items - we use it to understand the effects of data
sparsity on recommender performance.

Table 1: Number of items, editors and interactions in the two Datasets. Dataset | # editors | # items | # interactions
---|---|---|---
Wikidata-14M | 221,353 | 4,881,720 | 14,045,523
Wikidata-Active-Editors | 8,024 | 381,784 | 3,272,086
Table 2: Statistics of the distribution of editing activities among editors and items. | Wikidata-14M | Wikidata-Active-Editors
---|---|---
Median | Mean | Std | Median | Mean | Std
# items/editor | 1 | 72 | 534 | 900 | 1,244 | 2,197
# edits/editor | 2 | 143 | 1,413 | 1,045 | 4,666 | 32,338
# editors/item | 1 | 2 | 3.7 | _ | _ | _

((a))

((b))

((c))

Figure 2: Editing Activities Distribution in Wikidata-14M (a and b) and in
Wikidata-Active-Editors (c).

###  3.3 Datasets Description

Table 1 reports some key descriptive statistics of the two datasets, _i.e._
number of editors, items and interactions. Figure 2(a), 2(b) and 2(c)
characterise the distribution of edits in the two datasets. For Wikidata-14M,
Figure 2(a) shows the number of items edited by each editor - most editors
edited only a few items (¡$1,000$); while fewer than $100$ editors (out of
$221,353$) edited more than $1,500$ items. Table 2 illustrates this skewed
distribution: the median values are much smaller than the mean values, both
per editor and per item. It is in line with observations from previous studies
in Wikidata and other large-scale community-driven knowledge engineering
initiatives [16, 29, 34], which attest that a small core of contributors are
responsible for a majority of the work. This also gives us an opportunity to
increase retention in the long tail of editors (novice editors mostly) through
recommendations. We also examine items that are edited by multiple editors,
which is useful in collaborative filtering: Figure 2(b) shows that most items
have been edited only by small number of editors; the highest number of
editors per item is $20$. This implies the issue of cold-start for new items,
we address it by mining item content and relation information for
recommendation (see Section 4).

We observe similar, though less pronounced effects for Wikidata-Active-Editors
(see Figure 2(c) and Table 2): the interaction data between editors and items
are denser, e.g. the mean and median of the number of items per editor is
$1,244$ and $900$, respectively, showing a less skewed distribution. Notice we
do not have the corresponding Figure 2(b) (Editors vs. Items) for Wikidata-
Active-Editors, because all active editors are filtered and selected, making
this statistic no longer meaningful.

##  4 Wikidata Recommender System

###  4.1 Problem Statement

Our problem is defined as follows: given a set of $N$ editors, a set of $M$
items, and an interaction matrix $A^{N\times M}=\\{a_{ij}\\}$, where each
matrix entry $a_{ij}$ is either 1 or 0 indicating whether editor $i$ has
edited item $j$. The task is learning to estimate the preference scores of
editors to items so as to rank and recommend new items to editors. Solving
this problem is not straightforward: as discussed in Section 3, there exists a
high number of items but a low number of interactions between items and
editors; many items have only a few edits. Standard approaches using
collaborative filtering [35] rely primarily on the item-editor interaction
data and do not perform well in our scenario. Intuitively, there are two ways
to improve it: by including more information from either editors (e.g. their
interests or activities) or items (e.g. their content or relations). In this
paper we focus on the latter, as for the former, there is very little
descriptive information available for Wikidata editors, a known issue also
present in Wikipedia [5, 23]. Of course one could try to collect additional
information about editors from their activities in Wikidata discussions or
contributions to other WikiProjects. This however is not the scope of this
paper as we decide to steer away from it to avoid the potential ethical
implications.

###  4.2 Approach

We introduce a Wikidata recommender system WikidataRec which is a hybrid model
combining item content and relation information with collaborative filtering
[17] by means of mixture of experts (MoE) [13] (Figure 3). The matrix
factorization (MF) decomposes the item-editor interaction matrix $A$ into
editor-centric and item-centric edit representations, denoted by $e_{i}$ and
$v_{j}$ for editor $i$ and item $j$ respectively. Item content representation,
denoted by $c_{j}$, is obtained by the sentence embedding model ELMo [26]; and
item relation representation, denoted by $r_{j}$, is obtained by the graph
embedding model TransR [20] building upon the Wikidata knowledge graph. To
combine multiple item-based representations, we introduce a neural mixture of
representations (NMoR) inspired by MoE, in which we utilize a soft-gating to
assign different weights to each representation. More specifically, the
network takes four inputs $e_{i}$, $v_{j}$, $c_{j}$ and $r_{j}$, where
$v_{j}$, $c_{j}$ and $r_{j}$ are added with weights ($w_{v}$, $w_{c}$,
$w_{r}$) produced from a soft-gating branch whose input is the concatenation
of $v_{j}$, $c_{j}$ and $r_{j}$. The merged item representation is fed into an
element-wise dot product layer together with $e_{i}$ to predict the preference
score $x_{ij}$. At the training stage, $x_{ij}$ is optimized with the cross-
entropy loss over the ground truth label in $A$; while at testing, it is used
to recommend items to the editors. The whole process can be written as,

${x_{ij}}=e_{i}(w_{v}\cdot v_{j}+w_{c}\cdot c_{j}+w_{r}\cdot
r_{j})^{\intercal}.$ (1)

Notice item and editor representations are used in the network as fixed
representations (i.e., they are not updated during the training process). They
are learned in advance by MF, ELMo, and TransR, respectively. We decide not to
jointly tune these representations for efficiency and simplicity reasons, as
it is not lightweight to integrate all the models in a whole. We will work on
joint learning in our future work.

Below we first specify the generation of $e_{i}$, $v_{j}$, $c_{j}$ and $r_{j}$
and then introduce the neural mixture of representations for $w_{v}$, $w_{c}$
and $w_{r}$.

###  4.3 Editor-centric and Item-centric Edit Representations

Editing activities are summarized in the interaction matrix $A$. Inspired by
[17], we use matrix factorization to decompose $A$ into the editor’s latent
representation matrix $E^{N\times Z}$ and the item’s latent representation
matrix $V^{M\times Z}$. This is achieved by approximating the target matrix
$A$ via the matrix product of two low-rank matrices $E$ and $V$:

$\widehat{A}=EV^{\intercal}$ (2)

Each row $e_{i}$ in $E$ can be seen as a latent representation of editor $i$
in terms of its edits. Similarly, each row $v_{j}$ of $V$ describes an item
$j$ with respect to its edits by different editors. Thus, the prediction
formula from Eqn(2) can also be written as:

$\widehat{a_{ij}}=e_{i}v_{j}^{\intercal}$ (3)

In order to single out $E$, $V$ from $A$, we utilize Bayesian Personalized
Ranking (BPR) [32] to optimize $\widehat{A}$ over it. If we look at Eqn(1),
$\widehat{a_{ij}}$($e_{i}v_{j}^{\intercal}$) is its first term.

Figure 3: The architecture of WikidataRec.

###  4.4 Item Content Representation

To generate item content representation $c_{j}$, we need an embedding model
that can learn both semantic and syntactic representations from text. There
are various embedding models that learn item representations from words,
sentences, or paragraphs. Among them, word embedding models such as Word2Vec
[22], GloVe [25] and FastText [14] are widely used for many recommendation
tasks [8], but they have limitations: the order of words is ignored, which
leads to the loss of the syntactic and semantic meaning in sentences [41]. We
therefore considered two state-of-the-art sentence embedding models instead,
specifically Embeddings from Language Models (ELMo) and Sentence-BERT. ELMo
[26] employs a bi-directional deep LSTM network which takes an entire sentence
as input, assigns representation to each word in the sentence, then takes the
average of the vectors of words to output a $Z$-dimensional vector for the
input sentence. Sentence-BERT [31] relies on a bi-directional Transformer
network [31] which learns from fix-sized semantically meaningful sentences. We
preferred ELMo to Sentence-BERT to generate $c_{j}$ because of the high
computational costs of the latter, particularly on large corpora [30]. Each
item in Wikidata has a very short description which naturally serves as input
sentence to ELMo. The implementation details are in Section 5.1.

###  4.5 Item Relational Representation

Item relational reprentations are built upon the Wikidata graph over items. We
model item relations as a directed labeled graph, where it has a set of nodes
to represent items, a set of edges (unweighted) to represent relations between
items, and labels to capture the type of relations (see Section 2.1). Over
different graph embedding models [43, 3, 20], we adopt the TransR [20], a
representative approach for heterogeneous graphs, to learn low-dimensional
($Z$ in this work) relational embedding of items, $r_{j}$. It builds entity
and relation embeddings by regarding a relation as a translation from head
entity to tail entity [20]. Different from other graph embedding methods[43,
3] which normally assume the embedding of entities and relations within the
same space, TransR represents entities and relations in distinct spaces and
projects entities from entity space to relation space via a projection matrix.
For this reason, it is selected to model the heterogeneity of items and their
relations in the Wikidata graph. Items with similar relations have similar
embedding in TransR. The implementation details are in Sec. 5.1.

###  4.6 Neural Mixture of Representations

Having $e_{i}$, $v_{j}$, $c_{j}$, and $r_{j}$ ready, we present our neural
mixture of representations (NMoR) to combine them for Wikidata recommendation.
$e_{i}$, $v_{j}$, $c_{j}$, and $r_{j}$ are fed into NMoR in parallel as fixed
representations, where they are not updated during training but learned in
advance (as noted in Section 4.2). Item representations $v_{j}$, $c_{j}$ and
$r_{j}$ are added with weights $w_{v}$, $w_{c}$ and $w_{r}$ generated by an
additional soft-gating branch (see Figure 3). Soft-gating is used to
distinguish from hard-gating in MoE. In hard-gating, weights are either 0 or 1
for each expert. Soft-gating can be seen as probabilities combining different
experts. The motivation for assigning different weights to different item-
based representations is to control their respective contribution to the final
prediction $x_{ij}$ (Eqn. 1). These weights are also $Z$-dimensional vectors
as with $v_{j}$, $c_{j}$, and $r_{j}$ such that they are tailored to every
dimension of feature representations. The soft-gating branch takes the
concatenation of $v_{j}$, $c_{j}$ and $r_{j}$ of size 1 $\times$ Z $\times$ 3
(corresponding to rows, columns, and channels of the tensor) as input; three
1D convolutional layers (kernel size 1 x 1, filters $1024$) followed by ReLu
are applied to process the tensor along its channels; the number of their
output channel remains $3$. The output of the last convolutional layer is thus
of size 1 $\times$ Z $\times$ 3, where each channel corresponds to the weight
for certain item-based representation ($w_{v}$, $w_{c}$ or $w_{r}$). At every
column of the tensor, it is softmaxed over channels such that the three values
at each column are summed to $1$. The merged item representation $w_{v}\cdot
v_{j}+w_{v}\cdot c_{j}+w_{r}\cdot r_{j}$ is passed through the element-wise
dot product layer with $e_{i}$, their similarity value $x_{ij}$ is hence
computed. To optimize $x_{ij}$, we adopt the binary cross-entropy loss,

$L=-(y_{ij}\log(\sigma(x_{ij}))+(1-y_{ij})\log(1-\sigma(x_{ij}))),$ (4)

where $x_{ij}$ is transformed into a probability using the sigmoid activation
function $\sigma(\cdot)$; $y_{ij}$ is the ground truth label of either 0 or 1
in $A$ indicating whether interaction between the item $j$ and editor $i$ has
been observed (_i.e._ positive instances) or not (_i.e._ negative instances).
Since $A$ is sparse, we randomly select negative instances to make its number
the same to that of positive instances. $x_{ij}$ indicates how likely item $j$
is relevant to editor $i$, we can use it to recommend items to editors at
testing.

Traditional weight tuning approach like line search is only suitable for
optimizing a limited number of weights, for instance, one parameter for each
item-based representation. The proposed NMoR optimizes weights in a neural
network which enables a refined weight steering on every dimension of the
item-based representation.

##  5 Experiments

###  5.1 Experimental Setup and Implementation Details

We run experiments on the two datasets Wikidata-14M and Wikidata-Active-
Editors introduced in Section 3. Both datasets are structured in an editor-
item-edits format. For each dataset, similar to [11, 12], we follow the hold-
out strategy to select $80\%$ items associated with each editor to constitute
the training set and use the remaining $20\%$ as the test set. Within the
training set, we set aside $10\%$ as validation data for hyper-parameter
tuning. In Wikidata-14M dataset, $25\%$ of the editors edited only between 2
to 6 different items during their editing tenure on the selected dates; these
are cold-start editors. Therefore, we exclude these editors from the training
set and include them only in the test set. We do this because the editing data
for this type of editor are rather too sparse to be informative during
training. However, we use this data in the test set, as we want to evaluate
the performance of WikidataRec on cold-start editors. $Z$ for feature
representations is set to $1024$.

_NMoR:_ To learn NMoR, the batch size is set to $32$. The model is learned
with Adam optimizer with a learning rate of $0.001$ for 100 epochs.

_ELMo:_ To generate the item content representations, we start from the
Wikidata-items-content table in our datasets, and then follow the following
steps using Python’s Spacy: 1) tokenize and normalize the content (label and
description) of each item; 2) remove stopwords, punctuation and single-letter
words; 3) use part-of-speech tagging and retained only nouns and adjectives.
We pass the resulting corpus as input to the ELMo model. ELMo is pre-trained
on the $1$ Billion Word Benchmark that contains about $800M$ tokens of
general-purpose data from WMT 2011[4]. This is very comprehensive such that
fine-tuning on Wikidata is no longer necessary. Following [10], we simply
forward each Wikidata item descriptor to ELMo to obtain the item-content
representation. Notice ELMo produces multi-scale output which we average them.

_TransR:_ To generate the item relational representations, we first build the
labeled directed graph $G(V,E,P)$ from the Wikidata-items-relations (Sec.
3.1), where $V$ signifies Wikidata items (the head of the triplet), $E$ inter-
item relations (the tail of the triplet), and $P$ relations’ types (the
relation of the triplet). We use the graph as input to train the TransR model.
The original triplets in the graph are the positive instances. We sample a few
of them to replace either their heads or tails with wrong components to create
negative triplets. TransR is trained with both positive and negative triplets
using binary cross-entropy loss. It is optimized with the SGD optimizer for
$10$ epochs; the batch size is $128$ and learning rate is $0.01$.

###  5.2 Evaluation Protocols

For evaluation, in order to simulate the practical recommendation scenario in
Wikidata, we follow the spirit in [23] to carry out the test: for each editor
$i$ and the item-editor interaction matrix $A$ in the test set, we 1) split
the items of editor $i$ into half-half where the editor-centric edit
representation $e_{i}$ along with the representations of item-centric edit,
content, and relations $v_{j}$, $c_{j}$ and $r_{j}$ are obtained by excluding
the second-half items in $A$ in our model; 2) exclude the editor $i$ from $A$
to generate the representations of item-centric edits, item-content and item
relations for the second half of items in our model; 3) randomly select $200$
negative items that were not edited by editor $i$, where we can obtain their
corresponding sub-matrix of $A$ and obtain their item-based representations;
4) feed ($e_{i}$, $v_{j}$, $c_{j}$ and $r_{j}$) obtained from step 1 to our
NMoR along with the item-based representations obtained from steps 2 and 3 to
obtain their ranking scores;

The evaluation criteria is to measure how well _WikidataRec_ ranks the correct
items against the negative items for a given editor. Therefore, we employ
_Precision@k_ , _Recall@k_ , and _MAR@k_ (see Section 2.4). We are also
interested in the diversity of the recommended items, as editing a different
set of items is a noted behavior in Wikidata community [34, 27]. Having a
diverse recommendation would allow the editors to discover a wide range of
items for the editing.

Table 3: Precision@k and Recall@k results comparison between our model and state-of-the-art for _Wikidata-14M_. | Precision @k | Recall @k
---|---|---
| 5 | 10 | 50 | 100 | 200
GMF | 0.096 | 0.071 | 0.135 | 0.183 | 0.274
BPR-MF | 0.050 | 0.043 | 0.093 | 0.135 | 0.190
eALS | 0.048 | 0.027 | 0.087 | 0.112 | 0.154
YouTube-DNN | 0.105 | 0.082 | 0.142 | 0.204 | 0.305
WikidataRec | 0.120 | 0.113 | 0.215 | 0.243 | 0.337

###  5.3 Results on Wikidata-14M

Comparison to state of the art. We compare _WikidataRec_ against the following
methods on _Wikidata-14M_ : _BPR-MF_ [32] is a representative collaborative
filtering model that uses matrix factorization (MF) as the underlying
predictor and is optimized with a pairwise ranking loss. It is suitable for
recommendation scenarios with no explicit editor feedback and personalised
ranked recommendation results. _eALS_ [12] is also a MF-based method that is
optimized with square loss. It treats all unobserved interactions as negative
instances and weights them non-uniformly by the item’s popularity. _GMF_ [11]
is a neural network-based collaborative filtering which implements MF with
cross-entropy loss. It embeds each item and editor in the network and computes
their element-wise dot product to predict the relevance score. _YouTube-DNN_
[6] is a neural recommender for YouTube videos, using deep candidate
generation and ranking networks. It uses a hybrid of users’ activities and
content information of users and items and directly learns their low
dimensional representations. In this paper, we adapt YouTube-DNN with our item
content and relational representations.

Table 3 shows the results: _WikidataRec_ achieves the best performance over
all with both accurate and diverse recommendations. First, it outperforms the
collaborative-filtering (CF) methods GMF, eALS and BPR-MF by a large margin.
CF methods only rely on the item-editor interactions data without taking into
account the information of items themselves, whereas adding additional item
content and relation information significantly improves the recommendation
performance in our model (see Table 4). Our model also beats Youtube DNN.
Youtube-DNN learns item’s content and relational embedding in a neural network
with random initialization while we employ additional state-of-the-art
embedding models (i.e., ELMo and TransR) to generate them separately which
yields superior performance.

Ablation Study.  We ablate different components of _WikidataRec_ on the
_Wikidata-14M_ to understand the effect of each one.

_Item contents and relations._ To justify the importance of item content and
relations in _WikidataRec_ , we start with the original collaborative
filtering (CF) technique, BPR-MF, and gradually add item content and
relational representations using NMoR. The results are in Table 4 which show
that values in every metric are increased by considering item content and
relations. Particularly, item content representations provide rich information
in terms of words and semantic meanings in items, which contribute more in
performance increase; item relational representations in contrast contribute
less. We can note each feature’s contributions from the amount of increase
when each of them is added. We suggest the reason as these relations are
learned from the Wikidata graph, which is unevenly and sparingly connected, as
discussed in [29].

_Neural mixture of representations._ To ablate the proposed NMoR, Table 4
further illustrates the results of _WikidataRec_ with and without using it.
For the latter, the item-based representations are added with no weights.
WikidataRec with NMoR performs significantly better than WikidataRec without
NMoR.

Table 4: Ablation study of _WikidataRec_ on _Wikidata-14M_. | Precision @k | Recall @k | MAR@k | Diversity
---|---|---|---|---
| 5 | 10 | 50 | 100 | 200 | 5 | 10 | 10
CF (BPR-MF) | 0.050 | 0.043 | 0.093 | 0.135 | 0.190 | 0.069 | 0.107 | 0.252
\+ item content |  0.104 |  0.086 |  0.189 |  0.199 |  0.287 |  0.108 |  0.170 |  0.502
\+ item relations |  0.120 |  0.113 |  0.215 |  0.243 |  0.337 |  0.133 |  0.224 |  0.567
WikidataRec w/o NMoR |  0.085 |  0.072 |  0.165 |  0.209 |  0.284 |  0.090 |  0.191 |  0.521
WikidataRec w/ NMoR |  0.120 |  0.113 |  0.215 |  0.243 |  0.337 |  0.133 |  0.224 |  0.567
Table 5: Ablation study of embedding methods for item content on _Wikidata-14M_.  | Embedding Model | Precision@5 | Recall@50 | Recall@100
---|---|---|---|---
| Word2Vec | 0.060 | 0.123 | 0.158
BPR-MF + | FastText | 0.029 | 0.069 | 0.115
ELMo | 0.104 | 0.189 | 0.199

_Embedding models for item content._ We compare the ELMo model with two text
embedding models, Word2Vec [22] and FastText [14] to generate item content
representations. Word2Vec and FastText are lightweight models which we train
them from scratch using our Wikidata. The results are in Table 5: ELMo
outperforms Word2Vec and FastText clearly. ELMo employs the deep bi-
directional Language Model (biLM), which provides a very rich representation
about the word tokens and captures the semantic and syntactic meaning of
words. This is not the case in the Word2Vec and FastText who utilize shallow
neural networks.

###  5.4 Results on Wikidata-Active-Editors

We ran another ablation study of our model on the second dataset, _Wikidata-
Active-Editors_ , introduced in Section 3.3. _Wikidata-Active-Editors_ is a
subset of _Wikidata-14M_ , focusing on active editors and frequently edited
items. The results are in Table 6, in which we note improved results on
_WikidataRec_ by adding item content and relations. In particular, content-
based information contributed most towards the performance of the model. On
the other hand, the contribution of relational information is less than that
of content information. Furthermore, the results show that _WikidataRec_ with
NMoR performs better than _WikidataRec_ without NMoR.

_Dataset sparsity._ We further study the sparsity of the dataset in terms of
its item-editor interaction data. Wikidata is very sparse in nature: a small
number of interactions between items and editors are observed. _Wikidata-
Active-Editors_ is denser than _Wikidata-14M_ , but its sparsity is still
$99.90\%$, which means only 1% of interactions are observed over all the
possible connections from every item to every editor in the dataset. To study
the influence of the dataset sparsity, we extract three sub-datasets from
_Wikidata-Active-Editors_ with different sparsity yet roughly the same size
(about 6000 editors and 10,000 items): Wikidata-sparse-1 with $99.81\%$
sparsity, Wikidata-sparse-2 with $99.68\%$ sparsity, and Wikidata-sparse-3
with $99.27\%$. The editors in each dataset have edited more than $200$
different items, and each item has edited by more than $16$ different editors.
Figure 4 show the Precision@5 and Recall@10 on the three subsets. It shows
that when the sparsity decreases (from $99.81\%$ to $99.27\%$, data gets
denser), our model performance increases. This again verifies the conclusion
in [1].

###  5.5 Discussion and analysis of results on both datasets

Comparing the results of the two datasets (Table 4 and Table 6), we summarize
that: 1) adding item content and relations with NMoR improves the performance
on both datasets; 2) WikidataRec performs better on the _Wikidata-Active-
Editors_ than on the _Wikidata-14M_ ; the former is with denser editing data,
so having less sparse data is likely to improve the results. This finding is
consistent with the conclusion of [1].

Table 6: Ablation study of _WikidataRec_ on _Wikidata-Active-Editor_. | Precision @k | Recall @k | MAR@k | Diversity
---|---|---|---|---
| 5 | 10 | 50 | 100 | 200 | 5 | 10 | 10
CF (BPR-MF) | 0.079 | 0.055 | 0.139 | 0.260 | 0.236 | 0.115 | 0.193 | 0.353
\+ item content |  0.143 |  0.109 |  0.253 |  0.302 |  0.349 |  0.157 |  0.251 |  0.565
\+ item relations |  0.164 |  0.131 |  0.289 |  0.342 |  0.391 |  0.179 |  0.297 |  0.596
WikidataRec w/o NMoR |  0.132 |  0.073 |  0.196 |  0.295 |  0.323 |  0.134 |  0.245 |  0.553
WikidataRec w/ NMoR |  0.164 |  0.131 |  0.289 |  0.342 |  0.394 |  0.179 |  0.297 |  0.596
Figure 4: WikidataRec with various sparsity levels.

##  6 Conclusion and Future Work

We present _WikidataRec_ , a hybrid recommender model that recommends Wikidata
items to editors based on their past editing activities. The work is motivated
by Wikidata’s quest for more (and more engaged) editors to keep up with a
knowledge graph of growing size and complexity. As the first work of its kind,
our focus is on establishing technical feasibility and providing a benchmark
for future recommendation research in Wikidata. We do so with solid system
implementation and benchmark datasets. Our model is informed by related
research in content-based and collaborative filtering in similar verticals,
which cannot rely upon explicit feedback for the recommendation task. It uses
state-of-the-art models for representation learning and operates by means of a
mixture of experts [13]. We employ ELMo [26] for items’ content-based
representations and TransR [20] for items’ relations-based representations.
The results, though far from perfect, are promising and could be considered a
baseline for future Wikidata recommender work.

Based on our experiments, we make three claims for the recommendation task in
Wikidata: 1) Collaborative filtering is not enough to recommend Wikidata
items, as editing data is very sparse; however, adding item content and
relational representations significantly enhances performance; 2) Not each
item representation contributes to the final predictions equally. We show how
to optimise the weights with NMoR; 3) While our model works better on the
denser datasets extracted from _Wikidata-Active-Editors_ data, showing that
there is room for improvement.

We plan to extend the work in several directions. First, we plan to run editor
studies, including interviews with more or less experienced editors to learn
about their current ways to choose what they work on and perhaps uncover how
existing technical affordances and interfaces influence such decisions.
Second, we aim to experiment with other recommender models, particularly
sequential learning and time-sensitive models that encode the temporal aspect
and distinguish between older and more recent editor interests, as well as
with ways to elicit more information about the editors in a responsible way.
One option here would be to recruit new editors for a study in which they
would explicitly consent to us collecting such information or would provide
additional information about their interests themselves. Also, since the data
of Wikidata can be represented as item-item relation graph and editor-item
graph, exploring graph neural networks (GNNs) for a recommendation would be an
interesting future direction. The advantages provided by the GNNs would
provide great potential to advance our recommendation task. Third, topic-
recommendation is another area that might interests the editors in Wikidata
more than item recommendations. We plan to investigate this area. In addition,
we are going to conducting an editor-centric evaluation would provide a space
to examine and compare the feasibility and effectiveness of the two algorithms
(topic vs. item recommendations).

## References

  * [1] Adomavicius, G., Zhang, J.: Impact of data characteristics on recommender systems performance. ACM Transactions on Management Information Systems (TMIS) (2012)
  * [2] Aggarwal, C.C., et al.: Recommender systems (2016)
  * [3] Bordes, A., Usunier, N., Garcia-Duran, A., Weston, J., Yakhnenko, O.: Translating embeddings for modeling multi-relational data. In: Neural Information Processing Systems (2013)
  * [4] Chelba, C., Mikolov, T., Schuster, M., Ge, Q., Brants, T., Koehn, P., Robinson, T.: One billion word benchmark for measuring progress in statistical language modeling. arXiv preprint arXiv:1312.3005 (2013)
  * [5] Cosley, D., Frankowski, D., Terveen, L., Riedl, J.: Suggestbot: using intelligent task routing to help people find work in wikipedia. In: International conference on Intelligent user interfaces (2007)
  * [6] Covington, P., Adams, J., Sargin, E.: Deep neural networks for youtube recommendations. In: ACM Conference on Recommender Systems (2016)
  * [7] Dror, G., Koren, Y., Maarek, Y., Szpektor, I.: I want to answer; who has a question?: Yahoo! answers recommender system. In: ACM SIGKDD international conference on Knowledge discovery and data mining (2011)
  * [8] Esmeli, R., Bader-El-Den, M., Abdullahi, H.: Using word2vec recommendation for improved purchase prediction. In: International Joint Conference on Neural Networks (2020)
  * [9] Freyne, J., Jacovi, M., Guy, I., Geyer, W.: Increasing engagement through early recommender intervention. In: ACM Conference on Recommender Systems (2009)
  * [10] Hassan, H.A.M., Sansonetti, G., Gasparetti, F., Micarelli, A., Beel, J.: Bert, elmo, use and infersent sentence encoders: The panacea for research-paper recommendation? In: RecSys (Late-Breaking Results) (2019)
  * [11] He, X., Liao, L., Zhang, H., Nie, L., Hu, X., Chua, T.S.: Neural collaborative filtering. In: The International Conference on World Wide Web (2017)
  * [12] He, X., Zhang, H., Kan, M.Y., Chua, T.S.: Fast matrix factorization for online recommendation with implicit feedback. In: International ACM SIGIR Conference on Research and Development in Information Retrieval (2016)
  * [13] Jacobs, R.A., Jordan, M.I., Nowlan, S.J., Hinton, G.E.: Adaptive mixtures of local experts. Neural Computation (1991)
  * [14] Joulin, A., Grave, E., Bojanowski, P., Mikolov, T.: Bag of tricks for efficient text classification. arXiv preprint arXiv:1607.01759 (2016)
  * [15] Kaffee, L.A., Piscopo, A., Vougiouklis, P., Simperl, E., Carr, L., Pintscher, L.: A glimpse into babel: an analysis of multilinguality in wikidata. In: Proceedings of the 13th International Symposium on Open Collaboration. pp. 1–5 (2017)
  * [16] Kanza, S., Stolz, A., Hepp, M., Simperl, E.: What does an ontology engineering community look like? a systematic analysis of the schema. org community. In: European Semantic Web Conference (2018)
  * [17] Koren, Y., Bell, R., Volinsky, C.: Matrix factorization techniques for recommender systems. Computer (2009)
  * [18] Kurup, A.R., Sajeev, G.: Task recommendation in reward-based crowdsourcing systems. In: International Conference on Advances in Computing, Communications and Informatics (2017)
  * [19] Lave, J., Wenger, E.: Legitimate peripheral participation. Learning and Knowledge (1999)
  * [20] Lin, Y., Liu, Z., Sun, M., Liu, Y., Zhu, X.: Learning entity and relation embeddings for knowledge graph completion. In: AAAI Conference on Artificial Intelligence (2015)
  * [21] Liu, Z., Li, K., Qu, D.: Knowledge graph based question routing for community question answering. In: International Conference on Neural Information Processing (2017)
  * [22] Mikolov, T., Chen, K., Corrado, G., Dean, J.: Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781 (2013)
  * [23] Moskalenko, O., Parra, D., Saez-Trumper, D.: Scalable recommendation of wikipedia articles to editors using representation learning. arXiv preprint arXiv:2009.11771 (2020)
  * [24] Müller-Birn, C., Karran, B., Lehmann, J., Luczak-Rösch, M.: Peer-production system or collaborative ontology engineering effort: What is wikidata? In: International Symposium on Open Collaboration (2015) (2015)
  * [25] Pennington, J., Socher, R., Manning, C.D.: Glove: Global vectors for word representation. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) (2014)
  * [26] Peters, M.E., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., Zettlemoyer, L.: Deep contextualized word representations. arXiv preprint arXiv:1802.05365 (2018)
  * [27] Piscopo, A., Phethean, C., Simperl, E.: What makes a good collaborative knowledge graph: Group composition and quality in wikidata. In: International Conference on Social Informatics (2017)
  * [28] Piscopo, A., Phethean, C., Simperl, E.: Wikidatians are born: paths to full participation in a collaborative structured knowledge base (2017)
  * [29] Piscopo, A., Simperl, E.: Who models the world?: Collaborative ontology creation and user roles in wikidata. the ACM on Human-Computer Interaction (2018)
  * [30] Polignano, M., de Gemmis, M., Semeraro, G.: Contextualized bert sentence embeddings for author profiling: The cost of performances. In: International Conference on Computational Science and Its Applications. pp. 135–149. Springer (2020)
  * [31] Reimers, N., Gurevych, I.: Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084 (2019)
  * [32] Rendle, S., Freudenthaler, C., Gantner, Z., Schmidt-Thieme, L.: Bpr: Bayesian personalized ranking from implicit feedback. arXiv preprint arXiv:1205.2618 (2012)
  * [33] Safran, M., Che, D.: Efficient learning-based recommendation algorithms for top-n tasks and top-n workers in large-scale crowdsourcing systems. ACM Transactions on Information Systems (TOIS) 37(1), 1–46 (2018)
  * [34] Sarasua, C., Checco, A., Demartini, G., Difallah, D., Feldman, M., Pintscher, L.: The evolution of power and standard wikidata editors: comparing editing behavior over time to predict lifespan and volume of edits. Computer Supported Cooperative Work (CSCW) (2019)
  * [35] Schafer, J.B., Frankowski, D., Herlocker, J., Sen, S.: Collaborative filtering recommender systems. In: The Adaptive Web (2007)
  * [36] Shani, G., Gunawardana, A.: Evaluating recommendation systems. In: Recommender Systems Handbook (2011)
  * [37] Sun, J., Vishnu, A., Chakrabarti, A., Siegel, C., Parthasarathy, S.: Coldroute: effective routing of cold questions in stack exchange sites. Data Mining and Knowledge Discovery (2018)
  * [38] Sun, J., Zhao, J., Sun, H., Parthasarathy, S.: An end-to-end framework for cold question routing in community question answering services. arXiv preprint arXiv:1911.11017 (2019)
  * [39] Turki, H., Shafee, T., Taieb, M.A.H., Aouicha, M.B., Vrandečić, D., Das, D., Hamdi, H.: Wikidata: A large-scale collaborative ontological medical database. Journal of Biomedical Informatics (2019)
  * [40] Vrandečić, D., Krötzsch, M.: Wikidata: a free collaborative knowledge base (2014)
  * [41] Wang, B., Wang, A., Chen, F., Wang, Y., Kuo, C.C.J.: Evaluating word embedding models: methods and experimental results. APSIPA Transactions on Signal and Information Processing (2019)
  * [42] Wikipedia: Wikidata. https://en.wikipedia.org/wiki/Wikidata (2021), [Online; accessed 26-June-2021]
  * [43] Xiao, H., Huang, M., Hao, Y., Zhu, X.: Transa: An adaptive approach for knowledge graph embedding. arXiv preprint arXiv:1509.05490 (2015)
  * [44] Yang, L., Amatriain, X.: Recommending the world’s knowledge: Application of recommender systems at quora. In: ACM Conference on Recommender Systems (2016)