# Modeling of Speech-dependent Own Voice Transfer Characteristics for
Hearables with In-ear Microphones

Mattes Ohlenbusch1∗, Christian Rollwage1, Simon Doclo1,2

###### Abstract

Hearables often contain an in-ear microphone, which may be used to capture the
own voice of its user. However, due to ear canal occlusion the in-ear
microphone mostly records body-conducted speech, which suffers from band-
limitation effects and is subject to amplification of low frequency content.
These transfer characteristics are assumed to vary both based on speech
content and between individual talkers. It is desirable to have an accurate
model of the own voice transfer characteristics between hearable microphones.
Such a model can be used, e.g., to simulate a large amount of in-ear
recordings to train supervised learning-based algorithms aiming at
compensating own voice transfer characteristics. In this paper we propose a
speech-dependent system identification model based on phoneme recognition.
Using recordings from a prototype hearable, the modeling accuracy is evaluated
in terms of technical measures. We investigate robustness of transfer
characteristic models to utterance or talker mismatch. Simulation results show
that using the proposed speech-dependent model is preferable for simulating
in-ear recordings compared to a speech-independent model. The proposed model
is able to generalize better to new utterances than an adaptive filtering-
based model. Additionally, we find that talker-averaged models generalize
better to different talkers than individual models.

1 Fraunhofer Institute for Digital Media Technology IDMT, Oldenburg Branch for
Hearing, Speech and Audio Technology HSA, Germany

2 University of Oldenburg, Department of Medical Physics and Acoustics and
Cluster of Excellence Hearing4all, Germany

∗ Corresponding author<EMAIL_ADDRESS>
##  1 Introduction

Hearables, i.e. smart earbuds containing a loudspeaker and one or more
microphones, are often used in everyday noisy environments. Although hearables
are frequently used to enhance the voice of a person the hearable user is
communicating with in a noisy environment, the scenario we are considering in
this paper is to enhance the own voice of the user while talking in a noisy
environment (e.g., to be transmitted via a wireless link to a mobile phone or
another hearable). In-ear microphones may offer benefits for own voice pickup
since external noise is attenuated due to ear canal occlusion.

However, own voice recorded inside the occluded ear suffers from amplification
below 1 kHz and heavy attenuation above 2 kHz, leading to a limited bandwidth
[1]. In addition, the amount of occlusion can be observed to depend on device
properties like quality of the earmould fit or device insertion depth [2, 3].
In this work, we refer to this as own voice transfer characteristics. These
properties vary between individuals due to anatomical differences like the
length of the body transfer path, the residual ear canal volume, or the ear
canal shape [4]. However, human ear canal geometries are costly to measure,
e.g., using medical imaging methods [5]. Measurements of the occlusion effect
are also influenced by the ratio between the airborne and body-conducted
component of own voice, which can be observed to depend on phonemes used in
the measurement [6, 7], possibly due to . In this paper, we refer to this
behavior as own voice transfer characteristics. Mouth movements during
articulation [8] or body conduction from different places of excitation likely
influence these transfer characteristics as well.

Many communication applications for hearables have been already investigated
in which signal processing algorithms rely on accurate modeling of sound
transfer inside the head of the hearable user. Active noise cancellation (ANC)
for hearables strongly relies on the availability of accurate estimates of the
primary and secondary paths [9, 10]. In active occlusion cancellation (AOC),
own voice transfer inside the talkers head is modeled in order to compensate
for the occlusion effect [11]. It can be observed that both the occlusion
effect as well as AOC performance can vary based on phonemes uttered [12]. In
both of these applications, transfer paths are either modeled as time-
invariant linear filters [12, 11, 13] or using adaptive filters [14].

Modeling of sound transfer inside the head of the hearable user is not just
relevant for active systems, but also for approaches aiming bandwidth
extension, equalization, and noise reduction of the own voice for either
radio-based communication or speech recognition if in-ear microphones or body-
conduction sensors are employed. In these applications, these sensors provide
the benefit of only capturing a limited amount of external noise in noisy
environments [15]. To enhance the quality of the in-ear microphone signal,
several approaches have been proposed, either based on classical signal
processing [1] or supervised learning [16]. For supervised learning-based
approaches large amounts of training data are typically required, which may be
hard to obtain for realistic in-ear recordings. Transfer characteristic models
may be utilized to overcome these requirements.

Training data requirements have already been addressed in supervised learning-
based speech enhancement approaches using body-conduction sensors: In [17],
own voice transfer characteristics are modeled by a deep neural network (DNN).
This model and a multi-modal enhancement network are jointly trained within a
semi-supervised training scheme, resulting in reduced data requirements
compared to a fully supervised training. In [18], it has been proposed to
convert airborne to bone-conducted speech using a DNN model of own voice
transfer characteristics that accounts for individual differences between
talkers using a speaker identification system. In [19], the own voice transfer
characteristics of an in-ear device are simulated using a time-invariant, non-
individual linear zero-phase filter obtained from a single talker in order to
synthesize training and testing data. In previous work, we have proposed to
estimate the transfer characteristics between the entrance of the ear canal
and the in-ear microphone using a time-invariant linear model to simulate
short segments of in-ear speech for data augmentation in DNN training [16].

Results in our previous work indicate that these applications would benefit
from more accurate transfer characteristic models. It can generally be
observed that data augmentation using more accurate acoustic models yields a
benefit for trained system performance or generalization ability, e.g., in
acoustic parameter estimation [20, 21] or in synthesizing drone recordings for
detection or suppression of acoustic emissions [22]. In order to achieve a
similar performance gain in own voice enhancement systems, more accurate
models of own voice transfer characteristics are hence also of interest. We
note that for training data generation, there are no real-time or low-latency
processing requirements on models of own voice transfer characteristics as
e.g., in AOC applications. Meanwhile, when generating new in-ear recordings
from broadband speech corpora, there are often no parallel in-ear recordings
of the same utterance by the same talker available.

In this paper, we propose to model own voice transfer characteristics using a
phoneme-dependent system identification approach, where for each phoneme a
different linear filter is estimated. In simulation, a phoneme recognition
system is utilized to select a phoneme-specific filter based on speech
content. The proposed approach can be utilized to simulate speech at an in-ear
microphone from regular speech recordings. The simulation accuracy is assessed
using real own voice recordings of over 300 utterances by 18 talkers each from
a prototype hearable device. By comparing the proposed speech-dependent
approach to speech-independent modeling and an adaptive filtering-based
approach, we investigate the role of speech dependency on modeling in-ear own
voice recordings. Results show that the proposed speech-dependent model is
able to better simulate in-ear recordings than the speech-independent model.
Under utterance mismatch, we find that the adaptive filtering-based approach
fails to generalize to different utterances. We compare the performance of
individualized and average-talker models and investigate how well modeling
approaches are able to generalize to different talkers. In terms of distance
measures, the proposed talker-averaged speech-dependent model achieves the
best results when generalizing to utterances of new talkers.

This paper is an extended version of preliminary results published in [23]. We
extend our previous work by investigating utterance and talker mismatch
effects individually, proposing talker-averaged models, and compare models to
an adaptive filtering-based model with oracle utterance knowledge. Experiments
in this paper are conducted on a new, larger corpus of hearable recordings.

The structure of this paper is as follows: In Section 2, a signal model of own
voice transfer characteristics is formulated. In Section 3, a description of
the modeling task considered in this work is given and multiple models of own
voice transfer characteristics are formulated. In Section 4, transfer
characteristic models are evaluated using real own voice recordings for
different conditions.

##  2 Signal model

Figure 1 depicts the considered scenario, where a talker is wearing a hearable
device equipped with an in-ear microphone and an outer microphone, denoted by
subscript $\mathrm{i}$ and $\mathrm{o}$, respectively.

$Y_{\mathrm{o}}^{a}(k,l)$$Y_{\mathrm{i}}^{a}(k,l)$$S_{\mathrm{o}}^{a}(k,l)$$S_{\mathrm{i}}^{a}(k,l)$$V_{\mathrm{i}}^{a}(k,l)$$T_{S}^{a}$
Figure 1: The signal model of own voice transfer characteristics as considered
in this paper.

In the short time Fourier transform (STFT) domain the recorded own voice
signal at the outer microphone of talker $a$ is denoted by
$Y_{\mathrm{o}}^{a}(k,l)$ with $k$ the frequency bin index and $l$ the time
frame index. It is assumed that this signal does not contain any additive
noise. The signal recorded at the in-ear microphone $Y_{\mathrm{i}}^{a}$ is
assumed to contain an in-ear own voice speech component $S_{\mathrm{i}}^{a}$
and a body-produced noise component $V_{\mathrm{i}}^{a}$, i.e.

$Y_{\mathrm{i}}^{a}(k,l)=S_{\mathrm{i}}^{a}(k,l)+V_{\mathrm{i}}^{a}(k,l)$ (1)

where $S_{\mathrm{i}}^{a}$ and $V_{\mathrm{i}}^{a}$ are assumed to be
uncorrelated. Body-produced noise, such as breathing sounds, heartbeats, or
sounds originating from movement, may be recorded by an in-ear microphone
[15]. The own voice speech at the in-ear microphone $S_{\mathrm{i}}^{a}(k,l)$
is related to the own voice speech at the outer microphone
$Y_{\mathrm{o}}^{a}(k,l)$ by a transfer characteristic $T_{S}^{a}\\{\cdot\\}$,
i.e.

$S_{\mathrm{i}}^{a}(k,l)=T_{S}^{a}\left\\{Y_{\mathrm{o}}^{a}(k,l)\right\\}.$
(2)

This transfer characteristic can be observed to change based on the own voice
$S$, making it speech-dependent [6, 7, 12] (see also Figure 7). We therefore
assume this transfer characteristic to be linear, time-varying due to its
speech-dependency, and talker-specific due to individual anatomical
differences [4]. The goal in this paper is to obtain an accurate transfer
characteristic model which is robust against utterance and talker mismatch.

##  3 Modeling of own voice transfer characteristics

In this section, we present system identification-based approaches to model
own voice transfer characteristics (see Fig. 2). Several linear models are
described, some of which are time-varying.

We follow a system identification-based approach in which model parameters are
estimated in an identification step and later applied to estimate own voice
speech at the in-ear microphone in a simulation step as outlined in Figure 2.

Figure 2: Overview of the identification and simulation steps for modeling own
voice transfer characteristics.

During identification, recordings of talker $a$ are used to obtain the model
$\hat{T}^{a}$. During simulation, this model is used to obtain an estimate
$\hat{S}_{\mathrm{i}}^{b}(k,l)$ of the own voice of talker $b$. If $a=b$, the
model is applied to the same talker as in identification. If $a\neq b$, the
model is applied to a different talker, i.e. talker mismatch is present. In
Section 3.1, we present a time-invariant individual speech-independent model.
In Section 3.2 we propose a time-varying individual speech-dependent model
that takes into account the speech-dependency of own voice transfer
characteristics. In Section 3.3 we describe how to compute talker-averaged
models of the speech-independent and the speech-dependent models in order to
investigate whether talker-averaging can be employed to increase robustness to
talker mismatch. In Section 3.4, an adaptive filtering-based model requiring
oracle knowledge of the target utterance is described in order to be used as a
reference method in later experiments.

###  3.1 Speech-independent individual model

If own voice transfer characteristics are assumed speech-independent, the
individual transfer characteristics $T^{a}$ of talker $a$ can be modeled as a
linear time-invariant relative transfer function (RTF) $H^{a}(k)$ between the
outer microphone and the in-ear microphone:

$\hat{T}_{\mathrm{sp.-indep.}}^{a}=\left\\{\hat{H}^{a}(k)\leavevmode\nobreak\
\middle|\leavevmode\nobreak\ k=1,\leavevmode\nobreak\
\dots,\leavevmode\nobreak\ K\right\\},$ (3)

where $K$ denotes the STFT size. In order to obtain an RTF estimate, batch
estimation over a signal of $L$ STFT frames is carried out. Since the outer
microphone signal does not contain any additive noise, the RTF
$\hat{H}^{a}(k)$ can be estimated using the well-known least squares approach
[24]. By minimizing

$\hat{H}^{a}(k)=\arg\min_{H^{a}(k)}\sum_{l}|Y_{\mathrm{i}}^{a}(k,l)-H^{a}(k)\cdot
Y_{\mathrm{o}}^{a}(k,l)|^{2}$ (4)

the least-squares RTF estimate is obtained as

$\hat{H}^{a}(k)=\frac{\sum_{l}Y_{\mathrm{i}}^{a}(k,l)\cdot
Y_{\mathrm{o}}^{a,*}(k,l)}{\sum_{l}|Y_{\mathrm{o}}^{a}(k,l)|^{2}},$ (5)

where $\cdot^{*}$ denotes complex conjugation. For simulation with the speech-
independent individual model of talker $a$, own voice speech of talker $b$
recorded at the outer microphone is filtered in the STFT domain:

$\hat{S}_{\mathrm{i}}^{b}(k,l)=\hat{H}^{a}(k)\cdot Y_{\mathrm{o}}^{b}(k,l).$
(6)

The signal flow during simulation with the speech-independent individual model
is shown in Figure 3. A weighted overlap-add (WOLA) scheme is employed to
obtain a time-domain signal.

Figure 3: Simulation with the speech-independent individual model.

###  3.2 Speech-dependent individual model

Since own voice transfer characteristics likely depend on speech content, we
propose a time-varying speech-dependent individual model for the transfer
characteristics $T^{a}$ of talker $a$. Using a phoneme recognition system $R$,
we first obtain a frame-wise phoneme annotation $p(l)\in
1,\leavevmode\nobreak\ \dots,\leavevmode\nobreak\ P$ with $P$ possible phoneme
classes from a speech signal $y_{\mathrm{o}}^{a}[n]$ recorded at the outer
microphone, where $n$ is the discrete time index:

$p(l)=R\left\\{y_{\mathrm{o}}^{a}[n]\right\\}$ (7)

For each unique phoneme $p^{\prime}$, an RTF is then estimated over all
detected occurrences of this phoneme within the identification utterances of
talker $a$, i.e.

$\hat{H}_{p^{\prime}}^{a}(k)=\frac{\sum_{p(l)=p^{\prime}}Y_{\mathrm{i}}^{a}(k,l)\cdot
Y_{\mathrm{o}}^{a,*}(k,l)}{\sum_{p(l)=p^{\prime}}|Y_{\mathrm{o}}^{a}(k,l)|^{2}}.$
(8)

In total, the speech-dependent individual model hence consists of a database
of $P$ RTFs:

$\hat{T}_{\mathrm{sp.-dep.}}^{a}=\left\\{\hat{H}_{p}^{a}(k)\leavevmode\nobreak\
\middle|\leavevmode\nobreak\ p\in 1,\ \dots,\ P,\ k=1,\ \dots,\ K\right\\}.$
(9)

For simulation, the phoneme sequence $p^{b}(l)$ is first determined on the own
voice speech of talker $b$ recorded at the outer microphone. For each frame,
the corresponding phoneme-specific RTF $\hat{H}_{p(l)}^{a}(k)$ is selected.
Then recursive smoothing with smoothing constant $\alpha$ is applied by
computing the smoothed RTF as

$\tilde{H}_{p(l)}^{a}(k)=\alpha\cdot\tilde{H}_{p(l-1)}^{a}(k)+(1-\alpha)\cdot\hat{H}_{p(l)}^{a}(k)$
(10)

in order to prevent discontinuities in the RTFs during phoneme transitions.
The smoothed RTF $\tilde{H}_{p(l)}^{a}(k)$ is finally used for predicting the
own voice of talker $\mathrm{b}$ at the in-ear microphone:

$\hat{S}_{\mathrm{i}}^{b}(k,l)=\tilde{H}_{p^{b}(l)}^{a}(k)\cdot
Y_{\mathrm{o}}^{b}(k,l)$ (11)

Due to the use of the phoneme recognition system for frame-wise RTF selection,
the proposed model can be utilized to model speech-dependent behavior on new
utterances not used for model identification. Unlike the speech-independent
individual model, the speech-dependent individual model also accounts for
speech breaks by modeling them as separate phonemes.

The signal flow during simulation with the speech-dependent individual model
is shown in Figure 4. Similar to the speech-independent model in Section 3.1,
a WOLA scheme is employed to obtain a time-domain signal.

Figure 4: Simulation with the speech-dependent individual model.

###  3.3 Talker-averaged models

Since individualized models may not be sufficient for generalization to
different talkers, we propose to compute talker-averaged models instead. For
both the speech-independent and the speech-dependent model, talker-averaged
models are obtained by estimating a single RTF over all utterances by all
considered talkers. The RTFs of the speech-independent talker-averaged model
are computed as

$\hat{H}^{\mathrm{avg}}(k)=\frac{\sum_{a\neq
b}\sum_{l}Y_{\mathrm{i}}^{a}(k,l)\cdot Y_{\mathrm{o}}^{a,*}(k,l)}{\sum_{a\neq
b}\sum_{l}|Y_{\mathrm{o}}^{a}(k,l)|^{2}}.$ (12)

For speech-dependent modeling, the estimation of each phoneme RTF is carried
out over all occurrences of the phoneme from all considered talkers. The RTFs
of the speech-dependent talker-averaged model are computed as

$\hat{H}_{p}^{\mathrm{avg}}(k)=\frac{\sum_{a\neq
b}\sum_{p(l)=p^{\prime}}Y_{\mathrm{i}}^{a}(k,l)\cdot
Y_{\mathrm{o}}^{a,*}(k,l)}{\sum_{a\neq
b}\sum_{p(l)=p^{\prime}}|Y_{\mathrm{o}}^{a}(k,l)|^{2}}.$ (13)

For simulation with the speech-independent talker-averaged model, similarly as
in Section 3.1 RTFs are multiplied on the outer microphone STFT coefficients
as in (6) and a WOLA scheme is employed to obtain a time-domain signal. For
simulation with the speech-dependent talker-averaged model, similarly as in
Section 3.2 smoothing as in (10) is also applied before filtering as in (11)
and a WOLA scheme is employed to obtain a time-domain signal.

###  3.4 Adaptive filtering-based model

Alternatively, an adaptive filtering scheme can be utilized to model a time-
varying transfer path. For this task, we propose to use the well-known
normalized least mean squares (NLMS) algorithm [25]. The signal flow is
illustrated in Figure 5. In this case, the input signal vector to the NLMS
consists of $N$ time-domain samples of the outer microphone signal
$y_{\mathrm{o}}^{a}[n]$:

$\mathbf{y}_{\mathrm{o}}^{a}[n]=\begin{bmatrix}y_{\mathrm{o}}^{a}[n],&y_{\mathrm{o}}^{a}[n-1],&\dots,&y_{\mathrm{o}}^{a}[n-N-1]\end{bmatrix}^{T}$
(14)

During the identification step, the time-domain filter coefficient vector
$\mathbf{h}^{a}$ with $N$ taps is updated using the NLMS update equation:

$\mathbf{h}^{a}[n+1]=\mathbf{h}^{a}[n]+\frac{\mu}{\epsilon+\left(\mathbf{y}_{\mathrm{o}}^{a}[n]\right)^{T}\mathbf{y}_{\mathrm{o}}^{a}[n]}\mathbf{y}_{\mathrm{o}}^{a}[n]e[n]$
(15)

where $\mu$ is a step size parameter, $\epsilon$ a small regularization
constant and the error signal $e[n]$ is given by

$e[n]=s_{\mathrm{i}}^{a}[n]-\hat{s}_{\mathrm{i}}^{a}[n]=s_{\mathrm{i}}^{a}[n]-\left(\mathbf{h}^{a}[n]\right)^{T}\mathbf{y}_{\mathrm{o}}^{a}[n].$
(16)

Since the filter coefficients are subject to adaptation, the model parameters
of this model are

$\hat{T}_{\mathrm{adapt.}}^{a}=\\{\mathbf{h}^{a}[n]\forall n\\}$ (17)

and rely on the availability of input signal $y_{\mathrm{i}}^{a}[n]$ and
desired signal $y_{\mathrm{i}}^{a}[n]$. For simulation, the estimate is
computed as

$\hat{s}_{\mathrm{i}}^{b}[n]=\left(\mathbf{h}^{a}[n]\right)^{T}\mathbf{y}_{\mathrm{o}}^{b}[n].$
(18)

It should be noted that in case of utterance mismatch, the filter is applied
to a different input signal which likely results in estimation errors. Since
this model implicitly depends on a specific utterance, it is not possible to
obtain a talker-averaged model by following the procedure for the speech-
independent and speech-dependent models described in Section 3.3.

$y_{\mathrm{o}}^{a}[n]$$\mathbf{h}^{a}[n]$$\hat{s}_{\mathrm{i}}^{a}[n]$$+$$-$$y_{\mathrm{i}}^{a}[n]$$e[n]$$\mathbf{h}^{a}[n]$$y_{\mathrm{o}}^{b}[n]$$\hat{s}_{\mathrm{i}}^{b}[n]$$\hat{T}_{\mathrm{adapt}}^{a}[n]$IdentificationSimulation
Figure 5: The adaptive filtering scheme utilized for predicting in-ear speech
signals. The filter coefficients are transferred from identification to
simulation directly after each sample-wise adaptation step.

##  4 Evaluation

In this section, the previously described transfer characteristic models are
evaluated in terms of their accuracy in predicting own voice signals at an in-
ear microphone.

###  4.1 Evaluation data and conditions

A dataset of own voice speech from 18 native German talkers with approximately
25 to 30 minutes of speech per talker is utilized in the evaluation. The
hearable device used for recording is the closed-vent variant of the Hearpiece
[26]. 306 pre-determined sentences per talker were recorded: the Marburg and
Berlin sentences[27] consisting of 100 sentences each, 100 common everyday
German sentences for language learners [28] and the German version of the
well-known text The North Wind and The Sun consisting of 6 sentences.
Recordings were conducted in a sound-proofed listening booth using a Behringer
UMC1820 audio interface.

During estimation, model parameters were estimated on 150 utterances per
talker. During simulation, in-ear speech is predicted and evaluated per
utterance. Three different simulation conditions are investigated:

Same talker, same utterance

The models are evaluated on speech of the same talker $a$ they were estimated
on ($a=b$). The same utterances as in estimation are used to validate the
models. For the adaptive filtering-based approach, the same utterance that is
being predicted is also used in estimation so that the desired signal is equal
to the prediction target.

Same talker, utterance mismatch

The models are evaluated on speech of the same talker $a$ they were estimated
on ($a=b$). In order to investigate generalization ability of models for the
same talker, the remaining 156 utterances not used in estimation are
simulated. If there is a mismatch in utterance length, the adaptive filtering-
based model is estimated on concatenated estimation utterances of the same
talker, which are then cut to match the length of the target utterance.

Talker mismatch

The ability of models to generalize to different talkers is investigated by
using models estimated on a talker $a\neq b$ to estimate speech of a different
talker $b$. Matching of estimation talker $a$ to simulation talker $b$ is
carried out by random assignment. In this condition, there is also an implicit
utterance mismatch because the same sentence uttered by different talkers may
have differences w.r.t. speed, frequency content, pronunciation and other
speech attributes. Talker-averaged models are evaluated in this condition
only. For each simulation talker $b$, a talker-averaged model is computed from
utterances of the remaining 17 talkers.

We utilize Log-Spectral Distance (LSD) [29] and Mel-Cepstral Distance (MCD)
[30] between the real and simulated in-ear recordings as evaluation metrics.
In both cases, a lower value indicates a more accurate estimate. We focus on
numerical evaluation and do not utilize perceptual metrics such as PESQ [31],
which was found not to correlate well with subjective ratings of body-
conducted own voice recordings [32].

###  4.2 Model setup and parameter estimation

The experiments were carried out with a sampling rate of 5 kHz in order to
isolate the region in which a speech-dependent effect is expected. Above 2.5
kHz the in-ear signals are assumed here to mostly result from air-conducted
transmission through the device, which likely does not depend on speech
content. Model-specific settings were tuned empirically based on preliminary
experiments. For the speech-independent models and the speech-dependent
models, we use an STFT size of $K=128$. Additionally, a smoothing parameter of
$\alpha=0.8$ corresponding to an effective smoothing time of 64 ms is utilized
in the speech-dependent model. For the speech-dependent models, an in-house
proprietary phoneme recognition system with sufficient accuracy for speech in
quiet was utilized. It is trained on German speech exclusively and provides
labels distinguishing between $P=62$ unique phonemes. In the adaptive
filtering-based model, the filter length $N=128$, step size parameter
$\mu=0.5$, and regularization value $\epsilon=10^{-6}$ are used. The filter
coefficients are initialized as zeros. An STFT framework with a frame length
of $K=128$ corresponding to $25.6$ ms and an overlap of $50$ % is used, where
both in analysis and synthesis a square-root Hann window is utilized. No voice
activity detection was employed so that utterances may contain small pauses.

###  4.3 Example spectrograms and RTFs

Example spectrograms of the outer and in-ear microphone signals as well as the
in-ear speech signals predicted by the speech-independent and the speech
dependent individual models for the same talker, same utterance condition are
shown in Figure 6. We note that while the prediction of the speech-independent
model works well in the frequency region below 500 Hz, it underestimates
speech components for higher frequencies. The estimate of the speech-dependent
model appears to be more accurate in higher frequencies, although differences
are visible above 1 kHz. Also, the low-frequency body-produced noise appearing
in the recorded in-ear microphone signal is not present in the simulated in-
ear signals.

Figure 6: Example spectrograms for the same talker, same utterance condition:
The recorded outer (top left) and in-ear (top right) microphone signals and
the in-ear speech signals predicted by the speech-independent (bottom left)
and the speech dependent (bottom right) individual models.

A time-domain own voice speech signal with its phoneme annotation and the
corresponding talker-specific RTFs are shown in Figure 7. Different from other
experiments, these RTFs were estimated with a sampling frequency of $f_{s}=16$
kHz and an STFT size of $N=256$ to show the high-frequency region as well. It
can be seen that for different phonemes, the RTFs differ a lot in the low-
frequency region below $2.5\,$kHz while above the RTFs are very similar.

Figure 7: Example outer microphone own voice signal and phoneme annotation
(top) and corresponding RTFs (bottom) of a single talker for an utterance of
the beginning of the German version of The North Wind and The Sun.

A selection of RTF magnitudes estimated for the speech-independent and the
speech-dependent approaches for all talkers considered in the experiments are
shown in Figure 8. Different from the talker-averaged RTFs used in the talker
mismatch condition, averages here are computed over all 18 talkers. For the
speech-independent RTF shown in the upper subplot, we observe that for most
talkers the low frequency region below approximately 600 Hz is amplified at
the in-ear microphone relative to the outer microphone. For the frequency
region above approximately 1.5 kHz we observe relative attenuation. While half
of the estimated RTFs are very similar in magnitude, for some talkers there
appear to be larger deviations from the average. For the phoneme-specific RTFs
as utilized in the speech-independent model, we observe similar tendencies in
terms of inter-individual variance. However, it can be observed that the
talker-averaged RTFs differ from the one used in the speech-independent
approach. In particular, for the phoneme /Z/ the magnitude is considerably
higher in the frequency region between 500 to 1.5 kHz and above 2 kHz for the
majority of talkers. This is not true for the phoneme /o/ where especially in
the low frequency region the relative magnitude seems to be lower as for the
talker-averaged RTF used in the speech-independent model.

Figure 8: Relative transfer functions estimated for the speeech-independent
approaches (top) and for two phonemes within the speech-dependent approaches
(middle and bottom). Values between the quartiles Q1 and Q3 and minimum and
maximum values are indicated by the respective surrounding margins. Talker-
averages over all talkers are shown as solid black lines.

###  4.4 Same talker, same utterance

The results for the same talker, same utterance condition are shown in Figure
9. It can be observed that the speech-dependent individual model and the
adaptive filtering-based model predict in-ear speech signals much better than
the speech-independent individual model. The adaptive filtering-based model
performs slightly better than the speech-dependent individual model. These
observations can be made for both metrics. For all models, remaining errors
can be observed. Since in-ear recordings are assumed to also contain body
noise uncorrelated to the outer microphone signals, remaining error signals
are expected due to models not accounting for this signal component. These
results demonstrate that the in-ear speech signals can better be predicted
when time-varying or speech-dependent transfer characteristics are accounted
for. In addition, the speech-dependent individual model performs similar to
the adaptive filtering-based model with oracle knowledge of the target
utterance, which indicates that the proposed model is able to accurately model
time-varying behavior in own voice transfer characteristics.

It should be noted here that the modeling accuracy of the approaches could be
influenced by the use of a voice activity detection mechanism or lack thereof,
as the speech-dependent individual model implicitly accounts for speech breaks
by including them as a separate phoneme. Since speech varies in frequency
content and coherence, it is likely that important frequency regions for any
phoneme can be modeled better if only coherent speech is selected, as is the
case if RTFs are always estimated on the same phoneme.

(a) Log-Spectral Distance.

(b) Mel-Cepstral Distance.

Figure 9: Results for the same talker, same utterance condition with speech-
independent (SI), speech-dependent (SD) and adaptive filtering-based (AD)
models.

###  4.5 Same talker, utterance mismatch

The results for the same talker, utterance mismatch condition are shown in
Figure 10. We observe that the results for speech-dependent and speech-
independent individual models are very similar to the results in Section 4.4,
indicating that both models do not suffer from generalization to other
utterances. For the adaptive filtering-based model, there is a large increase
in distance values. This increase stems from the mismatch between estimation
and target utterance and demonstrates that this approach is no longer as
useful for prediction of new utterances than when oracle knowledge of the
target utterance is available.

(a) Log-Spectral Distance.

(b) Mel-Cepstral Distance.

Figure 10: Results for the same talker, utterance mismatch condition with
speech-independent (SI), speech-dependent (SD) and adaptive filtering-based
(AD) models.

###  4.6 Talker mismatch

The results for the talker mismatch condition are shown in Figure 11. In this
condition, the speech-dependent models yield a higher accuracy over the
speech-independent models. For both the speech-independent and the speech-
dependent models, using a talker-averaged over an individual model results in
an improvement in terms of both metrics in this condition. While the LSD
values in Figure 11(a) for the speech-dependent individual model are slightly
higher compared to the values achieved in previous conditions in Sections 4.4
and 4.5, the speech-independent individual model performs very similar with
talker mismatch as without. We note that in terms of the MCD in Figure 11(b),
using the speech-dependent individual model results in similar scores as using
the speech-independent talker-averaged model, but with a larger spread of
values for individual utterances. Since this effect does not occur in the
other conditions, it is likely a consequence of talker mismatch. We
hypothesize that for similar talkers, model of a different talker can achieve
similar accuracy as a model of the same talker, while for less similar talkers
the mismatch effects can also be much larger. Overall, the speech-dependent
talker-averaged model achieves the lowest distance values.

(a) Log-Spectral Distance.

(b) Mel-Cepstral Distance.

Figure 11: Results for the talker mismatch condition with speech-independent
(SI), speech-dependent (SD) and adaptive filtering-based (AD) models using
individual and talker-averaged (avg.) models.

##  5 Conclusion

In this paper, several approaches to model own voice transfer characteristics
in hearables have been proposed. In particular, proposed models take into
account inter-individual differences between talkers and time-varying speech-
dependent behavior. The models can be utilized to prediction own voice speech
at an in-ear microphone.

The influence of utterance and talker mismatch on accuracy of predicted in
have been investigated. Results show that using a speech-dependent model is
beneficial compared to using a speech-independent model. Although the adaptive
filtering-based approach is able to model the speech-dependency of the own
voice transfer characteristics well when the filter is adapted the the
identical in-ear own voice signal that is being simulated in the same talker,
same utterance condition, it fails when the in-ear signal of a different
utterance or of an utterance of a different talker is being simulated. We
observe that while individual models can achieve a higher accuracy when speech
of the same talker is being simulated, the use of talker-averaged models
results in lower simulation errors in talker mismatch.

## Conflict of interest

The authors declare no conflict of interest.

## Acknowledgments

The Oldenburg Branch for Hearing, Speech and Audio Technology HSA is funded in
the program »Vorab« by the Lower Saxony Ministry of Science and Culture (MWK)
and the Volkswagen Foundation for its further development. Part of this work
was funded by the German Ministry of Science and Education BMBF FK 16SV8811.
This work was partly funded by the Deutsche Forschungsgemeinschaft (DFG,
German Research Foundation) - Project ID 352015383 - SFB 1330 C1. The authors
wish to thank the talkers for their participation in the recordings.

## References

  * [1] Rachel E. Bouserhal, Tiago H. Falk and Jérémie Voix  “In-ear microphone speech quality enhancement via adaptive filtering and artificial bandwidth extension”  In _J. Acoust. Soc. Am._ 141.3, 2017, pp. 1321–1331  DOI: 10.1121/1.4976051
  * [2] Kichol Lee and John G. Casali  “Investigation of the Auditory Occlusion Effect with Implications for Hearing Protection and Hearing Aid Design”  In _Proc. of the Human Factors and Ergonomics Society Annual Meeting_ 55, 2011, pp. 1783–1787  DOI: 10.1177/1071181311551370
  * [3] Mie Østergaard Hansen  “Occlusion effects, Part II: A study of the occlusion effect mechanism and the influence of the earmould properties” Volume: Report 73, 1998
  * [4] Steffen Vogl and Matthias Blau  “Individualized prediction of the sound pressure at the eardrum for an earpiece with integrated receivers and microphones”  In _J. Acoust. Soc. Am._ 145.2, 2019, pp. 917–930  DOI: 10.1121/1.5089219
  * [5] Reinhild Roden and Matthias Blau  “The IHA database of human geometries including torso, head and complete outer ears for acoustic research” Issue: 2  In _Proc. Internoise_ 261, 2020, pp. 4226–4237
  * [6] Sabine Reinfeldt, Per Östli, Bo Håkansson and Stefan Stenfelt  “Hearing one’s own voice during phoneme vocalization - Transmission by air and bone conduction”  In _J. Acoust. Soc. Am._ 128.2, 2010, pp. 751–762  DOI: 10.1121/1.3458855
  * [7] Hugo Saint-Gaudens, Hugues Nélisse, Franck Sgard and Olivier Doutres  “Towards a practical methodology for assessment of the objective occlusion effect induced by earplugs”  In _J. Acoust. Soc. Am._ 151.6, 2022, pp. 4086–4100  DOI: 10.1121/10.0011696
  * [8] J. Richard, V. Zimpfer and S. Roth  “Effect of bone conduction microphone location and mouth opening on transfer function between oral cavity sound pressure and skin acceleration”  In _Proc. Convention of the European Acoustics Association (Forum Acusticum)_ , 2023
  * [9] Stefan Liebich, Johannes Fabry, Peter Jax and Peter Vary  “Signal Processing Challenges for Active Noise Cancellation Headphones”  In _Proc. ITG Conference on Speech Communication_ , 2018, pp. 11–15
  * [10] Piero Rivera Benois, Reinhild Roden, Matthias Blau and Simon Doclo  “Optimization of a Fixed Virtual Sensing Feedback ANC Controller For In-Ear Headphones with Multiple Loudspeakers”  In _Proc. International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2022, pp. 8717–8721  DOI: 10.1109/ICASSP43922.2022.9746327
  * [11] Stefan Liebich and Peter Vary  “Occlusion Effect Cancellation in Headphones and Hearing Devices—The Sister of Active Noise Cancellation”  In _IEEE/ACM Trans. on Audio, Speech, and Language Processing_ 30, 2022, pp. 35–48  DOI: 10.1109/TASLP.2021.3130966
  * [12] Christoph Weyer and Peter Jax  “Occlusion Effect Reduction Using a Vibration Sensor”  In _Proc. International Workshop on Acoustic Signal Enhancement (IWAENC)_ , 2022  DOI: 10.1109/IWAENC53105.2022.9914705
  * [13] Thomas Zurbruegg  “The Occlusion Effect - Measurements, Simulations and Countermeasures”  In _Proc. ITG Conference on Speech Communication_ , 2018, pp. 26–30
  * [14] Renata C. Borges, Márcio H. Costa, Júlio A. Cordioli and Luciana F.. Assuiti  “An adaptive occlusion canceller for hearing aids” ISSN: 2076-1465  In _Proc. European Signal Processing Conference (EUSIPCO)_ , 2013
  * [15] Rachel E. Bouserhal, Antoine Bernier and Jérémie Voix  “An in-ear speech database in varying conditions of the audio-phonation loop”  In _J. Acoust. Soc. Am._ 145.2, 2019, pp. 1069–1077  DOI: 10.1121/1.5091777
  * [16] Mattes Ohlenbusch, Christian Rollwage and Simon Doclo  “Training Strategies for Own Voice Reconstruction in Hearing Protection Devices Using An In-Ear Microphone”  In _Proc. International Workshop on Acoustic Signal Enhancement (IWAENC)_ , 2022  DOI: 10.1109/IWAENC53105.2022.9914801
  * [17] Heming Wang, Xueliang Zhang and DeLiang Wang  “Fusing Bone-Conduction and Air-Conduction Sensors for Complex-Domain Speech Enhancement”  In _IEEE/ACM Trans. on Audio, Speech, and Language Processing_ 30, 2022, pp. 3134–3143  DOI: 10.1109/TASLP.2022.3209943
  * [18] Michael Pucher and Thomas Woltron  “Conversion of Airborne to Bone-Conducted Speech with Deep Neural Networks”  In _Proc. Interspeech_ , 2021, pp. 1–5  DOI: 10.21437/Interspeech.2021-473
  * [19] Julien Hauret, Thomas Joubaud, Véronique Zimpfer and Éric Bavu  “EBEN: Extreme bandwidth extension network applied to speech signals captured with noise-resilient body-conduction microphones”  In _Proc. International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2023  DOI: 10.1109/ICASSP49357.2023.10096301
  * [20] Nicholas J. Bryan  “Impulse Response Data Augmentation and Deep Neural Networks for Blind Room Acoustic Parameter Estimation” ISSN: 2379-190X  In _Proc. International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2020, pp. 396–400  DOI: 10.1109/ICASSP40776.2020.9052970
  * [21] Prerak Srivastava, Antoine Deleforge and Emmanuel Vincent  “Realistic Sources, Receivers and Walls Improve The Generalisability of Virtually-Supervised Blind Acoustic Parameter Estimators”  In _Proc. International Workshop on Acoustic Signal Enhancement (IWAENC)_ , 2022  DOI: 10.1109/IWAENC53105.2022.9914740
  * [22] Kurt Heutschi, Beat Ott, Thomas Nussbaumer and Peter Wellig  “Synthesis of real world drone signals based on lab recordings” Number: 6 Publisher: EDP Sciences  In _Acta Acustica_ 4.6, 2020, pp. 24  DOI: 10.1051/aacus/2020023
  * [23] Mattes Ohlenbusch, Christian Rollwage and Simon Doclo  “Speech-dependent Modeling of Own Voice Transfer Characteristics for In-ear Microphones in Hearables”  In _Proc. Convention of the European Acoustics Association (Forum Acusticum)_ , 2023  DOI: 10.48550/arXiv.2309.08294
  * [24] Yekutiel Avargel and Israel Cohen  “On Multiplicative Transfer Function Approximation in the Short-Time Fourier Transform Domain”  In _IEEE Signal Processing Letters_ 14.5, 2007, pp. 337–340  DOI: 10.1109/LSP.2006.888292
  * [25] Simon Haykin  “Adaptive Filter Theory”  Prentice Hall, 1996
  * [26] Florian Denk, Miriam Lettau, Henning Schepker, Simon Doclo, Reinhild Roden, Matthias Blau, Jörg-Hendrik Bach, Jan Wellmann and Birger Kollmeier  “A one-size-fits-all earpiece with multiple microphones and drivers for hearing device research”  In _Proc. AES International Conference on Headphone Technology_ , 2019, pp. 1–9
  * [27] Adrian P Simpson, Klaus J Kohler and Tobias Rettstadt  “The Kiel corpus of read/spontaneous speech: Acoustic data base, processing tools, and analysis results”  In _Arbeitsberichte Institut für Phonetik und Digitale Sprachverarbeitung Universität Kiel_ 32 IPDS, 1997, pp. 243–247
  * [28] Andi Neustein  “100 Sätze reichen für ein ganzes Leben (Blog-post)”  In _deutschlernerblog_ , 2019  URL: https://deutschlernerblog.de/100-saetze-reichen-fuer-ein-ganzes-leben/
  * [29] Augustine Gray and John Markel  “Distance measures for speech processing”  In _IEEE Trans. on Acoustics, Speech, and Signal Processing_ 24.5, 1976, pp. 380–391  DOI: 10.1109/TASSP.1976.1162849
  * [30] Robert F. Kubichek  “Mel-cepstral distance measure for objective speech quality assessment”  In _Proc. IEEE Pacific Rim Conference on Communications Computers and Signal Processing_ , 1993, pp. 125–128  DOI: 10.1109/PACRIM.1993.407206
  * [31] International Telecommunications Union (ITU)  “ITU-T P.862, Perceptual evaluation of speech quality (PESQ): An objective method for end-to-end speech quality assessment of narrow-band telephone networks and speech codecs” Geneva, Switzerland  In _International Telecommunications Union_ , 2001
  * [32] J. Richard, V. Zimpfer and S. Roth  “Comparison of objective and subjective methods for evaluating speech quality and intelligibility recorded through bone conduction and in-ear microphones”  In _Applied Acoustics_ 211, 2023, pp. 109576  DOI: 10.1016/j.apacoust.2023.109576


# Wembedder: Wikidata entity embedding web service

Finn Årup Nielsen
Cognitive Systems  DTU Compute
Technical University of Denmark
Kongens Lyngby  Denmark
<EMAIL_ADDRESS>

###### Abstract

I present a web service for querying an embedding of entities in the Wikidata
knowledge graph. The embedding is trained on the Wikidata dump using Gensim’s
Word2Vec implementation and a simple graph walk. A REST API is implemented.
Together with the Wikidata API the web service exposes a multilingual resource
for over 600’000 Wikidata items and properties.

###### keywords:

Wikidata, embedding, RDF, web service.

##  1 Introduction

The Word2Vec model [7] spawned an interest in dense word representation in a
low-dimensional space, and there are now a considerable number of “2vec”
models beyond the word level.111https://github.com/MaxwellRebo/awesome-2vec
One recent avenue of research in the “2vec” domain uses knowledge graphs [13].
Such systems can take advantage of the large knowledge graphs, e.g., DBpedia
or Freebase, for graph embedding. Graph embedding in the simplest case would
map individual nodes of the network to a continuous low-dimensional space,
while embedding with knowledge graphs would typically handle the typed links
between knowledge items/nodes.

Wikidata https://www.wikidata.org/ [16] is a relatively new knowledge graph
resource. It is run by the Wikimedia Foundation that is also behind Wikipedia,
thus Wikidata can be regarded as a sister site to Wikipedia. While Wikipedia
has been extensively used as a data and text mining resource [6], Wikidata has
so far seen less use in machine learning contexts. There are several
advantages with Wikidata. Wikidata is not tied to a single language, but can
include labels for hundreds of languages for each item in the knowledge graph.
As such, an embedding that works from Wikidata items is in principle
multilingual (however, there is no guarantee that the item label for a
specific language is set). Another advantage with Wikidata is that each item
can provide extra contextual data from the Wikidata statements. Search in
Wikidata is enabled by string-based search engines in the Wikidata API as well
as the SPARQL-based Wikidata Query Service (WDQS). General functions for
finding related items or generating fixed-length features for machine learning
are to my knowledge not available.

There is some research that combines machine learning and Wikidata [9, 14],
e.g., Mousselly-Sergieh and Gurevych have presented a method for aligning
Wikidata items with FrameNet based on Wikidata labels and aliases [9].

Property suggestion is running live in the Wikidata editing interface, where
it helps editors recall appropriate properties for items during manual
editing, and as such a form of recommender system. Researchers have
investigated various methods for this process [17].

Scholia at https://tools.wmflabs.org/scholia/ is our web service that presents
scholarly profiles based on data in Wikidata extracted with WDQS [11].
Counting co-occurrence patterns with SPARQL queries, Scholia can list related
items based on a query item. For instance, Scholia lists related diseases
based on overlapping associated genes.222See, e.g., the page for schizophrenia
at https://tools.wmflabs.org/scholia/disease/Q41112. Other than these count-
and SPARQL-based methods Scholia has limited means to show related items to a
Scholia user.

Several research groups provide word embedding web services: GPL-licensed
_WebVectors_ uses a Flask and Gensim [5, 4], and instances for English and
Russian run at http://rusvectores.org/ and for English and Norwegian at
http://vectors.nlpl.eu/explore/embeddings/. A Turku BioNLP group provides a
Flask-based word embedding web service at http://bionlp-www.utu.fi/wv_demo/
based on the English Google News and Finnish Corpora. A web service for
handling multilingual word embeddings has also been announced [1]. Wembedder
is distinguished from these services by using the Wikidata entities (items and
properties) as the “words” in the embedding (rather than natural language
words) and by using the live Wikidata web service to provide multilingual
labels for the entities.

##  2 Wembedder

Figure 1: Wembedder’s output after a query on Q298 (the country Chile) with
the interface set to Swedish.

###  2.1 Model setup

The Wikimedia Foundation provides the Wikidata RDF dumps for download at
https://dumps.wikimedia.org/wikidatawiki/entities/. For the setup of the
initial model, I downloaded the so-called truthy dumps available in Notation3
format. The specific file was the 5.2 GB large compressed file
wikidata-20170613-truthy-BETA.nt.bz2. The truthy dumps only have a limited set
of all the triples in Wikidata: Those that are associated with the `wdt`
prefix. From this dump, I extracted the triples where both subject and object
were Wikidata items, i.e., leaving out triples where the object is a value
such as an identifier, a date, a name, etc. The generated file contains
88’941’173 lines each with a triple. The http://www.wikidata.org/entity/ and
http://www.wikidata.org/prop/direct/ prefixes were stripped, so the first few
lines of the generated file have the following content in a format similar to
Magnus Manske’s QuickStatements format:

    Q22 P1546 Q2016568
    Q22 P610 Q104674
    Q22 P1151 Q8143311
    Q22 P31 Q3336843
    Q22 P36 Q23436
    Q22 P47 Q21
    ...

Each line can be regarded as a very simple graph walk consisting of a single
step from one Wikidata item through a typed property to the next Wikidata
item. These triple data I now regard as a sentence of three “words” which can
be treated by standard word embedding implementations. I use the Word2Vec
model in the Gensim program [18]. The initial model trained used the CBOW
training algorithm, an embedding dimension on 100, a window of 1 and a minimum
count of 20, i.e., any “word” must appear 20 times or more to be included in
the model. The rest of the parameters in the Word2Vec model were kept at the
Gensim defaults. With this setup, the model ends up with a vocabulary of
609’471. This number includes 738 properties and 608’733 Wikidata items.
Gensim can store its model parameters in files with a combined size of 518
megabytes. A permanent version of the model parameters is available in Zenodo
under DOI 10.5281/zenodo.823195.

###  2.2 Web service

The web service was set up with the Python Flask web framework [3] with the
Apache-licensed code available at a GitHub repository:
https://github.com/fnielsen/wembedder. Figure 1 shows the interface. A version
of Wembedder runs from https://tools.wmflabs.org/wembedder/, i.e., from the
cloud service provided by the Wikimedia Foundation.

The Wembedder web service relies on the Wikidata API at
https://www.wikidata.org/w/api.php and its _wbsearchentities_ action for
searching for items in multiple languages in an implementation based on the
search facility on the Wikidata homepage. Labels for searching and labels for
the results are generated via ajax calls to the Wikidata API.

A REST API is implemented as part of Wembedder and returns JSON-formatted
results, e.g., `/api/most-similar/Q80` will return the most similar entities
for a query on Tim Berners-Lee (Q80), see also Figure 2. Similarity
computations are implemented with a URL such as `/api/similarity/Q2/Q313`.
Here the Earth and Venus are compared. The human interface of Wembedder uses
the REST API in a ajax fashion, returning an HTML page with an empty result
list and with JavaScript for the actual fetching of the results.

Figure 2: JSON output from Wembedder’s REST API with a query on Q2 (Earth)
rendered in a web browser. The first entities in the result list is Q313
(Venus).

##  3 Evaluation

The embedding in the current version of Wembedder is fairly simple compared to
the state of the art embeddings, that uses complex/holographic knowledge graph
embedding [10] or multiple knowledge graphs and pre-trained corpora-based
resources for building the embedding [15]. One should not expect Wembedder to
perform at the state of the art level,333An overview of the state of the art
performance in semantic relatedness task, including for the Wordsim-353 task,
is available at
https://aclweb.org/aclwiki/index.php?title=Similarity_(State_of_the_art) and a
comparison with the Wordsim-353 dataset for semantic relatedness evaluation
[2] shows poor performance with Pearson and Spearman correlations on just
0.13.

When used to evaluate the Wikidata graph embedding, a matching is needed
between English Wordsim-353 words and the Wikidata items. It is not
straightforward as there usually is a semantic difference between the words
and the items. It is often possible to find the word as the English label in a
Wikidata item, but for instance, for the Wordsim-353 word “Japanese” one must
decide whether it should be linked to Japanese as a language (Q5287), Japanese
as a people (Q161652), another item (e.g., the disambiguation page, Q346080)
or an average or sum over the items. I attempted to match the words with
items, but left several unmatched so only 278 of the word pairs of the 353
were possible in the analysis. The correlations were computed from these 278
word pairs. A skipgram trained model yielded even lower performance with
correlations of just 0.11 and 0.10 for Pearson and Spearman correlations,
respectively. A CBOW model trained with a higher number of iterations (DOI
10.5281/zenodo.827339) performed somewhat better with correlations of 0.21.

##  4 Discussion and future work

Wembedder—with its 100-dimensional Gensim model query—will usually be able to
return results in around one second, while the API call is considerably
faster. It means that it could be used for interactive “related items” search.
The SPARQL-based related items queries in Scholia usually takes several
seconds.

Wikidata at its current state is mostly an encyclopedic source having little
lexical information. State of the art relational modeling ConceptNet is setup
from both encyclopedic and lexical knowledge graphs as well as corpus-based
embeddings [15]. Embeddings based on Wikidata could presumably perform better
by using the link to Wikipedia with the different language versions of
Wikipedia acting as a corpora. There exist several works describing joint
models of words and entities from knowledge bases/graphs, see, e.g., [8] and
reference therein. There is work underway to enable Wikidata to represent
lexical information [12]. A Wikidata-based embedding may benefit from such
data.

##  5 Acknowledgment

The Danish Innovation Foundation supported this work through Danish Center for
Big Data Analytics driven Innovation (DABAI).

## References

  * [1] Ammar, W., Mulcaire, G., Tsvetkov, Y., Lample, G., Dyer, C., and Smith, N. A. Massively Multilingual Word Embeddings.
  * [2] Finkelstein, L., Gabrilovich, E., Matias, Y., Rivlin, E., Solan, Z., Wolfman, G., and Ruppin, E. Placing search in context: the concept revisited.  ACM Transactions on Information Systems 20 (January 2002), 116–131.
  * [3] Grinberg, M. Flask Web Development.
  * [4] Kutuzov, A., and Kuzmenko, E. Building Web-Interfaces for Vector Semantic Models with the WebVectors Toolkit.  Proceedings of the Software Demonstrations of the 15th Conference of the European Chapter of the Association for Computational Linguistics (April 2017), 99–103.
  * [5] Kutuzov, A., and Kuzmenko, E. WebVectors: A Toolkit for Building Web Interfaces for Vector Semantic Models.  Analysis of Images, Social Networks and Texts: 5th International Conference, AIST 2016 (December 2017), 155–161.
  * [6] Mehdi, M., Okoli, C., Mesgari, M., Nielsen, F. Å., and Lanamäki, A. Excavating the mother lode of human-generated text: A systematic review of research that uses the wikipedia corpus.  Information Processing & Management 53 (March 2017), 505–529.
  * [7] Mikolov, T., Chen, K., Corrado, G., and Dean, J. Efficient Estimation of Word Representations in Vector Space.
  * [8] Moreno, J. G., Besançon, R., Beaumont, R., D’hondt, E., Ligozat, A.-L., Rosset, S., Tannier, X., and Grau, B. Combining Word and Entity Embeddings for Entity Linking.  The Semantic Web (May 2017), 337–352.
  * [9] Mousselly-Sergieh, H., and Gurevych, I. Enriching Wikidata with Frame Semantics.  Proceedings of the 5th Workshop on Automated Knowledge Base Construction (June 2016), 29–34.
  * [10] Nickel, M., Rosasco, L., and Poggio, T. Holographic Embeddings of Knowledge Graphs.
  * [11] Nielsen, F. Å., Mietchen, D., and Willighagen, E. Scholia and scientometrics with Wikidata.
  * [12] Pintscher, L. Let’s move forward with support for Wiktionary.  Wikidata mailing list (September 2016).
  * [13] Ristoski, P., and Paulheim, H. RDF2Vec: RDF Graph Embeddings for Data Mining.  The Semantic Web – ISWC 2016 (December 2016), 498–514.
  * [14] Sorokin, D., and Gurevych, I. End-to-end Representation Learning for Question Answering with Weak Supervision.
  * [15] Speer, R., Chin, J., and Havasi, C. ConceptNet 5.5: An Open Multilingual Graph of General Knowledge.  Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence (December 2016), 4444–4451.
  * [16] Vrandečić, D., and Krötzsch, M. Wikidata: a free collaborative knowledgebase.  Communications of the ACM 57 (October 2014), 78–85.
  * [17] Zangerle, E., Gassler, W., Pichl, M., Steinhauser, S., and Specht, G. An Empirical Evaluation of Property Recommender Systems for Wikidata and Collaborative Knowledge Bases.  Proceedings of the 12th International Symposium on Open Collaboration (December 2016).
  * [18] Řehůřek, R., and Sojka, P. Software framework for topic modelling with large corpora.  New Challenges For NLP Frameworks Programme (May 2010), 45–50.