Copyright for this paper by its authors. Use permitted under Creative Commons
License Attribution 4.0 International (CC BY 4.0).

CLEF 2023: Conference and Labs of the Evaluation Forum, September 18–21, 2023,
Thessaloniki, Greece

[ orcid=0000-0002-9165-8718<EMAIL_ADDRESS>url=https://acmiyaguchi.me, ] [1]

[<EMAIL_ADDRESS>]

[ orcid=0009-0003-9818-496X<EMAIL_ADDRESS>url=https://linkedin.com/in/murilo-gustineli, ]

[<EMAIL_ADDRESS>]

[1]Corresponding author.

# Transfer Learning with Semi-Supervised Dataset Annotation for Birdcall
Classification

Anthony Miyaguchi  Nathan Zhong  Murilo Gustineli  Chris Hayduk  Georgia
Institute of Technology, North Ave NW, Atlanta, GA 30332

(2023)

###### Abstract

We present working notes on transfer learning with semi-supervised dataset
annotation for the BirdCLEF 2023 competition, focused on identifying African
bird species in recorded soundscapes. Our approach utilizes existing off-the-
shelf models, BirdNET and MixIT, to address representation and labeling
challenges in the competition. We explore the embedding space learned by
BirdNET and propose a process to derive an annotated dataset for supervised
learning. Our experiments involve various models and feature engineering
approaches to maximize performance on the competition leaderboard. The results
demonstrate the effectiveness of our approach in classifying bird species and
highlight the potential of transfer learning and semi-supervised dataset
annotation in similar tasks.

###### keywords:

Transfer Learning Dataset Annotation BirdNET Bird-MixIT CEUR-WS

##  1 Introduction

The BirdCLEF 2023 competition [1] focuses on classifying bird species in
10-minute-long soundscapes recorded in various parts of Africa as part of the
LifeCLEF lab [2]. We need to label each 5-second interval in the test
soundscape with the probability that each 264 target species is present. The
training dataset comprises 16,941 tracks spanning 192 hours of audio tagged by
the label of the target species present. While we have metadata about the
species in the tracks, we do not have concrete labels on which audio sections
contain calls. The dataset also poses significant challenges given the species
distribution, with many having only one or two examples.

We focus our efforts on utilizing existing off-the-shelf models as the basis
for our classification models. We hypothesize that we can reduce environmental
noise and cross-talk using a sound-separation model and re-purpose knowledge
in the embedding space learned by a domain-specific convolution neural
network. These models can address some of the most significant impediments in
the competition, including the need for labeled data through semi-supervision.

##  2 Embedding Space and Transfer Learning

BirdNET [3] classifies 48kHz 3-second audio clips into 3337 classes using a
convolutional neural network trained on scaled spectrograms computed by the
short-time Fourier transform. The classes are primarily composed of bird
species but include non-bird classes such as environmental noise or human
voices. We obtain embedding tokens by taking the values at the second-to-last
layer of the model, preceding a fully connected logit layer. The embedding
maps the audio in the time domain into a vector space $\mathcal{R}^{320}$ that
roughly preserves distances between points. We use embedding tokens as
features in a supervised machine-learning model to take advantage of the
compact representation of the audio data. We show a clear separation between
the embedding tokens by running them through a dimensional reduction technique
in figure 1. Clustering supports a hypothesis that we effectively utilize
representation learned by BirdNET in new contexts.

Figure 1:  We demonstrate the clustering properties of the BirdNET embeddings
by projecting them into $\mathcal{R}^{2}$ via UMAP [4] The projection
preserves Euclidean distance in 2D space. We take the embedding token across
each track with the most significant probability across the BirdNET prediction
vector and assign it a positive label. The left plot shows clustering across
the seven most common species in the training dataset. The right plot
demonstrates a clear separation between the seven most common species.

##  3 Semi-Supervised Dataset Annotation

Figure 2:  Bird-MixIT has improved the precision of downstream classifiers in
experimental settings. We demonstrate separation across entities in the mel-
spectrogram of track XC207767 containing a Red-chested Cuckoo. We observe
separating three sound signatures into sources 0, 1, and 3. Source 2 is an
amalgamation of the other sound signatures, an artifact of the model
separating into four channels. An automated process should choose source 3
containing the species of interest.  Figure 3:  We use Luigi to coordinate a
processing pipeline spanning days on an n2-standard-16 compute instance. We
prevent processing skew across workers by recursively training audio. The
audio is then source separated and embedded, resulting in a parquet file per
audio chunk. We consolidate the parquet files into the final dataset.

We propose a process to derive an annotated dataset to fit data to traditional
supervised learning algorithms. First, we chunk audio within the training
examples so that no track lasts 3 minutes by recursively splitting the tracks
until they are smaller than our threshold, padded to the nearest third second
with additive white noise. Chunking the audio solves the problem of batch
processing skew introduced by several examples that are longer than 30
minutes. We assume the upper bound on the track length is sufficient to model
temporal dependencies. We use Bird-MixIT [5] to isolate environmental noise
and bird vocalizations. We then process each of the tracks using BirdNET to
extract an embedding vector in $\mathbb{R}^{320}$ and a prediction logit
vector in $\mathbb{R}^{3337}$ for every 3-second interval over a 1-second
sliding window. Each interval is labeled with top prediction labels and the
energy of the original track. We save the results from each track to disk and
consolidate them into a parquet dataset. See table 1 for an example row of
this process. We only have to pay for the expensive process of running
TensorFlow models once by processing the audio training examples before
fitting models.

We had four versions of the embedding dataset (emb). In emb_v1, we encountered
missing entries and Docker permission problems. We fixed this in emb_v2 but
incorrectly mapped input to the wrong model layer. Our first usable dataset
was emb_v3, which fixed previous issues and addressed the skew of long tracks
by recursively chunking them. emb_v4 includes all possible 3-second intervals
at a 1-second resolution. With the embedding and prediction vectors in a
dataset, we apply a series of heuristics and feature engineering for our final
models.

column name | value
---|---
species | grecor
track_stem | XC629875_part003
track_type | source0
track_name | grecor/XC629875_part003_source0.mp3
embedding | [0.6731137633323669, 1.1389738321304321, 0.6284520626068115, …
prediction_vec | [-8.725186347961426, -7.3204827308654785, -9.82101821899414, …
predictions | [{0, 3026, Tetrastes bonasia_Hazel Grouse, hazgro1, 0.02235649898648262}, {1,…
start_time | 75
energy | 0.01598571054637432
Table 1:  Example output row from the embedding dataset (emb_v4). The emb_3
dataset is about 20 GB, and the emb_v4 dataset is about 60 GB.

##  4 Implementation and Workflow

We split our workflow into training and inference. Our training pipeline runs
on the Google Cloud Platform (GCP), while inference runs in a Kaggle notebook
optimized for offline usage.

We implement a shared Python package on GitHub. 111Implementation at
github.com/dsgt-birdclef/birdclef-2023 The package defines environment
dependencies to run the training and inference workflows. It contains helper
PySpark code [6], wrappers around BirdNET, and utilities for manipulating
audio samples into matrices representing sliding windows. We also define a
workflow package containing Luigi [7] scripts that implement dataset
processing and annotation. We build Docker images for BirdNet and MixIT and
integrate modified versions of the canonical inference script into our data
pipeline as per figure 3.

The inference pipeline is composed of three notebooks. The package sync
notebook downloads the shared Python package with all dependencies into a
local directory. The model sync notebook similarly downloads serialized models
and weights from object storage. The final inference notebook runs offline
after attaching the package and model sync notebooks as data sources. We read
the soundscapes, split them into chunks, obtained BirdNET embeddings, and
computed predictions submitted to the competition.

##  5 Experiments

We run experiments on the embedding dataset to maximize our performance on the
public leaderboard. The feature engineering process of embedding tokens and
prediction logits is part of the model-fitting process. We first focused on a
baseline with minimal modifications to the embedding dataset and then worked
toward overcoming the idiosyncrasies of the training dataset by more complex
feature engineering. The input interval of BirdNET and the output interval of
the competition do not match, so we had to consider this difference. The
former expects 3-second intervals, while the latter expects 5-second
intervals. Our two main approaches are to aggregate the output of models that
represent 3-second intervals and to aggregate input to represent 5-second
intervals.

While we use simple measures such as macro-precision and accuracy to assess
models against the derived dataset in hyperparameter searches, we note that
they did not reflect performance against the public leaderboard. The results
of our derived datasets and models are summarized in table 4 and table 4,
respectively.

column name | value
---|---
track_stem | XC116777
track_type | source1
start_time | 0
primary_label | ratcis1
metadata_species | [ratcis1]
probability | 0.348693
embedding | [0.64499, 0.45046, 0.36006, …
next_embedding | [0.86384, 0.81126, 0.26452, …
track_embedding | [0.67923, 0.53318, 0.36459, …

column name | value
---|---
track_stem | XC213642
track_type | original
start_time | 55
species | afmdov1
embedding | [1.76885, 0.83265, 0.49710, …
prediction_vec | [-14.44950, -12.09912, -15.69485, …

Table 2:  Example output row from the post v6 dataset on the left and v7 on
the right.

Dataset |  Source |  Description
---|---|---
post v1 |  emb v3 |  Simplified dataset where embedding tokens come from the channel with the highest number of positive classifications against the baseline model. Multi-label generated by averaging pairs and triplets of embedding tokens together.
post v2 |  emb v4 |  Tokens are now the average of the first and third tokens of each 5-second interval. We generate current, next, and track embeddings using only source-separated tracks. Tokens are multi-labeled by the primary and secondary species associated with the track.
post v3 |  post v2 |  Augments above but the top-20 tokens in each species are averaged against random no-call tokens to simplify train-test splits.
post v4 |  emb v4 |  Same methodology as post v2 and v3. Multi-label is generated by confident baseline predictions filtered by plausible primary and secondary metadata labels.
post v5 |  emb v4 |  Same as post v4, but it fixes a modulo bug in previous averaged-token datasets.
post v6 |  emb v4 |  Drops notion of multi-label prediction. It uses the original track and the best source-separated track to increase the number of training examples.
post v7 |  emb v4 |  Adds logic to assign the primary label and no-call labels. It uses concatenation instead of interpolation for a 5-second interval token and includes the prediction logits for the current interval.

Table 3:  An overview of changes in the post-processed dataset (post).

Model | Dataset |  Description |  Public Score |  Private Score
---|---|---|---|---
Logistic Regression | emb v3 |  Baseline |  0.78541 |  0.68369
MLP | emb v3 |  Baseline |  0.74014 |  0.62283
XGBoost | post v1 |  Multi-label one-vs-rest strategy. Interpolated token pairs and triplets. |  0.79068 |  0.68181
XGBoost | post v1 |  Same as above, but weighted samples and native multi-label training. |  0.78829 |  0.68053
XGBoost | post v5 |  Current interpolated-token. |  0.7692 |  0.65937
XGBoost | post v5 |  Current and track interpolated-token. |  0.7489 |  0.63059
XGBoost | post v3 |  Current, next, and track interpolated-token. |  0.75049 |  0.63877
XGBoost | post v3 |  Current and next interpolated-token. |  0.76484 |  0.65346
XGBoost | post v7 |  Current concatenated token. |  0.75997 |  0.64414
XGBoost | post v4 |  Ensemble of best logistic regression and boost model. |  0.75091 |  0.64242
Complement Naive Bayes | post v7 |  Current prediction logit softmax vector. |  0.71093 |  0.59652

Table 4: A summary of experiments and their associated datasets.

###  5.1 Baseline Model

The baseline model uses embedding tokens taken from the isolated track source
with the highest energy, assuming that the loudest voice in the track is
associated with the primary label. We label the tokens according to the
primary label if the max probability of the prediction vector exceeds a
threshold, e.g., 0.5; otherwise, we label the token as "no-call". We fit the
data to logistic regression, multi-layer perceptron (MLP), support vector
machine (SVM), and gradient-boosted decision tree (GBDT) classifiers. We use
scikit-learn [8] for the first three of these models and the scikit-learn
compatible interface against XGBoost [9] for the latter. When applicable, we
perform a hyper-parameter search using sci-kit-optimize [10], which performs
sequential optimization using Bayesian methods.

We hypothesize that classes cluster together in the low-dimensional embedding
learned by BirdNET and that linear models can effectively learn to
discriminate between new classes. Our baseline logistic regression classifier
trained on the embedding tokens reaches a public/private score of 0.78/0.68,
notably better than the starter Kaggle notebook using the Google Research Bird
Vocalization Classifier with a score of 0.72/0.61. We find that SVM and GBDT
are comparable to the logistic regression and that MLP models require
significant tuning to reach good performance.

GBDT via XGBoost is our preferred model because it trains quickly with a GPU
with relatively high predictive performance. While logistic regression has
fewer parameters and performs just as well, training can be slow as the number
of examples increases. In table 5, logistic regression takes 12x as long to
train as XGBoost with GPU-based histogram binning.

Model | GPU | Fit time | Predict time
---|---|---|---
Logistic Regression (Newton-Cholesky) | No | 59 min 17 s | 1.5 s
SVC | No | 90 min + | -
MLP | No | 4 min 14 s | 3.5 s
XGBoost (hist) | No | 48 min 20 s | 14.4 s
XGBoost (gpu_hist) | Yes | 5 min | 15.3 s
ComplementNB | No | 4.2 s | 1.5 s
Table 5:  A comparison between fit and predict the time for various models fit
on a GCP n1-standard-4 compute instance with a Telsa T4 GPU. We fit the
post-v7 dataset, which has 255,372 rows.

####  5.1.1 Baseline Binary No-call Model

We explore and analyze the performance of a binary classifier to further our
understanding of the embedding space. While we do not use this model directly
in the competition, it helps quantify the quality of our automated labeling
process.

We construct a binary dataset with positive and negative embedding samples.
Positive signifies the presence of a birdcall in the sample, while negative
denotes its absence. The distribution of positive and negative samples within
the dataset is well-balanced, with 49.7% being positive and 50.3% being
negative. About half of the available training data is empty across original
and sound-separated tracks. A logistic regression classifier achieved an
accuracy of 0.88 on this binary dataset.

We create a second binary dataset from a subset of the first using the top
three most common species, with 49.8% of samples being positive and 50.2%
being negative. A logistic regression classifier is trained and predicted 1.0
accuracy on the smaller dataset. We hypothesize that this behavior is due to
the large number of species in the dataset. The distribution of some species
is highly skewed, as shown in Figure 4.

Furthermore, we utilized the trained logistic regression classifier to make
inferences a background audio embeddings based on the freefield1010 dataset
[11]. The classifier predictions are probability scores indicating the
presence or absence of birdcalls in each sample. It predicted 77.3% of the
samples as having no birdcalls (no-call) and 22.7% as having birdcalls (call).
However, if the classifier’s predictions were perfect, the no-call percentage
should have been 1.0, as the entire dataset consisted only of background
noise.

Figure 4:  We demonstrate the clustering properties of the binary no-call
embeddings by projecting them into $\mathcal{R}^{2}$ via UMAP. The clustering
compares the binary dataset with the overall species versus the top-3 species.

###  5.2 Interpolated Embedding Models

We build upon the baseline embedding model by interpolating embedding tokens
to synthesize new examples and labels. We assume that positive examples from
each class tend to cluster together in the high-dimensional embedding space.
We also assume that embedding space takes on a Euclidean geometry or admits
some approximation. By interpolating examples, we hypothesize that the
resulting coordinate lies between the clusters and, therefore, closer to the
decision boundary of the sources.

We first use interpolation to address the alignment problem between the
3-second interval of BirdNET and the 5-second interval of the competition by
generating a new feature. In addition, we also construct a method that
generates pairs and triplets of tokens sampled evenly across classes. These
interpolated tokens are assigned multiple labels used in a multi-label
classifier.

We experiment with interpolation to add contextual information to each
example. For each token in a track from our dataset $v_{t}$, we generate the
token that follows it directly in time $v_{t+1}$ and the token that represents
the entire track $\sum_{i=0}^{n}v_{i}$. We generate features by concatenating
($\oplus$) each of these tokens and evaluate performance relative to each
other using the same set of labels.

$\displaystyle\hat{y}$ $\displaystyle\sim M_{1}(v_{t})$ (1)
$\displaystyle\hat{y}$ $\displaystyle\sim M_{2}(v_{t}\oplus v_{t+1})$ (2)
$\displaystyle\hat{y}$ $\displaystyle\sim
M_{3}(v_{t}\oplus\sum_{i=0}^{n}v_{i})$ (3) $\displaystyle\hat{y}$
$\displaystyle\sim M_{4}(v_{t}\oplus v_{t+1}\oplus\sum_{i=0}^{n}v_{i})$ (4)

Interpolation may provide some value, in particular around multi-labeling. We
found that our model trained on a dataset does not perform worse than our
baseline model while avoiding issues related to having a small number of
training examples for the class. Most models trained using a form of
interpolated tokens resulted in lower leaderboard scores. However, these
models did not include interpolated pair and triplet tokens.

###  5.3 Concatenated Embedding Model

Another class of models that handles the time-interval discrepancy involves
concatenating the embedding tokens. We must train a classifier that accepts
input in $\mathcal{R}^{2x320}$. This model performs worse than the
interpolated model. The increased dimensional of the underlying feature also
increases the model fit time, which leads us to skip augmented feature sets.

###  5.4 Ensemble Embedding Model

Our last embedding model uses the best models from our baseline and
interpolated embedding experiments. We train an XGBoost model trained on the
outputs from the best classifiers. This model is likely affected by the
quality of the training dataset and the differences in embedding token
semantics.

###  5.5 Probability Logit Model

Our final experiment uses the outputs of the final logit layer in the BirdNET
model to determine a class’ presence directly. We generate a probability
vector by taking a softmax of the logit layer. We ran into out-of-memory
issues on our GPUs fitting XGBoost models using the probability vector in
$\mathcal{R}^{3337}$ and found the scikit-learn logistic regression
implementation needed to be faster. We fit the data using a Complement Naive
Bayes classifier instead of a GBDT or logistic regression classifier. Naive
Bayes assumption works well in this problem, where each feature is treated
independently toward the classification goal. It is also swift because it
simply computes counts over features. We use the Complement Naive Bayes model
to address the heavy skew in the class distribution but also find comparable
performance across this family of classifiers. The predictive performance is
far worse than the baseline, with a public/private score of 0.71/0.59.

##  6 Discussion

###  6.1 Semi-Supervised Annotation Quality

The data quality is an aspect of our training dataset that we would like to
explore more deeply because the data quality affects the model’s quality. We
have built a training workflow that enables flexibility in labeling timestamps
across the entire training audio to 1-second granularity. While we succeeded
in our baseline transfer learning experiment, having a human-labeled test
dataset independent of the competition leaderboard would be helpful. We can
use ground-truth annotations to test how well an automated labeling process
does at assigning primary, secondary, and no-call labels.

It would also be worth exploring human annotation in the sound separation
process. While we have listened to several examples to judge the quality of
sound separation, we need a method to quantify quality. In the case of sorting
by the highest energy source, we may confuse a source with a significant
amount of noise for the true birdcall just by the nature of higher entropy in
the source. In the case of sorting by the highest number of matching
classifications from an existing model, we may need to produce a classifier
that can distinguish between noise and birdcall for rare classes. In this
case, we continue to propagate uncertainty into the resulting labels, leading
to poor performance, mainly when the number of examples is small.

We could introduce a metric to measure the resulting separation quality
rigorously. We could create many positive birdcall clips and have a human
determine which channel mainly contains the primary species. These clips let
us see how well our automated channel selection process does at choosing the
right channel based on human-generated labels. However, this metric would not
allow us to determine the separation quality in degenerate cases where a
single separated source contains multiple bird vocalizations.

###  6.2 Audio Source Separation

The MixIT source separation model plays a significant role in the embedding
pipeline used for our experiments. One of the most significant benefits is
noise suppression. We also relied on sound-separated channels to generate
training examples and semi-supervised labels. Running an ablation study by
evaluating the labeling process without access to the sound separation model
would have been helpful.

We wanted to explore the 8-source model, which could have different
performance characteristics than the 4-source model. However, we decided to
ignore this model because the training examples generally have few distinct
vocalizations, and creating more source channels increases the necessary disk
space for the intermediate files.

We are also interested in the effect of the sound separation model during
inference. However, we observed that the sound separation stage in training
was a significant fraction of the computation budget. There are also issues
with different sample rates. BirdNET expects audio sampled at 48kHz, while
MixIT expects audio at 32KHz. We did not attempt to integrate the model into
this project because it would have put us over the submission time limit and
required significant engineering effort to run inside the competition
environment.

###  6.3 Embedding Space and Transfer Learning

Our experiments with interpolated embeddings in section 5.2 had mixed results
with respect to the baseline embedding model. Including embedding context from
other time intervals had substantially lower performance than our baseline.
The new labeling process may have overshadowed potential positive effects. On
the other hand, we saw a slight increase in our model performance when using
the mean of pairs and triplets during model fitting. Further research is
necessary to determine whether it is valuable to manipulate embeddings
similarly to mix up [12] that interpolates between training examples in the
time domain to increase and augment training data.

We want to experiment with the embeddings from another model, such as the
Google Research Bird Vocalization Classifier. This classifier has seen more of
the species in this competition than BirdNET. It would be helpful to see how
these two embeddings compare, and we could try this out as another feature.

Sequential models could be helpful in the competition by capturing dynamics
and imbuing contextual information between embedding tokens. The simplest
model would be an auto-regressive linear model using an embedding from a
single timestep to predict the next timestep optimized by a squared error
loss. We made initial forays into attention-based sequence-to-sequence models
to address the output time-interval issue, but we needed more time to complete
our experimentation. Future work might explore data-driven methods like HAVOC
[13] to analyze the dynamics of birdcall audio and their embeddings.

We would also like to explore the relationship between the sound-separated
tracks and the embedding. The separation model is constrained so that the sum
of the sources results in the original track. It would be interesting to
verify a relationship between embeddings of various tracks by fitting a
predictive model that takes tokens from each source to predict the embedding
of the original track. This line of thought does not directly help with model
performance on the final task, but it does help understand the nature of the
classifier embedding space.

##  7 Conclusion

In summary, our approach leveraged the embedding space learned by BirdNET to
address the representation and labeling challenges in the competition. We
developed a pipeline that included sound separation with MixIT, extraction of
embedding tokens using BirdNET, and the creation of annotated datasets. Our
results showcased the competitive performance of our logistic regression
baseline model as an effective method on unseen species and the comparative
performance of various feature engineering work. Our approach demonstrated the
potential of transfer and semi-supervised learning for bird species
classification in soundscapes.

###### Acknowledgements.

Thanks to the Data Science at Georgia Tech (DS@GT) club for hosting our Kaggle
competition team. Thanks to DS@GT leadership for publicizing recruitment,
particularly Krishi Manek as the Director of Projects. Thanks to Erin
Middlemas and Grant Williams for their support and engagement as initial team
members.

## References

  * Kahl et al. [2023] S. Kahl, T. Denton, H. Klinck, H. Reers, F. Cherutich, H. Glotin, H. Goëau, W.-P. Vellinga, R. Planqué, A. Joly,  Overview of BirdCLEF 2023: Automated bird species identification in eastern africa,  Working Notes of CLEF 2023 - Conference and Labs of the Evaluation Forum (2023).
  * Joly et al. [2023] A. Joly, C. Botella, L. Picek, S. Kahl, H. Goëau, B. Deneu, D. Marcos, J. Estopinan, C. Leblanc, T. Larcher, R. Chamidullin, M. Šulc, M. Hrúz, M. Servajean, H. Glotin, R. Planqué, W.-P. Vellinga, H. Klinck, T. Denton, I. Eggel, P. Bonnet, H. Müller,  Overview of LifeCLEF 2023: evaluation of AI models for the identification and prediction of birds, plants, snakes and fungi,  in: International Conference of the Cross-Language Evaluation Forum for European Languages, Springer, 2023.
  * Kahl et al. [2021] S. Kahl, C. M. Wood, M. Eibl, H. Klinck,  Birdnet: A deep learning solution for avian diversity monitoring,  Ecological Informatics 61 (2021) 101236.
  * McInnes et al. [2020] L. McInnes, J. Healy, J. Melville, Umap: Uniform manifold approximation and projection for dimension reduction, 2020. arXiv:1802.03426.
  * Denton et al. [2022] T. Denton, S. Wisdom, J. R. Hershey,  Improving bird classification with unsupervised sound separation,  in: ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE, 2022, pp. 636–640.
  * Armbrust et al. [2015] M. Armbrust, R. S. Xin, C. Lian, Y. Huai, D. Liu, J. K. Bradley, X. Meng, T. Kaftan, M. J. Franklin, A. Ghodsi, et al.,  Spark sql: Relational data processing in spark,  in: Proceedings of the 2015 ACM SIGMOD international conference on management of data, 2015, pp. 1383–1394.
  * Lui [2023] Luigi 2.8.13 documentation, https://luigi.readthedocs.io/en/stable/, 2023\. Accessed: 2023-06-07.
  * Pedregosa et al. [2011] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, E. Duchesnay,  Scikit-learn: Machine learning in Python,  Journal of Machine Learning Research 12 (2011) 2825–2830.
  * Chen and Guestrin [2016] T. Chen, C. Guestrin,  XGBoost: A scalable tree boosting system,  in: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’16, ACM, New York, NY, USA, 2016, pp. 785–794. URL: http://doi.acm.org/10.1145/2939672.2939785. doi:10.1145/2939672.2939785.
  * Head et al. [2018] T. Head, MechCoder, G. Louppe, I. Shcherbatyi, fcharras, Z. Vinícius, cmmalone, C. Schröder, nel215, N. Campos, T. Young, S. Cereda, T. Fan, rene rex, K. K. Shi, J. Schwabedal, carlosdanielcsantos, Hvass-Labs, M. Pak, SoManyUsernamesTaken, F. Callaway, L. Estève, L. Besson, M. Cherti, K. Pfannschmidt, F. Linzberger, C. Cauet, A. Gut, A. Mueller, A. Fabisch, scikit-optimize/scikit-optimize: v0.5.2, 2018\. URL: https://doi.org/10.5281/zenodo.1207017. doi:10.5281/zenodo.1207017.
  * Stowell and Plumbley [2013] D. Stowell, M. D. Plumbley,  An open dataset for research on audio field recording archives: freefield1010,  arXiv preprint arXiv:1309.5275 (2013).
  * Zhang et al. [2018] H. Zhang, M. Cisse, Y. N. Dauphin, D. Lopez-Paz, mixup: Beyond empirical risk minimization, 2018. arXiv:1710.09412.
  * Brunton et al. [2017] S. L. Brunton, B. W. Brunton, J. L. Proctor, E. Kaiser, J. N. Kutz,  Chaos as an intermittently forced linear system,  Nature communications 8 (2017) 19.


11institutetext: University of Bonn 11email<EMAIL_ADDRESS>11email:
<EMAIL_ADDRESS>22institutetext: National Chung Hsing University, Taiwan
22email<EMAIL_ADDRESS>33institutetext: University of Bonn & Fraunhofer
IAIS, Germany 33email<EMAIL_ADDRESS>
# Mapping Large Scale Research Metadata to Linked Data: A Performance
Comparison of HBase, CSV and XML

Sahar Vahdati 11 Farah Karim 11 Jyun-Yao Huang 22 Christoph Lange 33

###### Abstract

OpenAIRE, the Open Access Infrastructure for Research in Europe, comprises a
database of all EC FP7 and H2020 funded research projects, including metadata
of their results (publications and datasets). These data are stored in an
HBase NoSQL database, post-processed, and exposed as HTML for human
consumption, and as XML through a web service interface. As an intermediate
format to facilitate statistical computations, CSV is generated internally. To
interlink the OpenAIRE data with related data on the Web, we aim at exporting
them as Linked Open Data (LOD). The LOD export is required to integrate into
the overall data processing workflow, where derived data are regenerated from
the base data every day. We thus faced the challenge of identifying the best-
performing conversion approach. We evaluated the performances of creating LOD
by a MapReduce job on top of HBase, by mapping the intermediate CSV files, and
by mapping the XML output.

##  1 Introduction

11todo: 1CL: re-enable keywords if required

The European Commission emphasizes open access as a key tool to bring together
people and ideas in a way that catalyses science and innovation. More than
ever before, there is a recognized need for digital research infrastructures
for all kinds of research outputs, across disciplines and countries. OpenAIRE,
the Open Access Infrastructure for Research in Europe
(http://www.openaire.eu), (1) manages scientific publications and associated
scientific material via repository networks, (2) aggregates Open Access
publications and links them to research data and funding bodies, and (3)
supports the Open Access principles via national helpdesks and comprehensive
guidelines.

Data related to those in the OpenAIRE information space exist in different
places on the Web. Combining them with OpenAIRE will enable new use cases. For
example, understanding changes of research communities or the emergence of
scientific topics not only requires metadata about publications and projects,
as provided by OpenAIRE, but also data about events such as conferences as
well as a knowledge model of research topics and subjects (cf. [1]).

The availability of data that is free to use, reuse and redistribute (i.e.
_open data_) is the first prerequisite for analysing such information
networks. However, the diverse data formats and means to access or query data,
the use of duplicate identifiers, and the heterogeneity of metadata schemas
pose practical limitations on reuse. Linked Data, based on the RDF graph data
model, is now increasingly accepted as a lingua franca to overcome such
barriers [2].

The University of Bonn is coordinating the effort of publishing the OpenAIRE
data as Linked Open Data (LOD) and linking it to related datasets in the
rapidly growing LOD Cloud111http://lod-cloud.net. This effort is further
supported by the Athena Research and Innovation Center and CNR-ISTI. Besides
data about scientific events and subject classification schemes, relevant data
sources include public sector information (e.g., to find research results
based on the latest employment statistics, or to answer questions such as ‘how
do the EU member states’ expenses for health research compare to their health
care spendings?’) and open educational resources (‘how soon do emergent
research topics gain wide coverage in higher education?’).

Concrete steps towards this vision are (1) mapping the OpenAIRE data model to
suitable standard LOD vocabularies, (2) exporting the objects in the OpenAIRE
information space as a LOD graph and (3) facilitating integration with related
LOD graphs. Expected benefits include

  * •

enabling semantic search over the outputs of European research projects,

  * •

simplifying the way the OpenAIRE data can be enriched by third-party services,
and consumed by interested data or service providers,

  * •

facilitated outreach to related open content and open data initiatives, and

  * •

enriching the OpenAIRE information space itself by exploiting how third
parties will use its LOD graph.

The specifically tailored nature of the OpenAIRE infrastructure, its large
amount of data (covering more than 11 million publications) and the frequent
updates of the more than 5000 repositories from which the data is harvested
pose high requirements on the technology chosen for mapping the OpenAIRE data
to LOD. We therefore compared in depth three alternative mapping methods, one
for each source format in which the data are available: HBase, CSV and XML.

Section 2 introduces the OpenAIRE data model and the three existing data
sources. Section 3 presents our specification of the OpenAIRE data model as an
RDF vocabulary. Section 4 establishes requirements for the mapping. Section 5
presents the state of the art for each of the three mapping approaches.
Section 6 explains our three implementations. In section 7 we evaluate them in
comparison, with regard to different metrics induced by the requirements.
Section 8 reviews work related to our overall approach (comparing mappings and
producing research LOD). Section 9 concludes and outlines future work.

##  2 Input Data

The data model of OpenAIRE infrastructure is specified as an entity
relationship model (ERM) [3, 4] with the following entity categories:

  * •

Main entities (cf. figure 1)222https://issue.openaire.research-
infrastructures.eu/projects/openaire2020-wiki/wiki/Core_Data_Model: Result
(Publication or Dataset), Person, Organization, Projects, and DataSource (e.g.
Repository, Dataset Archive or CRIS333Current research information system, a
system to manage information about the research activity of an institution).
Instances of these are continuously harvested from data providers.

  * •

Structural entities representing complex information about main entities:
Instances (of a Result in different DataSources), WebResources, Titles, Dates,
Identities, and Subjects.

  * •

Static entities, whose metadata do not change over time: Funding. E.g., once a
funding agency has opened a funding stream, it remains static.

  * •

Linking entities represent relationships between entities that carry further
metadata; e.g., an entity of type Person_Result whose property ranking has the
value 1 indicates the first author.

Figure 1: OpenAIRE Data Model: core entities and relationships

So far, the OpenAIRE data have been available in three formats: HBase, CSV and
XML.

###  2.1 HBase

Currently, the master source of all OpenAIRE data is kept in HBase, a column
store based on HDFS (Hadoop Distributed File System). HBase was introduced in
2012 when data integration efforts pushed the original PostgreSQL database to
its limits: joins became inefficient and parallel processing, as required for
deduplication, was not supported. Each row of the HBase table has a unique row
key and stores a main entity and a number of related linked entities. The
attribute values of the main entities are stored in the <family>:body column,
where the <family> is named after the type of the main entity, e.g., result,
person, project, organization or datasource. The attribute values of linked
entities, indicating the relationship between main entities, are stored in
dedicated column families <family>:<column>, where <family> is the class of
the linked entity and <column> is the row key of the target entity. Both
directions of a link are represented. Cell values are serialized as byte
arrays according to the Protocol Buffers [5] specification; for example:

⬇

message Person {

optional Metadata metadata = 2;

message Metadata {

optional StringField firstname = 1;

repeated StringField secondnames = 2;

optional Qualifier nationality = 9; ... }

repeated Person coauthors = 4; }

The following table shows a publication and its authors. For readability, we
abbreviated row keys and spelled out key-value pairs rather than showing their
binary serialization.

RowKey |  result: |  person: | …hasAuthor: |  …isAuthorOf:
---|---|---|---|---
|  body |  body |  30|…001::9897… |  30|…001::ef29… |  50|…001::39b9…
50|…0 01::39 b9… |  resulttype= “publication”; title=“The Data Model of …”; dateofacceptance= “2012-01-01”; language=“en”; publicationDate= “2012”; publisher= “Springer”; |  |  ranking=1; |  ranking=2; |
30|…0 01::98 97… |  |  firstname=“Paolo”; lastname=“Manghi”; |  |  |  ranking=1;
30|…0 01::ef 29… |  |  firstname=“Nikos”; lastname=“Houssos”; |  |  |  ranking=2;

###  2.2 CSV

CSV files aid the computation of statistics on the OpenAIRE information space.
HBase is a sparse key value-store designed for data with little or no internal
relations. Therefore, it is impossible to run complex queries directly on top
of HBase, for example a query to find all results of a given project. It is
thus necessary to transform the data to a relational representation, which is
comprehensible for statistics tools and enables effective querying. Via an
intermediate CSV representation, the data is imported into a relational
database, which is queried for computing the statistics.

In this generation process, each main entity type (result, project, person,
organization, datasource) is mapped to a CSV file of the same name, which is
later imported into a relational database table. Each single-valued attribute
of an entity (id, title, publication year, etc.) becomes a field in the
entity’s table. Multi-valued attributes, such as the publication languages of
a result, are mapped to relation tables (e.g. result_languages) that represent
a one-to-many relation between entity and attributes. Linked entities, e.g.
the authors of a result, are represented similarly. As the data itself
includes many special characters, for example commas in publication titles,
the OpenAIRE CSV files use ! as a delimiter and wrap cell values into leading
and trailing hashes:

⬇

#dedup_wf_001::39b91277f9a2c25b1655436ab996a76b#!#The Data Model of the
OpenAIRE

Scientific Communication
e-Infrastructure#!#null#!#null#!#Springer#!#null#!#null

#!#null#!#null#!#2012#!#2012-01-01#!#Open Access#!#Open
Access#!#Access#!#null#!#

0#!#null#!#nulloai:http://helios-
eie.ekt.gr:!#publication#10442/13187oai:pumaoai.

isti.cnr.it:cnr.isti/cnr.isti/2012-A2-040#!#1#!

Finally, using CSV has the advantage that existing tools such as Sqoop can be
used, thus reducing the need to develop and maintain customly implemented
components on the OpenAIRE production system.

###  2.3 XML

OpenAIRE features a set of HTTP APIs444http://api.openaire.eu/ for exporting
metadata as XML for easy reuse by web services. These APIs use an XML Schema
implementation of the OpenAIRE data model called OAF (OpenAIRE
Format)555https://www.openaire.eu/schema/0.2/doc/oaf-0.2.html, where each
record represents one entity. There is one API for searching, and one for bulk
access. For example, the listing below comes from
http://api.openaire.eu/search/publications
?openairePublicationID=dedup_wf_001::39b91277f9a2c25b1655436ab996a76b and
shows the metadata of a publication that has been searched for.

⬇

<oaf:result>

<title schemename="dnet:dataCite_title" classname="main title"

schemeid="dnet:dataCite_title" classid="main title">The Data Model of the

OpenAIRE Scientific Communication e-Infrastructure</title>

<dateofacceptance>2012-01-01</dateofacceptance>

<publisher>Springer</publisher>

<resulttype schemename="dnet:result_typologies" classname="publication"

schemeid="dnet:result_typologies" classid="publication"/>

<language schemename="dnet:languages" classname="English"

schemeid="dnet:languages" classid="eng"/>

<format>application/pdf</format>

...

</oaf:result><

The API for bulk access uses OAI-PMH (The Open Archives Initiative Protocol
for Metadata
Harvesting)666http://www.openarchives.org/OAI/openarchivesprotocol.html to
publish metadata and its corresponding endpoint is at
http://api.openaire.eu/oai_pmh. The bulk access API lets developers fetch the
whole XML files step by step. For our experiments, we obtained the XML data
directly from the OpenAIRE server, as an uncompressed Hadoop
SequenceFile777http://wiki.apache.org/hadoop/SequenceFile comprising 500
splits of $\sim$300 MB each.

##  3 Implementing the OpenAIRE Data Model in RDF

As the schema of the OpenAIRE LOD we specified an RDF vocabulary by mapping
the entities of the ER data model to RDF classes and its attributes and
relationships to RDF properties. We reused suitable existing RDF vocabularies
identified by consulting the Linked Open Vocabularies search
service888http://lov.okfn.org and studying their specifications. Reused
vocabularies include Dublin Core for general metadata,
SKOS999http://www.w3.org/2004/02/skos/ for classification schemes and
CERIF101010Common European Research Information Format; see
http://www.eurocris.org/cerif/main-features-cerif for research organizations
and activities. We linked new, OpenAIRE-specific terms to reused ones, e.g.,
by declaring Result a superclass of http://purl.org/ontology/bibo/Publication
and http://www.w3.org/ns/dcat#Dataset.

We keep the URIs of the LOD resources (i.e. entities) in the
http://lod.openaire.eu/data/ namespace. We modelled them after the HBase row
keys. In OpenAIRE, these are fixed length identifiers of the form
{typePrefix}|{namespacePrefix} ::md5hash. typePrefix is a two digit code, 10,
20, 30, 40 or 50, corresponding to the main entity types datasource,
organization, person, project and result. The namespacePrefix is a unique
12-character identifier of the data source of the entity. For each row,
md5hash is computed from the entity attributes. The resulting URIs look like
http://lod.openaire.eu/data/result/dedup_wf_001::39b9127
7f9a2c25b1655436ab996a76b.

The following listing shows our running example in RDF/Turtle syntax.

⬇

@prefix oad: <http://lod.openaire.eu/data/> .

@prefix oav: <http://lod.openaire.eu/vocab#> .

# further prefixes omitted; see http://prefix.cc for their standard bindings.

oad:result/...001::39b9... rdf:type oav:Result, bibo:Publication;

dcterms:title "The Data Model of the OpenAIRE Scientific Communication

e-Infrastructure"@en ;

dcterms:dateAccepted "2012-01-01"^^xsd:date ;

dcterms:language "en";

oav:publicationYear 2012 ;

dcterms:publisher "Springer";

dcterms:creator oad:person/...001::9897..., oad:person/...001::ef29... .

oad:person/...001::9897... rdf:type foaf:Person;

foaf:firstName "Paolo"; foaf:lastName "Manghi";

oav:isAuthorOf oad:result/...001::39b9... .

oad:person/...001::ef29... rdf:type foaf:Person;

foaf:firstname "Nikos"; foaf:lastName "Houssos";

oav:isAuthorOf oad:result/...001::39b9... .

##  4 Requirements

In cooperation with the other technical partners in the OpenAIRE2020
consortium, most of whom had been working on the infrastructure in previous
projects for years, we established the following requirements for the LOD
export:

  1. R1

The LOD output must follow the vocabulary specified in section 3.

  2. R2

The LOD must be generated from one of the three existing data sources, to
avoid extra pre-processing costs.

  3. R3

The mapping to LOD should be maintainable w.r.t. planned extensions of the
OpenAIRE data model (such as linking publications and data to software) and
the evolution of linked data vocabularies.

  4. R4

The mapping to LOD should be orchestrable together with the other existing
OpenAIRE data provision workflows, always exposing a consistent view on the
information space, regardless of the format.

  5. R5

To enable automatic and manual checks of the consistency and correctness of
the LOD before its actual publication, it should be made available in
reasonable time in a private space.

To prepare an informed decision on the preferred input format to use for the
LOD export, we realised one implementation for each of HBase, CSV and XML.

##  5 Technical State of the Art

For each possible approach, i.e. mapping HBase, CSV or XML to RDF, we briefly
review the state of the art to give an overview of technology we could
potentially reuse or build on, whereas section 8 reviews work related to our
overall approach. We assess reusability w.r.t. the OpenAIRE-specific
requirements stated above.

HBase, being a sparse, distributed and multidimensional persistent sorted map,
provides dynamic control over the data format and layout. Several works have
therefore explored the suitability of HBase as a triple store for semi-
structured and sparse RDF data. Sun et al. adopted the idea of the Hexastore
indexing technique for storing RDF in HBase [6]. Khadilkar et al. focused on a
distributed RDF storage framework based on HBase and Jena to gain
scalability[7]. Others have provided MapReduce implementations to process
SPARQL queries over RDF stored in HBase [8, 9].

We are only aware of one work on exposing data from column-oriented stores as
RDF. Kiran et al. provide a method for generating a SPARQL endpoint, i.e. a
standardized RDF query interface, on top of HBase [10]. They map tables to
classes, rows to resources, and columns to properties. Their approach do not
scale well with increasing numbers of HBase entries, as the results show that
the time taken to map HBase data to RDF is in hours for a few million rows
[10].

CSV is widely used for publishing tabular data [11]. The CSV on the Web W3C
Working Group111111http://www.w3.org/2013/05/lcsv-charter.html provides
technologies for data dependent applications on the Web working with CSV.
Several existing implementations, including that of Anything To Triples
(any23)121212http://any23.apache.org, map CSV to a generic RDF representation.
Customizable mappings are more suitable for our purpose. In Tarql
(Transformation SPARQL)131313https://tarql.github.io, one can define such
mappings in SPARQL; Tabels (Tabular
Cells)141414http://idi.fundacionctic.org/tabels and
Sparqlify151515https://github.com/AKSW/Sparqlify [12] use domain-specific
languages similar to SPARQL. Tabels provides auxiliary machinery to filter and
compare data values during the transformation process. Sparqlify is mainly
designed to map relational databases to RDF but also features the sparqlify-
csv module.

XML is used for various data and document exchange purposes. Like for
CSV$\to$RDF, there are generic and domain-specific XML$\to$RDF approaches.
Breitling implemented a direct, schema-independent transformation, which
retains the XML structure [13]. Turning this generic RDF representation into a
domain-specific one requires post-processing on the RDF side, e.g.,
transformations using SPARQL CONSTRUCT queries. On the other hand, the current
version of Breitling’s approach is implemented in XSLT 1.0, which does not
support streaming and is therefore not suitable for the very large inputs of
the OpenAIRE setting. Klein uses RDF Schema to map XML elements and attributes
to RDF classes and properties [14]. It does not automatically interpret the
parent-child relation between two XML elements as a property between two
resources, but a lot of such relationships exist in the OpenAIRE XML. XSPARQL
can transform XML to RDF and back by combining the XQuery and SPARQL query
languages to [15]; authoring mappings requires good knowledge of both. By
supporting XQuery’s expressive mapping constructs, XSPARQL requires access to
the whole XML input via its DOM (Document Object Model), which results in
heavy memory consumption. A subset of XQuery161616cf. ‘Streaming in XQuery’,
http://www.saxonica.com/html/documentation/sourcedocs/streaming/streamed-
query.html is suitable for streaming but neither supported by the XSPARQL
implementation nor by the free version of the Saxon XQuery processor required
to run XSPARQL.

##  6 Implementation

As the only existing HBase$\to$RDF implementation does not scale well (cf.
section 5), we decided to follow the MapReduce paradigm for processing massive
amounts of data in parallel over multiple nodes. We implemented a single
MapReduce job. Its mapper reads the attributes and values of the OpenAIRE
entities from their protocol buffer serialization and thus obtains all
information required for the mapping to RDF. Hence no reducer is required. The
map-only approach performs well thanks to avoiding the computationally
intensive shuffling. RDF subjects are generated from row keys, predicates and
objects from attribute names and cell values or, for linked entities, from
column families/qualifiers.

Mapping the OpenAIRE CSV$\to$RDF is straightforward: files correspond to
classes, columns to properties, and each row is mapped to a resource. We
initially implemented mappings in Tarql, Sparqlify and Tabels (cf. section 5)
and ended up preferring Tarql because of its good performance171717Tabels
failed to handle large CSV files because it loads all the data from the CSV
into main memory; Sparqlify works similar to Tarql but with almost doubled
execution time (7,659 s) and more than doubled memory usage. and the most
flexible mapping language – standard
SPARQL181818http://www.w3.org/TR/sparql11-query/ with a few extensions. As we
_map_ CSV$\to$RDF, as opposed to _querying_ CSV like RDF, we implemented
CONSTRUCT queries, which specify an RDF template in which, for each row of the
CSV, variables are instantiated with the cell values of given columns.

To enable easy maintenance of XML$\to$RDF mappings by domain experts, and
efficient mapping of large XML inputs, we implemented our own
approach191919See source code and documentation at
https://github.com/allen501pc/XML2RDF.. It employs a SAX parser and thus
supports streaming. Our mapping language is based on RDF triple templates and
on the XPath202020http://www.w3.org/TR/xpath20/ language for addressing
content in XML. XPath expressions in the subjects or objects of RDF triple
templates indicate where in the XML they obtain their values from. To keep
XPath expressions simple and intuitive, we allow them to be ambiguous, e.g.,
by saying that oaf:result/publisher/text() (referring to the text content of
the publisher element of a result) maps to the dcterms:publisher property of
an oav:Result, and that oaf:result/dateofacceptance/text() maps to
dcterms:dateAccepted. In theory, any combination of publisher and
dateofacceptance elements would match such a pattern; however in reality only
those nodes that have the shortest distance in the XML document tree represent
attributes of the _same_ OpenAIRE entity. XML Filters [16] efficiently
restrict the XPath expressions to such combinations.

##  7 Evaluation

###  7.1 Comparison Metrics

The time it takes to transform the complete OpenAIRE input data to RDF is the most important performance metric (requirement R4). The main memory usage of the transformation process is important because OpenAIRE2020 envisages the development of further services sharing the same infrastructure, including deduplication, data mining to measure research impact, classification of publications by machine learning, etc. One objective metric for maintainability is the size of the mapping’s source code – after stripping comments and compression, which makes the comparison ‘independent of arbitrary factors like lengths of identifiers and amount of whitespace’ [17].212121We used tar cf - <input files> | xz -9. For HBase, we considered the part of the Java source code that is concerned with declaring the mapping, whereas our CSV and XML mappings are natively defined in high-level mapping languages. The ‘cognitive dimensions of notation’ (CD) evaluation framework provides further criteria for systematically assessing the ‘usability of information artefacts’ [18]. The following dimensions are straightforward to observe here: _closeness_ of the notation to the problem (here: mapping HBase/CSV/XML to RDF), _terseness_ (here measured by code size; see above), _error-proneness_ , _progressive evaluation_ (i.e. whether one can start with an incomplete mapping rule and evolve it to further completeness), and _secondary notation and escape from formalism_ (e.g. whether reading cues can be given by non-syntactic means such as indentation or comments).

###  7.2 Evaluation Setup

The HBase$\to$RDF evaluation ran on a Hadoop cluster of 12 worker nodes
operated by CNR.222222https://issue.openaire.research-
infrastructures.eu/projects/openaire/wiki/Hadoop_Clusters#section-3 As our
CSV$\to$RDF and XML$\to$RDF implementations required dependencies not yet
installed there, we evaluated them locally: on a virtual machine on a server
with an Intel Xeon E5-2690 CPU, having 3.7 GB memory and 250 GB disk space
assigned and running Linux 3.11 and JDK 1.7. As we did not have a cluster
available, and as the tools employed did not natively support parallelization,
we ran the mappings from CSV and XML sequentially.

###  7.3 Measurements and Observations

The following table lists our measurements; further observations follow below.

Objective Comparison Metrics | HBase | CSV |  XML Mapping Time(s) | 1,043 | 4,895 | 45,362 Memory (MB) | 68,000 | 103 | 130 Compressed Mapping Source Code (KB) | 4.9 | 2.86 | 1.67 Number of Input rows/records | 20,985,097 | 203,615,518 | 25,182,730 Number of Generated RDF Triples | 655,328,355 | 654,193,273 | 788,953,122
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---

For HBase$\to$RDF, the peak memory usage of the cluster was 68 GB, i.e.
$\sim$5.5 GB per worker node. No other MapReduce job was running on the
cluster at the same time; however, the usage figure includes the memory used
by the Hadoop framework, which schedules and monitors job execution.

The 20 CSV input files correspond to different entities but also to
relationships. This, plus the way multi-valued attributes are represented (cf.
section 2.2), causes the high number of input rows. The size of all files is
33.8 GB. The XML$\to$RDF memory consumption is low because of stream
processing. The time complexity of our mapping approach depends on the number
of rules (here: 118) and the size of the input (here: 144 GB). With the
complexity of the XML representation, this results in an execution time of
more than 12 hours. The size of the single RDF output file is $\sim$91 GB.
Regarding _cognitive dimensions_ , the different notations expose the
following characteristics; for lack of space we focus on selected highlights.
_Terseness_ : the high-level CSV$\to$RDF and XML$\to$RDF languages fare better
than the Java code required for HBase$\to$RDF. Also, w.r.t. _closeness_ , they
enable more intuitive descriptions of mappings. As the CSV$\to$RDF mappings
are based on SPARQL, which uses the same syntax for RDF triples than the
Turtle RDF serialization, they look closest to RDF. _Error-proneness_ :
Syntactically correct HBase$\to$RDF Java code may still define a semantically
wrong mapping. In Tarql’s CSV$\to$RDF mappings, many types of syntax and
semantics errors can be detected easily. _Progressive evaluation_ : one can
start with an incomplete Tarql mapping rule CSV$\to$RDF mapping rule and
evolve it towards completeness. _Secondary notation_ : Tarql and Java support
flexible line breaks, indentation and comments, whereas our current
XML$\to$RDF mapping implementation requires one (possibly long) line per
mapping rule. Overall, this strongly suggests that CSV$\to$RDF is the most
maintainable approach.

##  8 Related Work

Comparisons of different approaches of mapping data to RDF have mainly been
carried out for relational databases as a source [19, 20]. Similarly to our
evaluation criteria, the reference comparison framework of the W3C RDB2RDF
Incubator Group covers mapping creation, representation and accessibility, and
support for data integration [21]. Hert et al. compared different RDB2RDF
mapping languages w.r.t. syntactic features and semantic expressiveness [22].

For other linked datasets about research, we refer to the ‘publication’ and
‘government’ sectors of the LOD Cloud, which comprises, e.g., publication
databases such as DBLP, as well as snapshots of funding databases such as
CORDIS. From this it can be seen that OpenAIRE is a more comprehensive data
source than those published as LOD before.

##  9 Conclusion and future work

We have mapped a recent snapshot of the OpenAIRE data to RDF. A preliminary
dump as well as the definitions of the mappings are available online at
http://tinyurl.com/OALOD. Mapping from HBase is fastest, whereas mapping from
CSV promises to be most maintainable. Its slower execution time is partly due
to the less powerful hardware on which we ran it; comparing multiple
CSV$\to$RDF processes running in parallel to the HBase$\to$RDF implementation
on the CNR Hadoop cluster seems promising. Based on these findings the
OpenAIRE2020 LOD team will decide on the preferred approach for providing the
OpenAIRE data as LOD; we will then make the data available for browsing from
their OpenAIRE entity URIs, and for querying via a SPARQL endpoint.

Having implemented almost the whole OpenAIRE data model, future steps include
interlinking the output with other existing datasets. E.g., we so far output
countries and languages as strings, whereas DBpedia and Lexvo.org are suitable
linked open datasets for such terms. Link discovery tools will further enable
large-scale linking against existing ‘publication’ and ‘government’ datasets.

#### Acknowledgements.

We would like to thank the partners in the OpenAIRE2020 project, in particular
Claudio Atzori, Alessia Bardi, Glykeria Katsari and Paolo Manghi, for their
help with accessing the OpenAIRE data. This work has been partially funded by
the European Commission under grant agreement no. 643410.

## References

  * [1] F. Osborne and E. Motta, “Understanding research dynamics,” in Semantic Web Evaluation Challenges 2014, no. 457 in CCIS, Springer, 2014.
  * [2] F. Scharffe, G. Atemezing, R. Troncy, F. Gandon, S. Villata, B. Bucher, F. Hamdi, L. Bihanic, G. Képéklian, and F. Cotton, “Enabling linked data publication with the Datalift platform,” in Proc. AAAI workshop on semantic cities, 2012.
  * [3] P. Manghi, M. Mikulicic, and C. Atzori, “Openaire data model specification,” deliverable.
  * [4] P. Manghi, N. Houssos, M. Mikulicic, and B. Jörg, “The data model of the openaire scientific communication e-infrastructure,” in Metadata and Semantics Research, Springer, 2012.
  * [5] “Protocol buffers,” 2015.
  * [6] J. Sun and Q. Jin, “Scalable rdf store based on hbase and mapreduce,” in Advanced Computer Theory and Engineering (ICACTE), vol. 1, IEEE, 2010.
  * [7] V. Khadilkar, M. Kantarcioglu, B. Thuraisingham, and P. Castagna, “Jena-hbase: A distributed, scalable and efficient rdf triple store,” in ISWC Posters & Demonstrations, 2012.
  * [8] N. Papailiou, I. Konstantinou, D. Tsoumakos, and N. Koziris, “H2rdf: adaptive query processing on rdf data in the cloud.,” in International conference on World Wide Web, ACM, 2012.
  * [9] A. Haque and L. Perkins, “Distributed rdf triple store using hbase and hive,” University of Texas at Austin, 2012.
  * [10] K. V. K. and D. G. S. Sadasivam, “A novel method for dynamic sparql endpoint generation in nosql databases,” Australian Journal of Basic and Applied Sciences, vol. 9, no. 6, 2015.
  * [11] T. Lebo and G. T. Williams, “Converting governmental datasets into linked data,” in I-Semantics, ACM, 2010.
  * [12] I. Ermilov, S. Auer, and C. Stadler, “Csv2rdf: User-driven csv to rdf mass conversion framework,” in I-Semantics, 2013.
  * [13] F. Breitling, “A standard transformation from xml to rdf via xslt,” Astronomische Nachrichten, vol. 330, no. 7, 2009.
  * [14] M. Klein, “Interpreting xml documents via an rdf schema ontology,” in DEXA, 2002.
  * [15] S. Bischof, S. Decker, T. Krennwallner, N. Lopes, and A. Polleres, “Mapping between rdf and xml with xsparql,” Journal on Data Semantics, vol. 1, no. 3, 2012.
  * [16] Y. Diao, P. Fischer, M. J. Franklin, and R. To, “Yfilter: Efficient and scalable filtering of xml documents,” in Data Engineering, IEEE, 2002.
  * [17] F. Wiedijk, “The “de Bruijn factor”,” 2012.
  * [18] A. Blackwell and T. Green, “Cognitive dimensions of notations resource site,” 2010\.
  * [19] M. Svihla and I. Jelinek, “Benchmarking rdf production tools,” in DEXA, Springer, 2007.
  * [20] F. Michel, J. Montagnat, and C. Faron-Zucker, “A survey of rdb to rdf translation approaches and tools,” 2014.
  * [21] S. S. Sahoo, W. Halb, S. Hellmann, K. Idehen, T. T. Jr, S. Auer, J. Sequeda, and A. Ezzat, “A survey of current approaches for mapping of relational databases to rdf,” 2009.
  * [22] M. Hert, G. Reif, and H. C. Gall, “A comparison of rdb-to-rdf mapping languages,” in I-Semantics, ACM, 2011.