# Correcting Turbulence-induced Errors in Fiber Positioning for the Dark
Energy Spectroscopic Instrument

E. F. Schlafly Space Telescope Science Institute, 3700 San Martin Drive,
Baltimore, MD 21218, USA  J. Guy Lawrence Berkeley National Laboratory, 1
Cyclotron Road, Berkeley, CA 94720, USA  K. Honscheid Center for Cosmology and
AstroParticle Physics, The Ohio State University, 191 West Woodruff Avenue,
Columbus, OH 43210, USA  Department of Physics, The Ohio State University, 191
West Woodruff Avenue, Columbus, OH 43210, USA  S. Kent Department of Astronomy
and Astrophysics, University of Chicago, 5640 South Ellis Avenue, Chicago, IL
60637, USA  Fermi National Accelerator Laboratory, PO Box 500, Batavia, IL
60510, USA  S. E. Koposov Institute for Astronomy, University of Edinburgh,
Royal Observatory, Blackford Hill, Edinburgh EH9 3HJ, UK  Institute of
Astronomy, University of Cambridge, Madingley Road, Cambridge CB3 0HA, UK
Kavli Institute for Cosmology, University of Cambridge, Madingley Road,
Cambridge CB3 0HA, UK  J. Aguilar  Lawrence Berkeley National Laboratory, 1
Cyclotron Road, Berkeley, CA 94720, USA  S. Ahlen Physics Dept., Boston
University, 590 Commonwealth Avenue, Boston, MA 02215, USA  S. Bailey Lawrence
Berkeley National Laboratory, 1 Cyclotron Road, Berkeley, CA 94720, USA  D.
Brooks  Department of Physics & Astronomy, University College London, Gower
Street, London, WC1E 6BT, UK  T. Claybaugh  Lawrence Berkeley National
Laboratory, 1 Cyclotron Road, Berkeley, CA 94720, USA  K. Dawson  Department
of Physics and Astronomy, The University of Utah, 115 South 1400 East, Salt
Lake City, UT 84112, USA  P. Doel  Department of Physics & Astronomy,
University College London, Gower Street, London, WC1E 6BT, UK  K. Fanning
Kavli Institute for Particle Astrophysics and Cosmology, Stanford University,
Menlo Park, CA 94305, USA  SLAC National Accelerator Laboratory, Menlo Park,
CA 94305, USA  D. P. Finkbeiner Department of Physics, Harvard University, 17
Oxford St., Cambridge, MA 02138, USA  Harvard-Smithsonian Center for
Astrophysics, 60 Garden St., Cambridge, MA 02138, USA  A. Font-Ribera
Department of Physics & Astronomy, University College London, Gower Street,
London, WC1E 6BT, UK  Institut de Física d’Altes Energies (IFAE), The
Barcelona Institute of Science and Technology, Campus UAB, 08193 Bellaterra
Barcelona, Spain  J. E. Forero-Romero Departamento de Física, Universidad de
los Andes, Cra. 1 No. 18A-10, Edificio Ip, CP 111711, Bogotá, Colombia
Observatorio Astronómico, Universidad de los Andes, Cra. 1 No. 18A-10,
Edificio H, CP 111711 Bogotá, Colombia  S. Gontcho A Gontcho Lawrence Berkeley
National Laboratory, 1 Cyclotron Road, Berkeley, CA 94720, USA  G. Gutierrez
Fermi National Accelerator Laboratory, PO Box 500, Batavia, IL 60510, USA  D.
Kirkby Department of Physics and Astronomy, University of California, Irvine,
92697, USA  T. Kisner Lawrence Berkeley National Laboratory, 1 Cyclotron Road,
Berkeley, CA 94720, USA  A. Kremin Lawrence Berkeley National Laboratory, 1
Cyclotron Road, Berkeley, CA 94720, USA  J. Lasker Department of Physics,
Southern Methodist University, 3215 Daniel Avenue, Dallas, TX 75275, USA  M.
Landriau Lawrence Berkeley National Laboratory, 1 Cyclotron Road, Berkeley, CA
94720, USA  L. Le Guillou Sorbonne Université, CNRS/IN2P3, Laboratoire de
Physique Nucléaire et de Hautes Energies (LPNHE), FR-75005 Paris, France  M.
E. Levi Lawrence Berkeley National Laboratory, 1 Cyclotron Road, Berkeley, CA
94720, USA  A. de la Macorra Instituto de Física, Universidad Nacional
Autónoma de México, Cd. de México C.P. 04510, México  P. Martini Center for
Cosmology and AstroParticle Physics, The Ohio State University, 191 West
Woodruff Avenue, Columbus, OH 43210, USA  Department of Astronomy, The Ohio
State University, 4055 McPherson Laboratory, 140 W 18th Avenue, Columbus, OH
43210, USA  The Ohio State University, Columbus, 43210 OH, USA  A. Meisner NSF
NOIRLab, 950 N. Cherry Ave., Tucson, AZ 85719, USA  R. Miquel  Institució
Catalana de Recerca i Estudis Avançats, Passeig de Lluís Companys, 23, 08010
Barcelona, Spain  Institut de Física d’Altes Energies (IFAE), The Barcelona
Institute of Science and Technology, Campus UAB, 08193 Bellaterra Barcelona,
Spain  J. Moustakas Department of Physics and Astronomy, Siena College, 515
Loudon Road, Loudonville, NY 12211, USA  G. Niz Departamento de Física,
Universidad de Guanajuato - DCI, C.P. 37150, Leon, Guanajuato, México
Instituto Avanzado de Cosmología A. C., San Marcos 11 - Atenas 202. Magdalena
Contreras, 10720. Ciudad de México, México  F. Prada Instituto de Astrofísica
de Andalucía (CSIC), Glorieta de la Astronomía, s/n, E-18008 Granada, Spain
G. Rossi  Department of Physics and Astronomy, Sejong University, Seoul,
143-747, Korea  E. Sanchez CIEMAT, Avenida Complutense 40, E-28040 Madrid,
Spain  M. Schubnell  Department of Physics, University of Michigan, Ann Arbor,
MI 48109, USA  University of Michigan, Ann Arbor, MI 48109, USA  R. Sharples
Centre for Advanced Instrumentation, Department of Physics, Durham University,
South Road, Durham DH1 3LE, UK  Institute for Computational Cosmology,
Department of Physics, Durham University, South Road, Durham DH1 3LE, UK  D.
Sprayberry  NSF NOIRLab, 950 N. Cherry Ave., Tucson, AZ 85719, USA  G. Tarlé
University of Michigan, Ann Arbor, MI 48109, USA  B. A. Weaver  NSF NOIRLab,
950 N. Cherry Ave., Tucson, AZ 85719, USA  H. Zou National Astronomical
Observatories, Chinese Academy of Sciences, A20 Datun Rd., Chaoyang District,
Beijing, 100012, P.R. China

###### Abstract

Highly-multiplexed, robotic, fiber-fed spectroscopic surveys are observing
tens of millions of stars and galaxies. For many systems, accurate positioning
relies on imaging the fibers in the focal plane and feeding that information
back to the robotic positioners to correct their positions. Inhomogeneities
and turbulence in the air between the focal plane and the imaging camera can
affect the measured positions of fibers, limiting the accuracy with which
fibers can be placed on targets. For the Dark Energy Spectroscopic Instrument,
we dramatically reduced the effect of turbulence on measurements of positioner
locations in the focal plane by taking advantage of stationary positioners and
the correlation function of the turbulence. We were able to reduce positioning
errors from $7.3~{}\micron$ to $3.5~{}\micron$, speeding the survey by 1.6%
under typical conditions.

instrumentation: spectrographs — techniques: spectroscopic

††facilities: Mayall††software: astropy (Astropy Collaboration et al., 2013,
2018, 2022)

##  1 Introduction

Massively-multiplexed, robotically-positioned, fiber-fed spectroscopic systems
routinely survey the night sky, measuring the spectra of tens of millions of
stars and galaxies. Current and upcoming systems include LAMOST, SDSS-V, PFS,
and MOONS (Cui et al., 2012; Kollmeier et al., 2017; Takada et al., 2014;
Cirasuolo et al., 2014), and we focus in this work on the Dark Energy
Spectroscopic Instrument (DESI) (Levi et al., 2013). The robotic fiber
positioners on these instruments can quickly position fibers at the locations
of target astronomical sources. Precise positioning of these fibers is
important in order to maximize the amount of light received from targets.

Accurate positioning of fibers in the focal plane is challenging, however, due
to tight tolerances: for example, the DESI design requires that positioners be
placed within 10 $\mathrm{\mu m}$ of targets. Most systems contain guide
cameras, which are used to bring the telescope to the correct position and the
focal plane to the correct rotation. The locations of stars in the guide
cameras are then used to determine the target locations for fibers in the
focal plane, using knowledge of the metrology of the focal plane and the
mapping from the focal plane to the sky. For DESI, our approach to positioning
fibers is laid out in Kent et al. (2023). The accuracy of the positioning is
determined by a fiber dithering approach described in Schlafly et al. (2024),
which we also use to improve the mapping between the focal plane and the sky.

Most current and upcoming multi-object spectrographs feature cameras that
image the fibers in the focal plane. These images are used to measure the
locations of fibers, which are then used to improve their centering on target
stars and galaxies. These systems provide critical feedback about the quality
of the positioning, but they introduce another set of optics into the problem,
and require that the mapping between the focal plane and those cameras is well
understood.

When fibers are imaged in the focal plane for use in improving the positioning
of fibers, turbulence from convective cells in the volume of air between the
imaging camera and focal plane can distort the measured positions. For DESI,
inhomogeneities in the air imprint characteristic patterns on the observed
positions of positioners, adding roughly 7 $\mathrm{\mu m}$ of noise (0.1″) in
the focal plane. This is small compared to the DESI fiber diameter of 107
$\mathrm{\mu m}$ (1.5″), but still reduces survey speed meaningfully.
Moreover, this value is large compared to the intrinsic precision of the
motors (1.6 $\mathrm{\mu m}$, Schubnell et al. 2018), so if we could correct
the turbulence, we could position the fibers with much better accuracy. Figure
1 shows an example of these turbulence-induced changes in the measured
positions of DESI fibers. The measurement errors are highly coherent from
fiber to fiber, and therefore imprint correlated trends in the signal-to-noise
ratio obtained on observed stars and galaxies if not corrected.

Figure 1: An example of turbulence-induced variations in the measured
positions of fibers as seen by the Fiber View Camera (FVC) in DESI. Arrows
point from the average position over a long series of exposures to the
measured position in a particular exposure. The root-mean-square variation
$\sigma=\sqrt{\langle x^{2}+y^{2}\rangle}$ of $7.7~{}\mathrm{\mu m}$ is given
in the upper right, and an arrow of length $10~{}\mathrm{\mu m}$ is shown for
comparison in the upper left.

This work describes the approach taken by DESI to correct for the effects of
turbulence on the measured positions of fibers. The basic idea is to take
advantage of the correlation function of the turbulence in combination with
measurements of the locations of stationary points in the focal plane. The
stationary points provide measurements of the turbulence-induced errors in the
positions, and a Gaussian process predicts the turbulent contribution to the
measured positions at other locations. Using this approach, we obtain a
typical positioning error of 3.5 $\mathrm{\mu m}$, compared with 7.3
$\mathrm{\mu m}$ before correction. This corresponds to 0.8% more flux from a
point source entering a fiber in typical 1″ seeing, speeding the survey by
1.6% in the background limited regime.

This work is organized as follows. In §2, we describe the DESI instrument and
some of its data products, as well as the specific observations taken for this
study. In §3, we describe the approach we take here to correct for the effect
of turbulence on the measured positions of fibers. In §4, we discuss the
results for DESI, the correlation function of turbulence observed, and the
performance as a function of the available number of stationary points.
Finally, we conclude in §5. The code and data used to produce the figures in
this work are available at https://zenodo.org/doi/10.5281/zenodo.12701420.

##  2 Data

The DESI instrument resides at the prime focus of the Mayall telescope on Kitt
Peak in Arizona. The 4 m primary mirror focuses light through a wide-field
optical corrector onto the focal plane (Miller et al., 2023). The focal plane
is outfitted with 5000 fibers which can be robotically positioned in a 1.4′
patrol radius to collect light from target stars and galaxies (Silber et al.,
2023; Myers et al., 2023). This light is then piped to ten spectrographs which
cover 360 nm – 980 nm (DESI Collaboration et al., 2022). The resulting spectra
are extracted and calibrated to determine the spectral energy distributions of
astronomical sources (Guy et al., 2023).

The DESI collaboration is currently performing a survey of tens of millions of
stars and galaxies (Levi et al., 2013; DESI Collaboration et al., 2016;
Schlafly et al., 2023), selected from imaging from the DESI Legacy Imaging
Survey (Dey et al., 2019). Primary target classes include Milky Way stars
(Cooper et al., 2023), bright galaxies (Hahn et al., 2023), luminous red
galaxies (Zhou et al., 2023), emission line galaxies (Raichoor et al., 2023),
and quasars (Chaussidon et al., 2023). The first results from survey
validation (DESI Collaboration et al., 2023a) and DESI’s Early Data Release
(DESI Collaboration et al., 2023b) are now publicly available. The survey
recently reported measurements of the baryon-acoustic oscillation signal from
the first year of DESI observations of galaxies (DESI Collaboration et al.,
2024a) and the Lyman-$\alpha$ forest (DESI Collaboration et al., 2024b), and
the associated cosmological implications (DESI Collaboration et al., 2024c).

When DESI observes a field, the telescope slews into place and, in parallel,
the DESI positioners begin moving to their target locations in the focal
plane. Next the guide-focus array cameras (GFAs) identify bright stars and the
telescope boresight is adjusted to precisely center and rotate the field.
After the initial move is complete and the field has been acquired, the fibers
are back-illuminated from the spectrographs and imaged by the Fiber-View
Camera (FVC). The FVC is located in the hole in the center of the Mayall
primary mirror and looks through the corrector at the focal plane (Baltay et
al., 2019). The current locations of fibers are measured using the FVC image,
and used together with updated information about the mapping of the focal
plane to the sky from the GFAs to determine how best to adjust the location of
each fiber to center it on target sources. The DESI positioners then execute a
second move to bring the positioners closer to their target locations. A final
image of the back-illuminated fibers is taken to measure the locations of the
positioners, and then spectroscopic observations begin. For a more complete
description of the positioning of DESI fibers, see Kent et al. (2023).

As described above, each time DESI positions fibers to observe a field, it
moves the fibers twice. The first move is termed the “blind” move because the
positioners are simply commanded to go to a specific target location. The
second move is termed the “correction” move because it relies on feedback from
the FVC to slightly improve the positioning of each fiber. The typical blind
move has an accuracy of 30 $\mathrm{\mu m}$, while correction moves have an
accuracy of roughly 7 $\mathrm{\mu m}$, in the absence of turbulence
correction. The FVC measures centroids with an accuracy of 2 $\mathrm{\mu m}$,
a small portion of the 7 $\mathrm{\mu m}$ offsets (Baltay et al., 2019).

Critically for the purpose of this work, the correction move depends on the
measured position of each fiber. The turbulence in the volume of air between
the primary mirror and the corrector imprints correlated errors in the
measurements of those positions. Additionally, the measurements of the
ultimate positions of the fibers will be subject to an independent realization
of the turbulence, leading to measurements of positional offsets that are on
average a factor of $\sqrt{2}$ larger than the actual offset of the fiber from
the target location.

This work focuses on using measurements of stationary points to measure and
correct for the turbulence present in images from the FVC. These stationary
points include 120 fiducials (DESI Collaboration et al., 2022) exclusively
used for calibration purposes, as well as several hundred non-functional
positioners that are rarely used in operations. For this work, we used a total
of 793 stationary points, provided by the fiducials and 673 non-functional
positioners. The non-functional positioners have a fairly random distribution
in the focal plane. Most of the non-functional positioners can move, but their
movement is erratic and can lead to collisions with neighboring positioners,
and so we have chosen to keep their locations fixed in DESI operations. We
expect to be able to return many of these positioners to operations in the
future. Fortunately, however, these stationary positioners provide good
measurements of the sky background for DESI spectroscopy and can also be used
to probe the turbulent positioning errors that are the focus of this work.

Every DESI observation results in a “coordinates” file111See the DESI data
model at
https://desidatamodel.readthedocs.io/en/latest/DESI_SPECTRO_DATA/NIGHT/EXPID/coordinates-
EXPID.html for details. which contains the measurements of the locations of
fibers in each FVC exposure, both before and after turbulence correction.
These measurements are the focus of the analysis of this paper.

During poor conditions, while the Mayall dome was closed, we took a series of
400 FVC images of the back-illuminated focal plane in order to better measure
the correlation function of the turbulence observed by DESI. These images used
the standard FVC exposure time (2 s) with the Mayall pointed directly at
zenith. This data is the basis for the analysis of §4.2.

##  3 Method

We model the turbulence observed by the FVC using a Gaussian process, as we
will describe in §3.1. The Gaussian process uses measurements of turbulence
from a number of stationary points (§3.2) which are not moved during normal
survey operations. Thus, their apparent motion in FVC images can be largely
attributed to turbulence, and the Gaussian process modeling conceptually
averages and interpolates among them in an optimal way. Some additional
calibration of the stationary points is needed to make this procedure work
well; we describe the calibration we perform in §3.3.

###  3.1 Gaussian Process Modeling

Gaussian processes naturally describe the turbulence seen by DESI fibers. We
have measurements of the positions $\boldsymbol{x}_{n}$ of roughly 5000 fibers
$n$ in two dimensions222Technically, we are counting here both the actual
fibers and the fiducials.. These measurements derive from the centroids of the
light of the back-illuminated fibers in images from the FVC. The centroids are
converted into locations in the focal plane using information about the
current setting of the atmospheric dispersion corrector, the measured
locations of 120 fiducial locations, and a model for the optics of the DESI
corrector lens (Kent et al., 2023). Using the nominal locations of stationary
points (§3.2), we can compute the displacement $\boldsymbol{d}$ of the
stationary points due to turbulence and noise. The displacements
$\boldsymbol{d}$ have a correlation function $C_{nm}$ which we model as the
sum of a turbulent kernel $K$ and a diagonal noise term:

$C_{nm}=\langle\boldsymbol{d}_{n}\boldsymbol{d}_{m}\rangle=K(\boldsymbol{x}_{n},\boldsymbol{x}_{m})+\delta_{nm}\sigma^{2}\,,$
(1)

where $\delta_{nm}$ is the Kronecker delta and $n$ and $m$ index over fibers.
Given the correlation function $C$ and measured displacements
$\boldsymbol{d}_{n}$, the predicted turbulence at any position
$\boldsymbol{x}_{s}$ is given by

$\boldsymbol{d}_{s}=\sum_{nm}K(\boldsymbol{x}_{s},\boldsymbol{x}_{m})(C^{-1})_{nm}\boldsymbol{d}_{n}\,,$
(2)

as described in Rasmussen & Williams (2006). Here $n$ and $m$ both loop over
the indices of the measured fibers. We follow Algorithm 2.1 of Rasmussen &
Williams (2006) to solve for the maximum likelihood turbulence measurements
$\boldsymbol{d}_{s}$.

This procedure relies on knowledge of the turbulence kernel
$K(\boldsymbol{x}_{n},\boldsymbol{x}_{m})$, which describes how the turbulent
offset at one location is correlated with the turbulent offset at another
location. We discuss this quantity more in §4.2. For DESI, we took the
approach of treating the turbulent offsets in the $x$ and $y$ direction as two
separate, independent Gaussian fields, and chose a fixed squared exponential
kernel for $K$:

$K(\boldsymbol{x}_{n},\boldsymbol{x}_{m})=A^{2}\exp\left({-\frac{(\boldsymbol{x}_{n}-\boldsymbol{x}_{m})^{2}}{2l^{2}}}\right)\,,$
(3)

In principle we could have chosen to fit the parameters $A$, $l$, and $\sigma$
for each DESI exposure, but we opted to fix the terms of $C$ in order to speed
the computation and simplify the algorithm. We adopted $A=5~{}\mathrm{\mu m}$,
$l=50~{}\mathrm{mm}$, and $\sigma=5~{}\mathrm{\mu m}$ as a rough approximation
to the observed correlation function and uncertainties. However, in retrospect
we overestimated the noise $\sigma$ and underestimated the amplitude $A$; see
§4.2 for more discussion.

Though we chose to model the turbulent offsets in the $x$ and $y$ components
of $\boldsymbol{d}$ as independent Gaussian processes, other approaches better
reflect the physics of the problem. Because the absolute deflections are
small, the location of a source in an image is given by the gradient of the
projection of the density fluctuations in the air column onto a surface.
Accordingly, a different approach would be to model the surface as a scalar
field with correlated variations across the focal plane. The observed
centroids would then be computed as the gradient of that scalar. In that case,
we have instead

$K((\boldsymbol{x}_{n})_{i},(\boldsymbol{x}_{m})_{j})=\partial_{i}\partial_{j}K^{\prime}(\boldsymbol{x}_{m},\boldsymbol{x}_{n})$
(4)

where $i$ and $j$ index over the $x$ and $y$ components of
$\boldsymbol{x}_{n}$, $K$ is the kernel for the observed displacements, and
$K^{\prime}$ is the kernel for the scalar field whose gradient gives the
observed displacements. We do not use this approach for DESI operations, but
we find in §4 that it performs slightly better than the independent approach.
We again use a squared exponential kernel for $K^{\prime}$ in this case, with
$\sigma=2~{}\mathrm{\mu m}$, $A=1~{}\mathrm{mm}^{2}$, and
$l=100~{}\mathrm{mm}$ for $K^{\prime}$. The values for $A$ and $l$ are are
different from those for the independent case because they describe the
correlation of the scalar field rather than its derivative. The value of
$\sigma$ is different because it was developed later after better estimates of
the uncertainties had been developed.

Occasionally DESI positioners may collide with their neighbors. When a
stationary positioner is bumped, its location can be significantly changed. We
do not want this movement to be interpreted as turbulence. To avoid this, we
added outlier rejection to the turbulence modeling. We do this by first
computing a simple median and clipped standard deviation of the displacements
$d$. We discard any points more than 5 standard deviations from the median. We
do an initial Gaussian process analysis, and further remove any points with
measured locations more than 10 $\mathrm{\mu m}$ from the predictions from the
turbulence modeling. Finally, we repeat the Gaussian process modeling with the
remaining points. This is ad hoc but has delivered good performance even
during rare occurrences when we inadvertently bumped several nearby stationary
positioners.

###  3.2 Stationary Points

The key measurements needed for the Gaussian process modeling of §3.1 are the
displacements of the stationary points due to turbulence. Because the
stationary points do not move, in principle these measurements can be made by
averaging together their observed locations over a long series of ordinary
observations. In practice, we have instead dedicated a small number of nights
to measuring the locations of the stationary points. These measurements can be
made when bad weather prevents scientific observations. We observe small
changes to the “stationary” points ($<2~{}\mathrm{\mu m}$) from month to
month, but these offsets are not large enough to meaningfully affect the
survey speed. Additionally, although infrequently, a handful of positioners
are bumped and have significantly altered positions. These positions are
corrected as part of the routine updates to the stationary points, and are
usually large enough that the outlier clipping of §3 identifies and removes
them.

For DESI, the stationary points come in two varieties. Most of the stationary
points (673 in this work) come from back-illuminated fibers on non-functional
positioners that are fixed in location and measured by the FVC. The remaining
stationary points (120) come from fiducials, which contain an LED which shines
through four pinholes with a characteristic pattern to the FVC. DESI reports
an average position for each fiducial in each image, averaging over the four
pinholes. The fiducials in principle provide somewhat better positional
measurements than the non-functional fibers, since they are mechanically more
stable and produce four points in the FVC images, but we do not find
significant differences in accuracy between the fiducials and non-functional
positioners.

###  3.3 Variation with Telescope Position

Though we never intentionally move the stationary points, we find that their
apparent positions move systematically depending on the direction the
telescope is pointing on the sky. The root-mean-square motion is 4.5
$\mathrm{\mu m}$ and is clearly detected in the motions of the stationary
points following long slews. These motions are spatially coherent across the
focal plane. Moreover, both non-functional positioners and fiducials show the
same coherent patterns, and so we expect that these apparent motions are an
effect in the optics rather than mechanical sag of individual positioners or
petals.

To measure the apparent motion of stationary points, we took special FVC
observations when the dome was closed due to bad weather. We took a series of
FVC images across a broad range of telescope hour angles and declinations,
using ten times the normal FVC exposure time in order to reduce the effect of
turbulence. We then fit the apparent focal plane coordinates of each fiber
with a cubic function of hour angle and declination. The resulting fits have
root-mean-square residuals around 2.7 $\mathrm{\mu m}$. These fits were
removed from the observed positions of the stationary points prior to fitting
for turbulence, for all FVC location measurements.

##  4 Results and Discussion

We began correcting positioning for the effects of turbulence in December
2021. Since then, we have corrected more than 10,000 positioning sequences. We
discuss the results of this correction in §4.1, and then investigate the
turbulence correlation function in §4.2. In §4.3, we look at how the accuracy
of the turbulence correction depends on the number of stationary points.

###  4.1 Turbulence & its Correction for DESI

The effect of turbulence on positioning is immediately apparent in FVC
measurements. Figure 2 shows three examples of images from a sequence of 400
FVC images taken with all positioners fixed in place. The top row shows how
the positions appear to move from their mean positions over the sequence due
to turbulence. The middle row shows the inferred turbulence field using the
method of §3. Finally, the bottom row shows the residuals after removing the
turbulence field. While the observed displacements can be 10 $\mathrm{\mu m}$
in size, after correction they are close to 2 $\mathrm{\mu m}$; most of the
turbulence is fit and removed. The residuals do show measurable remaining
correlations at small spatial scales that we are not correcting, though 2
$\mathrm{\mu m}$ residuals are too small to impact survey speed meaningfully.

Figure 2: Three examples of turbulence. Each column corresponds to a different
exposure and therefore realization of the turbulence-induced position offsets.
The top row shows the measurements; the middle row shows the turbulence
derived using the DESI turbulence algorithm; and the bottom row shows the
residuals. The turbulence modeling explains most of the variation and is able
to reduce the residuals to roughly $2~{}\mathrm{\mu m}$. The root-mean-square
displacement is given in the upper right of each panel, and a 10 $\mathrm{\mu
m}$ scale bar is shown in the upper left.

This level of turbulence is fairly typical of DESI observations. Figure 3
shows the positioning performance of DESI over 10,000 positioning sequences
since DESI began computing turbulence corrections in December 2021. The shaded
regions show the 16th through 84th percentiles of the rms displacements in
total (blue), due to turbulence (orange), and after correction for turbulence
(green). The dashed lines show the overall median of each quantity over all of
the positioning sequences.

Figure 3: Fiber positioning performance for DESI. The blue, orange, and green
regions show the range between the 16th and 84th percentiles for DESI
positioning accuracy over roughly 10,000 DESI positioning sequences. Darks,
biases, and other exposures that do not involve a positioning loop are not
used. Absent any correction, the blue region shows that we would obtain
typical precision of $7.3~{}\mathrm{\mu m}$. We however fit for the turbulence
in each image (orange), which contributes most of the variance. The final
positioning errors after correction for turbulence (green) have typical values
of $3.5~{}\mathrm{\mu m}$. Dashed horizontal lines show the median RMS
positioning error over all of the exposures.

Figure 3 shows that turbulence is the dominant contribution to the positioning
error in each exposure, and that we remove most of it. The total positioning
error in the absence of turbulence corrections has a typical value of 7.3
$\mathrm{\mu m}$, of which 6.1 $\mathrm{\mu m}$ is due to turbulence. After
correction for turbulence, the remaining residuals are 3.5 $\mathrm{\mu m}$.
This remaining error is due to a combination of imperfect motion of the
positioners, noise in the measurements of the FVC, uncorrected turbulence, and
unmodeled motions of the positioners when the telescope is moved (§3.2). When
the positioners are not moved, we can obtain residuals of $\sim 2$
$\mathrm{\mu m}$, which includes noise in the FVC measurements and uncorrected
turbulence. The DESI motors have a precision of $\approx 1.6$ $\mathrm{\mu m}$
(Schubnell et al., 2018). Our expectation then is that the total positioning
RMS in operations should be the quadrature sum of the zenith performance
($2~{}\mathrm{\mu m}$), the motor performance ($1.6~{}\mathrm{\mu m}$), and
the performance at different hour angles and declinations (2.7 $\mathrm{\mu
m}$). The result of 3.7 $\mathrm{\mu m}$ is close to the observed 3.5
$\mathrm{\mu m}$ RMS performance.

###  4.2 The Turbulence Correlation Function

The optimal correction for turbulence depends on the correlation function of
the displacements it induces in measurements. The correlation function of
turbulence has been studied extensively. For homogeneous, isotropic turbulent
displacement fields, the correlation function of the displacement vector
$\boldsymbol{d}$ as a function of the separation vector $\boldsymbol{r}$ can
be fully characterized by the longitudinal ($\Psi_{\parallel}$) and transverse
($\Psi_{\perp}$) correlation functions (Monin & Iaglom, 1975). Note that
“parallel” and “perpendicular” in this context refer to the direction of the
turbulent displacements seen by two fibers relative to the separation between
those fibers.

We measure $\Psi_{\parallel}$ and $\Psi_{\perp}$ for DESI by using a sequence
of 400 DESI FVC observations for which the fibers were not moved, so that all
of their apparent motion can be attributed to turbulence and uncertainty in
the centroids. We then have roughly 5000 measured displacements
$\boldsymbol{d}_{n}$ for each of 400 exposures, where the separations between
fibers is essentially constant. We compute the correlation functions
$\Psi_{\parallel}$ and $\Psi_{\perp}$ by considering all pairs of
displacements $\boldsymbol{d}_{n}$ and $\boldsymbol{d}_{m}$, and computing
their separation $\boldsymbol{r}_{nm}$. The displacements have components
parallel to $\boldsymbol{r}_{rm}$ ($d_{\parallel}$) and perpendicular to it
($d_{\perp}$). We compute those components and then average their product
together for all pairs with separations falling into a particular bin. We end
up with $\Psi_{\parallel k}$ and $\Psi_{\perp k}$, a binned approximation to
the correlation functions $\Psi(r)$ we want to measure. Mathematically, this
computation is described as follows:

$\displaystyle\Psi_{\parallel k}$
$\displaystyle=\frac{1}{N_{k}}\sum_{n>m}{d}_{\parallel,n}(\boldsymbol{r}_{nm}){d}_{\parallel,m}(\boldsymbol{r}_{nm})I_{k}(\boldsymbol{r}_{nm})$
(5) $\displaystyle\Psi_{\perp k}$
$\displaystyle=\frac{1}{N_{k}}\sum_{n>m}{d}_{\perp,n}(\boldsymbol{r}_{nm}){d}_{\perp,m}(\boldsymbol{r}_{nm})I_{k}(\boldsymbol{r}_{nm})$
(6) $\displaystyle\boldsymbol{r}_{nm}$
$\displaystyle=\boldsymbol{x}_{n}-\boldsymbol{x}_{m}$ (7)
$\displaystyle{d}_{\parallel}(\boldsymbol{r})$
$\displaystyle=\frac{\boldsymbol{d}\cdot\boldsymbol{r}}{|\boldsymbol{r}|}$ (8)
$\displaystyle{d}_{\perp}(\boldsymbol{r})$
$\displaystyle=\frac{|\boldsymbol{d}\times\boldsymbol{r}|}{|\boldsymbol{r}|}$
(9) $\displaystyle N_{k}$ $\displaystyle=\sum_{n>m}I_{k}(\boldsymbol{r}_{nm})$
(10) $\displaystyle I_{k}(\boldsymbol{r})$
$\displaystyle=\begin{cases}1\quad\text{if }|\boldsymbol{r}|\text{ falls in
bin }k\\\ 0\quad\mathrm{otherwise}\end{cases}\quad.$ (11)

The bins are labeled by $k$ and the indicator function $I_{k}$ defines each
bin. We choose to make bins 1 mm in size, such that $I_{k}$ corresponds to
separations from $k-1$ mm to $k$ mm. In simple terms, we examine all pairs of
stationary points with separations of a given distance. We then calculate the
average of the product of the parallel and perpendicular components of the
displacements for each pair. This process yields the parallel and
perpendicular correlation functions.

Figure 4 shows the correlation functions in the parallel and perpendicular
directions obtained from the average of 400 FVC observations. The amplitude of
the correlation function at zero separation is $(7.7~{}\micron)^{2}$,
consistent with expectations given the total standard deviation of the
measurements in the sample. The correlation falls rather slowly, becoming
negative near 300 mm for the longitudinal and 600 mm for the transverse
correlation. The parallel correlation falls roughly twice as quickly as the
perpendicular correlation, and the parallel correlation halves near 100 mm.
This is a rather large scale; the radius of the DESI focal plane is only 420
mm, so not many turbulence measurements are needed to correct for most of the
turbulence.

Figure 4: The correlation between positioning offsets along the direction of
separation ($\Psi_{\parallel}$, longitudinal) and perpendicular to it
($\Psi_{\perp}$, transverse). The quantity $\frac{d}{dr}(r\Psi_{\perp})$ is
shown by a dashed line, and should match $\Psi_{\parallel}$ if the turbulence
is sourced by a scalar field. There is good agreement at small separations
though the agreement becomes worse at large separations; see text for
discussion. The shaded regions corresponds to the 16th to 84th percentile of
the correlation function at each separation over the 400 images considered,
while the solid lines give the average.

It may seem peculiar for the correlation function to become negative. This is
likely because the positions we measure have already had some filtering
applied before our analysis. DESI’s PlateMaker package (Kent et al., 2023)
fits out an overall scale, translation, and rotation in order to account for
any shifts in the focus, position, or rotation of the FVC or focal plane.
These terms also naturally absorb any very large scale turbulence signal.

When the vector field can be expressed as the gradient of a scalar function,
then it can be shown that $\frac{d}{dr}(r\Psi_{\perp})=\Psi_{\parallel}$ (e.g.
Gorski, 1988). For DESI, we expect that the observed displacements are the
gradient of the surface of projected air density fluctuations, so we expect
good agreement here. Figure 4 shows the prediction for $\Psi_{\parallel}$ from
this relation as the dashed line. It matches the observed measurement closely
at small separations but diverges at large separations, likely because of the
extra filtering PlateMaker applies on large scales. As a further test, we took
several images with large turbulence and decomposed the displacement pattern
into E and B modes using code that was developed for modeling distortions in
the corrector optics (Kent et al., 2023). As expected, virtually all the
effect (98% to 99%) is in E modes, and the small amount in B modes is likely
due to noise.

###  4.3 How many turbulence probes are needed?

The accuracy of the turbulence correction depends on the number of stationary
points that are available to probe the turbulence field. How many are needed
to obtain a given level of accuracy? Judging by the correlation function
(§4.2) we expect most of the turbulence to be corrected even when including
only a small number of points, on the order of tens. This follows because the
correlation scale is roughly 100 mm and the focal plane is roughly 400 mm, and
$(\frac{400}{100})^{2}=16$. The total impact of turbulence on DESI’s survey
speed is only about $2\%$, and so a hypothetical correction which eliminated
all turbulence would be worth about 2% of all of DESI’s fibers—100 fibers. So
were we able to pick a number of fibers to deactivate and use as turbulence
probes, we would likely choose a number on the order of 20, and anything in
excess of 100 would be clearly wasteful.

We investigate how well we can correct for the effect of turbulence as a
function of the number of stationary points in Figure 5, for a variety of
different turbulence correction schemes.

We consider four approaches for turbulence correction.

  1. 1.

None: no correction is performed

  2. 2.

Simple: the measured value from the nearest turbulence probe is used

  3. 3.

Independent: a Gaussian process with a squared exponential kernel is used,
separately for the $x$ and $y$ components of the displacement field

  4. 4.

Gradwavefront: a Gaussian process with a squared exponential kernel is used,
where a scalar field has the correlations and the observed displacements are
given by its gradient.

These are roughly intended to scale up in terms of physical accuracy and
complexity. The “independent” approach (3) is what is used in DESI operations.

For this test, we used the same sequence of 400 FVC images also used for §4.2.
We could therefore estimate the “true” position of each positioner extremely
well from the mean position over the 400 exposures. We then tested the
accuracy of the different correction schemes according to how well they
predicted this true position on the basis of a single FVC exposure. We varied
the number of stationary points available to the different approaches by
randomly subselecting among the 793 total stationary points available to DESI
to 30, 100, and 300 stationary points.

Figure 5: The positioning accuracy as a function of the number of positioners
used to measure the turbulence, for a variety of different turbulence
correction schemes. Even when only 30 positioners are available to probe the
turbulence, turbulence-induced errors can be reduced by half for DESI, even
with a simple correction scheme. Using a Gaussian process and taking advantage
of the curl-free nature of the turbulence field provides the best performance
(gradwavefront), though when a large number of turbulence probes are available
(e.g., for DESI), the improvement with respect to the independent scheme is
negligible.

As shown in Figure 5, all of these approaches do well (except for the trivial
“none” option). With the large number of turbulence probes available to DESI,
just looking up the nearest stationary point corrects positions to within 3
$\mathrm{\mu m}$. The Gaussian-process based approaches do better, $\sim 2$
$\mathrm{\mu m}$, presumably because they are able to average over the small
amount of centroid noise in the measurement of the positions of DESI fibers
($\sim 2$ µm, Baltay et al. 2019). The approach using the gradient of a
wavefront has the best performance, especially when the available number of
turbulence probes is small; for example, when only 100 probes are available,
it achieves 2.5 $\mathrm{\mu m}$ accuracy while the nearest neighbor approach
gives 4 µm.

The “independent” approach we actually adopted for DESI is nearly as good as
the full gradient approach for the large numbers of turbulence probes
available to DESI, but it actually performs worse than the simple approach
when the number of turbulence probes is small. This is because we chose to
approximate the turbulence correlation function with a squared exponential
kernel with a length scale of 50 mm, while a better choice would have been a
kernel that falls less steeply in the wings. Still, given the large number of
turbulence probes DESI actually has, our choice performs very well, within a
tenth of a micron of the best performing option.

DESI has 120 fixed fiducials intended solely for measuring the locations of
different fixed points in the focal plane; this already allows the nearest-
neighbor approach to reach 4 $\mathrm{\mu m}$ accuracy and the gradient
approach to reach 3 $\mathrm{\mu m}$. Practically, this performance is more
than adequate and additional turbulence probes are not needed. In DESI we do
include many more measurements, because they are available and large numbers
make the system more robust. But DESI would prioritize targeting additional
astronomical sources to probing turbulence in between the primary mirror and
corrector lenses.

##  5 Conclusions

The Dark Energy Spectroscopic Instrument has observed millions of stars and
galaxies in its first three years of operations. These observations depend on
accurately placing fibers on sources. To accomplish this, DESI images back-
illuminated fibers in the focal plane to measure their positions, and then
adjusts those positions to place them closer to their intended locations. This
correction dramatically improves the accuracy of DESI’s positioning, but has
the potential to introduce correlated positioning errors. These errors stem
from turbulence in the volume of air through which DESI images the focal plane
and the effect of that turbulence on the observed centroids of fibers. We
model and remove these turbulence-induced errors by taking advantage of
stationary points in the focal plane, whose apparent motions can be attributed
to turbulence. By interpolating between and averaging these measured motions
in an optimal way using a Gaussian process, we reduce the typical 7.3
$\mathrm{\mu m}$ root-mean-square positioning error to about 3.5 $\mathrm{\mu
m}$, improving the survey speed by roughly 1.6%.

We study the correlation function of the measured centroid offsets and find
that they are well modeled as the gradient of a scalar field, consistent with
expectations that ultimately turbulence is distorting the wavefront of the
light seen by the fiber view camera. We find a characteristic scale of roughly
100 mm in the focal plane, which corresponds to $\approx 35$ mm halfway
between the FVC and the first lens in the corrector.

We study how the performance of the turbulence correction depends on the
number of stationary points available to measure the turbulence. We find that
only a relatively small number are needed to obtain good performance. In the
data set used for this study, DESI had 793 stationary points used for
measuring turbulence. Even with only 100 points, we could have corrected
turbulence to $<3~{}\mathrm{\mu m}$, in practical terms no worse than the
$\approx 2~{}\mathrm{\mu m}$ we obtain with the full complement. Since DESI
has 120 fiducials, we need not consider any smaller numbers, but we note that
even with only 30 stationary points we could still correct turbulence to
better than $<5~{}\mathrm{\mu m}$ with DESI-like data. This accuracy is made
possible by excellent, high signal-to-noise images from the fiber view camera
and precise modeling of the centroids in those images (Baltay et al., 2019).

Turbulence will affect other large robotically-positioned multi-object
spectrographs like LAMOST, SDSS-V, PSF, and MOONS in a similar way. We expect
that these surveys would benefit from using stationary points to model and
remove turbulence from the measured locations of fibers.

Chris Blake suggested using the approach of Gorski (1988) and Monin & Iaglom
(1975) to investigate the turbulence correlation function. His guidance was
critical to §4.2. SK acknowledges support from the Science & Technology
Facilities Council (STFC) grant ST/Y001001/1. For the purpose of open access,
the author has applied a Creative Commons Attribution (CC BY) licence to any
Author Accepted Manuscript version arising from this submission. This material
is based upon work supported by the U.S. Department of Energy (DOE), Office of
Science, Office of High-Energy Physics, under Contract No. DE–AC02–05CH11231,
and by the National Energy Research Scientific Computing Center, a DOE Office
of Science User Facility under the same contract. Additional support for DESI
was provided by the U.S. National Science Foundation (NSF), Division of
Astronomical Sciences under Contract No. AST-0950945 to the NSF’s National
Optical-Infrared Astronomy Research Laboratory; the Science and Technology
Facilities Council of the United Kingdom; the Gordon and Betty Moore
Foundation; the Heising-Simons Foundation; the French Alternative Energies and
Atomic Energy Commission (CEA); the National Council of Humanities, Science
and Technology of Mexico (CONAHCYT); the Ministry of Science, Innovation and
Universities of Spain (MICIU/AEI/10.13039/501100011033), and by the DESI
Member Institutions: https://www.desi.lbl.gov/collaborating-institutions. Any
opinions, findings, and conclusions or recommendations expressed in this
material are those of the author(s) and do not necessarily reflect the views
of the U. S. National Science Foundation, the U. S. Department of Energy, or
any of the listed funding agencies. The authors are honored to be permitted to
conduct scientific research on Iolkam Du’ag (Kitt Peak), a mountain with
particular significance to the Tohono O’odham Nation.

## References

  * Astropy Collaboration et al. (2013) Astropy Collaboration, Robitaille, T. P., Tollerud, E. J., et al. 2013, A&A, 558, A33
  * Astropy Collaboration et al. (2018) Astropy Collaboration, Price-Whelan, A. M., Sipőcz, B. M., et al. 2018, AJ, 156, 123
  * Astropy Collaboration et al. (2022) Astropy Collaboration, Price-Whelan, A. M., Lim, P. L., et al. 2022, ApJ, 935, 167
  * Baltay et al. (2019) Baltay, C., Rabinowitz, D., Besuner, R., et al. 2019, PASP, 131, 065001
  * Chaussidon et al. (2023) Chaussidon, E., Yèche, C., Palanque-Delabrouille, N., et al. 2023, ApJ, 944, 107
  * Cirasuolo et al. (2014) Cirasuolo, M., Afonso, J., Carollo, M., et al. 2014, in Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, Vol. 9147, Ground-based and Airborne Instrumentation for Astronomy V, ed. S. K. Ramsay, I. S. McLean, & H. Takami, 91470N
  * Cooper et al. (2023) Cooper, A. P., Koposov, S. E., Allende Prieto, C., et al. 2023, ApJ, 947, 37
  * Cui et al. (2012) Cui, X.-Q., Zhao, Y.-H., Chu, Y.-Q., et al. 2012, Research in Astronomy and Astrophysics, 12, 1197
  * DESI Collaboration et al. (2016) DESI Collaboration, Aghamousa, A., Aguilar, J., et al. 2016, arXiv e-prints, arXiv:1611.00036
  * DESI Collaboration et al. (2022) DESI Collaboration, Abareshi, B., Aguilar, J., et al. 2022, AJ, 164, 207
  * DESI Collaboration et al. (2023a) DESI Collaboration, Adame, A. G., Aguilar, J., et al. 2023a, arXiv e-prints, arXiv:2306.06307
  * DESI Collaboration et al. (2023b) —. 2023b, arXiv e-prints, arXiv:2306.06308
  * DESI Collaboration et al. (2024a) —. 2024a, arXiv e-prints, arXiv:2404.03000
  * DESI Collaboration et al. (2024b) —. 2024b, arXiv e-prints, arXiv:2404.03001
  * DESI Collaboration et al. (2024c) —. 2024c, arXiv e-prints, arXiv:2404.03002
  * Dey et al. (2019) Dey, A., Schlegel, D. J., Lang, D., et al. 2019, AJ, 157, 168
  * Gorski (1988) Gorski, K. 1988, ApJ, 332, L7
  * Guy et al. (2023) Guy, J., Bailey, S., Kremin, A., et al. 2023, AJ, 165, 144
  * Hahn et al. (2023) Hahn, C., Wilson, M. J., Ruiz-Macias, O., et al. 2023, AJ, 165, 253
  * Kent et al. (2023) Kent, S., Neilsen, E., Honscheid, K., et al. 2023, AJ, 166, 177
  * Kollmeier et al. (2017) Kollmeier, J. A., Zasowski, G., Rix, H.-W., et al. 2017, arXiv e-prints, arXiv:1711.03234
  * Levi et al. (2013) Levi, M., Bebek, C., Beers, T., et al. 2013, arXiv e-prints, arXiv:1308.0847
  * Miller et al. (2023) Miller, T. N., Doel, P., Gutierrez, G., et al. 2023, arXiv e-prints, arXiv:2306.06310
  * Monin & Iaglom (1975) Monin, A. S., & Iaglom, A. M. 1975, Statistical fluid mechanics: Mechanics of turbulence. Volume 2 /revised and enlarged edition/
  * Myers et al. (2023) Myers, A. D., Moustakas, J., Bailey, S., et al. 2023, AJ, 165, 50
  * Raichoor et al. (2023) Raichoor, A., Moustakas, J., Newman, J. A., et al. 2023, AJ, 165, 126
  * Rasmussen & Williams (2006) Rasmussen, C. E., & Williams, C. K. I. 2006, Gaussian Processes for Machine Learning
  * Schlafly et al. (2023) Schlafly, E. F., Kirkby, D., Schlegel, D. J., et al. 2023, AJ, 166, 259
  * Schlafly et al. (2024) Schlafly, E. F., Schlegel, D., BenZvi, S., et al. 2024, arXiv e-prints, arXiv:2403.05688
  * Schubnell et al. (2018) Schubnell, M., Aguilar, J., Ameel, J., et al. 2018, in Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, Vol. 10706, Advances in Optical and Mechanical Technologies for Telescopes and Instrumentation III, ed. R. Navarro & R. Geyl, 107062A
  * Silber et al. (2023) Silber, J. H., Fagrelius, P., Fanning, K., et al. 2023, AJ, 165, 9
  * Takada et al. (2014) Takada, M., Ellis, R. S., Chiba, M., et al. 2014, PASJ, 66, R1
  * Zhou et al. (2023) Zhou, R., Dey, B., Newman, J. A., et al. 2023, AJ, 165, 58


Copyright for this paper by its authors. Use permitted under Creative Commons
License Attribution 4.0 International (CC BY 4.0).

IEEE International Conference on Advanced Learning Technologies (ICALT23)

[orcid=0000-0002-2921-4809<EMAIL_ADDRESS>] [1]

[ ]

[ ]

[ ]

[ ]

[ ]

[1]Corresponding author.

# Building Contextual Knowledge Graphs for Personalized Learning
Recommendations using Text Mining and Semantic Graph Completion

Hasan Abu-Rasheed  University of Siegen, Siegen, Germany  Leibniz University
Hannover, Hannover, Germany  Mareike Dornhöfer  Christian Weber  Gábor
Kismihók  Ulrike Buchmann  Madjid Fathi

(2023)

###### Abstract

Modelling learning objects (LO) within their context enables the learner to
advance from a basic, remembering-level, learning objective to a higher-order
one, i.e., a level with an application- and analysis objective. While
hierarchical data models are commonly used in digital learning platforms,
using graph-based models enables representing the context of LOs in those
platforms. This leads to a foundation for personalized recommendations of
learning paths. In this paper, the transformation of hierarchical data models
into knowledge graph (KG) models of LOs using text mining is introduced and
evaluated. We utilize custom text mining pipelines to mine semantic relations
between elements of an expert-curated hierarchical model. We evaluate the KG
structure and relation extraction using graph quality-control metrics and the
comparison of algorithmic semantic-similarities to expert-defined ones. The
results show that the relations in the KG are semantically comparable to those
defined by domain experts, and that the proposed KG improves representing and
linking the contexts of LOs through increasing graph communities and
betweenness centrality

###### keywords:

Knowledge graphs Graph-based database models Learning context Personalized
learning Text mining

##  1 Introduction

The rapid progress of digitalization in education increased the reachability
of learners to online, open educational recourses (OER), e.g., through massive
open online courses (MOOCs). While this provides more learning opportunities,
an approach of one-size-fits-all on online learning platforms may not work for
the wide spectrum of learners, their learning goals, and learning contexts.
Personalized learning is proven as an essential requirement for better
educational development and learning performance. Moreover, personalization
becomes more essential in situations like self-learning, on-the-job training,
or vocational education and training (VET), where initial training that
learners get, e.g., in formal education, is no longer adequate to the job
requirements [1, 2]. This is due to learning taking place within a context,
where not only multiple learning objectives may overlap based on the job’s
requirements, but also the learner’s background and current state of knowledge
might differ considerably. Modern digital learning platforms need to be more
contextualized and personalized environments to realize an educational setting
that is meaningful to learners, related to their domain, and tailored towards
their preferences and needs, to enhance their capacity-development [1, 2]. To
account for the learning context, personalization approaches, such as
recommender systems (RS), have been developed as context-aware algorithms.
Their efficiency, however, is also bound to the contextual information of
learning materials that are being recommended, not only to the learner
profiles. A personalized, context-aware recommendation is offered when RS
algorithms can match a learner’s context to a context of learning objects (LO)
and materials. Digital learning platforms still often organize learning
content into groups of materials that deploy a hierarchical nature, e.g., as
curricula, which are organized into courses, lectures, topics, and finally
materials. While this organization is suited to represent the details of a
learning objective, it offers no strategy for learning individualization, or
model a cross-domain contextualization, which may indicate a learning
recommendation, e.g., of courses from different curricula that are required in
one application area or one learning context. To consider this context, data
models need to be designed in a flexible and semantically enhanced way, which
can be realized as a knowledge graph (KG) structure [3]. Unlike hierarchical
data structures, KG structures offer the potential to consider relations among
LOs. Transferring a hierarchical data model into a KG is possible by
identifying relations between elements, based on their semantic relatedness
and independent of hierarchical limitations. This identification can be done
by text mining methods. In KGs, LOs are defined as nodes, and the relations
among them are defined as edges. Relations in the KG define one or more types
of relevancies between LOs, thus creating a contextual semantic relation based
on e.g., the domain of the LOs or their textual similarity. In this paper, we
extract information about the shared context of LOs by analyzing their textual
semantics. Contextual information is then integrated into the KG as additional
relations. Groups of contextually related LOs in the KG may be utilized by a
RS, to generate contextual recommendations [4, 5]. We construct the KG
utilizing the relations of a hierarchical data model and the semantic
relations between LOs, representing the context of those objects in the graph.
Semantic relations are extracted with a customized text mining pipeline (TMP),
which is designed to analyze and capture the similarities between LOs based on
their textual descriptions. A language-dependent similarity algorithm is
utilized to mine the semantic similarities between the two description
languages of the input data, namely English and German. Our contributions in
this paper are: 1) Designing a semantic approach for transferring hierarchical
data models in digital learning platforms to graph models. 2) Developing an
approach for capturing and modelling the context of LOs to realize semantic KG
completion based on a semantic TMP. With the later contribution, present
hierarchical structures for LOs can be transformed into contextualizing KGs.
In the following sections, an overview of the research background is discussed
before we describe the KG construction and semantic relation extraction (RE)
among LOs. Then, the evaluation process and results are presented before
concluding the paper.

##  2 Background

Historically, learning is founded in “didactic- and lecture-based methods”
which oftentimes are focused only on memorizing content instead of supporting
“learning transfer” and engendering “problem-solving” [6]. Wilcke, Bloem and
de Boer identified a lack of general data models allowing the representation
of more than one domain simultaneously and are thus unsuitable to incorporate
heterogeneous data, therefore proposing KGs as a default data model [3]. KGs
have been utilized in technology-enhanced learning (TEL) [7, 8, 9] in recent
years. However, the role of the data model in representing the context of LOs
in TEL is still less visited in literature, in comparison to e.g. user
profiling. Here, we define the context based on situative and subject-oriented
learning theories [2], where context is defined as the situation in which
learning happens. This includes the learner’s situation (location, previous
knowledge, etc.) and the LO’s situation (type, length, implementation
scenario, etc.). Context is described by factors that can relate two or more
LOs to each other if they serve the same learning situation [8, 10].
Ontologies and KGs have been identified as effective methods for embedding LO
context into knowledge representation [7, 11]. Furthermore, research shows
that the embedding approaches of contextual information in the KG model and
algorithms enable capturing the complex relations that learners follow to
solve problems [6, 12, 13]. Several approaches have been followed to construct
graph models from LOs [14]. Based on a job-skill hierarchy, authors in [15,
16] utilize the job titles as textual content to create relations between job
representations in the KG. To link jobs and their corresponding skills, Dave
et al. [17] create three graphs, connecting jobs and skills, using the textual
content and existing hierarchical levels. Similarly, de Groot et al. [18]
create a text-based skill matching, with the support of the ESCO framework.
The previous approaches, however, do not explicitly address the role of LOs’
context in the KG and do not go far enough to mine further semantic and
contextualizing relations between LOs.

##  3 Contextual Knowledge Graph Construction

We develop a KG as a contextual model that represents the semantic relations
among LOs. We adopt the taxonomical structure of the digital learning system
in [19], eDoer. This 5-level taxonomy is sufficiently simple and semantically
distinctive for the construction of a solid graph data model. Its levels are
1) Journey: which represents a learning goal, 2) Course: a container that
represents one aspect of the learning goal, 3) Topic: a concrete knowledge
element of a course, 4) Educational Package: a group of related and ordered
OERs, and 5) Educational Content: a concrete OER that can be studied. Semantic
relations in the KG connect LOs within one or more taxonomical levels.
Hereafter, we use the term LO to represent all levels except the Journey. The
resulting KG enables modelling the context of a LO through its connections to
other objects, which appear in the same learning scenario.

###  3.1 Text Mining for Semantic Relation Extraction

When creating new Journeys and their content, content-creators provide
detailed textual descriptions of each object. These descriptions offer the
potential to mine semantic similarities between the objects [5]. Capturing
textual semantics requires text-mining algorithms to consider word meanings
and sentence structures. We designed a customized TPM, shown in Fig.1, to
maximize the use of LO’s textual metadata. The TMP handles objects’ titles
(short texts) and descriptions (long texts), written in two languages (English
and German). To detect the language, we use a pre-trained model for language
detection [20], with a precision of 99% over 53 languages. A text-cleaning
step then removes special characters and strings that can affect the text
similarity algorithm. The structure of the sentences is kept intact, since
text similarities are calculated on a semantic level, not as a bag-of-words
(BOW). Titles and descriptions follow two processing paths in the TMP, due to
the higher volume of description texts, which may include multiple topics.
Such topics provide information about potential relevancies to other LOs.
Therefore, the next step for titles is creating title embeddings, while
descriptions have an additional step of extracting the integrated topics
before the embeddings are created for each extracted topic. To generate title
embeddings that are accurate in English and German, we use the Sentence-BERT
model [21] and the Spacy library [22] for natural language processing (NLP) in
Python. Title embeddings are used to calculate semantic textual similarities
between titles using a cosine similarity algorithm [6]. Titles with different
languages have embeddings created in different corpus spaces. Therefore, we
translate German titles before calculating the cosine similarity. The
translation is done automatically using DeepL API translation service [23].
After extracting the main topics from the description using the KeyBERT model
[24], text embeddings are created. To compare two descriptions, an
intersection matrix is created to determine which topic-pairs from the two
topic sets are compared. The final similarity average is then calculated
across all topics of both LOs. The resulting scores from titles and
descriptions are used to determine if a semantic relation is created between
the two LOs, based on similarity thresholds, which are defined experimentally
and then fine-tuned through expert validation.

Figure 1: Text mining pipeline (TMP) and semantic similarity calculation for
title and description properties of the learning objects.

###  3.2 Knowledge Graph Construction

The structure of our KG includes the original hierarchical model and extends
it with semantic relations calculated by the TMP, see Fig.2. We preserve the
hierarchical relations since they are created by the content creators, and we
add the “has_semantic_relation_to” type to them. In the KG, node types
represent taxonomy levels. The new relation type creates direct and indirect
connections between learning goals (Journeys) and the LOs. For example,
although the Journey “Dementia in elderly care” is not directly connected with
the Journey “Communication in elderly care", several indirect paths were
semantically found through connected Courses and Topics that appear in a
similar learning context. This context appears in Fig.2 in the form of a
densely connected partial network of Topics (area A), whose textual
descriptions revealed that they serve the same scenario. As discussed in [4],
such a network can enable a learner to achieve a higher knowledge level that
enables comprehension, inferencing, and problem-solving. This is accomplished
by learning LOs within their context, in contrast to learning isolated LOs
that may only enable remembering.

Figure 2: First three levels of the KG structure, representing the role of
semantic relations in creating the KG from the hierarchical one.

##  4 Evaluation and Results

The evaluation of KG’s quality utilizes a wide range of metrics, which may
have different indications in different domains. In TEL and VET, it is
important to consider the metrics, whose indications can reflect the role of
the KG in the learning process. KG evaluation should also address the quality
of the extracted relations, as a foundation for evaluating the quality of the
KG structure. Therefore, our evaluation strategy is implemented quantitatively
and qualitatively. Qualitative evaluation is conducted with domain experts to
validate the relation-extraction process and results, as well as the KG’s role
in contextualizing the learning recommendations. Quantitative evaluation is
designed two-fold: The first part evaluates the quality of extracted relations
in the KG. It focuses on the TMP and evaluates its ability to connect LOs
semantically in comparison to expert-created links. The second part considers
network metrics [25] and quality-control methods, as surveyed in [26], where
those are classified in the domain of linked open data (LOD) into six
dimensions: representation, accessibility, intrinsic, dataset dynamicity,
contextual, and trust. To select the metrics that are relevant and meaningful
in the scope of this research, we utilize [6, 27, 13]. They show that
effective evaluation of KG’s ability to represent learning context and the
learner’s mental model can be accomplished through metrics that correspond to
contextual dimension (relevancy and completeness) and intrinsic dimension
(interlinking, density, average degree). We build on those findings and use
the metrics of average degree centrality, clustering coefficient, weakly
connected components, and betweenness centrality.

###  4.1 Evaluation Dataset

For the evaluation, we use a dataset of expert-curated OERs. The data set is
provided by Tavakoli et al. [19]. Expert-curated content provides a high level
of credibility to the baseline model, which is essential for a fair
evaluation. After data exploratory analysis (DEA), we filter out LOs that do
not have educational content associated with them and remove the duplicated
and isolated ones. The resulting hierarchical model we use for creating the KG
features 122 Journeys, 432 Courses, 767 Topics, 2565 Educational packages, and
7358 OERs.

###  4.2 Evaluation of the Semantic Relation Extraction

The goal of semantic relation extraction is to connect LOs that appear in the
same or a similar learning context. This goal aligns with the best practices
of content creators, who connect an LO to another one if both are needed in
the same learning situation. We utilize this logic to evaluate the semantic
similarity scores that our TMP calculates among Courses and Topics. We first
calculate the average of text cosine similarity scores among the LOs in each
Journey (i): $J_{i}^{sim}$, by comparing each LO to all other LOs in Journey
i. Then, we compare the similarity score of each semantic relation
$SR_{i,j}^{sim}$, to the average similarity of the journeys i,j that it
connects $J_{i,j}^{avg_{sim}}=\frac{J_{i}^{sim}+J_{j}^{sim}}{2}$.

If the TMP is able to extract semantic relations with similarity scores that
are comparable to the expert-curated relations, we assume that the new
semantic relation is meaningful. We define comparability here as being within
the same range of both Journeys’ similarity $J^{sim}$. This means that
$SR_{i,j}^{sim}>=J_{i,j}^{avg_{sim}}$. Fig.3 shows the scores of a random
sample of 240 semantic relations in the KG, alongside the semantic similarity
of each Journey-pair they connect. From the results, we can conclude: 1) the
similarity scores calculated within each Journey range between 86% and around
90%. This confirms the basic assumption that expert-curated LOs in Journeys
are semantically similar. 2) 79% of our semantic similarity scores are either
equal or higher than score averages calculated within Journeys, while the
remaining 21% is slightly below the average. This indicates that the semantic
relations between different Journeys are as meaningful as the relations that
experts created within each Journey.

Figure 3: Semantic-relation textual-similarity scores (Purple), compared to
the Journeys (Blue and Orange) connected by those relations through the TMP.

###  4.3 KG Evaluation Metrics

From the wide range of graph quality metrics, our selection is based on the
meaning of each metric in the scope of TEL and VET. Selected metrics reflect
certain aspects of our KG, when interpreted from a pedagogical point of view
in TEL and VET. The comparison here is conducted between the hierarchical
structure and the proposed KG, in the light of the value-added that KG
structure offers based on the interpretation of each graph quality metric. In
the following, we elaborate on the selected metrics and their domain
interpretation. Table 1 shows our KG results in comparison to the original
hierarchical model.

####  4.3.1 Average Degree Centrality (ADC)

Degree centrality (DC) measures a node’s popularity in a graph [28]. It
evaluates the connectedness of a node through its incoming and outgoing
relations. In an educational use-case, ADC reflects LO’s connectedness to
other objects that take place in the learning scenario. The reason is that LOs
are either connected by experts, or through a semantic relation. While DC in a
hierarchical model represents the LO’s relation to other LOs from the same
Journey, the increase of that object’s DC in the graph model shows that new
relations have been found to other Journeys, meaning that the LO may appear in
more learning contexts. After calculating the DC for all graph nodes, the
average (ADC) is determined, which has doubled for our KG.

####  4.3.2 Clustering Coefficient (CC)

CC reflects how strongly a node belongs to a community in a graph. Communities
are clusters of nodes that are densely inter-connected. A community is
considered more modular if it is easy to separate from other communities.
Communities of LOs in our KG represent their context. Therefore, the higher
the community count is, the more contexts are represented. Modularity, on the
other hand, reflects the potential to separate a learning context from
another. Therefore, a lower modularity score is preferred, since it means that
learning contexts are well connected to each other. Evaluation results show
that our KG increased the detected communities, due to the semantic relations,
and decreased graph modularity.

####  4.3.3 Weakly Connected Components (WCC)

WCC detects sets of nodes in the graph that are loosely connected to other
graph parts. Therefore, a high WCC score reflects poor connectedness of LO
groups in the data model. Using Monge and Elkan’s algorithm [29], we find an
effective reduction of the loosely connected node groups in our KG.

####  4.3.4 Betweenness Centrality (BC)

BC describes the impact of a node on the flow of information within that
graph. It is especially relevant to VET and TEL, since LOs with high BC work
as bridges that link individual learning goals to each other, to solve a more
complex problem [4]. Using Brandes and Pich’s algorithm [30], our results show
that the KG has an average BC value of 15.1, which is about 10 times the BC
score of the original hierarchical model.

Table 1: KG Quality Evaluation Against Hierarchical Model. Preferred Value-Trends in VET and TEL Use-Case are Explained  Evaluation Metric | Hierarchical data model | KG | Preferred value trends
---|---|---|---
Average Degree Centrality | 1.079 | 2.262 | increasing
Clustering Coefficient (Number of communities) | 253 | 541 | increasing
Clustering Coefficient (Average modularity score) | 0.779 | 0.636 | decreasing
Weakly Connected Components | 63 | 35 | decreasing
Betweenness Centrality | 1.57 | 15.1 | increasing

###  4.4 Qualitative expert-evaluation

We also evaluated our KG construction and contextualization approach with
domain experts in two focus groups. The groups included experts in VET and
researcher-training programs. A total of nine experts participated and
answered three questions on 1) The role of connecting learning goals in
learning contexts, 2) KGs as contextual data models in their domain, and 3)
Using LO’s textual description to mine contextual relations. All participants
emphasized the need for connecting individual learning goals for solving real-
life, job-related problems. Six participants pointed out that highly connected
nodes in the graph can represent transferrable skills among different domains.
Three experts found a direct use of connecting multi-lingual LOs in their
daily work. They also pointed out that a recommendation of multi-lingual
content should avoid repetition of the same content written in different
languages. Experts also agreed that textual content and description of LOs are
important sources for contextual information. They indicated that content
creators should roughly understand the way algorithms extract semantic
relations, so that they can enrich the descriptions of their LO content with
useful contextual information, allowing better connectivity in the KG. The
pedagogical experts also addressed the dynamic change of the learner’s
context, and the development of the learning domain, which requires a
continuous update of the KG.

##  5 Conclusion

In this paper, we introduced a semantic approach for KG completion to enhance
the contextual representation of LOs for personalized learning systems. A
concept and a text-mining pipeline for relation extraction are proposed, to
transfer hierarchical data models into graph ones, thus enhancing the
structural and contextual quality of the data model. Our findings from the TMP
and KG evaluation suggest that the KG was able to enhance LOs connectivity on
a semantic level. Increased connectivity allowed the KG to represent a context
around an LO, through its relations to other similar LOs, which appear in the
same or similar learning contexts. Proposed TMP can be used with different
hierarchical structures of LOs, such as those generated from other digital
learning platforms, since it utilizes the commonly used title- and description
properties of LOs. A limitation here is the dependency of our TMP on the
volume and quality of LO’s textual metadata. The multilingualism of the
proposed solution corresponded to a real-world challenge in VET, but it also
raised the concern about repetitive content in a KG-based RS. Although our
solution is not responsible for the decision-making process in that scenario,
it can still be further developed to expand the similarity description,
reflecting potential identical content in multiple languages. Further steps of
this research aim to increase the robustness of TMP against textual data
sparsity and enrich LOs contextualization with additional domain-specific
features.

## References

  * Buchmann [2011] U. Buchmann, Subjektbildung und Qualifikation. Ein Beitrag zur Entwicklung berufsbildungswissenschaftlicher Qualifikationsforschung, 2nd ed., Frankfurt a.M., 2011.
  * Buchmann [2022] U. Buchmann,  Vocational-scientific education in qualification design – conditional factors for quality assurance and development of qualifications across educational sectors,  in: V. Rein, J. Wildt (Eds.), Professional-scientific education: discourses, perspectives, implications and options for science and practice, Verlag Barbara Budrich, Opladen Berlin Toronto, 2022, pp. 341–371.
  * Wilcke et al. [2017] X. Wilcke, P. Bloem, V. de Boer,  The knowledge graph as the default data model for learning on heterogeneous knowledge,  DS 1 (2017) 39–57. URL: https://content.iospress.com/articles/data-science/ds007. doi:10.3233/DS-170007.
  * Trumpower et al. [2014] D. L. Trumpower, M. Filiz, G. S. Sarwar,  Assessment for learning using digital knowledge maps,  in: D. Ifenthaler, R. Hanewald (Eds.), Digital Knowledge Maps in Education, Springer New York, New York, NY, 2014, pp. 221–237. doi:10.1007/978-1-4614-3178-7_12.
  * Abu Rasheed et al. [2021] H. Abu Rasheed, C. Weber, J. Zenkert, P. Czerner, R. Krumm, M. Fathi,  A text extraction-based smart knowledge graph composition for integrating lessons learned during the microchip design,  in: K. Arai, S. Kapoor, R. Bhatia (Eds.), Intelligent Systems and Applications, Springer International Publishing, Cham, 2021, pp. 594–610.
  * Giabbanelli et al. [2019] P. J. Giabbanelli, A. A. Tawfik, V. K. Gupta,  Learning analytics to support teachers’ assessment of problem solving: A novel application for machine learning and graph algorithms,  in: D. Ifenthaler, D.-K. Mah, J. Y.-K. Yau (Eds.), Utilizing Learning Analytics to Support Study Success, Springer International Publishing, 2019, pp. 175–199. doi:10.1007/978-3-319-64792-0_11.
  * Ilkou et al. [2021] E. Ilkou, H. Abu-Rasheed, M. Tavakoli, S. Hakimov, G. Kismihók, S. Auer, W. Nejdl,  Educor: An educational and career-oriented recommendation ontology,  in: The Semantic Web – ISWC 2021: 20th International Semantic Web Conference, ISWC 2021, Virtual Event, October 24–28, 2021, Proceedings, Springer-Verlag, Berlin, Heidelberg, 2021, p. 546–562. doi:10.1007/978-3-030-88361-4_32.
  * Verbert et al. [2012] K. Verbert, et al.,  Context-aware recommender systems for learning: A survey and future challenges,  IEEE Trans. Learning Technol. 5 (2012) 318–335. doi:10.1109/TLT.2012.11.
  * Visvizi and Daniela [2019] A. Visvizi, L. Daniela,  Technology-enhanced learning and the pursuit of sustainability,  Sustainability 11 (2019) 4022. doi:10.3390/su11154022.
  * Hemmler and Ifenthaler [2022] Y. M. Hemmler, D. Ifenthaler,  Indicators of the learning context for supporting personalized and adaptive learning environments,  in: 2022 International Conference on Advanced Learning Technologies (ICALT), IEEE, Bucharest, Romania, 2022, pp. 61–65. doi:10.1109/ICALT55010.2022.00026.
  * Berri et al. [2006] J. Berri, R. Benlamri, Y. Atif,  Ontology-based framework for context-aware mobile learning,  in: Proceeding of the 2006 international conference on Communications and mobile computing - IWCMC ’06, ACM Press, Vancouver, British Columbia, Canada, 2006, p. 1307. doi:10.1145/1143549.1143811.
  * Giabbanelli et al. [2022] P. J. Giabbanelli, A. A. Tawfik, B. Wang,  Designing the next generation of map assessment systems: Open questions and opportunities to automatically assess a student’s knowledge as a map,  Journal of Research on Technology in Education (2022). doi:10.1080/15391523.2022.2119449.
  * Kim and McCarthy [2021] M. K. Kim, K. S. McCarthy,  Using graph centrality as a global index to assess students’ mental model structure development during summary writing,  Education Tech Research Dev 69 (2021) 971–1002. doi:10.1007/s11423-021-09942-1.
  * Fettach et al. [2022] Y. Fettach, M. Ghogho, B. Benatallah,  Knowledge graphs in education and employability: A survey on applications and techniques,  IEEE Access 10 (2022) 80174–80183. doi:10.1109/ACCESS.2022.3194063.
  * Decorte et al. [2021] J.-J. Decorte, J. V. Hautte, T. Demeester, C. Develder,  Jobbert: Understanding job titles through skills,  in: arXiv, 2021. doi:10.48550/arXiv.2109.09605.
  * Luo et al. [2019] H. Luo, S. Ma, A. J. B. Selvaraj, Y. Sun,  Learning job representation using directed graph embedding,  in: Proceedings of the 1st International Workshop on Deep Learning Practice for High-Dimensional Sparse Data, ACM, Anchorage Alaska, 2019, pp. 1–5. doi:10.1145/3326937.3341263.
  * Dave et al. [2018] V. S. Dave, B. Zhang, M. A. Hasan, K. AlJadda, M. Korayem,  A combined representation learning approach for better job and skill recommendation,  in: Proceedings of the 27th ACM International Conference on Information and Knowledge Management, ACM, Torino Italy, 2018, pp. 1997–2005. doi:10.1145/3269206.3272023.
  * de Groot et al. [2021] M. de Groot, J. Schutte, D. Graus,  Job posting-enriched knowledge graph for skills-based matching,  arXiv (2021). doi:10.48550/arXiv.2109.02554.
  * Tavakoli et al. [2022] M. Tavakoli, A. Faraji, M. Molavi, S. T. Mol, G. Kismihók,  Hybrid human-ai curriculum development for personalised informal learning environments,  in: LAK22: 12th International Learning Analytics and Knowledge Conference, Association for Computing Machinery, New York, NY, USA, 2022, pp. 563–569. doi:10.1145/3506860.3506917.
  * Nakatani [2010] S. Nakatani, Language detection library for java, 2010. URL: http://code.google.com/p/language-detection/, accessed: Feb. 02, 2023.
  * Reimers and Gurevych [2019] N. Reimers, I. Gurevych,  Sentence-bert: Sentence embeddings using siamese bert-networks,  arXiv (2019). doi:10.48550/arXiv.1908.10084.
  * spa [2023] Library architecture · spacy api documentation, 2023. URL: https://spacy.io/api, accessed Feb. 03, 2023.
  * dee [2023] Deepl api, 2023. URL: https://www.deepl.com/en/docs-api, accessed Feb. 03, 2023.
  * Grootendorst [2021] M. Grootendorst, Maartengr/keybert: Bibtex, 2021. doi:10.5281/zenodo.4461265.
  * Guéret et al. [2012] C. Guéret, P. Groth, C. Stadler, J. Lehmann,  Assessing linked data mappings using network measures,  in: E. Simperl, et al. (Eds.), The Semantic Web: Research and Applications, Springer Berlin Heidelberg, 2012, pp. 87–102. doi:10.1007/978-3-642-30284-8_13.
  * Zaveri et al. [2016] A. Zaveri, et al.,  Quality assessment for linked data: A survey: A systematic literature review and conceptual framework,  SW 7 (2016) 63–93. URL: https://content.iospress.com/articles/semantic-web/sw175. doi:10.3233/SW-150175.
  * Shin and Jeong [2021] H. S. Shin, A. Jeong,  Modeling the relationship between students’ prior knowledge, causal reasoning processes, and quality of causal maps,  Computers & Education 163 (2021) 104113. doi:https://doi.org/10.1016/j.compedu.2020.104113.
  * Freeman [1978] L. C. Freeman,  Centrality in social networks: Conceptual clarification,  Social Networks 1 (1978) 215–239. doi:10.1016/0378-8733(78)90021-7.
  * Monge and Elkan [1997] A. E. Monge, C. Elkan,  An efficient domain-independent algorithm for detecting approximately duplicate database records,  in: Workshop on Research Issues on Data Mining and Knowledge Discovery, DMKD 1997 in cooperation with ACM SIGMOD’97, Tucson, Arizona, USA, 1997.
  * Brandes and Pich [2007] U. Brandes, C. Pich,  Centrality estimation in large networks,  International Journal of Bifurcation Chaos 17 (2007) 2303–2318. doi:10.1142/S0218127407018403.