# Improving AlphaFlow for Efficient Protein Ensembles Generation

Shaoning Li  Mingyu Li  Yusong Wang  Xinheng He  Nanning Zheng  Jian Zhang
Pheng Ann Heng

###### Abstract

Investigating conformational landscapes of proteins is a crucial way to
understand their biological functions and properties. AlphaFlow stands out as
a sequence-conditioned generative model that introduces flexibility into
structure prediction models by fine-tuning AlphaFold under the flow-matching
framework. Despite the advantages of efficient sampling afforded by flow-
matching, AlphaFlow still requires multiple runs of AlphaFold to finally
generate one single conformation. Due to the heavy consumption of AlphaFold,
its applicability is limited in sampling larger set of protein ensembles or
the longer chains within a constrained timeframe. In this work, we propose a
feature-conditioned generative model called AlphaFlow-Lit to realize efficient
protein ensembles generation. In contrast to the full fine-tuning on the
entire structure, we focus solely on the light-weight structure module to
reconstruct the conformation. AlphaFlow-Lit performs on-par with AlphaFlow and
surpasses its distilled version without pretraining, all while achieving a
significant sampling acceleration of around 47 times. The advancement in
efficiency showcases the potential of AlphaFlow-Lit in enabling faster and
more scalable generation of protein ensembles.

Machine Learning, ICML

##  1 Introduction

Exploring conformational landscapes is essential to capture the dynamic nature
of protein structures, offering insights into their flexibility, biological
function, and interactions. Traditionally, ensembles of conformational changes
are collected through molecular dynamics (MD) simulations (Karplus & McCammon,
2002). While MD-base methods adhere to physical laws and will theoretically
explore the entire landscape, they are time- and resource-intensive. To
expedite this process, some methods focus on increasing the diversity of
AlphaFold (Jumper et al., 2021), which is a powerful deep learning model for
crystal structure prediction but falls short in accounting for conformational
divergence. Specifically, these methods sample different multiple sequence
alignments (MSAs) as input (Wayment-Steele et al., 2024) or enable the dropout
function in AlphaFold (Wallner, 2023) during the inference process. Despite
these inference interventions indeed bring some diversity to AlphaFold, they
fall significantly short of generating enough conformational heterogeneity to
thoroughly explore the protein landscape.

Figure 1:  Model architecture of sequence-conditioned AlphaFlow (left) and
feature-conditioned AlphaFlow-Lit (right). $T$: Denoising steps; $x_{t}$:
Noisy structure; $\tilde{x}_{0}$: Predicted structure.

Recently, (Jing et al., 2024) harnessed the power of generative methods flow
matching, and integrated this framework into AlphaFold, called AlphaFlow. To
be concrete, it treats AlphaFold as a powerful sequence-conditioned denoising
model, which receives the noisy structures as templates and samples the
protein ensembles from harmonic prior under a flow field. AlphaFlow inherits
the weights of AlphaFold, and was trained on general PDB then fine-tuned on
different protein MD trajectories as a regression model, using loss functions
similar to those in the original AlphaFold. Due to these enhancements,
AlphaFlow is much more flexible and diverse than the aforementioned inference
intervention methods. It is the first method to ingeniously combine both the
advantages of accurate structure prediction and the generative capability for
conformation sampling.

However, limitations persist in sampling consumption. As shown in Fig. 1,
since AlphaFlow is trained by fully fine-tuning AlphaFold, generating the
final structure $\hat{x}_{0}$ requires $T$ denoising steps, which means
running $T$ times AlphaFold with additional embedders. Although flow matching
method is relatively faster compared with other diffusion methods, as shown in
Fig. 2(A), AlphaFlow showcases cubic growth with the chain length, which leads
to unacceptable time consumption and hinders its application for generating
larger set of protein ensembles. While AlphaFlow adopts diffusion distillation
to reduce the generative process to a single forward pass, this approach
compromises the level of the sampling performance.

To address this issue, we propose a feature-conditioned generative model
called AlphaFlow-Lit, which can be treated as an efficient and lighter version
of AlphaFlow. As demonstrated in AlphaMissense (Cheng et al., 2023), features
derived from MSAs encoder could be further utilized for variants
classification. The lastest AlphaFold3 also employs these features to train a
non-equivariant denoiser (Abramson et al., 2024) for the adaption of multi-
modality such as nucleic acids, small molecules, ions, and modified residues.
Also, as shown in (Jing et al., 2024), the MSAs have relatively minor impact
on structural diversity compared with the flow matching framework. Inspired by
these findings, as illustrated in Fig. 1, AlphaFlow-Lit retains the original
AlphaFold embedder and Evoformer blocks in a frozen state and is directly
conditioned on computed single and pair features. Given that the remaining
structure module and auxiliary heads are significantly lighter than the
Evoformer block, compared to AlphaFlow, AlphaFlow-Lit can achieve a faster
sampling process (around 47 times speedup) with the same number of denoising
steps. When directly trained on ATLAS dataset (Vander Meersche et al., 2024)
of protein MD trajectories, AlphaFlow-Lit performs on-par with AlphaFlow and
surpasses its distilled version. We also provide the analysis of the generated
MD ensembles from AlphaFlow-Lit, including protein dynamics analysis, local
arrangements within residue, and long-range correlations among residues to
illustrate the diverse capabilities of AlphaFlow-Lit.

Figure 2:  Visualization of MD evaluation from MD, AlphaFlow-Lit and
AlphaFlow. (A) Runtime comparison corresponding to the sequence length and
their fitted curves. (B) Principal components analysis (PCA) for 6q9c_A
ensembles. The representative structures are pointed out. (C, D) Ensembles of
PDB ID 7buy_A with C$\alpha$ RMSF by residue index shown in insets, and their
Dynamic cross-correlation matrix (DCCM).

##  2 Preliminary

In this section, we briefly introduce the flow matching framework and some
details of AlphaFlow.

### Flow matching

The flow matching framework begins with the continuous normalizing flow (CNF)
$\psi_{t}$, defined as the solution of an ordinary differential equation (ODE)
governed by a time-dependent vector field
$u_{t}:\frac{d}{dt}\psi_{t}=u_{t}(\psi_{t})$. Let $x$ be a data point on a
specific manifold, the CNF has an initial condition $\psi_{0}(x)=x$. Given two
distributions $p_{0}$ and $p_{1}$, we can define a probability path $p_{t}$ as
their interpolation, which can be viewed as paths generated by $u_{t}$. To
effectively learn the CNF, we make $u_{t}$ tractable and by adopting the
conditional probability path $p_{t}(x|x_{1})$, which samples $x_{0}$ from
prior distribution $p_{0}$ and interpolating it linearly with the data point
$x_{1}$:

$x_{t}=(1-t)\cdot x_{0}+t\cdot x_{1}$ (1)

with the corresponding vector field:

$u_{t}(x_{t}|x_{1})=(x_{1}-x_{t})/(1-t)$ (2)

This method is referred as conditional flow matching (CFM). We employ a neural
network $v_{t}^{\theta}$ to learn the vector field. The objective of CFM can
be written as:

$\mathcal{L}_{\operatorname{CFM}}=\mathbb{E}_{t,p(x_{0}),p(x_{1})}\left\|v_{t}^{\theta}(x_{t})-u_{t}(x_{t}|x_{1})\right\|^{2}$
(3)

### AlphaFlow

To integrate the AlphaFold, which is a regression model that directly outputs
$x_{1}$, into the flow matching framework, AlphaFlow reparameterizes the
neural vector field as:

$v_{t}^{\theta}(x_{t})=(\operatorname{AlphaFold}(x_{t})-x_{t})/(1-t)$ (4)

It allows the objective to be rewritten as learning the expectation of
$x_{1}$. Consequently, AlphaFlow can employ the similar regression loss
function (e.g. FAPE) to optimize the neural network. AlphaFlow introduces two
key innovations: (a) It employs the 3D coordinates of its $\beta$-carbons
($\alpha$-carbon for glycine) to describe the noisy structure and the prior
distribution is defined over the $\beta$-carbons coordinates as a harmonic
prior (Jing et al., 2023); (b) AlphaFlow treats $x_{t}$ as features (similar
to templates), and the denoising process does not directly apply to the
spatial domain as in prevailing SE(3) generative models (Yim et al., 2023;
Bose et al., 2023; Li et al., 2024). Instead, it starts from the identity
rigids, which is the same as AlphaFold. These contributions make AlphaFlow as
a new paradigm for utilizing AlphaFold within different frameworks or
applications.

##  3 Method

AlphaFlow-Lit follows the same ideas of AlphaFlow but introduces some
modifications to the input pipeline. AlphaFlow-Lit is a feature-conditioned
generative model, that is to say, it is conditioned on the single and pair
features after the Evoformer blocks to generate diverse conformations. As
illustrated in Fig. 1, the AlphaFold embedders (including the original input
embedder, recycling embedder, extra MSA embedder, and extra MSA stack) and
Evoformer are kept frozen. The input embedding module for noisy structures
$x_{t}$ and timesteps is similar to that of AlphaFlow but with little
modifications. The single and pair output of input embedding module are
derived from the torsion angles (if designated) and contact map of the noisy
structure, respectively. These outputs are followed by a Linear layer
initialized with zeros before summation with the features after Evoformer
blocks. This is similar to the zero convolution in ControlNet (Zhang et al.,
2023), designed to minimally disrupt the pretrained weights at the outset. The
detailed algorithm for input embedding module is described in Appendix A
Algorithm 2. It is worth noting that the torsion angles in AlphaFold are
represented in 8 rigids groups with $\sin-\cos$ formats, indicating rotation
towards the coordinates of the former group. As a result, these angles are
invariant to rigid transformations, eliminating the need to rotate the
predicted structure after RMSD alignment. The training procedure for
AlphaFlow-Lit is the same as for AlphaFlow, and the inference procedure is
provided in Algorithm 1. We keep the Algorithm notations same as (Jing et al.,
2024). The acceleration primarily results from the pre-computation of single
and pair features. In contrast to AlphaFlow, AlphaFlow-Lit does not
necessitate running Evoformer blocks at each denoising step but only conducts
this process once at the inception. Under this circumstances, the denoising
network in AlphaFlow-Lit is a lightweight structure module that is conditioned
on single and pair features rather than MSAs. It distinguishes AlphaFlow-Lit
as feature-conditioned and contributes to its efficiency and speed in
generating protein emsembles.

Table 1: Evaluation on MD ensembles. We compare the predicted ensemble from
AlphaFlow-Lit and AlphaFlow with the ground truth MD ensemble according to
various metrics. For Pairwise RMSD and Per-target RMSF, the ground truth
values (from the MD ensembles) are in parenthesis. Following (Jing et al.,
2024), the median across the 82 test ensembles is reported. $p$: Pearson
correlation; $JSD$: Jensen-Shannon divergence.

|  | AlphaFlow-Full | AlphaFlow-Lit | AlphaFlow-Distilled
---|---|---|---|---
Protein dynamics | Pairwise RMSD ($=2.90$) | 2.89 | 2.43 | 1.94
Pairwise RMSD $p$ $\uparrow$ | 0.49 | 0.58 | 0.49
PCA $C_{\alpha}$ coordinates $JSD$ $\downarrow$ | 0.43 | 0.46 | 0.51
PCA $C_{\alpha}$ pairwise distance $JSD$ $\downarrow$ | 0.48 | 0.52 | 0.56
Local arrangements within residues | Per-target RMSF ($=1.94$) | 1.88 | 1.65 | 1.34
Per-target RMSF $p$ $\uparrow$ | 0.75 | 0.77 | 0.71
Stable contacts $JSD$ $\downarrow$ | 0.84 | 0.83 | 0.79
Dihedral distributions $JSD$ $\downarrow$ | 0.47 | 0.51 | 0.57
Long-range correlations among residues | DCCM $p$ $\uparrow$ | 0.78 | 0.78 | 0.74

Algorithm 1 Inference

Input: Sequence and MSA $(A,M)$

Output: Sampled all-atom structure $\hat{S}$

Sample
$\mathbf{x}_{0}\sim\operatorname{HarmonicPrior}(\operatorname{length}(A))$

$\mathbf{f}^{\text{tor}}_{0}=\emptyset$

$\mathbf{s}_{i}^{\text{evo}},\mathbf{z}_{ij}^{\text{evo}}\leftarrow\operatorname{Evoformer}(\operatorname{AlphaFoldEmbedder}(A,M))$

for $n\leftarrow 0$ to $N-1$ do

Let $t\leftarrow n/N$ and $s\leftarrow t+1/N$

$\mathbf{s}_{i},\mathbf{z}_{ij}\leftarrow\operatorname{InputEmbedder}(\mathbf{x}_{t},\mathbf{f}^{\text{tor}}_{t},t)$

$\mathbf{s}_{i},\mathbf{z}_{ij}\mathrel{+}=\mathbf{s}_{i}^{\text{evo}},\mathbf{z}_{ij}^{\text{evo}}$

Predict
$\hat{S}\leftarrow\operatorname{StructureModule}(\mathbf{s}_{i},\mathbf{z}_{ij},A)$

if $n=N-1$ then

return $\hat{S}$

end if

Extract $\hat{\mathbf{x}}_{1}\leftarrow\operatorname{BetaCarbons}(\hat{S})$

Align
$\mathbf{x}_{t}\leftarrow\operatorname{RMSDAlign}(\mathbf{x}_{t},\hat{\mathbf{x}}_{1})$

Interpolate
$\mathbf{x}_{s}\leftarrow\frac{s-t}{1-t}\cdot\hat{\mathbf{x}}_{1}+\frac{1-s}{1-t}\cdot\mathbf{x}_{t}$

if embed_angles then

Extract
$\mathbf{f}^{\text{tor}}_{t}\leftarrow\operatorname{TorsionAngle}(\hat{S})$

else

$\mathbf{f}^{\text{tor}}_{t}\leftarrow\emptyset$

end if

end for

##  4 Experiments

We directly train AlphaFlow-Lit on ALTAS MD trajectories (Vander Meersche et
al., 2024) without pretraining on the PDB. Similar to AlphaFlow, we use
1265/39/82 ensembles for the training, validation, and test splits,
respectively. All multiple sequence alignments (MSAs) are derived from
OpenProteinSet (Ahdritz et al., 2024). For sequences not present in
OpenProteinSet, we use MMseqs2 (Steinegger & Söding, 2017) to search the
UniRef30 and ColabDB databases (Mirdita et al., 2022). The initial weight of
AlphaFlow-Lit is from the AlphaFold’s publicly available weights. ALTAS
provides three parallel trajectories with 10,001 frames for each protein. We
subsample the trajectories with a stride of 100 frames to create the training
set (300 frames in total). During training, we uniformly sample one frame at
each step. Since the Evoformer blocks are frozen, we set the weight of masked
MSA loss to 0.

We generate 250 samples for 82 targets in the test set. AlphaFlow-Full refers
to AlphaFlow with 10 consecutive denoising steps. AlphaFlow-Distilled denotes
its distilled with a single forward denoising step. AlphaFlow-Lit employs the
full denoising steps. The protein ensembles of AlphaFlow-Full and AlphaFlow-
Distilled are downloaded from its public
repository111https://github.com/bjing2016/alphaflow. We assess the sampling
runtime based on the protein length for these methods and approximate their
consumption curve. To evaluate the effectiveness of each method, we first
investigate the protein dynamics, considering both the general dynamics
indicated by the pairwise root-mean-square deviation (RMSD) and the essential
dynamics uncovered through principal components analysis (PCA) (Amadei et al.,
1993). Furthermore, we assess the detailed capability of each method at the
residue resolution of protein dynamics by systematic comparisons of local
arrangements within residues and motional correlations among residues. The
results are presented in Table 1 and Fig. 2.

### Runtime comparison

In Figure 2, we depict the relation between runtime of sampling and sequence
length ranging from 100AA to 1,000AA in increments of 100. AlphaFlow-Lit
demonstrate superior scalability, maintaining consistently low runtime across
increasing protein lengths. AlphaFlow exhibit cubic growth in runtime,
indicating its inefficiency for longer chains. This inefficiency could be
attributed to the cubic complexity of attention in the Evoformer block. While
AlphaFlow-Distilled performs better than AlphaFlow-Full due to its single
forward inference, it still shows moderate increases in runtime as protein
length grows. In summary, AlphaFlow-Lit surpasses AlphaFlow by 6 to 51 times
(47 times in average) and AlphaFlow-Distilled by 2 to 4 times (3.8 times in
average), making it the most efficient configuration and highlighting its
potential for generating a larger set of protein ensembles.

### Protein dynamics analysis

For each conformational ensemble, the general dynamics are quantified as the
average $C_{\alpha}$-RMSD between any pair of conformations. Using this
measurement, the AlphaFlow-Lit ensembles demonstrate the strongest Pearson
correlation with the ground truth ensembles produced by classic MD, while
maintaining a comparable level of diversity in the conformational ensembles
relative to AlphaFlow-Full. In contrast, AlphaFlow-Distilled loosely match the
general dynamics with the ground truth and does not achieve the same level of
diversity. Also, we assess the essential dynamics of proteins by projecting
the ensembles onto the first two principal components (PCs) derived from PCA.
Two common featurization methods for proteins are utilized: aligned
$C_{\alpha}$ absolute coordinates and pairwise $C_{\alpha}$ internal
distances. The differences in the distributions are quantified using the mean
Jensen-Shannon divergence (JSD) for each PC between the predicted and true
ensembles. Likewise, AlphaFlow-Lit exhibits essential dynamics distributions
that are comparable to those of AlphaFlow-Full, surpassing the performance of
AlphaFlow-Distilled. We visualize an example 6q9c_A in Fig. 2(B) and extract
the representative structures. Both AlphaFlow-Full and AlphaFlow-Lit highly
align with one of the principal component distributions of molecular dynamics
(MD) However, they do not capture the other distribution, which, although
relatively minor, is still significant.

### Local arrangements analysis

Allostery, which has been coined the second secret of life after genetic
codes, is a fundamental mechanism underlying most protein dynamics (Fenton,
2008). To further evaluate the practical effects of each method in identifying
residues that undergo conformational changes in the local environment—changes
that are likely critical for allostery—we performed a multifaceted analysis at
the residue resolution, including thermally averaged flexibility, residual
contact probabilities, and key dihedral angles distribution. In terms of
thermally averaged flexibility, we calculate the root-mean-square fluctuation
(RMSF) at the residue level, represented by $C_{\alpha}$. AlphaFlow-Lit
achieves a strong Pearson correlation of 0.77 between the predicted and actual
ensembles within a target, whereas AlphaFlow-Distilled only exhibits a
moderate Pearson correlation of 0.71. We visualize the RMSF of 7buy_A in Fig.
2(C). We observe that the decrease in Pearson correlation of AlphaFlow is due
to the high diversity of the structure’s end point. If we exclude the first 5
residues and recompute the RMSF Pearson correlation (values in parentheses),
both AlphaFlow and AlphaFlow-Lit will yield identical results. In addition,
contact probability analysis is utilized to elucidate the conformational
rearrangements concerning the relative positions and orientations of
structural motifs. Meanwhile, their internal conformational displacements are
more accurately represented through variations in key dihedral angles. For
contact probability analysis, a stable contact is defined as $C_{\alpha}$
pairs that maintain contact (with a threshold of 7 Å) in over 85% of the
conformational ensembles. The Jaccard similarity (JS) between the contact
residue pairs is calculated between the predicted and the ground truth sets.
Regarding key dihedral angles, in addition to the backbone phi and psi angles,
the chi1 values are also included, as they may indicate significant side chain
reorientations relevant to the formation of key polar or non-polar
interactions. The results show that contact and dihedral distributions exhibit
moderate consistency between the actual and predicted ensembles generated by
AlphaFlow-Full and AlphaFlow-Lit, unlike the larger inconsistencies observed
with AlphaFlow-Distilled.

### Long-range correlations analysis

Finally, we analyze the motional correlations similarity among residues by
calculating the dynamic cross-correlation map (DCCM) using the conformational
ensembles generated by different methods (Hünenberger et al., 1995). Such
correlations could reveal pivot residues that mediate long-range allosteric
coupling (McClendon et al., 2009). AlphaFlow-Lit shows higher similarity in
these matrices compared to AlphaFlow-Distilled, underscoring its superior
ability to capture couplings even among long-range residues. We visualize the
DCCM of 7buy_A in Fig. 2(D).

##  5 Conclusion

We propose AlphaFlow-Lit, an improved version of AlphaFlow for efficient
protein ensembles generation. Compared with AlphaFlow, AlphaFlow-Lit is a
feature-conditioned generative model that eliminates the heavy reliance on
MSAs encoding blocks and utilizes computed features to produce a diverse range
of conformations. By directly training on ATLAS, AlphaFlow-Lit performs on-par
with AlphaFlow while outperforming its distilled version, all while achieving
a substantial acceleration in sampling speed of around 47 times. In addition,
we conduct a thorough analysis of protein dynamics, local arrangements, and
long-range coupling within the generated ensembles. The advantages of
AlphaFlow-Lit make it capable of generating a larger set of protein ensembles,
enabling us to more effectively explore the protein landscape using deep
learning techniques.

### Limitation and future work

As illustrated in the Experiment section, AlphaFlow-Lit exhibits less
diversity compared to AlphaFlow-Full, likely due to the absence of pretraining
on the PDB or insufficient training on MD trajectories. This will be addressed
in future work. Additionally, in the PCA analysis of example 6q9c_A, both
models fail to capture the additional conformation present in the ground truth
MD distribution. Enhancing their capability to capture such nuances will be a
focus of our future research.

## Acknowledgement

This paper is supported by National Key Research and Development Program of
China (2023YFF1205103), National Natural Science Foundation of China
(81925034), National Natural Science Foundation of China (62088102) and a
grant from the Hong Kong Innovation and Technology Fund (Project No.
ITS/241/21). We thank the open-source codebases from OpenFold Team and Bowen
Jing (AlphaFlow). We thank Dr. Le Zhuo for valuable discussions.

## References

  * Abramson et al. (2024) Abramson, J., Adler, J., Dunger, J., Evans, R., Green, T., Pritzel, A., Ronneberger, O., Willmore, L., Ballard, A. J., Bambrick, J., et al.  Accurate structure prediction of biomolecular interactions with alphafold 3.  _Nature_ , pp. 1–3, 2024.
  * Ahdritz et al. (2024) Ahdritz, G., Bouatta, N., Floristean, C., Kadyan, S., Xia, Q., Gerecke, W., O’Donnell, T. J., Berenberg, D., Fisk, I., Zanichelli, N., et al.  Openfold: Retraining alphafold2 yields new insights into its learning mechanisms and capacity for generalization.  _Nature Methods_ , pp. 1–11, 2024.
  * Amadei et al. (1993) Amadei, A., Linssen, A. B., and Berendsen, H. J.  Essential dynamics of proteins.  _Proteins: Structure, Function, and Bioinformatics_ , 17(4):412–425, 1993.
  * Bose et al. (2023) Bose, J., Akhound-Sadegh, T., FATRAS, K., Huguet, G., Rector-Brooks, J., Liu, C.-H., Nica, A. C., Korablyov, M., Bronstein, M. M., and Tong, A.  Se (3)-stochastic flow matching for protein backbone generation.  In _The Twelfth International Conference on Learning Representations_ , 2023.
  * Cheng et al. (2023) Cheng, J., Novati, G., Pan, J., Bycroft, C., Žemgulytė, A., Applebaum, T., Pritzel, A., Wong, L. H., Zielinski, M., Sargeant, T., et al.  Accurate proteome-wide missense variant effect prediction with alphamissense.  _Science_ , 381(6664):eadg7492, 2023.
  * Fenton (2008) Fenton, A. W.  Allostery: an illustrated definition for the ‘second secret of life’.  _Trends in biochemical sciences_ , 33(9):420–425, 2008.
  * Hünenberger et al. (1995) Hünenberger, P., Mark, A., and Van Gunsteren, W.  Fluctuation and cross-correlation analysis of protein motions observed in nanosecond molecular dynamics simulations.  _Journal of molecular biology_ , 252(4):492–503, 1995.
  * Jing et al. (2023) Jing, B., Erives, E., Pao-Huang, P., Corso, G., Berger, B., and Jaakkola, T. S.  Eigenfold: Generative protein structure prediction with diffusion models.  In _ICLR 2023-Machine Learning for Drug Discovery workshop_ , 2023.
  * Jing et al. (2024) Jing, B., Berger, B., and Jaakkola, T.  Alphafold meets flow matching for generating protein ensembles.  _arXiv preprint arXiv:2402.04845_ , 2024.
  * Jumper et al. (2021) Jumper, J., Evans, R., Pritzel, A., Green, T., Figurnov, M., Ronneberger, O., Tunyasuvunakool, K., Bates, R., Žídek, A., Potapenko, A., et al.  Highly accurate protein structure prediction with alphafold.  _Nature_ , 596(7873):583–589, 2021.
  * Karplus & McCammon (2002) Karplus, M. and McCammon, J. A.  Molecular dynamics simulations of biomolecules.  _Nature structural biology_ , 9(9):646–652, 2002.
  * Li et al. (2024) Li, S., Wang, Y., Li, M., Shao, B., Zheng, N., Jian, Z., and Tang, J.  F3low: Frame-to-frame coarse-grained molecular dynamics with se(3) guided flow matching.  In _ICLR 2024 Workshop on Generative and Experimental Perspectives for Biomolecular Design_ , 2024.
  * McClendon et al. (2009) McClendon, C. L., Friedland, G., Mobley, D. L., Amirkhani, H., and Jacobson, M. P.  Quantifying correlations between allosteric sites in thermodynamic ensembles.  _Journal of chemical theory and computation_ , 5(9):2486–2502, 2009.
  * Mirdita et al. (2022) Mirdita, M., Schütze, K., Moriwaki, Y., Heo, L., Ovchinnikov, S., and Steinegger, M.  Colabfold: making protein folding accessible to all.  _Nature methods_ , 19(6):679–682, 2022.
  * Steinegger & Söding (2017) Steinegger, M. and Söding, J.  Mmseqs2 enables sensitive protein sequence searching for the analysis of massive data sets.  _Nature biotechnology_ , 35(11):1026–1028, 2017.
  * Vander Meersche et al. (2024) Vander Meersche, Y., Cretin, G., Gheeraert, A., Gelly, J.-C., and Galochkina, T.  Atlas: protein flexibility description from atomistic molecular dynamics simulations.  _Nucleic Acids Research_ , 52(D1):D384–D392, 2024.
  * Wallner (2023) Wallner, B.  Afsample: improving multimer prediction with alphafold using massive sampling.  _Bioinformatics_ , 39(9):btad573, 2023.
  * Wayment-Steele et al. (2024) Wayment-Steele, H. K., Ojoawo, A., Otten, R., Apitz, J. M., Pitsawong, W., Hömberger, M., Ovchinnikov, S., Colwell, L., and Kern, D.  Predicting multiple conformations via sequence clustering and alphafold2.  _Nature_ , 625(7996):832–839, 2024.
  * Yim et al. (2023) Yim, J., Campbell, A., Foong, A. Y., Gastegger, M., Jiménez-Luna, J., Lewis, S., Satorras, V. G., Veeling, B. S., Barzilay, R., Jaakkola, T., et al.  Fast protein backbone generation with se (3) flow matching.  _arXiv preprint arXiv:2310.05297_ , 2023.
  * Zhang et al. (2023) Zhang, L., Rao, A., and Agrawala, M.  Adding conditional control to text-to-image diffusion models.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pp. 3836–3847, 2023.

##  Appendix A Method Details.

We highlight the difference with AlphaFlow in yellow.
$\operatorname{init}=\operatorname{`final^{\prime}}$ indicates the weight and
bias (if has) of $\operatorname{Linear}$ is initialized with 0. Other
notations are kept same as (Jumper et al., 2021) and (Jing et al., 2024).

Algorithm 2 InputEmbedding

Input: Beta carbon coordinates $\mathbf{x}\in\mathbb{R}^{N\times 3}$, Torsion
angles $\mathbf{f}^{\text{tor}}$, time $t\in[0,1]$

Output: Input pair embedding $\mathbf{z}\in\mathbb{R}^{N\times N\times 64}$

$\mathbf{z}_{ij}\leftarrow\lVert\mathbf{x}_{i}-\mathbf{x}_{j}\rVert$

$\mathbf{z}_{ij}\leftarrow\operatorname{Bin}(\mathbf{z}_{ij},{\min}=3.25\text{
\AA},{\max}=50.75\text{ \AA},N_{\text{bins}}=39)$

$\mathbf{z}_{ij}\leftarrow\operatorname{Linear}(\hbox{\pagecolor{yellow}$\displaystyle\operatorname{concat}(\operatorname{OneHot}(\mathbf{z}_{ij}),\mathbf{f}_{ij}^{\operatorname{mask\\_2d}})$})$

for $l\leftarrow 1$ to $N_{\text{blocks}}=4$ do

$\\{\mathbf{z}\\}_{ij}\mathrel{+}=\operatorname{TriangleAttentionStartingNode}({\mathbf{z}_{ij}},c=64,N_{\text{head}}=4)$

$\\{\mathbf{z}\\}_{ij}\mathrel{+}=\operatorname{TriangleAttentionEndingNode}({\mathbf{z}_{ij}},c=64,N_{\text{head}}=4))$

$\\{\mathbf{z}\\}_{ij}\mathrel{+}=\operatorname{TriangleMultiplicationOutgoing}({\mathbf{z}_{ij}},c=64)$

$\\{\mathbf{z}\\}_{ij}\mathrel{+}=\operatorname{TriangleMultiplicationIncoming}({\mathbf{z}_{ij}},c=64)$

$\\{\mathbf{z}\\}_{ij}\mathrel{+}=\operatorname{PairTransition}({\mathbf{z}_{ij}},n=2)$

end for

$\mathbf{z}_{ij}\leftarrow\operatorname{Linear}(\mathbf{z}_{ij},\hbox{\pagecolor{yellow}$\displaystyle\operatorname{init}=\operatorname{`final^{\prime}}$})$

$\mathbf{z}_{ij}\mathrel{+}=\operatorname{Linear}(\operatorname{GaussianFourierEmbedding}(t,\hbox{\pagecolor{yellow}$\displaystyle
d=128$}),\hbox{\pagecolor{yellow}$\displaystyle\operatorname{init}=\operatorname{`final^{\prime}}$})$

if embed_angles and $\mathbf{f}^{\text{tor}}\neq\emptyset$ then

$\displaystyle\mathbf{s}_{i}\leftarrow\operatorname{Linear}(\operatorname{concat}(\operatorname{OneHot}(\mathbf{f}^{\text{tor}}_{i},\mathbf{f}^{\text{mask\\_tor}})))$

$\displaystyle\mathbf{s}_{i}\leftarrow\operatorname{Linear}(\mathbf{s}_{i},\operatorname{init}=\operatorname{`final^{\prime}})$

else

$\mathbf{s}_{i}\leftarrow\mathbf{0}$

end if

##  Appendix B Runtime comparison

Table 2: Sampling runtime across AlphaFlow, AlphaFlow-Lit and AlphaFlow-
Distilled in Fig. 2(A). Proteins are selected from ATLAS. All methods are
conducted on a single A100 GPU. Runtime values are reported in second.

PDB ID | Seq. length | AlphaFlow-Full | AlphaFlow-Distilled | AlphaFlow-Lit
---|---|---|---|---
5h6x_A | 100 | 6.63 | 0.86 | 0.76
2q9r_A | 200 | 14.75 | 1.48 | 0.81
2v4b_B | 300 | 27.74 | 2.76 | 0.85
1ru4_A | 400 | 44.98 | 4.46 | 1.10
2d5b_A | 500 | 68.97 | 6.82 | 1.50
6zsl_B | 603 | 108.96 | 10.75 | 2.20
6lrd_A | 705 | 153.57 | 14.93 | 3.00
4ys0_A | 824 | 192.23 | 19.06 | 3.94
3nci_A | 903 | 283.16 | 29.83 | 5.44
1gte_D | 1025 | 403.68 | 40.97 | 7.89


11institutetext: University of Applied Sciences of the Grisons, 7000 Chur,
Switzerland 11email<EMAIL_ADDRESS>22institutetext: webLyzard
technology, 1090 Vienna, Austria

# Slot Filling for Extracting Reskilling and Upskilling Options from the Web

Albert Weichselbraun 1122 0000-0001-6399-045X Roger Waldvogel 11
0000-0001-5191-9816 Andreas Fraefel 11 Alexander van Schie 11 Philipp
Kuntschik 11

###### Abstract

Disturbances in the job market such as advances in science and technology,
crisis and increased competition have triggered a surge in reskilling and
upskilling programs. Information on suitable continuing education options is
distributed across many sites, rendering the search, comparison and selection
of useful programs a cumbersome task.

This paper, therefore, introduces a knowledge extraction system that
integrates reskilling and upskilling options into a single knowledge graph.
The system collects educational programs from 488 different providers and uses
context extraction for identifying and contextualizing relevant content.
Afterwards, entity recognition and entity linking methods draw upon a domain
ontology to locate relevant entities such as skills, occupations and topics.
Finally, slot filling integrates entities based on their context into the
corresponding slots of the continuous education knowledge graph.

We also introduce a German gold standard that comprises 169 documents and over
3800 annotations for benchmarking the necessary content extraction, entity
linking, entity recognition and slot filling tasks, and provide an overview of
the system’s performance.

###### Keywords:

content extraction knowledge extraction knowledge base population entity
recognition entity classification entity linking slot filling gold standard

##  1 Introduction

The automated extraction of structured knowledge from Web content for
knowledge base population is a challenging task, since it requires combining
content extraction and context aware knowledge extraction. In the continuing
education domain, for example, Web pages promoting courses and degree programs
often contain information on learning outcomes and course prerequisites which
use entities of similar types (e.g., skills, degrees, etc.). Correctly
interpreting these entities requires contextual knowledge of the section in
which they appear.

The presented research has been motivated by an industry project, which aims
at creating a knowledge graph of national and international educational
programs relevant to the Swiss reskilling and upskilling market. To improve
the performance of knowledge extraction tasks, the industry partner
contributed a comprehensive domain ontology which formalizes domain knowledge
such as occupations, skills, topics and positions.

Once completed, the system will cover almost 100,000 educational programs
spanning a heterogeneous set of sources such as academic programs, continuing
education certificates, courses, seminars, and international online courses.
The created knowledge graph will power a search platform and recommender
system that will support users in locating suitable reskilling and upskilling
programs.

The main contributions of this paper are: (i) the application of content
extraction and knowledge extraction methods to a complex industry-driven
setting that requires building a knowledge graph of educational offerings
suitable for reskilling and upskilling; (ii) the introduction of methods
capable of performing the required complex slot filling task on Web pages
retrieved from numerous different education providers; (iii) the integration
of background knowledge available in the industry partner’s domain ontology
with state-of-the-art content extraction and knowledge extraction methods;
(iv) the creation of an infrastructure for assessing the overall task and
subtask performance which can be used for benchmarking slot filling on
educational content. This infrastructure comprises a German gold standard
dataset that contains two partitions of 169 and 75 documents, and six
evaluation tasks that consider content extraction, entity extraction, and slot
filling.

##  2 Related Work

In recent years, industry-scale knowledge graphs such as the Google Knowledge
Graph, Microsoft’s Bing knowledge graph, and the knowledge graphs deployed by
Facebook, eBay and IBM Watson have emerged [8]. Creating and maintaining such
comprehensive knowledge graphs requires significant resources, which has
further accelerated research in automated knowledge extraction methods.

Knowledge base population, for instance, applies knowledge extraction
techniques to discover facts in unstructured textual resources to integrate
them into a knowledge base or knowledge graph. DBpedia, for example, is
constructed by extracting knowledge from Wikipedia Web pages and storing them
in the form of (subject, predicate, object) triples [4]. Other approaches
operate on more heterogeneous document collections, as reflected in the
composition of evaluation datasets covering News articles [6], question
answering [2], and even general Web documents [3].

Slot filling is a knowledge extraction technique that extracts information on
predefined slots (e.g., a person’s occupation, age, etc.). When applied to
open-world scenarios, slot filling is very challenging, as demonstrated by the
TAC 2017 Cold Start Slot Filling Task in which even the winning systems only
obtained F-measures below 20% [5]. If applied to a single domain, considerably
better scores are achieved, as demonstrated by Ritze et al. [10] who use slot
filling for augmenting knowledge bases. Most state-of-the-art systems such as
Coach [7], RSZ[11] and LEONA [12] deploy deep learning to improve system
performance, and transfer learning to allow adaptation to new domains.

The research introduced in this paper, in contrast, customizes slot filling to
the target domain by combining deep learning with domain knowledge encoded in
a skill and education ontology.

##  3 Method

The presented knowledge extraction system populates the continuing education
knowledge graph by drawing upon a skill and education ontology to extract
knowledge on educational offerings from websites published by educational
institutions such as universities, schools, and course providers.

###  3.1 System Overview and Formalization

Figure 1 illustrates the developed knowledge extraction and knowledge base
population process, which operates on Web pages from 488 different education
providers.

Figure 1: Overview of the automatic knowledge base population process.

The system’s knowledge graph population pipeline expands the continuing
education knowledge graph by analyzing the Web pages $doc_{i}$ of educational
offerings $i$. A page segmentation and classification component extracts
relevant page segments $seg^{type}_{i}\in doc_{i}$ with $type\in$ _{ target
group,_ _prerequisite,_ _learning objective,_ _course content,_ _certificates
& degree}_ from these pages. Afterwards, entity linking identifies known
entities $e^{known}_{ij}$ such as skills and occupation within the segments
and links them to the skill and education domain ontology. We complement
entity linking with entity recognition which is capable of identifying
entities $e^{new}_{ij}$ that are not yet available in the domain ontology and,
therefore, significantly improves the coverage of the entity extraction
process.

Figure 2: Annotated example course description taken from sae.edu. Blue
highlighting indicates identified entities, and the red border outlines the
corresponding page segment.

Finally, the slot filling component fills for each educational offering $i$
the slots outlined in Table 1 by contextualizing the extracted entities
$e_{ij}=e^{known}_{ij}\cup e^{new}_{ij}$ with the information on the page
segment $seg^{type}_{i}$ from which they have been extracted. New entities
that have been discovered by the ER component are assigned unique identifiers
that can be used for linking them to the domain ontology at a later stage.

In the real-world example shown in Figure 2, the slot value with surface form
“Programmierung” (edu:prog) extracted from the page segment “Lehrinhalte”
(edu:course_content), for example, fills the course content slot, while the
same entity extracted from the segment ‘Voraussetzungen” (edu:prerequisites)
would be considered a prerequisite for visiting the course.

Table 1: Target slot, valid entity types and cardinalities of course entities. slot | (entity) type | cardinality
---|---|---
|  | (min, max)
school | school | (1, 1)
target group | topic, occupation, degree, education, industry, position | (0, *)
prerequisite | topic, skill, occupation, position, education | (0, *)
learning objective | topic, skill, occupation | (1, *)
course content | topic, skill | (1, *)
certificates | degree, education | (0, *)

After the completion of the slot filling process, the system integrates the
information on each educational offering into the knowledge graph by forming
the corresponding triples (e.g., <https://sae.edu#c01> edu:content edu:prog.).

###  3.2 Knowledge extraction

####  3.2.1 Page segmentation.

Most websites structure educational offerings in sections, such as course
prerequisites and learning objectives. Extracting the content from these
sections and correctly labeling it allows contextualization of mined entities,
which helps to distinguish slots such as prerequisites, learning outcomes and
certificates.

The developed page segmentation system deploys the following three-step
process for identifying page segments within the documents: (i) _Text
segmentation_ compares the document’s HTML structure with the corresponding
text representation generated by the Inscriptis content extraction framework
[13]. Based on the HTML structure, the text is split into segments whenever
separator elements $m=$_{ div, p, li, td, th, dt, dd, summary, legend, h1, h2,
h3, h4, h5, h6}_, which enclose a text and usually do not contain any other
element $m$, occurs. (ii) _Titles and text clusters_ are identified by
combining HTML elements with a fallback heuristic which annotates non-standard
titles based on their length (max. 3 words) and a task-specific list of
commonly used title terms (e.g., ‘prerequisite’, ‘content’, and ‘degree’)
obtained from the domain-ontology. After identifying the title element, it is
used to determine the text cluster by merging all segments until the next
title is reached. (iii) The final _cluster classification_ step compares the
title terms with word sequences from the skill & education ontology to
determine the cluster type.

####  3.2.2 Entity linking.

The system uses a graph-based entity linking approach [14] that draws upon the
project’s skill and education database. We customized the component to
differentiate between the project-specific entity types (i) education, (ii)
function (i.e., occupation and position), (iii) skill and (iv) topic.

Figure 3: Overview of the EL process: data preparation transforms the data
into a Linked Data repository, training mines the data repository to create a
serializable EL profile, and evaluation uses the profile to annotate new and
unknown documents.

Figure 3 illustrates the utilization of the industry partner’s skill and
education ontology within the EL component. The ontology covers most areas
relevant to the human resource sector by organizing domain knowledge into 42
tables which combine custom schemas with industry standards such as the
_Standard Classification of Occupations_ 111https://www.bls.gov/soc/ and
multiple industry directories. The given EL task utilizes tables covering
classes, relations and instances on skills, educations, occupations, topics,
industries and schools. The system’s data preparation step deploys Protege
Ontop222https://github.com/ontop/ontop, a framework that allows translating
SPARQL into SQL queries, to transforms these relational data into a Linked
Data repository.

The training step utilizes SPARQL queries to mine, pre-process, and classify
relevant entities and context information from the created Linked Data
repository. The EL Profile Builder further applies multiple pre-processors and
analyzers to the query results which create artificial name variations such as
plural, possessive forms and abbreviations to maximize the EL profile’s
coverage, and determine whether the mined surface forms are unambiguous (e.g.,
“hedge fund manager”), ambiguous (e.g., “wolf of wallstreet”) or provide
additional context (e.g., “occupation”). The EL Profile Builder concludes the
training by serializing an EL profile containing all information required for
the EL process. The EL Web service deploys the profile in conjunction with a
graph-based disambiguation algorithm [14] that identifies mentions of entities
and links them to the ontology concepts.

####  3.2.3 Entity recognition and entity classification

serves as a fallback for entities that have not yet been included into the
industry partner’s skill and education ontology. Entity recognition not only
provides an efficient mitigation strategy for identifying these missing
entities, but also suggests concepts for inclusion into the skill & education
ontology, which helps in improving its coverage over time.

Figure 4: Transformer model used for entity recognition and entity
classification.

We model entity recognition and entity classification as a token
classification problem, where a deep learning model takes a sequence of tokens
(e.g., a sentence) and then provides labels for each of them. The entity
classification component draws upon the _distilbert-base-german-cased_ model
provided by the popular transformers library [15], using the Adam solver with
a learning rate of $5\cdot 10^{-5}$, and a dropout of 0.1. Figure 4 outlines
the model structure and the training process.

The domain adaptation step draws upon a domain corpus of 28,000 documents that
has been enriched with silver standard annotations obtained from the
previously described EL component. Afterwards, fine-tuning draws upon the gold
standard documents to further improve the model’s capabilities of capturing
new entities and to generate the final model. During benchmarking, a five-fold
cross evaluation procedure ensures that no training documents are used within
the evaluation step.

##  4 Evaluation

###  4.1 Gold standard

The CareerCoach 2022 gold standard is available for download in the NIF and
JSON format333https://github.com/fhgr/careercoach2022, and draws upon
documents from a corpus of over 99,000 education courses which have been
retrieved from 488 different education providers. The project’s industry
partner classified 20 of these education providers “high priority”, since they
play a key role in the Swiss continuing education market. The list of high
priority education providers contains ten academic institutions (universities
and universities of applied sciences), five companies focusing on IT education
certification, and the five largest education providers not fitting into any
of these categories.

####  4.1.1 Gold standard partitions.

The corpus contains two partitions. Partition (P1) supports the content
extraction tasks and comprises 169 documents and gold standard annotations for
page segments. Since the structure of HTML pages frequently differs even
within educational offerings, this corpus contains up to five documents per
website and also considers a higher total number of education providers. P1
has been created by sampling (i) five random documents per “high priority”
education provider and (ii) one random document taken from a selection of 69
randomly selected standard priority education providers.

The second partition (P2) only contains 75 documents but also a significantly
richer set of annotations that considers page segments, entities and slots.
It, therefore, supports benchmarking knowledge extraction tasks such as entity
extraction and slot filling on top of the content extraction task. Since these
tasks draw upon the extracted and normalized content, a higher document
variability within an education provider seems to be less beneficial,
particularly when considering that creating this much richer set of
annotations is also very time intensive.

####  4.1.2 Annotation guidelines.

The gold standard annotation process involved two researchers and two industry
experts who also outlined guidelines specifying: (i) _relevant classes_ for
the page segment recognition and classification tasks, and information on how
to identify them; and (ii) the _entities_ supported by the entity extraction
tasks and detailed instructions for identifying and distinguishing these
entities. This has been particularly important for the ’topic’ and ’skill’
entity types, which provide valuable context information but are less
intuitive to annotators. Entities are mapped to unique identifiers, and the
entities of type ’occupation’, ’degree’, ’education’, ’industry’, ’school’ and
’position’ have also been linked to the corresponding _DBpedia_ , _Wikidata_
and _European Skill/Competences, qualifications and Occupations (ESCO)_
444https://ec.europa.eu/esco/portal/download concepts.

During the annotation process, the guidelines have been revisited and further
optimized. We drew upon the introduced knowledge extraction pipeline to create
candidate annotations that have then been manually corrected and improved by
domain experts. In a final step, two additional domain experts from the
industry partner manually validated the gold standard annotations based on 28
documents that comprised 1533 annotations, reported potential problems and
triggered further improvements to the gold standard and annotation guidelines.
At this point, the inter-rater-agreement between domain experts from the
research institution and the industry experts for partition P2 has been 0.85
(Cohen’s Kappa), and 0.88 (pairwise F1 score) [1] for annotated entities.
Within the sample, the experts agreed on all page segment annotations,
suggesting that the page partitioning task can be easily performed by humans
based on the provided annotation guidelines. The substantial inter-rater-
agreement also indicates that pre-annotating the corpus is unlikely to
introduce any significant bias.

####  4.1.3 Gold standard properties.

Table 2 provides basic statistics and summarizes information on the number of
annotated classes, and annotations per document. The table also gives an
impression on the differences to expect between education providers. For
instance, there are documents that do not contain any relevant contents (i.e.,
no relevant page segment) as well as cases where page segments appear multiple
times. The entity extraction and slot filling tasks are also confronted with a
large variety of entities, ranging from 2 entities per document to a total of
235 entities identified in a single document.

Table 2: Corpus properties | P1: Content | P2: Entity extraction
---|---|---
| extraction | and slot filling
Number of documents | 169 | 75
Number of education providers | 89 | 55
Number of annotation classes | 5 | 8
Min. annotations in document | 0 | 2
Avg. annotations in document | 3.4 | 43.9
Median annotations in document | 3 | 30
Max. annotations in document | 17 | 235

###  4.2 Evaluation tasks

The evaluation draws upon the gold standard and supports benchmarking the
following six tasks performed by the introduced knowledge extraction system:

####  4.2.1 Content extraction.

Content extraction identifies and classifies text segments relevant for the
slot filling tasks. We distinguish between,

  1. 1.

_T1: page segment recognition_ \- identifies page segments within HTML pages
$doc$ and extracts the text strings $seg^{j}_{i}\in doc_{i}$ from these
segments.

  2. 2.

_T2: page segment classification_ \- assigns each extracted text segment
$seg^{j}_{i}$ to a class $type\in$ {_target group_ , _prerequisite_ ,
_learning objective_ , _course content_ , _certificates & degree}_.

We evaluate the page segment recognition task (T1) by comparing the tokens in
the extracted page segments $t_{i}\in seg^{j}_{i}$ with the tokens in the gold
standard segments $t_{g}\in s^{j}_{g}$. The evaluation of the page segment
classification also requires that the types of the gold standard page segment
$t_{g}^{type}$ and of the extracted segment $t_{g}^{type}$ match.

####  4.2.2 Entity extraction.

The entity extraction tasks aim at identifying mentions of entities of type
$tp_{i}\in\mathcal{T}_{i}$ within the extracted page segments. The corpus
contains annotations of the following entity types: ’skill’, ’occupation’,
’topic’, ’position’, ’school’, ’industry’, ’education’, ’degree’.

  1. 1.

_T3: entity recognition_ \- locates mentions $m_{i}$ of entities within text
segments.

  2. 2.

_T4: entity classification_ \- assigns each mention $m_{i}$ to the
corresponding entity type $tp_{i}\in\mathcal{T}_{i}.$

  3. 3.

_T5: entity linking_ \- links mentions $m_{i}$ to the appropriate entity
$e_{i}$ in the knowledge graph $KG$. Entities that are not yet available in
the knowledge graph are handled as NIL entities (i.e., they are assigned a
temporary identifier that is unique for all mentions which refer to the same
entity).

We distinguish between two evaluation settings: strict and relaxed. In the
strict settings, mentions $m_{i}$ identified by the entity recognition
component for task T3 are considered true positives (TP), if they are
identical to a gold standard mention $m_{g}$. The entity classification task
(T4) also requires that both entities have been assigned to the same entity
type $tp_{i}$, and the linking task (T5) requires linking the mention to the
correct knowledge base entity $e_{i}$.

The _relaxed setting_ eases these conditions by also considering mentions that
overlap a gold standard mention as correct.

Entities that do not appear in the gold standard are considered false
positives (FP), and false negatives (FN) refer to gold standard entities which
have been missed by the entity extraction task.

####  4.2.3 Slot filling.

The slot filling task (T6) combines all the tasks above. The page segment
recognition (T1) identifies page segments. Afterwards, the page segment
classification (T2) assigns them to the corresponding segment cluster, and
entity recognition (T3), entity classification (T4) and entity linking (T5)
are performed. Finally, we assign the extracted entities $e_{i}$, which have
been contextualized based on the classification of the page segment in which
they have been identified, to the corresponding slot.

###  4.3 Experiments and discussion.

Table 3 summarizes the results of a five-fold cross evaluation on the
CareerCoach 2022 gold standard. For the entity linking and slot filling task,
the evaluation also distinguishes between the strict and the relaxed setting.

Table 3: Slot filling and per component evaluation results. Component | P | R | F1
---|---|---|---
T1: page segment recognition | 0.82 | 0.84 | 0.83
T2: page segment classification | 0.82 | 0.84 | 0.83
T3: entity recognition | 0.82 | 0.66 | 0.73
T4: entity classification | 0.78 | 0.63 | 0.70
T5: entity linking (strict) | 0.67 | 0.80 | 0.73
T5: entity linking (relaxed) | 0.67 | 0.82 | 0.74
T6: slot filling (strict) | 0.48 | 0.60 | 0.54
T6: slot filling (relaxed) | 0.50 | 0.62 | 0.55

A comparison of the page segment recognition and page segment classification
performance reveals the same scores for both tasks. This confirms that the
developed simple segment classification heuristic has been very effective and
has classified all page segments correctly.

The results indicate that both entity recognition and entity classification
have been optimized towards a higher precision, to spare domain experts, which
need to confirm new entity types in the production process. Entity linking, in
contrast, has been optimized towards recall, as outlined in the evaluation
results.

The evaluation of the overall slot filling process only considers correctly
assigned slots (i.e., course, slot and slot value are correct) as true
positives. All other values are considered false positives, and missing values
as false negatives. The F1 score of the slot filling process indicates that
the system is not yet suitable for fully automated knowledge population, but
rather enables a semi-automated process that significantly improves throughput
when compared to the prior deployed manual approaches.

###  4.4 Automatic Knowledge Graph Population

Running the presented system on 55 course descriptions (one per unique
education provider) from the gold standard, extends the knowledge base by 453
unique statements. Most of these statements (222) describe the course content,
followed by target groups (90), learning objectives (61), course prerequisites
(51) and certificates (29). In addition, 511 slot values have been marked as
“related” since the system hasn’t been able to unequivocally resolve their
slot, due to shortcomings in the page partitioning process. This result
indicates, that further improving the page partitioning process will be key
towards enhancing the system’s recall.

    <https://sae.edu#c01>  edu:prerequisite    edu:matura, edu:student;
                           edu:course_content  edu:Design, edu:Podcasting;
                           skos:related        edu:Web_Design, edu:Publishing.

Figure 5: Snippet of six RDF triples that have been integrated into the
knowledge graph.

##  5 Outlook and conclusions

Performing slot filling tasks on third-party websites is a challenging task,
requiring content extraction and knowledge extraction methods to work in
concert.

This paper introduces a slot filling system that mines education provider
websites for a wide range of educational offerings such as academic programs,
continuing education programs, courses, seminars and online courses.
Integrating background knowledge from a proprietary ontology allows the
application of graph-based entity linking methods for the identification of
known entities, which are complemented by entity recognition to mitigate
coverage issues within the ontology. The slot filling component contextualizes
entities to distinguish between ambiguous slots such as _prerequisites_ versus
_learning outcomes_ , and afterwards integrates them into a continuing
education knowledge graph.

An evaluation framework comprising six evaluation tasks and a publicly
available German gold standard allow benchmarking the content extraction and
knowledge extraction methods utilized within the slot filling system. The
framework did not only yield information on the components’ performance, but
also guided the system development by providing rapid feedback on the impact
of changes and improvements. In addition, it offers a reliable benchmark to
third-party researchers interested in the described slot filling task.

Future work will focus on: (i) improving the slot filling performance by
enhancing page segmentation, increasing the coverage of the proprietary
knowledge graph used for entity linking, and fine-tuning the entity
recognition component. Given the importance of the created benchmarking
framework for the research and development process, we plan on (ii) further
increasing its size and coverage; and (iii) integrating the gold standard with
explainable benchmarking frameworks such as Orbis [9] to make it more
accessible to third-party researchers.

### Acknowledgments

The research presented in this paper has been conducted within the CareerCoach
project (www.fhgr.ch/CareerCoach) which is funded by Innosuisse.

## References

  * [1] Brandsen, A., Verberne, S., Wansleeben, M., Lambers, K.: Creating a Dataset for Named Entity Recognition in the Archaeology Domain. In: Proceedings of the 12th Language Resources and Evaluation Conference. pp. 4573–4577. Marseille, France (May 2020)
  * [2] Dubey, M., Banerjee, D., Abdelkawi, A., Lehmann, J.: LC-QuAD 2.0: A Large Dataset for Complex Question Answering over Wikidata and DBpedia. In: The Semantic Web – ISWC 2019. pp. 69–78. Cham (2019)
  * [3] Glass, M., Gliozzo, A.: A Dataset for Web-Scale Knowledge Base Population. In: Gangemi, A., Navigli, R., Vidal, M.E., Hitzler, P., Troncy, R., Hollink, L., Tordai, A., Alam, M. (eds.) The Semantic Web. pp. 256–271. Springer (2018)
  * [4] Lehmann, J., Isele, R., Jakob, M., Jentzsch, A., Kontokostas, D., Mendes, P.N., Hellmann, S., Morsey, M., van Kleef, P., Auer, S., Bizer, C.: DBpedia – A large-scale, multilingual knowledge base extracted from Wikipedia. Semantic Web 6(2), 167–195 (2015)
  * [5] Lim, S., Kwon, S., Lee, S., Choi, J.: UNIST SAIL System for TAC 2017 Cold Start Slot Filling. In: TAC (2017)
  * [6] Lin, X., Chen, L.: Canonicalization of Open Knowledge Bases with Side Information from the Source Text. In: 2019 IEEE 35th International Conference on Data Engineering (ICDE). pp. 950–961 (Apr 2019)
  * [7] Liu, Z., Winata, G.I., Xu, P., Fung, P.: Coach: A Coarse-to-Fine Approach for Cross-domain Slot Filling. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. pp. 19–25. Online (Jul 2020)
  * [8] Noy, N., Gao, Y., Jain, A., Narayanan, A., Patterson, A., Taylor, J.: Industry-scale Knowledge Graphs: Lessons and Challenges. Communications of the ACM 62(8), 36–43 (2019)
  * [9] Odoni, F., Brasoveanu, A.M., Kuntschik, P., Weichselbraun, A.: Introducing Orbis: An Extendable Evaluation Pipeline for Named Entity Linking Drill-Down Analysis. In: 82nd Annual Meeting of The Association for Information Science (ASIS&T 2019). vol. 56, pp. 468–471. Melbourne, Australia (2019)
  * [10] Ritze, D., Lehmberg, O., Oulabi, Y., Bizer, C.: Profiling the Potential of Web Tables for Augmenting Cross-domain Knowledge Bases. In: Proceedings of the 25th International Conference on World Wide Web. pp. 251–261. WWW ’16, Montréal, Canada (2016)
  * [11] Shah, D., Gupta, R., Fayazi, A., Hakkani-Tur, D.: Robust Zero-Shot Cross-Domain Slot Filling with Example Values. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. pp. 5484–5490. Florence, Italy (Jul 2019)
  * [12] Siddique, A., Jamour, F., Hristidis, V.: Linguistically-Enriched and Context-AwareZero-shot Slot Filling. In: Proceedings of the Web Conference 2021. pp. 3279–3290. WWW ’21, New York, NY, USA (2021)
  * [13] Weichselbraun, A.: Inscriptis - A Python-based HTML to text conversion library optimized for knowledge extraction from the Web. Journal of Open Source Software 6(66), 3557 (2021)
  * [14] Weichselbraun, A., Kuntschik, P., Brasoveanu, A.M.P.: Mining and Leveraging Background Knowledge for Improving Named Entity Linking. In: 8th International Conference on Web Intelligence, Mining and Semantics (WIMS 2018). Novi Sad, Serbia (Jun 2018)
  * [15] Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., Davison, J., Shleifer, S., von Platen, P., Ma, C., Jernite, Y., Plu, J., Xu, C., Scao, T.L., Gugger, S., Drame, M., Lhoest, Q., Rush, A.M.: Transformers: State-of-the-art natural language processing. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations. pp. 38–45. Online (Oct 2020)