# Wavelength-tunable open double-microcavity to enhance two closely spaced
optical transitions

S. Seyfferle  Institut für Halbleiteroptik und Funktionelle Grenzflächen,
Center for Integrated Quantum Science and Technology (IQ${}^{\text{ST}}$) and
SCoPE, University of Stuttgart, Allmandring 3, 70569 Stuttgart, Germany  T.
Herzog  Institut für Halbleiteroptik und Funktionelle Grenzflächen, Center for
Integrated Quantum Science and Technology (IQ${}^{\text{ST}}$) and SCoPE,
University of Stuttgart, Allmandring 3, 70569 Stuttgart, Germany  R. Sittig
Institut für Halbleiteroptik und Funktionelle Grenzflächen, Center for
Integrated Quantum Science and Technology (IQ${}^{\text{ST}}$) and SCoPE,
University of Stuttgart, Allmandring 3, 70569 Stuttgart, Germany  M. Jetter
Institut für Halbleiteroptik und Funktionelle Grenzflächen, Center for
Integrated Quantum Science and Technology (IQ${}^{\text{ST}}$) and SCoPE,
University of Stuttgart, Allmandring 3, 70569 Stuttgart, Germany  S.L.
Portalupi  Institut für Halbleiteroptik und Funktionelle Grenzflächen, Center
for Integrated Quantum Science and Technology (IQ${}^{\text{ST}}$) and SCoPE,
University of Stuttgart, Allmandring 3, 70569 Stuttgart, Germany  P. Michler
Institut für Halbleiteroptik und Funktionelle Grenzflächen, Center for
Integrated Quantum Science and Technology (IQ${}^{\text{ST}}$) and SCoPE,
University of Stuttgart, Allmandring 3, 70569 Stuttgart, Germany

###### Abstract

Microcavities have long been recognized as indispensable elements in quantum
photonic research due to their usefulness for enhanced light extraction and
light-matter interaction. A conventional high-Q cavity structure typically
allows only a single optical transition to be tuned into resonance with a
specific mode. The transition to a more advanced double-cavity structure,
however, introduces new and interesting possibilities such as enhancing two
spectrally close optical transitions at the same time with two distinct cavity
modes. Here, we investigate a cavity structure composed of a monolithic planar
cavity enclosed between two semiconductor distributed Bragg reflectors (DBR)
and a top dielectric mirror deposited on a fiber tip. While the bottom cavity
is formed by the two DBRs, the mirror on the fiber tip and the top DBR of the
semiconductor chip create a second tunable cavity. These coupled cavities
exhibit mode hybridization when tuned into resonance and their splitting can
be adjusted to match with the spectral separation of closely spaced optical
transitions by a suitable sample design. Furthermore, we report on the
simultaneous resonance tuning of the exciton and biexciton transition of a
semiconductor quantum dot, each to a separate mode of the open fiber-based
double cavity. Decay time measurements at simultaneous resonance showed a
Purcell-factor of $F_{P}^{X}$=1.9$\pm$0.4 for the exciton transition.

††preprint: AIP/123-QED

##  I Introduction

The combination of optical microcavities with semiconductor quantum dots (QDs)
looks back on a long-standing success story of achievements in enhanced light
extraction efficiency [1], strong light-matter coupling [2] and lasing [3]
among others. The ability of the cavity to influence the spontaneous emission
of optical transitions known as Purcell enhancement [4] is widely used in
cavity quantum electrodynamics to boost the device efficiency and coherence of
QD single photon emission and will greatly aid nascent quantum photonic
technologies[5]. Some of those, notably quantum repeaters[6], make use of
entangled photon pairs that can be generated e.g. via the cascaded emission of
the biexciton and exciton transitions in a QD [7, 8] that possess slightly
different emission energies ($\Delta E$=1-3 meV). The extraction efficiency
and optical quality of these entangled pairs would greatly benefit from
cavity-induced Purcell enhancement, however, the simultaneous resonance tuning
of both transitions to a cavity mode poses a challenge.

Successful efforts to this end have been made using circular Bragg cavities
[9, 10]. This approach utilizes a spectrally broad cavity mode (Q-factors of
150-300 [11, 12]) which enables to encompass exciton and biexciton within the
same mode. But due to the spectral separation between exciton and biexciton it
is likely that the two lines will be situated at the far side of the Gaussian
mode field, thus the light-matter interaction will be reduced as compared to a
spectral matching of cavity center wavelength and QD transition.

Another approach [13] utilized the coupling of a twin micropillar system to
the QDs exciton and biexciton and achieved excellent results in Purcell
enhanced entangled photon pairs. However, this approach requires very accurate
spectral matching of the micropillar modes to the exciton and biexciton
transition, respectively, together with spatial matching of the emitter with
the cavity field which makes this method difficult to realize.

Here, we utilize an open tunable fiber-based double-cavity [14] that provides
two spectrally separate modes to be tuned into resonance with the QD
transitions individually. Additional benefits are the free choice around which
emitter to form the resonator as well as considerable fine-tuning
possibilities of the cavity length by displacement of the fiber tip with
nanometric precision.

##  II Resonance tuning scheme & cavity design

The fiber-based double cavity is constituted of the combination of a planar
cavity sample consisting of a $\lambda$-cavity sandwiched between DBRs and a
fiber tip placed above the sample surface that serves as the external mirror
via its high reflectivity coating. This forms two separate cavities, the
monolithic cavity of the sample (bottom cavity) and the tunable open cavity
(top cavity). Fig. 1a illustrates this cavity design.

This geometry allows for the interaction of the modes of each separate cavity
that manifests itself in the appearance of the avoided crossing phenomenon
[15, 14] where modes of adjacent mode number approach each other spectrally
when tuned into resonance but avoid taking the same energy values due the
energy splitting caused by their interaction. This behaviour is the central
element of the scheme to enhance two optical transitions simultaneously since
the spectral separation of two longitudinal modes in the avoided crossing
region can be engineered by a suitable sample design to be on the order of the
spectral separation of an exciton/biexciton-pair for instance [14]. The scheme
for the dual resonance condition is visualized in Fig. 1b.

Figure 1: Schematic depiction of the double-cavity structure and the
simultaneous resonance tuning scheme. a Cross-section of the fiber double-
cavity geometry and the sample structure. b Schematic depiction of the
simultaneous resonance tuning scheme of exciton ($X$) and biexciton ($XX$) to
longitudinal modes in the anti-crossing region. The spectral distance of two
longitudinal modes of the double-cavity structure is reduced to the scale of
the binding energy of an exciton/biexciton-pair in the avoided crossing region
as the cavity length $L$ is altered.

The magnitude of the energy splitting in the avoided crossing region depends
on the coupling strength of the involved modes. The larger the coupling
strength, the larger the mode separation. In the fiber-based double-cavity
this interaction strength is mediated by the respective cavity lengths as well
as the reflectivities of the mirrors, but especially by the amount of middle
DBR pairs (cf. Fig. 1a). An increased number of layer pairs will raise the
reflectivity and thereby decrease the mode coupling and their separation as a
consequence [14].

The basis of the sample grown by metal-organic vapor-phase epitaxy (MOVPE) is
a GaAs-substrate onto which the bottom-DBR consisting of 37 pairs of
alternating AlAs and GaAs layers is deposited. This is followed by a
$\lambda$-cavity of GaAs containing the active layer of InAs QDs. The
structure is completed by 17 AlAs/GaAs DBR-layer pairs, which will act as the
middle-DBR in the final double cavity setup. The upper cavity is formed by 11
pairs of SiO2/Ta2O5 constituting the high reflectivity coating attached to the
fiber end facet. The concave imprint on the fiber produced by focused ion beam
milling serves as a parabolic resonator mirror and has a curvature radius of
20 µm [16]. The sample is held in a vacuum/helium environment at 4 K and is
attached to piezo actuators to effect its movement relative to the fiber tip
in all three directions. The fiber tip is mounted in a custom made holder for
its orthogonal alignment with respect to the sample surface and serves as the
excitation as well as the detection path. An above band-gap continuous wave
diode laser ($\lambda$=650 nm) or a pulsed Ti:Sa laser ($\lambda$=800 nm) for
decay time measurements are utilized as excitation sources. The QDs embedded
in the sample serve as an internal light source of the structure to make the
cavity modes visible.

Figure 2: Mode separation in the avoided crossing regime. a Cavity length scan
and b extracted mode trajectories for a sample with 17 middle DBR pairs. The
data points of the mode trajectories have been evaluated with a polynomial fit
in b and their minimum separation is calculated to 1.38 nm.

##  III Experimental results

###  III.1 Simultaneous resonance tuning

As shown in a previous study [14], the amount of middle-DBR layer pairs have a
distinct influence on the mode separation in the avoided crossing regime,
specifically, an increased number of layer pairs reduces the mode splitting.
As demonstrated in Ref. 14 the increase from 9 to 12 middle DBR-pairs reduced
the minimal mode splitting from 10 nm to 5.5 nm. Here, a sample of 17 middle
DBR pair layers has been fabricated and used for the formation of the fiber-
based double-cavity. The mode trajectories in dependence of the cavity length
$L$ is investigated in Fig. 2a. The cavity length has been continuously
altered by stepwise movement of the fiber tip above the sample accompanied by
the acquisition of a spectrum at every increment. The resulting scan can be
depicted as an intensity map (Fig. 2a) that reveals the mode propagation
through the avoided crossing region. The modes have been successively fitted
with a Gaussian function and their extracted trajectory is evaluated with a
polynomial fit in Fig. 2b. The calculation of the difference of the polynomial
fit of the two respective modes yields a minimum separation value of
$\Delta\lambda_{\text{min}}$=1.38 nm and demonstrates the reduction of the
spectral distance between modes of adjacent mode number to align with closely
spaced optical transitions in the range of 1-2 nm.

Figure 3: Simultaneous resonance tuning of an exciton/biexciton-pair and two
modes of the fiber-based double cavity. a Spectrum of the utilized
exciton/biexciton-pair acquired during a pre-selection process step. b
Polarization dependent energy measurement of the emission lines designated as
$X$ (blue) and $XX$ (black) in a. The anti-phase oscillations and very similar
energy splitting of $\Delta E_{X}$=38.9 µeV and $\Delta E_{XX}$=42.1 µeV
support the assignment of the transitions as exciton and biexciton of the same
QD. c Cavity length scan of the $X/XX$-pair in a in a fiber-based double
cavity at a temperature of 4 K. The appearance of the exciton is designated in
the plot, the biexciton comes to lie within the avoided crossing region and is
thus not supported by the modes as indicated by the upper red line. Increasing
the temperature red-shifts the QD transitions and establishes the simultaneous
resonance tuning as visualized by the lower red solid line and the white solid
guide to the eye lines roughly representing the mode trajectories. d
Comparison of the pre-selection spectrum (red) from a with the scan (grey)
from c. e Cavity length scan of the dual resonance configuration at 35 K. The
red bar indicates the cavity length of simultaneous resonance. g Spectrum of
the dual resonance in red at the cavity length marked by the red solid line in
e superimposed with the stacked spectra of the cavity length scan from sub-
figure e indicates the resonance tuning of $X$ and $XX$ each to one of the
longitudinal modes ($LM$).

In the next step, exciton/biexciton-pairs have been pre-selected via the
deposition of marker structures in a deterministic low-temperature optical
lithography process [17, 18]. This allows for the later retrieval of the
selected emitter in the fiber cavity set-up by locating the markers via an in-
plane reflectivity scan [16]. Selection requirements are the appearance of the
emission lines in question in the wavelength range of the anti-crossing region
as well as their splitting of $\sim$1.4 nm. A suitable choice is depicted in
Fig. 3a. The spectral distance of the transitions marked as $X$ and $XX$ in
the graph amounts to 1.6 nm and Fig. 3b indicates the nature of these emission
lines being exciton and biexciton by their anti-phase peak oscillations under
polarization dependent energy detection [19]. The fiber-based double cavity is
then formed around the designated QD and a cavity length scan is conducted at
4 K to investigate the spectral relations of the exciton/biexciton transitions
and the anti-crossing. The result is displayed in Fig. 3c. One of the
intensity spots (marked in the plot as $X$) can be identified as the exciton
of the pre-selected QD with the help of Fig. 3d. Here, the scan from sub-
figure c is reduced to one dimension by stacking every single spectrum on top
of each other and superimposed with the pre-selection spectrum from Fig. 3a.
The biexciton happens to be located within the intensity depleted region of
the anti-crossing and is therefore not supported by mode enhancement as
indicated by the upper red line in Fig. 3c. Increasing the temperature red-
shifts the transitions and moves the biexciton into the cavity mode at the
higher wavelength side of the avoided crossing resulting in a situation
represented by the lower red line and the white lines as guides to the eye for
the mode trajectories in Fig. 3c. The simultaneous resonance condition can be
established at 35 K as depicted in Fig. 3e. The mode emission intensity has
increased due to the higher rate of phonon-assisted cavity feeding at elevated
temperatures. Due to this broad enhancement singular spots that would indicate
the QD transitions in the cavity cannot be distinguished in Fig. 3e. The solid
red line drawn on top of the data in Fig. 3e marks the position of the exciton
at the left end of the line and the position of the biexciton on the other
end. Fig. 3g gives a comparison of the stacked plots of the scan of Fig. 3e
and the spectrum taken at the simultaneous resonance at the cavity length
marked by the red line in sub-figure e. The spectrum shows two separate
emission lines representing the longitudinal mode (LM) including the exciton
emission at lower wavelengths and the biexciton emission in the corresponding
other mode. The peak separation is 1.6 nm and corresponds exactly to the
spectral distance of exciton and biexciton as seen in Fig. 3a. In order to
demonstrate the dominant contribution of exciton and biexciton emission to the
mode emission an analysis of the decay dynamics of the modes ($X+LM$, $XX+LM$)
is discussed in the following.

###  III.2 Decay dynamics

Based on the dual resonance configuration the decay dynamics have been
measured via time-correlated single photon counting (TCSPC). The resulting
histogram is plotted in Fig. 4a. It can be noted that the increase of the
exciton coincidences occurs with a time delay in relation to the rise of the
biexciton coincidences and is less steep in comparison. After the biexciton
decay sets in the exciton follows, which is an indicator of the cascaded
emission typical for these two transitions. An exponential fit to the data
after deconvolution with the instrument response function yielded decay time
constants of $\tau_{X}$=0.47$\pm$0.07 ns and $\tau_{XX}$=0.38$\pm$0.09 ns for
the exciton and biexciton transitions, respectively. The ratio of the decay
times is unlike the expected 2:1 relations that exciton and biexciton usually
exhibit. It appears that the decay times are modified in a way that the
exciton recombination is enhanced at a higher rate than the biexciton decay.
This points towards the fact that each transition is subject to a different
cavity environment that accelerates the decay at a different rate. Since both
measurements have been made at the same cavity elongation the mode volume is
the same for both. So, the reason has to be found in the non-linear dependence
of the Q-factor on the cavity elongation as simulations in a preceeding study
[14] showed. Thus, it can be argued that the biexciton is resonant to a
segment of the mode where the Q-factor is smaller as compared to the Q-factor
at the spectral position of the exciton and consequently, the exciton
transition is enhanced at a higher rate.

Figure 4: Decay dynamics at the simultaneous resonance condition. a TCSPC
measurement of the exciton (blue) and biexciton (black) each simultaneously
tuned to resonance with a cavity mode on the basis of the spectrum of Fig. 3g.
The resulting decay constants of an exponential fit (red curves) amount to
$\tau_{X}$=0.47$\pm$0.07 ns and $\tau_{XX}$=0.38$\pm$0.09 ns. b The measured
decay traces are compared to numerical solutions of a rate equation model (see
supplementary) based on the fit results from sub-figure a. The simulations
(orange curves) are well in accordance with the data, thus supporting the
assumed exciton/biexciton decay behaviour. c TCSPC measurement of the exciton
transition in the planar cavity environment to estimate the Purcell-factor
$F_{P}$. The decay time is extracted to $\tau^{X}_{\text{plan}}$=0.91$\pm$0.07
ns.

A treatment of the data with a rate equation model (detailed in the
supplementary material) suggests itself in order to confirm the nature of the
exciton-biexciton dynamic in the decay behaviour. A numerical solution of the
model convolved with the instrument response function using the parameters
obtained from the fit in Fig. 4a ($\tau_{X}$=0.47 ns, $\tau_{XX}$=0.38 ns)
results in the decay curves plotted as orange solid lines in Fig. 4b. A
secondary decay time constant of $\tau_{2}$=8 ns inserted into the model has
been found necessary to best reproduce the decay traces. This hints at the
influence of the elevated temperature of 35 K which likely causes refilling by
a nearby trap state. The comparison of the calculated decay curves (solid
orange lines) with the measured decay dynamics displayed as data points in
blue (exciton) and black (biexciton) in Fig. 4b shows a very good agreement
between model and measurement. This underlines the assumption of a dominating
exciton/biexciton decay behaviour in the measurement.

An estimation of the Purcell-factor achieved in the simultaneous resonance
configuration requires the knowledge of the bare decay time. To this end, the
exciton transition has been measured without a fiber installed to obtain its
decay time in the planar cavity. The sample temperature was fixed to 35 K to
grant a direct comparison to the conditions of Fig. 3g. The resulting
histogram is seen in Fig. 4c. The decay constant of the exciton transition can
be determined with an exponential fit after deconvolution and amounts to
$\tau_{\text{plan}}^{X}$=0.91$\pm$0.07 ns. Note, that the biexciton decay
could not be measured at 35 K due to an encountered strong intensity depletion
of the emission line at high temperatures which is why a reasonable estimation
of the Purcell-factor of the biexciton transition is not provided. The ratio
of the decay times in the planar and double cavity environment of the exciton
transition allows to estimate the Purcell-factor which amounts to
$F_{P}^{X}$=1.9$\pm$0.4.

##  IV Conclusions

In summary, a study of the properties of a tunable fiber-based double-cavity
in view of supporting two closely spaced optical transitions via two separate
modes has been presented. The sample design can be engineered with respect to
mode interaction to enable a mode separation in the anti-crossing regime on
the order of $\sim$1-2 nm. A realization of the simultaneous resonance tuning
has been shown with the exciton and biexciton transitions of a semiconductor
QD individually brought to resonance with either one of the modes. A first
estimation of the Purcell-factor under these conditions amounts to
$F_{P}^{X}$=1.9$\pm$0.4 for the exciton transition. Future progress should
involve the implementation of strain tuning to establish the simultaneous
resonance condition without having to rely on temperature tuning.
Additionally, strain tuning can modify the biexciton binding energy which
would provide more flexibility in terms of spectral matching of the QD
transitions and the cavity modes .

###### Acknowledgements.

We gratefully acknowledge the support from DFG MI 500/31-1. Parts of this work
were financially supported by the program Competence Network Quantum
Technology in BW (QTBW.net) from the Baden-Württemberg Stiftung. Furthermore,
we thank L. Engel and S. Kolatschek for their support in the clean-room and
sample processing steps and F. Hornung for the help in data evaluation.

## References

## References

  * Unsleber  _et al._ [2016] S. Unsleber, Y.-M. He, S. Gerhardt, S. Maier, C.-Y. Lu, J.-W. Pan, N. Gregersen, M. Kamp, C. Schneider, and S. Höfling, “Highly indistinguishable on-demand resonance fluorescence photons from a deterministic quantum dot micropillar device with 74% extraction efficiency,” Optics Express 24, 8539–8546 (2016).
  * Reithmaier  _et al._ [2004] J. P. Reithmaier, G. Sek, A. Löffler, C. Hofmann, S. Kuhn, L. V. Reitzenstein, S. Keldysh, V. D. Kulakovskii, T. L. Reinecke, and A. Forchel, “Strong coupling in a single quantum dot-semiconductor microcavity system,” Nature 432, 197–200 (2004).
  * Michler  _et al._ [2000] P. Michler, A. Kiraz, L. Zhang, C. Becher, E. Hu, and A. Imamoglu, “Laser emission from quantum dots in microdisk structures,” Applied Physics Letters 77, 184–186 (2000), https://doi.org/10.1063/1.126918 .
  * Purcell [1946] E. M. Purcell, “Spontaneous emission probabilities at radio frequencies,” Physical Review 69, 681 (1946).
  * Reiserer and Rempe [2015] A. Reiserer and G. Rempe, “Cavity-based quantum networks with single atoms and optical photons,” Reviews of Modern Physics 87, 1379–1418 (2015).
  * Simon  _et al._ [2007] C. Simon, H. de Riedmatten, M. Afzelius, N. Sangouard, H. Zbinden, and N. Gisin, “Quantum repeaters with photon pair sources and multimode memories,” Physical Review Letter 98, 190503 (2007).
  * Young  _et al._ [2006] R. J. Young, R. M. Stevenson, P. Atkinson, K. Cooper, D. A. Ritchie, and A. J. Shields, “Improved fidelity of triggered entangled photons from single quantum dots,” New Journal of Physics 8, 29–29 (2006).
  * Akopian  _et al._ [2006] N. Akopian, N. H. Lindner, E. Poem, Y. Berlatzky, J. Avron, D. Gershoni, B. D. Gerardot, and P. M. Petroff, “Entangled photon pairs from semiconductor quantum dots,” Physical Review Letters 96, 130501 (2006).
  * Liu  _et al._ [2019] J. Liu, R. Su, Y. Wei, B. Yao, S. F. Covre da Silva, Y. Yu, J. Iles-Smith, K. Srinivasan, A. Rastelli, J. Li, and X. Wang, “A solid-state source of strongly entangled photon pairs with high brightness and indistinguishability,” Nature Nanotechnology 14, 586–593 (2019).
  * Wang  _et al._ [2019] H. Wang, H. Hu, T.-H. Chung, J. Qin, X. Yang, J.-P. Li, R.-Z. Liu, H.-S. Zhong, Y.-M. He, X. Ding, Y.-H. Deng, Q. Dai, Y.-H. Huo, S. Höfling, C.-Y. Lu, and J.-W. Pan, “On-demand semiconductor source of entangled photons which simultaneously has high fidelity, efficiency, and indistinguishability,” Physical Review Letters 122, 113602 (2019).
  * Kolatschek  _et al._ [2019] S. Kolatschek, S. Hepp, M. Sartison, M. Jetter, P. Michler, and S. L. Portalupi, “Deterministic fabrication of circular Bragg gratings coupled to single quantum emitters via the combination of in-situ optical lithography and electron-beam lithography,” Journal of Applied Physics 125, 045701 (2019), https://doi.org/10.1063/1.5050344 .
  * Kolatschek  _et al._ [2021] S. Kolatschek, C. Nawrath, S. Bauer, J. Huang, J. Fischer, R. Sittig, M. Jetter, S. L. Portalupi, and P. Michler, “Bright Purcell enhanced single-photon source in the telecom O-band based on a quantum dot in a circular Bragg grating,” Nano Letters 21, 7740–7745 (2021), pMID: 34478316, https://doi.org/10.1021/acs.nanolett.1c02647 .
  * Dousse  _et al._ [2010] A. Dousse, J. Suffczynski, A. Beveratos, O. Krebs, A. Lemaitre, I. Sagnes, J. Bloch, P. Voisin, and P. Senellart, “Ultrabright source of entangled photon pairs,” Nature 466, 217 (2010).
  * Herzog  _et al._ [2020] T. Herzog, S. Böhrkircher, S. Both, M. Fischer, R. Sittig, M. Jetter, S. L. Portalupi, T. Weiss, and P. Michler, “Realization of a tunable fiber-based double cavity system,” Physical Review B 102, 235306 (2020).
  * Wiersig [2006] J. Wiersig, “Formation of long-lived, scarlike modes near avoided resonance crossings in optical microcavities,” Physical Review Letters 97, 253901 (2006).
  * Herzog  _et al._ [2018] T. Herzog, M. Sartison, S. Kolatschek, S. Hepp, A. Bommer, C. Pauly, F. Mücklich, C. Becher, M. Jetter, S. L. Portalupi, and P. Michler, “Pure single-photon emission from In(Ga)As QDs in a tunable fiber-based external mirror microcavity,” Quantum Science and Technology 3, 034009 (2018).
  * Dousse  _et al._ [2008] A. Dousse, L. Lanco, J. Suffczyński, E. Semenova, A. Miard, A. Lemaître, I. Sagnes, C. Roblin, J. Bloch, and P. Senellart, “Controlled light-matter coupling for a single quantum dot embedded in a pillar microcavity using far-field optical lithography,” Physical Review Letters 101, 267404 (2008).
  * Sartison  _et al._ [2017] M. Sartison, S.-L. Portalupi, T. Gissibl, M. Jetter, H. Giessen, and P. Michler, “Combining in-situ lithography with 3D printed solid immersion lenses for single quantum dot spectroscopy,” Scientific Reports 7, 39916 (2017).
  * Kowalik  _et al._ [2005] K. Kowalik, O. Krebs, A. Lemaître, S. Laurent, P. Senellart, P. Voisin, and J. A. Gaj, “Influence of an in-plane electric field on exciton fine structure in inas-gaas self-assembled quantum dots,” Applied Physics Letters 86, 041907 (2005), https://doi.org/10.1063/1.1855409 .


#  JobSkape: A Framework for Generating Synthetic Job Postings to Enhance
Skill Matching

Antoine Magron*1 Anna Dai*1 Mike Zhang2
Syrielle Montariol1 Antoine Bosselut1
1EPFL, Switzerland
2IT University of Copenhagen, Denmark
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Recent approaches in skill matching, employing synthetic training data for
classification or similarity model training, have shown promising results,
reducing the need for time-consuming and expensive annotations. However,
previous synthetic datasets have limitations, such as featuring only one skill
per sentence and generally comprising short sentences. In this paper, we
introduce JobSkape, a framework to generate synthetic data that tackles these
limitations, specifically designed to enhance skill-to-taxonomy matching.
Within this framework, we create SkillSkape, a comprehensive open-source
synthetic dataset of job postings tailored for skill-matching tasks. We
introduce several offline metrics that show that our dataset resembles real-
world data. Additionally, we present a multi-step pipeline for skill
extraction and matching tasks using large language models (LLMs), benchmarking
against known supervised methodologies. We outline that the downstream
evaluation results on real-world data can beat baselines, underscoring its
efficacy and adaptability. 111Code and data available at
https://github.com/magantoine/JobSkape

JobSkape: A Framework for Generating Synthetic Job Postings to Enhance Skill
Matching

Antoine Magron*1 Anna Dai*1 Mike Zhang2 Syrielle Montariol1 Antoine Bosselut1
1EPFL, Switzerland 2IT University of Copenhagen, Denmark
<EMAIL_ADDRESS><EMAIL_ADDRESS>

**footnotetext: Equal contribution.

##  1 Introduction

In the dynamic modern labor market, understanding job demands at scale is
crucial for informed decision-making by policymakers, businesses, and other
stakeholders. One way of measuring job market demand lies in _skill matching_
: the extraction and alignment of skills from job descriptions to their
disambiguated forms (i.e., a knowledge base or taxonomy). This process
facilitates the investigation of current labor market dynamics and the
quantification of labor market demands, addressing the occupational skill
matching problem.

Regardless of their predictive effectiveness, supervised learning methods for
skill matching require regularly collecting and annotating up-to-date data
Zhang et al. (2022b), a process that is both expensive and time-consuming.
Synthetic data circumvents the need for such costly annotations. However,
despite efforts in generating synthetic training data Clavié and Soulié
(2023); Decorte et al. (2023) and real-world benchmarks Zhang et al. (2022a);
Decorte et al. (2022), challenges like incoherent sentences and over-
simplified setups exist in existing datasets. To address these shortcomings,
we introduce JobSkape, a framework for generating realistic skill matching
datasets that can be used for training and benchmarking.

JobSkape facilitates the creation of diverse labeled textual datasets that
align closely with actual job postings, ensuring cleaner and more coherent
data. We demonstrate its practical application by generating SkillSkape, a
large-scale dataset linking coherent sets of skills to corresponding job
descriptions. JobSkape uses generative large language models (LLMs) to curate
meaningful skill combinations and generate appropriate job descriptions
containing these combinations. A self-refinement step using LLMs Madaan et al.
(2023) ensures label quality in the refined SkillSkape dataset, assessed
through offline metrics. Finally, we challenge traditional supervised skill
matching methods with an LLM-based, in-context learning (ICL) pipeline, to
circumvent re-training the supervised model given new data. We evaluate skill
matching performance on our synthetic dataset and real-world annotated data
(Decorte et al., 2022), comparing our proposed extraction and matching
pipeline with supervised matching models trained on our dataset as well as
previous generation attempts (Decorte et al., 2023) in a controlled setting.

##### Contributions.

In this work, we contribute the following: (1) we propose JobSkape, a
framework for generating a synthetic dataset of job descriptions for skill
matching with existing skill taxonomies, (2) using our framework, we release a
synthetic train and evaluation dataset (SkillSkape) for skill matching, (3) we
show that SkillSkape has higher textual quality measured in perplexity and
implicitness compared to previous synthetic datasets, (4) lastly, we introduce
an ICL-based approach to extract and match skills from job descriptions to a
taxonomy and show that this method can outperform supervised baselines on
real-world benchmarks.

##  2 Related Work

##### Synthetic Data Generation.

Traditional synthetic data generation relies on language models, where a
generator model is trained on an existing dataset and then employed to
generate new data (Mohapatra and Mohapatra, 2022; Kumar et al., 2020). More
recent unsupervised methods, such as Wang et al. (2021), leverage pre-trained
language models like GPT-3 Brown et al. (2020) without the need for explicit
supervision. Other examples include Ye et al. (2022); Gao et al. (2023), who
use carefully designed prompts for data generation. Honovich et al. (2022)
generate synthetic instructions for fine-tuning large language models, while
Shao et al. (2023) create synthetic demonstrations to enhance the performance
of prompting LLMs.

##### Synthetic Data for Job Postings.

In the job market domain, Decorte et al. (2023); Clavié and Soulié (2023) both
employ GPT-3.5/4 to generate synthetic training data for skill matching.
Specifically, Decorte et al. (2023) prompt GPT-4 to generate ten examples for
each ESCO skill, while Clavié and Soulié (2023) use GPT-3.5 to generate 40
examples for each ESCO skill. In this work, we compare our dataset with the
one from Decorte et al. (2023), referred to hereafter as the Decorte dataset.

##### Skill Matching.

Earlier works focus on standardizing skills through matching with taxonomies.
For supervised methods, Gnehm et al. (2022) extract skills from Swiss-German
job descriptions and match them with the ESCO taxonomy in a two-step process.
Zhang et al. (2022b) assume pre-extracted skills and classify spans into their
respective taxonomy codes using multiclass classification. Decorte et al.
(2022) use distant supervision with the ESCO taxonomy to obtain labels, employ
binary classifiers for each ESCO skill and enhance training through negative
sampling strategies. Decorte et al. (2023); Clavié and Soulié (2023) employ
LLMs for skill matching with ESCO. Decorte et al. (2023) generate a synthetic
training set using GPT-3.5 and optimize a bi-encoder through contrastive
training for matching. Clavié and Soulié (2023) use a similar approach,
generating synthetic training data and employing a linear classifier for each
skill with a negative sampling strategy. Additionally, they use sentence
embedders Reimers and Gurevych (2019) to measure the similarity between
extracted skills and ESCO.

##  3 The JobSkape Framework

Our goal is to create a synthetic dataset comprising job posting sentences
associated with lists of skills from a taxonomy that closely aligns with real-
world job posting sentences. We initiate the process by generating
combinations of skills, derived from a given taxonomy, that are likely to
coexist in a job description. Leveraging LLMs and refinement techniques, we
produce diverse, realistic, and accurate job description sentences. To
evaluate the quality of our synthetic data generation, we define a set of
offline metrics and compare the generated sentences with real job postings.

###  3.1 The Label Space

In this study, we use the European Skills, Competences, Qualifications, and
Occupations (ESCO; le Vrang et al., 2014) taxonomy as the label space. ESCO
comprises 13,890 competencies categorized into _Skill_ , _Knowledge_ , and
_Attitudes_. Knowledge, according to ESCO, involves assimilating information
through learning, encompassing facts, principles, theories, and practices in a
specific field of work or
study.222https://ec.europa.eu/esco/portal/escopedia/Knowledge For example,
acquiring proficiency in the Python programming language through learning
represents a _knowledge_ component, classified as a _hard skill_. Conversely,
the application of this knowledge to perform tasks is considered a _skill_
component, defined by ESCO as the ability to apply knowledge and use know-how
to accomplish tasks and solve
problems.333https://ec.europa.eu/esco/portal/escopedia/Skill For the synthetic
sentence generation task at hand, we do not distinguish between skill and
knowledge components.

Our synthetic dataset creation framework generates sentences containing
multiple skills listed in the ESCO taxonomy. To reduce the data generation
cost (and to facilitate a fair comparison with prior work), we use the same
subset of 514 ESCO skills used in SkillSpan-M, an annotated set of real-world
job postings.

###  3.2 Formal Approach

Previous efforts (Decorte et al., 2023; Clavié and Soulié, 2023) focused on
generating synthetic training sentences with a single skill. In contrast, we
advocate for sentences containing multiple relevant skills to resemble
sentences from real job postings. We initiate the process by creating
combinations of skills, guided by three main conditions:

  1. 1.

Varying Lengths of Skill Combination: Recognizing the heterogeneity in real-
world job postings, we incorporate varying numbers of skills per sentence. By
doing so, our sentences will show higher diversity similar to real job
advertisements.

  2. 2.

Semantic Closeness in Skill Pairing: In real job postings, skills that are
mentioned in the same sentence are often related to each other. Aligning with
the logical grouping of skills, we construct more realistic and contextually
coherent sentences.

  3. 3.

Minimum Skill Representation: While our dataset aims to reflect the real-world
frequency of skill occurrences, we also want to ensure that each skill appears
enough times for training. This guarantees that even less common skills are
adequately represented, creating a more effective training dataset.

To achieve variety, we introduce two distributions, $\mathcal{N}$
(distribution of combination size) and $\mathcal{F}$ (distribution describing
skill frequency in job postings, akin to skill popularity). We iteratively
process skills $s_{i}\in\mathcal{S}$, the set of skills in our taxonomy,
ensuring each skill has the same minimum number of samples. For each skill, we
identify its $k$ nearest neighbors $\\{s_{j}^{\prime}\\}_{j=1}^{k}$ based on
cosine similarity between embeddings obtained from JobBERT, a language model
fine-tuned on domain-specific data Zhang et al. (2022a). Neighbors with a
similarity above threshold $T$ are retained, forming the set of nearest
neighbors $\mathcal{S}_{i}$:

$\mathcal{S}_{i}=\\{s_{j}:s_{j}\in\\{s_{i}^{\prime}\\}_{i=1}^{k}\wedge\text{sim}(s_{j},s_{i})\\}T\\}.$
(1)

This set is used for skill combination selection. We draw a sample size $n$
from distribution $\mathcal{N}$, setting the combination size to
$min(n,|\mathcal{S}_{i}|)$. However, sampling skills directly from
$\mathcal{S}_{i}$ is not straightforward. For instance, in analyzing the top
nearest neighbors of SQL, a frequently occurring skill in job postings, we
find THC Hydra, which is much less common. To accurately replicate the real-
world frequency distribution of skills in our synthetic dataset, we adjust
sampling probabilities to reflect actual skill popularity. Hence, we introduce
distribution $\mathcal{F}$ to compute the probability of selection over
$\mathcal{S}_{i}$ using softmax:

$\mathbf{P}(s_{j})=\frac{e^{\mathbf{P}_{\mathcal{F}}(s_{j})}}{\sum_{l=1}^{k}e^{\mathbf{P}_{\mathcal{F}}(s_{l})}}.$
(2)

where $\mathbf{P}_{\mathcal{F}}(s_{i})$ is the popularity of the skill
$s_{i}$. We then select $min(n,|\mathcal{S}_{i}|)$ skills from
$\mathcal{S}_{i}$ using the computed probability distribution.

For dataset creation, we form skill combinations from our ESCO subset, with
$\mathcal{N}$ set to $U(1,5)$. We employ JobBERT to obtain domain-specific
embeddings for job descriptions and skills. The distribution $\mathcal{F}$ is
computed as the average of standardized negative perplexities across sentences
generated with GPT-2 (Radford et al., 2019). These sentences include
variations like ‘‘I want a job that involves {skill}’’, ‘‘For my job, I want
to learn [to] {skills}’’, ‘‘At my job, my main is skill is [to] {skill}’’,
ensuring grammatical correctness. We set a similarity threshold $T$ to $0.83$
to be closer to SkillSpan distribution, using $k=20$.

###  3.3 Prompt Tuning for Generation

Given a skill combination, we generate synthetic job description sentences. A
candidate for this hypothetical job would need to be proficient to some extent
in each of these skills. We use GPT-3.5 as the text generator. We describe two
types of generations:

  * •

Dense: For a combination of four or less skills we generate a short job
description of at most one sentence. This is done to minimize the number of
hallucinated skills that could appear when generating a long job description
with a small set of skills.

  * •

Sparse: For a combination of more than four skills, we generate a job
description paragraph containing multiple sentences. The information is more
“sparse”.

Our prompt follows Clavié and Soulié (2023), it is used to make the mentions
of the skills as implicit as possible (i.e., skill does not have an exact
string match in the text). We further enhance the diversity by prompting the
model to vary the openings of the descriptions and avoid the examples starting
with ‘‘We are looking’’ or ‘‘We are searching’’ (see Appendix A.3.1).

We add additional instructions to the prompt to reduce ambiguity. To each
prompt, we add a list of synonyms of each inputted skill that are in the
taxonomy and instruct the model to not refer to SQL as MySQL since MySQL is a
separate skill in the taxonomy. Each skill is also given along with their
respective definitions to give more context to the model and avoid
miscomprehension.

Figure 1: Three-step Skill Extraction and Matching Pipeline. We show our in-
context learning pipeline for end-to-end skill matching. We use an LLM to
extract skills from job ads, then do candidate selection using heuristics, and
last, do skill matching with a constrained taxonomy.

###  3.4 Refinement of SkillSkape

At this step, the dataset comprises exclusively of positive samples, which
means that every generated sentence contains at least one associated skill. To
train a supervised classifier for real job descriptions, negative samples –
sentences containing unknown skills or no skills – are required. To create
negative samples with unknown skills, we apply the same generation method as
positive samples but draw from a broader pool of skills. This pool includes
skills that are not in our selected list but known in the wider skill
universe. We also generate negative samples containing no skills to represent
sentences in real-life job ads that do not mention skills required from the
candidate. To do so, we use two separate prompts to generate (1) sentences
describing the company: its reach, domain, location, et cetera and (2)
sentences detailing the salary and perks of a job (see Appendix A.3.2). To
guide the model in the generation, we provide two demonstrations. For the
SkillSkape dataset, we generate 500 negative samples with unknown skills as
well as 500 additional negative samples containing no skills.

We then apply self-refinement Madaan et al. (2023), involving feeding the
generated sentences back into the same model for feedback. The model is asked
to extract skills using the pipeline described in Appendix A.4.1, matching
them with the taxonomy. We compare the generated set of skills with the gold
set of skills, adding to the gold list all skills that were found in the
sentence. We do this because the LLM can extrapolate during sentence
generation, thereby adding related skills on top of the original list that was
fed to it. The list of skills, along with their associated spans in the
sentences, is filtered to include only pairs of skills and spans that have a
cosine similarity above a specified threshold. For this refined dataset
version, we use JobBERT as a span encoder, and the threshold is empirically
set to $0.7$ cosine similarity. One of the main reasons for this low
similarity is the LLM not reflecting accurately the gold skill during the
sentence generation step. In that case, we wish to remove that skill from the
gold set of skills associated with the sentence.

##### Span Extraction.

We use GPT-3.5 to label skill sequences in the sentences. Each mention,
whether implicit or explicit, is surrounded by @@ and ## following Wang et al.
(2023). In case the language model fails to label the span, it is asked to
self-correct, as outlined in Appendix A.4.2. We showcase two examples
extracted from the training set of the refined SkillSkape dataset.

##### Positive example.

This 28-word example, average in length for our dataset, contains three key
skills as annotated spans required of the applicant.

> Sentence: The ideal candidate will effectively @@engage with upper-level
> management##, @@maintain strong communication channels with key
> stakeholders##, and @@collaborate with peers## to ensure seamless
> coordination throughout the organization.
>
> Label: ’liaise with managers’, ’communicate with stakeholders’, ’liaise with
> colleagues’

In the example, the inter-skill similarity is high, showcasing the efficiency
of the skill combination selection method.

##### Negative example.

This sentence mentions information about the hiring company instead of the job
itself, and therefore, contains no skills.

> Sentence: Embrace a challenging and fulfilling career with us, where your
> hard work is recognized through a salary range of $80,000 to $90,000,
> reflecting our appreciation for your contributions.
>
> Label: NO LABEL

|  | Avg. |  | Avg. |
---|---|---|---|---|---
Dataset | Split | # Skills | % UNKs | # Words | # Samples
SkillSpan-M | Dev. | 2.0 | 47.0 | 15.0 | 178
Test | 1.9 | 47.3 | 16.3 | 751
Decorte | Train | 1.0 | 0.2 | 15.7 | 5,120
| Train | 2.6 | 7.9 | 28.2 | 6,352
SkillSkape | Dev. | 2.1 | 8.3 | 27.8 | 1,316
| Test | 2.6 | 8.4 | 28.1 | 1,272

Table 1: Datasets’ Statistics. Average # skills and words refer to the average
per sample (job posting sentence(s) and % UNKs refer to the percentage of
skill labels are under the unknown UNK label.

###  3.5 Summary and Comparison

The final version of SkillSkape has 8940 samples, split into training,
development and test sets ($\sim$ 70-15-15 split). We provide several
descriptive statistics in Table 1. To assess the quality of our generations,
we compare the generated dataset SkillSkape with two other datasets from the
literature: (1) a manually annotated benchmark, created by Decorte et al.
(2022), based on the SkillSpan-M(atch) dataset Zhang et al. (2022a), which
contains over 14.5K job posting sentences scraped from various sources, and
(2) the Decorte dataset Decorte et al. (2023), synthetically generated from
ESCO using GPT-4. By design, we created SkillSkape to cover the same label
space as SkillSpan-M, which has only a development and a test set. In that
dataset, two labels are used to indicate skills without an adapted label in
the taxonomy: UNDERSPECIFIED and LABEL NOT PRESENT. We map these to the UNK
label used in SkillSkape. Decorte associates ten synthetic sentences to each
skill in the ESCO taxonomy. It is only used as training data. It covers all of
ESCO (13.9K skills), but we restrict it to sentences with skills occurring in
SkillSpan-M, leading to 5,120 samples (we add 10 random UNK sentences).

##  4 Experimental Setup

In this section, we introduce several benchmarks for skill matching tasks. We
train a supervised multi-label classifier and present an LLM-based approach
with in-context learning.

###  4.1 Supervised Multi-label Classifier

For the supervised baseline, we use a pre-trained BERTbase_uncased model
Devlin et al. (2019) to extract contextualized embeddings from the input text
$t=\\{w_{1},w_{2},...,w_{n}\\}$. These embeddings are then input into a multi-
label classifier with a sigmoid activation applied independently to each
output logit. Let $y=\\{y_{1},y_{2},...,y_{k}\\}$ represent binary labels for
the $k$ classes. The model predicts the labels using:

$\hat{y}_{i}=\sigma(f_{i}(\text{BERT}(\boldsymbol{t}))),$ (3)

where $f_{i}(\cdot)$ is a function that maps the output embeddings from BERT
to a logit for class $i$, $\sigma(\cdot)$ is the sigmoid activation, and
$\hat{y}_{i}$ is the predicted probability for class $i$. The probability
threshold can be tuned; we empirically found that 0.2 works well for this
task.

We train the BERTbase model for 100 epochs with a learning rate of $3\times
10^{-5}$ and select the best-performing epoch. We use a batch size of 16 and a
maximum sequence length of 128. The model is trained for five different seeds.

###  4.2 In-context Learning with LLMs

We leverage an LLM to match skills in synthetic job posting sentences to the
ESCO taxonomy. This pipeline has three steps, visualized in Figure 1: 1) skill
extraction from the sentence, 2) candidate selection from the taxonomy, and 3)
skill matching to the list of candidates. Here, we first extract relevant
skills using LLM-prompting, pre-select viable candidates from our taxonomy,
and then match the skills to candidates in the taxonomy through LLM-prompting
again. We adopt a three-step approach to overcome the limited context window
of LLMs, specifically 4K for GPT-3.5-turbo OpenAI (2023), which makes feeding
large taxonomies directly to the model impractical.

##### (1) Skill Extraction.

For each job posting, the LLM identifies key skills and tasks within the job
ad while omitting irrelevant information. The LLM is directed to respond by
repeating the sentence and tagging the skills by surrounding them with @@ and
##, following Wang et al. (2023), as shown in our prompt below.

> System: You are an expert human resource manager. You need to analyse skills
> in a job posting.
>
> Instruction: You are an expert human resource manager. You are given an
> extract from a job description. Highlight all the skills, competencies and
> tasks that are required from the candidate applying for the job, by
> surrounding them with tags @@ and ##. Make sure you don’t highlight job
> titles, nor elements related to the company and not to the job itself. Make
> sure to rewrite the sentence with all the tags.
>
> {Demontrations}
>
> Sentence: {Sentence}
>
> Answer:

We provide seven demonstrations in a few-shot setting to assist the model in
understanding the task and following the instructions. To select few-shot
examples, we use $k$NN retrieval from a training set composed of sentences
along with their spans and labels Liu et al. (2022). The closest samples from
our dataset are selected as few-shot examples (see Appendix A). Finally, We
process the output by extracting the tagged sections as skills.

##### (2) Candidate Selection.

Matching extracted skills with skills defined in the taxonomy is crucial. Each
skill in the taxonomy is associated with a tiered structure of names, each
providing different levels of detail, and a definition. To provide richer
context to the model, we concatenate the most detailed (or granular) name of
the skill with its definition. We use two methods for pre-selecting viable
candidates from the taxonomy for each skill:

  * •

rule-based: Through string matching, we seek full or approximate matches of
the extracted skill within the taxonomy. If the exact string of the extracted
skill is present in the name or definition of a skill in the taxonomy, it is
considered a good candidate for a match. We randomly select five entries if
more than five candidates are found. If the exact strings do not match, we
calculate the token_set_ratio using
TheFuzz,444https://github.com/seatgeek/thefuzz a similarity score based on
Levenshtein’s distance Levenshtein et al. (1966). The top five candidates with
the highest scores are chosen.

  * •

embedding-based: Using a pre-trained language model (JobBERT; Zhang et al.,
2022a), we compare the extracted skills with taxonomy entries. We obtain the
contextualized embeddings of the extracted skill by embedding the sentences
and averaging the vector representation of the tokens of the extracted skill.
These embeddings are then compared to the representation of each skill in the
ESCO taxonomy if the extracted skill is a substring of the sentences it was
extracted from. Otherwise, the embedding of the extracted skill itself is
compared to the skills in the taxonomy. The top five most similar candidates
are selected based on cosine similarity.

While effective, the rule-based method may miss synonyms and context. On the
other hand, the embedding-based method addresses the limitations of the rule-
based method but risks selecting contextually similar yet factually dissimilar
candidates (e.g., software vs. hardware). Therefore, we adopt a hybrid
approach, retaining candidates from both rule-based and embedding-based
methods.

##### (3) Skill Matching.

The final step involves matching extracted skills to one of the selected
candidate skills. We present the LLM with formatted candidates as options, and
request the best match, resembling a ranking task. The model outputs the most
fitting option as a matched skill or provides no match if none are found. To
assist the model without overloading the prompt, we provide a one-shot example
in the following format:

  * Sentence: {generated sentence}

  * Skill: {extracted span}

  * A: {candidate 1}

  * ...

  * J: {candidate 10}

  * Answer: {selected candidate}

The full prompt with one-shot demontration can be found in Appendix A.2.

To conduct the experiments for in-context learning with LLM, we retrieve
demonstrations from the training set to provide examples for both the
extraction and matching steps. We conduct an ablation study on SkillSpan’s
validation set to select the best number of shots for both tasks. Experiments
are described in Appendix B, in Table 4, and Figure 2. The matching step is
performed with 10 candidates using the mixed setting (5 embedding candidates
and 5 string matching candidates). The best setting uses 7 demonstrations for
the extraction step and one demonstration for the matching step. Matching step
demonstrations have a large number of tokens due to the list of candidates
along with their definitions, which can explain the decreased performance
associated with adding more demonstrations.

| Perplexity ($\downarrow$) | S2SIM ($\uparrow$) | Explicitness (%, $\downarrow$)
---|---|---|---
SkillSpan-M | 178.2 | 0.662 | 5.0
Decorte | 65.1 | 0.739 | 22.4
SkillSkape | 44.3 | 0.744 | 6.9

Table 2: Offline Metrics. We show the offline metrics as described in
Subsection 4.3. ($\uparrow$) indicates higher the better, ($\downarrow$)
indicates lower is better.

| Supervised |  | Few-Shot ICL
---|---|---|---
$\downarrow$Train / Test$\rightarrow$ | SkillSkape | SkillSpan-M |  | SkillSkape | SkillSpan-M
Decorte | 28.0 $\pm$ 0.8 | 23.0 $\pm$ 0.7 |  | 36.8 $\pm$ 0.2 | 26.9 $\pm$ 0.5
SkillSkape | 68.0 $\pm$ 0.5 | 22.2 $\pm$ 0.9 |  | 37.6 $\pm$ 0.2 | 26.9 $\pm$ 0.3
Both | 67.2 $\pm$ 1.0 | 26.1 $\pm$ 1.2 |  | 37.6 $\pm$ 0.2 | 27.3 $\pm$ 0.4

Table 3: Supervised and Few-Shot ICL Results. _Both_ indicates the
concatenation of Decorte and SkillSkape. The scores are micro-$F_{1}$.

###  4.3 Offline Quality Metrics

We design a set of metrics to evaluate the quality and diversity of the data
at hand. Our intention is not to mirror metrics of SkillSpan-M, which is
untidy by nature of scraped data, but to produce high-quality training data
for downstream skill matching tasks.

  1. 1.

First, we consider Perplexity, i.e., how realistic the data is from the point
of view of a language model. We compute the perplexity of each of the
sentences using GPT-2 Radford et al. (2019), where lower is better.

  2. 2.

Second, we consider Skill-Sentence Similarity (S2SIM), the average cosine
similarity between a skill and the associated sentence. The higher this
metric, the closer the generated sentence will be semantically close to the
associated skills. We aim to maximize this metric. The embeddings are computed
using JobBERT, and BERT model fine-tuned on English job postings with the
masked language modeling objective.

  3. 3.

Finally, we measure Explicitness by counting the number of entities that
appear exactly in the sample, using string matching.

Table 2 shows offline metrics for SkillSpan-M, Decorte, and SkillSkape.
SkillSkape has a lower perplexity, and outperforms SkillSpan and Decorte in
terms of S2SIM. The main reason for SkillSpan-M’s low skill-sentence
similarity is its noisiness, leading to sentences often being cut mid-way and
lacking coherence. Around 7% of SkillSkape skills are fully explicit (the
label can be found exactly in the sentence), much closer to SkillSpan-M than
Decorte. A higher explicitness leads to an easier task; a skill matching model
needs to be trained on enough implicit examples to allow it to generalize to
implicit skills.

Overall, SkillSkape demonstrates similar statistics in perplexity and S2SIM
characteristics as Decorte. However, it notably exhibits a significant
($3\times$) enhancement in implicitly representing skills within each
sentence.

##  5 Results and Analysis

To assess the label refinement method (Section 3.4), we apply it to the
development set of the SkillSpan-M benchmark that has annotated skills and
associated spans. $40\%$ of our extracted spans match exactly with the
annotated span. $60\%$ of our extracted spans are either a perfect match or
contain the annotated span. In general, extracted spans have a Jaccard
similarity of $62\%$ with the annotated spans.

###  5.1 Supervised vs. Few-shot ICL Matching

In Table 3, we show the results of the skill matching task on the SkillSpan-M
test set and SkillSkape test set. We compare the performance of supervised and
in-context learning methods trained on the Decorte training set, SkillSkape
training set, or the concatenation of both. The supervised approach uses
training data to train a supervised multi-label classifier, whereas the few-
shot ICL approach uses it as a demonstration pool to retrieve $k$NN
demonstrations. For simplicity, we refer to both training and few-shot
learning with demonstrations as training in the remainder of this section.

Comparing across supervised and few-shot in-context learning settings, we
observe that both supervised and ICL approaches achieve higher performance on
both real-world data (SkillSpan-M) and our synthetic dataset (SkillSkape) when
trained on SkillSkape training set or a combination of SkillSkape and Decorte
training sets. This increase in matching performance is likely due to a higher
textual diversity in the SkillSkape dataset. Across both matching approaches,
we also observe that training on both SkillSkape and Decorte consistently
achieves the highest test $F_{1}$ scores on real-world data. However, the
difference in performance is greater for the supervised approach than ICL,
highlighting that the ability to generate high-quality data is most impactful
for supervised approaches.

Additionally, we observe an interesting result in the large difference between
the supervised and ICL performance on the SkillSkape test set, 68.0/67.2 and
37.6 micro-$F_{1}$ respectively when trained on SkillSkape or a combination of
SkillSkape and Decorte. We suspect that this difference could largely be due
the characteristics of our training and test data. Supervised models tend to
perform well when the training and test data follow the same distributions. In
contrast, the few-shot ICL method consistently outperforms than the supervised
approach on the SkillSpan-M test set. Given the minimal tuning required for
the ICL method, the ICL approach can be better suited to flexibly handle messy
real-world data. These results suggest that, for use cases when we have a
sample of annotated data from the same distribution as the data we want to
predict, we can combine it with synthetic training data and leverage
supervised models. Otherwise, the in-context learning approach is less
dependent on the training data.

In Appendix Table 6, we show several qualitative examples of predictions of
both the multi-label classifier and LLM. Several noticeable patterns are
underprediction for the multi-label classifier and overprediction of the LLM.
Additionally, we notice that the predictions of both models are rather close
“semantically” to the gold labels, but are deemed incorrect by the evaluation.

In summary, the results underscore the significance of both the quantity and
diversity of training data in the development of effective skill matching
dataset generators.

###  5.2 Effect of In-context Demonstrations

We evaluate the sensitivity of our method to the number and candidate
selection methodology of in-context learning examples.

##### Demonstrations.

We perform an ablation study on the number of demonstrations for both skill
extraction and matching. Results in Appendix Table 4 show that 7 shots for
extraction with 1 shot for matching leads to the best performance.

##### Candidate Selection.

Candidate selection using the hybrid method for $n=5$ candidates from each of
the rules- and embedding-based methods (i.e., 10 candidates in Figure 3)
presents the best trade-off between performance and computational cost. While
we do observe a higher $F_{1}$ score as we increase the number of candidates,
the increase in performance appears to be marginal while it would more than
double the number of input tokens.

Finally, an ablation study on the matching step of the pipeline (see
AppendixB.3) shows that directly selecting the top-1 candidate (rule-based) as
skill prediction lags behind the performance of using GPT-3.5 as a re-ranker
by around 8% $F_{1}$.

###  5.3 Effect of Sentence Length

The sentence length distribution is heavily skewed toward shorter sentences in
the Decorte and SkillSpan-M test sets, with 50% of sentences being 13–19 words
in Decorte and 7–20 words in SkillSpan-M. In contrast, 50% of the sentences in
SkillSkape are between 23–33 words (see Figure 4 in Appendix for a
visualization of the length distribution for each dataset).

When splitting the SkillSpan-M test set into two equal-sized sets depending on
the size of the sequence (less than 12 words, or more than 12 words), training
on Decorte leads to slightly higher performance than SkillSkape for shorter
sentences (0.26 vs. 0.24 $F_{1}$). For longer sentences, however, SkillSkape
reaches an $F_{1}$-score of 0.18 while Decorte’s $F_{1}$ is 0.17.

##  6 Conclusion

We introduce JobSkape, a general framework for generating synthetic job
posting sentences for skill matching. Using our framework, we release
SkillSkape a large dataset of synthetic job posting sentences labeled with
ESCO skills. Our analysis shows that SkillSkape contains more implicit skills,
has longer sentences, and is overall closer to real-world data, compared to
alternative synthetic dataset from the literature. Using our dataset, we
conducted several skill matching experiments by training a supervised multi-
label classifier and using in-context learning with an LLM, and showed that
both methods achieved comparable results when evaluated on real-world data
($F_{1}$ of 26.1 and 27.3 respectively). Furthermore, we note that the
potential applications of JobSkape extend beyond its current scope. Its
application in creating synthetic CVs, for instance, can enhance job matching
algorithms and facilitate skill-gap analysis in various industries. The
framework’s adaptability to different skill taxonomies also opens up
possibilities for use across multiple sectors. While promising, these extended
applications require further exploration to fully assess their impact.

##  7 Limitations

##### Closed model.

One of the primary limitations comes from our use of Large Language Models
(LLMs) that are closed. This restricts our ability to understand, modify, or
customize the underlying mechanisms of these models. The closed nature of the
LLMs used in our study also limits the transparency, adaptability, and
reproducibility of our system.

##### English only.

Our method is limited to processing and understanding English language
content. This language-specific focus narrows the scope of our system’s
applicability, excluding non-English speaking demographics.

##### Bias inherited from LLMs.

Another significant limitation is the potential bias inherited from the LLMs.
Since these models are trained on large datasets that may contain biases,
there is a risk that our system may inadvertently perpetuate these biases in
its generations. This could manifest in various forms, such as gender,
cultural, or industry-specific biases, and could affect the fairness and
neutrality of the job postings generated. Furthermore, if biased postings are
used extensively, they could adversely influence downstream tasks. For
example, biased job postings could skew job recommendation algorithms, leading
to unfair job suggestions that do not treat all individuals equally. This
highlights the need for careful consideration and mitigation of biases in our
approach to ensure equitable outcomes in all applications.

##### Subset of the Taxonomy.

Due to limited resources, we restricted the generation of our synthetic
dataset to $\sim$8K samples, with a fraction of the ESCO taxonomy that is also
used in the SkillSpan-M dataset. Consequently, the multi-class classifier is
also trained to classify with a limited set of skills. Scaling up to the full
taxonomy might modify the behavior of the supervised classification model,
while it should have little to no impact on the ICL skill-matching pipeline.

##  8 Ethics Statement

In this work, we strictly used publicly available data and generated synthetic
datasets, avoiding the use of sensitive or private information. This approach
aligns with ethical standards concerning data privacy and security.

However, our system can be used to extract information from personal
documents, or be used for sensitive applications in the human resources
domain, notably pre-selecting candidates to hire. It shall not be used without
the supervision of a human. In this work, we focus on the development of a
framework to reduce reliance on real-world annotated data. Extended to
resumes, it could allow users to perform the skill extraction and matching
task without requiring personal data to be anonymized. Given the limited
performance of anonymization tools, generating data following similar
distribution would greatly reduce privacy issues for such applications.

## Acknowledgements

We thank Jibril Frej (EPFL) for fruitful discussions and feedback on the first
version of the paper. MZ is supported by the Independent Research Fund Denmark
(DFF) grant 9131-00019B and in parts by ERC Consolidator Grant DIALECT
101043235. We also gratefully acknowledge the support of the Swiss National
Science Foundation (No. 215390), Innosuisse (PFFS-21-29), the EPFL Science
Seed Fund, the EPFL Center for Imaging, Sony Group Corporation, and the Allen
Institute for AI.

## References

  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901.
  * Clavié and Soulié (2023) Benjamin Clavié and Guillaume Soulié. 2023.  Large language models as batteries-included zero-shot ESCO skills matchers.  In _Proceedings of the 3rd Workshop on Recommender Systems for Human Resources (RecSys in HR 2023) co-located with the 17th ACM Conference on Recommender Systems (RecSys 2023), Singapore, Singapore, 18th-22nd September 2023_ , volume 3490 of _CEUR Workshop Proceedings_. CEUR-WS.org.
  * Decorte et al. (2022) Jens-Joris Decorte, Jeroen Van Hautte, Johannes Deleu, Chris Develder, and Thomas Demeester. 2022.  Design of negative sampling strategies for distantly supervised skill extraction.  _arXiv preprint arXiv:2209.05987_.
  * Decorte et al. (2023) Jens-Joris Decorte, Severine Verlinden, Jeroen Van Hautte, Johannes Deleu, Chris Develder, and Thomas Demeester. 2023.  Extreme multi-label skill extraction training using large language models.  _CoRR_ , abs/2307.10778.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Gao et al. (2023) Jiahui Gao, Renjie Pi, Lin Yong, Hang Xu, Jiacheng Ye, Zhiyong Wu, Weizhong Zhang, Xiaodan Liang, Zhenguo Li, and Lingpeng Kong. 2023.  Self-guided noise-free data generation for efficient zero-shot learning.  In _International Conference on Learning Representations (ICLR 2023)_.
  * Gnehm et al. (2022) Ann-sophie Gnehm, Eva Bühlmann, Helen Buchs, and Simon Clematide. 2022.  Fine-grained extraction and classification of skill requirements in German-speaking job ads.  In _Proceedings of the Fifth Workshop on Natural Language Processing and Computational Social Science (NLP+CSS)_ , pages 14–24, Abu Dhabi, UAE. Association for Computational Linguistics.
  * Honovich et al. (2022) Or Honovich, Thomas Scialom, Omer Levy, and Timo Schick. 2022.  Unnatural instructions: Tuning language models with (almost) no human labor.  _arXiv preprint arXiv:2212.09689_.
  * Kumar et al. (2020) Varun Kumar, Ashutosh Choudhary, and Eunah Cho. 2020.  Data augmentation using pre-trained transformer models.  _arXiv preprint arXiv:2003.02245_.
  * le Vrang et al. (2014) Martin le Vrang, Agis Papantoniou, Erika Pauwels, Pieter Fannes, Dominique Vandensteen, and Johan De Smedt. 2014.  Esco: Boosting job matching in europe with semantic interoperability.  _Computer_ , 47(10):57–64.
  * Levenshtein et al. (1966) Vladimir I Levenshtein et al. 1966.  Binary codes capable of correcting deletions, insertions, and reversals.  In _Soviet physics doklady_ , volume 10, pages 707–710. Soviet Union.
  * Liu et al. (2022) Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2022.  What makes good in-context examples for GPT-3? In _Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures_ , pages 100–114, Dublin, Ireland and Online. Association for Computational Linguistics.
  * Madaan et al. (2023) Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Sean Welleck, Bodhisattwa Prasad Majumder, Shashank Gupta, Amir Yazdanbakhsh, and Peter Clark. 2023.  Self-refine: Iterative refinement with self-feedback.  _CoRR_ , abs/2303.17651.
  * Mohapatra and Mohapatra (2022) Sovesh Mohapatra and Somesh Mohapatra. 2022.  Sentiment is all you need to win US presidential elections.  In _Proceedings of the 2nd International Workshop on Natural Language Processing for Digital Humanities_ , pages 15–20, Taipei, Taiwan. Association for Computational Linguistics.
  * OpenAI (2023) OpenAI. 2023.  GPT-3.5 Documentation.  https://platform.openai.com/docs/models.
  * Radford et al. (2019) Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019.  Language models are unsupervised multitask learners.
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019.  Sentence-BERT: Sentence embeddings using Siamese BERT-networks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3982–3992, Hong Kong, China. Association for Computational Linguistics.
  * Shao et al. (2023) Zhihong Shao, Yeyun Gong, Yelong Shen, Minlie Huang, Nan Duan, and Weizhu Chen. 2023.  Synthetic prompting: Generating chain-of-thought demonstrations for large language models.  _arXiv preprint arXiv:2302.00618_.
  * Wang et al. (2023) Shuhe Wang, Xiaofei Sun, Xiaoya Li, Rongbin Ouyang, Fei Wu, Tianwei Zhang, Jiwei Li, and Guoyin Wang. 2023.  Gpt-ner: Named entity recognition via large language models.  _arXiv preprint arXiv:2304.10428_.
  * Wang et al. (2021) Zirui Wang, Adams Wei Yu, Orhan Firat, and Yuan Cao. 2021.  Towards zero-label language learning.  _arXiv preprint arXiv:2109.09193_.
  * Ye et al. (2022) Jiacheng Ye, Jiahui Gao, Qintong Li, Hang Xu, Jiangtao Feng, Zhiyong Wu, Tao Yu, and Lingpeng Kong. 2022.  ZeroGen: Efficient zero-shot learning via dataset generation.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 11653–11669, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Zhang et al. (2022a) Mike Zhang, Kristian Jensen, Sif Sonniks, and Barbara Plank. 2022a.  SkillSpan: Hard and soft skill extraction from English job postings.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 4962–4984, Seattle, United States. Association for Computational Linguistics.
  * Zhang et al. (2022b) Mike Zhang, Kristian Nørgaard Jensen, and Barbara Plank. 2022b.  Kompetencer: Fine-grained skill classification in Danish job postings via distant supervision and transfer learning.  In _Proceedings of the Thirteenth Language Resources and Evaluation Conference_ , pages 436–447, Marseille, France. European Language Resources Association.

##  Appendix A Prompts

###  A.1 Extraction Demonstrations

In our in-context learning pipeline, we provide seven demonstrations to guide
the LLM in performing extractions. Below is one example.

> Sentence: we are looking for a team leader with strong communication skills
> to foster collaboration and information sharing within the team.
> Answer: We are looking for a team leader with strong @@communication
> skills## to foster collaboration and information sharing within the team.
>
> Sentence: the ability to work collaboratively across disciplines is a key
> criterion for this position.
> Answer: @@ability to collaborate across disciplines## is a key criterion for
> this position.
>
> Sentence: As a Java Senior Software Engineer with experience, you will be a
> member of a Scrum team.
> Answer: As a Java Senior Software Engineer with experience, you will be a
> member of a Scrum team.
>
> Sentence: In her role as a team leader, she has continuously supported the
> professional development of her employees.
> Answer: In her role as a team leader, she has continuously fostered the
> professional @@development of her employees##.
>
> Sentence: He is a resilient employee who has been able to set proper
> priorities and organize tasks thoughtfully during periods of heavy workload.
> Answer: He is a resilient employee who has been able to set @@correct
> priorities and organize tasks thoughtfully## during periods of high
> workload.
>
> Sentence: Highly qualified, flexible employees from the insurance and IT
> industry develop them further.
> Answer: Highly qualified, flexible employees from the insurance and IT
> industries continue to develop them.
>
> Sentence: Over the past few years, it has succeeded in continuously
> developing itself in a rapidly changing environment.
> Answer: Over the past few years, he has succeeded in @@continuously
> developing## himself in a rapidly changing environment##.

###  A.2 Matching

####  A.2.1 Prompt

Here we provide the prompt used to match each extracted skill to one of the
pre-selected candidates. The one-shot demonstration used in this prompt is
provided in section A.2.2.

> System: You are an expert human resource manager. You need to analyse skills
> in a job posting.
>
> Instruction: You are given a sentence from a job description, and a skill
> extracted from this sentence. Choose from the list of options the one that
> best match the skill in the context. Answer with the associated letter.
>
> {Demonstration}
>
> Sentence: {Sentence}
> Skills: {Extracted}
> A: {Candidate 1}
> ...
> J: {Candidate 10}
>
> Answer:

####  A.2.2 Demonstration

The demonstration we use in the matching step (Section A.2.1) of our in-
context learning pipeline.

> Sentence: Understand basic provisions of copyright and privacy.
>  Skill: Data protection.
>  Options:
>  A: "Respect privacy principles"
>  B: "Understand data protection"
>  C: "Ensure data protection in aviation operations"
>  D: "Data protection"
>  Answer: b, d.

###  A.3 Generation of dataset

####  A.3.1 Positive samples

We use this prompt to generate samples containing ESCO skills.

> System: You are the leading AI Writer at a large, multinational HR agency.
> You are considered as the world’s best expert at expressing required skills
> and knowledge in a variety of clear ways. You are particularly proficient
> with the ESCO Occupation and Skills framework. As you are widely lauded for
> your job posting writing ability, you will assist the user in all job-
> posting, job requirements and occupational skills related tasks.
>
> Instruction: You work in collaboration with ESCO to gather rigid standards
> for job postings. Given a list of ESCO skills and knowledges, you’re asked
> to produce a single example of exactly one sentence that could be found in a
> job ad and refer to all skill or knowledge component. Ensure that your
> sentence is well written and could be found in real job advertisement. Use a
> variety of styles. You’re trying to provide a representative sample of the
> many, many ways real job postings would evoke skills. All the skills in :
> {skillList} must be integrated. A candidate should have different degrees of
> expertise in all the given skills. This degree should be specified for each
> skills in the sentence. You must not include any skills in ESCO that were
> not given to you. Try to be as implicit as possible when mentionning the
> skill. Try not to use the exact skill string {wordsToAvoid}. Avoid
> explicitly using the wording of this extra information in your examples.
> Your sentence must not start with ’We are seeking’, ’We are looking’ or ’We
> are searching’. Generate stricly only one example.

####  A.3.2 Negative samples

We use two different prompts to generate negative samples: This first prompt
generates negative samples that describe the company.

> System: You are the leading AI Writer at a large, multinational HR agency.
> You are considered as the world’s best expert at writing introductions of
> job posting.
>
> Instruction: You are the leading AI Writer at a large, multinational HR
> agency. You are considered as the world’s best expert at writing
> introductions of job posting. You should write {nExamples} examples of the
> first line of the job posting. It should consists in introducing the
> company, its localization, the number of employees, and any information
> relevant to a future candidates who wants to learn about the company. The
> description should be concise, specify the potential growth of the company
> and a domain of action. You shouldn’t mentoin anything about the actual job,
> no skills required for the candidate and shouldn’t mention the candidate at
> all. You should mention a wide range of company field, size, and
> localization in each of the examples.

This second prompt generates sentences detailing the salary and perks of a
job.

> System: You are the leading AI Writer at a large, multinational HR agency.
> You are considered as the world’s best expert at specifying administrative
> information in job posting.
>
> Instruction: You are the leading AI Writer at a large, multinational HR
> agency. You are considered as the world’s best expert at specifying
> administrative information in job posting. You should produce {nExamples}
> descriptions of the salary and the perks a candidate to a certain job would
> have. You shouldn’t mention the actual job and the candidate itself. You
> could add diversity by varying the salary and the perks. You must write a
> salary range between 40k and 100k according to the job in half of your
> generation.

###  A.4 Refinement of dataset

####  A.4.1 Initial prompt

> System: You are an expert human resource manager. You need to analyse skills
> in a job posting.
>
> Instruction: You are an expert human resource manager. You are given an
> extract from a job description and a skill coming from ESCO. Highlight all
> the parts of the job description that relates to the given skill, by
> surrounding them with tags ’@@’ at the beginning and ’##’ at the end. You
> should rewrite the entire sentence. The highlighted parts should precisely
> talk about the given skills and only this skills. The higlighted parts must
> precisely be about the given skills. Do not highlight parts not related to
> it. The sentence should be rewritten perfectly, using the same exact same
> words. You must highlight at least one part in the sentence that you will
> rewrite. The highlighted part should be as short as possible.

####  A.4.2 Refining shots

In case of incorrectly bound annotations :

> In your response, you highlighted some parts using @@ at the beginning and
> @@ at the end. Please use @@ at the beginning of the parts and ## at the end
> of the part you want to highlight. Annotate the previous sentence, but with
> the correct highlighting.

When there is a lack of annotations :

> In your response, you highlighted nothing. Please annotate the previous
> sentence, and highlight at least one part linked to the skill.

##  Appendix B Ablation studies - Few-Shot ICL

###  B.1 Demonstrations

To conduct the experiments for the In-context Learning with LLM, we will use
the demonstrations retrieval from the training set to provide few shots for
both the extraction and the matching. We need to determine the number of
demonstration to use for both parts. For this purpose we conduct an ablation
study on SkillSpan-M ’s validation test trying different configuration of
number of shots. We try the following experiments :

  * •

baseline : Same shots for all the sentences A.1 A.2.2

  * •

$M_{1}$ : 1 demonstration for the matching part, baseline shot for extraction

  * •

$E_{5}$ : 5 demonstration for the extraction part, baseline shot for matching

  * •

$E_{7}$ : 7 demonstration for the extraction part, baseline shot for matching

  * •

$E_{10}$ : 10 demonstration for extraction, baseline shot for matching

  * •

$E_{7}M_{1}$ : 1 demonstration for the matching part and 7 for the extraction
part

  * •

$E_{7}M_{3}$ : 3 demonstrations for the matching part and 7 for the extraction
part

| Recall | Precision | $F_{1}$
---|---|---|---
baseline | 0.260 | 0.303 | 0.280
$E_{5}$ | 0.279 | 0.296 | 0.287
$E_{7}$ | 0.282 | 0.301 | 0.291
$E_{10}$ | 0.282 | 0.298 | 0.289
$M_{1}$ | 0.267 | 0.305 | 0.284
$E_{7}M_{1}$ | 0.289 | 0.298 | 0.2934
$E_{10}M_{3}$ | 0.283 | 0.293 | 0.288

Table 4: Ablation study for In-context Learning: Selecting optimal number of
demonstrations for extraction and matching with GPT-3.5 Figure 2: Ablation
study for In-context Learning

Given the stats in Table 4, displayed on Figure 2 we see the adding the
demonstration retrieval for the extraction part yields a significative
improvement on the recall. We will run the subsequent experiments with 7
demonstrations for the extraction part and one demonstration for the matching
part.

###  B.2 Candidate Selection

Figure 3 shows shows the $F_{1}$ scores of the ICL when we vary the number of
candidates selected using the rule-based, embedding-based, and hybrid
candidate selection methods, holding other elements constant. Looking at our
results, we elect to use 10 candidates ($n=5$) with the hybrid method.

While further increasing the candidates can increase matching performance
slightly, we find that providing too many candidates can lead to a noticeable
increase in inference time.

Figure 3: Rule-based, embedding-based, and hybrid candidate selection methods
to select $n$ candidates. Note, since the hybrid method takes the union of
rule-based and embedding-based methods, $n=5$ using the hybrid method would
approximate $n\times 2$ actual number of actual candidates selected

###  B.3 Matching Step

We conduct an ablation study on the matching. We remove the matching step from
the pipeline and we only extract the spans from the inputted sentences and use
rule-based to find matches. We focus on the rule-based method that yields the
best results when extracting a small amount of candidates. Table 5 shows that
the top-selected candidates are behind the performance of using GPT-3.5 as a
re-ranker by around 8%. Therefore, we continue our experiments using the full
three-step pipeline.

# of Candidates | Precision | Recall | $F_{1}$
---|---|---|---
1 | 24.2 | 16.9 | 19.9
2 | 17.1 | 23.5 | 19.8

Table 5: Ablation study of the matching step: Performance of the ICL pipeline
when taking only the top 1 or 2 candidates using the rule-based selection
methods.

##  Appendix C Qualitative Analysis

We present various qualitative examples of predictions from the test set of
SkillSkape in Table 6. These examples include outcomes from both the
supervised multi-label classifier and the in-context learning results using
GPT-4. A key observation is the relatively lower number of skills predicted by
the supervised classifier, which operates with a threshold of 0.15. Generally,
these predictions are feasible and align closely with the gold standard label.
However, it should be noted that the evaluation process tends to penalize
these predictions for their limited scope.

Sentence |  Multi-label Classifier |  In-context Learning |  Gold
---|---|---|---
(1) Seeking a highly skilled individual with extensive expertise in overseeing and optimizing the operation and maintenance of various technical components and systems on board maritime vessels. |  shipping industry |  overseeing and optimizing the operation and maintenance of various technical components and systems on board maritime vessels |  manage vessel engines and systems
(2) As an integral part of our team, the ideal candidate should possess a deep understanding of coordinating the alignment and seamless interaction of various system components, while executing rigorous testing and implementing an overarching strategy for the integration of ICT systems |  ICT system integration, define integration strategy |  coordinating the alignment and seamless interaction of various system components, rigorous testing, integration of ICT systems |  ICT system integration, define integration strategy, define software architecture, manage ICT data architecture
(3) Ability to effectively adapt to changing circumstances while maintaining a vigilant attitude, maintaining composure in challenging situations, and efficiently managing workload and responsibilities. |  handle stressful situations |  effectively adapt to changing circumstances, vigilant attitude, composure, efficiently managing workload and responsibilities |  exercise patience, adjust priorities, stay alert
(4) Are you an experienced professional with a proven track record in designing and implementing comprehensive technology testing frameworks, ensuring the seamless integration of software applications and systems? |  develop ICT test suite, execute software tests |  designing and implementing comprehensive technology testing frameworks, seamless integration of software applications and systems |  develop ICT test suite

Table 6:  We show several qualitative examples of predictions on the test set
of SkillSkape using the supervised multi-label classifier and in-context
learning results with GPT-4.

##  Appendix D Other Summary Statistics on SkillSkape

####  D.0.1 Skill Groups

We show in Table 7 the skill groups and counts of skills in each ESCO skill
group that is included in the label spaced used for the SkillSkape dataset.

Skill Group | skill count
---|---
agriculture, forestry, fisheries and veterinary | 4
arts and humanities | 8
assisting and caring | 13
business, administration and law | 40
communication, collaboration and creativity | 111
constructing | 3
education | 3
engineering, manufacturing and construction | 22
generic programmes and qualifications | 6
handling and moving | 15
health and welfare | 7
information and communication technologies (icts) | 71
information skills | 57
management skills | 65
natural sciences, mathematics and statistics | 10
services | 5
social sciences, journalism and information | 1
working with computers | 35
working with machinery and specialised equipment | 14
TOTAL | 514
Table 7: ESCO skill groups present in SkillSkape dataset

####  D.0.2 Sentence Length

Looking at Figure 4, we can see that SkillSkape has longer sentences and
contains more variation in sentence length than Decorte. The distribution of
SkillSkape resembles more that of real-world data (SkillSpan-M).

Figure 4: Sentence length distribution in the three datasets. SkillSkape has
much longer sentences. Decorte has very short sentences and low length
variance.