# Reliable compositional analysis of airborne particulate matter beyond the
quantification limits of total reflection X-ray fluorescence

Yves Kayser  [<EMAIL_ADDRESS>János Osán  [  Philipp Hönicke  Burkhard
Beckhoff  [

###### Abstract

Knowledge on the temporal and size distribution of particulate matter (PM) in
air as well as on its elemental composition is a key information for source
appointment, for the investigation of their influence on environmental
processes and for providing reliable data for climate models. While cascade
impactors allow for time- and size-resolved collection of airborne PM, total
reflection X-ray fluorescence (TXRF) allows for element-sensitive
investigation of minute sample amounts thanks to its detection sensitivity.
But during quantification by means of TXRF it is crucial to be aware of the
linear calibration limits of TXRF in order to identify situations where
collection times or pollution levels in the different size partitions were
exceedingly long or high. Indeed, TXRF can only be reliably used when the
amount of matter collected on the top of substrate is sufficiently dilute. By
means of grazing incidence X-ray fluorescence (GIXRF), where the excitation
conditions are varied in a controlled and reliable manner and include also the
TXRF regime, a self consistent quantification of elemental mass depositions
can be performed in order to validate or falsify TXRF quantification results.
For low mass depositions an agreement within few percent for the different
excitation conditions was found, while for increasing amounts of material
relative errors of up to factor of 4 were found for TXRF as compared to GIXRF.
Thus, TXRF cannot be applied to all samples regardless of their coverage and
threshold values for the validity of quantification results need to be
determined. As a flexible solution, GIXRF allows extending the dynamic range
of reliably quantifiable mass depositions beyond the linear regime of TXRF, an
important advantage when variable amounts of airborne PM need to be quantified
as in the case of collection with cascade impactors. Furthermore, the
presented more reliable quantification approach can be transferred to mobile
tabletop instrumentation as well. This aspect is highly relevant for air
quality monitoring in terms supporting the definition of appropriate
legislation and measures for health and climate protection as well as for
supporting their enforcement.

###### keywords:

TXRF, GIXRF, aerosols, particulate matter

PTB]Physikalisch-Technische Bundesanstalt, Abbestraße 2-12, 10587 Berlin,
Germany EK]Environmental Physics Department, Centre for Energy Research,
Konkoly-Thege M. út 29-33., 1121 Budapest, Hungary PTB]Physikalisch-Technische
Bundesanstalt, Abbestraße 2-12, 10587 Berlin, Germany

##  1 Introduction

Aerosols present in the environment affect our daily life at multiple levels.
For example, airborne particulate matter (PM) in air can impact health due to
inhalation 1, 2, 3 or can influence atmospheric processes 4, more precisely
the climate and environmental ecosystems through impacting cloud formation 5
or reflecting and scattering sunlight 6. The chemical composition, which
requires element-sensitive analytical methods, as well as the chemical
speciation of the elements contained in airborne PM is of interest for a
correct comprehension of its physical and chemical properties 7. With regard
to health concerns, the smallest particles, so called fine and ultrafine
particles with sizes in the sub-micrometer and sub-100 nanometer range, are
the most concerning for epidemiology as they can penetrate into the airways of
the lungs and may be held responsible for health-averse effects on the
respiratory and cardio-vascular system upon long-term exposure 8, 9, 10, 11.
In particular anthropogenic emissions result in a noticeably higher generation
of ultrafine particles 12. While toxicological studies have to assess possible
health risks of different nanomaterials 13, parallel efforts have to be
undertaken to quantify the presence of the different elements in air and trace
back the physical processes airborne PM is undergoing under different weather
conditions. The compositional analysis of aerosols is often addressed by
regulated analytical techniques requiring moderate amounts of substance
collected in fiber filters. Improved time-resolved and size-fractionated
information is, however, relevant for accurate modeling of climate changes,
for regulatory bodies to impose preventive measures and for legal entities to
enforce regulations on air quality and to correctly pinpoint anthropogenic or
natural sources. Adding to this the requirement to not only detect but to
quantify reliably trace levels of aerosols contained in air in order to
achieve good time resolution during environmental monitoring campaigns, highly
sensitive and accurate techniques need to be used.

Among different available techniques 14, 15 X-ray fluorescence (XRF) based
methods are promising contributors to the field by delivering ensemble
information on the chemical composition due to the advantages provided in
terms of sample preparation, consumption and sensitivity 14, 16. The best
possible detection limits with XRF based techniques can be achieved by means
of total reflection XRF (TXRF) 17, 18. In combination with cascade impactors,
where the PM is collected on different impaction stages and discriminated by
their size using the principle on inertia, all relevant information for a
time-dependent, element- and size-sensitive quantification of airborne PM in a
defined volume of air down to the range of few ng/m3 are at hand 19, 20, 21,
22, 23, 24, 25. But varying environmental conditions and human activities
result in unpredictable pollution levels which may in addition severely vary
between the different impaction stages for each collection interval. Thus, it
can not be ensured beforehand that the samples from the different impaction
stages and for the different collection times are all within the linear regime
of TXRF where it is assumed that the X-ray standing wavefield (XSW) created is
not or only to a small amount disturbed by the material on the top of the
substrate. Since the impact of airborne PM on the XSW can not be known
beforehand, quantification by means of TXRF, which is a single-point
measurement technique without parameter variation, risks to fail and must be
assessed during outdoor campaigns.

In the present work, the impact of different mass depositions on the
reliability of the quantification of the elemental mass deposition by means of
TXRF is investigated. This assessment is being done by analyzing samples from
field campaigns with different collection times by both, standard TXRF at a
single, fixed incidence angle and by grazing incidence X-ray fluorescence
(GIXRF) where the incidence angle is varied in a controlled and reproducible
manner. Indeed, during a GIXRF measurement the X-ray standing wavefield (XSW)
created on the top of the sample is modified, respectively can be neglected
for incidence angles far above the critical angle of total external reflection
such that the excitation conditions are fundamentally modified during the
measurement. For any given sample the mass deposition of the airborne PM
collected depends on the pollution level and collection time and does not vary
during analysis. Hence, it must be expected that the quantification of the
different elements yields the same result for each incidence angle used during
the GIXRF measurement. If this is the case, the quantification result can be
considered as validated in a self-consistent approach. Hereafter, it will be
assessed whether quantification of elemental mass depositions using TXRF and a
self-consistent validation of the results can be achieved independently of the
amount of airborne PM collected.

##  2 Quantification by means of TXRF

TXRF employs a specific geometry in which the X-ray beam used for the
excitation of the XRF signal impinges the sample at a very shallow angle
beneath the critical angle for total external reflection 26. Hence, TXRF
demands for collimated and monochromatic excitation conditions but offers
advantages such as the illumination of large sample areas and a large solid
angle of detection. Further benefits offered by TXRF are twofold. On the one
side, the penetration of the incident beam into the substrate is reduced to an
evanescent wave and any background signal, XRF or scattering, originating from
it is suppressed. On the other side, the reflection of the incident X-ray beam
at the substrate surface leads to the creation of a XSW due to interference
between incident and reflected X-rays. Consequently, enhanced excitation
conditions for XRF originating from the particulate matter deposited on the
top of the substrate can be achieved. A prerequisite to profit from total
external reflection is to use substrates which are flat on a macroscopic scale
and characterized on a microscopic scale by a roughness smaller than the
wavelength of the incident X-rays for best possible reflectivity.

In the field of environmental sciences, this necessity for TXRF measurements
prohibits analyzing foils or membrane filters as it is realized with different
instruments used for the collection of airborne PM. In this case, the sampled
material needs first to be transferred onto an adequate substrate which is
realized usually via different digestion techniques 27, 28, 29, 25 or slurry
techniques 30, 31. However, this approach discards TXRF based analytics of its
main advantages since such a time-consuming preparation step may involve
sample digestion, material loss or contamination issues. A more suitable
approach is to use the TXRF substrates directly, as can be done in cascade
impactors, to collect the particulate matter, either as they are 32, 33, 23 or
after applying a coating (to prevent bounce-off effects for example) and
removing it after the sample collection 34, 28, 22, 35. This additional a
posteriori treatment prior to TXRF investigation 22 is detrimental if other,
complementary analytical techniques shall be used as well.

Regardless of the sample preparation, the knowledge of the XSW is of
importance for quantitative measurements independently if external
standardization, internal standardization or reference-free quantification
schemes are applied. When using internal standardization the XSW created needs
to be identical throughout the sample area illuminated, while in case of
external standardization the same XSW needs to be created in a reproducible
manner for all samples investigated. External standardization means that for
each element of interest in an experimental campaign a calibration curve is
established by means of a set of reference samples with different mass
depositions of the selected elements. This approach requires adequate
reference samples which are sufficiently representative of the samples
investigated 36, 37. For samples collected during outdoor campaigns the
criteria include elemental composition (sample matrix), mass deposition
(concentration), particle size range and morphology as well as deposition
pattern. Hence, the production and selection of adequate calibration samples
for outdoor sampling campaigns requires additional a priori information. The
characterization of actual samples from the measurement campaign via
complementary techniques in order to use these samples as a kind of standards
is often impeded by the sensitivity, i.e., the amount of sample required, of
these techniques.

For this reason approaches based on internal standardization were developed as
an alternative that can be used with digested samples or in conjunction with
substrates prepared for sampling 20, 38. Internal standardization means that
on each sample to be analyzed a known quantity of a reference element is added
beforehand of the measurement while assuming that the excitation and detection
conditions at the position where the standard is deposited is representative
for the whole sample. However, the standard which is added needs to fulfill
the requirements of non-toxicity, not being ubiquitous and having XRF energies
which do not overlap with the XRF lines to be contained within the sample. The
goal of both approaches, external and internal standardization, is to extract
combined information on instrumental factors in order to allow quantifying the
elemental content of the material deposited on the top of the substrates.

In the reference-free XRF quantification scheme 17, information on the
different experimental and fundamental parameters is used to calculate the
mass deposition of different elements from the measured count rate of the
corresponding fluorescence line. This approach requires the use of
(radiometrically) calibrated instrumentation, e.g. apertures for an accurate
knowledge of the solid angle of detection as well as efficiency of diodes and
the silicon drift detector (SDD) for the incident photon flux and the detected
XRF intensity, and the knowledge of atomic fundamental parameters (FPs), i.e.
photoionization cross-sections and fluorescence yields which are element-
dependent and in a large part also energy-dependent.

##  3 Samples

The size-fractionated sampling of PM was realized by means of a 9-stage
extension of the May-type cascade impactor 39. The time-resolved aspect is
subject to the collection time and should be kept within a suitable range of
collected mass for TXRF analysis in order to best profit from the sensitivity
offered by this technique and avoid biasing the quantification as discussed in
this work. The aerodynamic cut-off diameters of the stages 1 to 9 are
respectively 17.9 $\mu$m, 8.9 $\mu$m, 4.5 $\mu$m, 2.25 $\mu$m, 1.13 $\mu$m,
0.57 $\mu$m, 0.29 $\mu$m, 0.18 $\mu$m and 0.07 $\mu$m at a constant flow rate
of 16.7 L$/$min. The cut-off diameter is defined as the dimension of the PM
which is collected with 50$\%$ efficiency, smaller particles escaping with a
higher probability. A well-known and constant airflow is required during the
collection of airborne PM. The first two stages with the largest particle
sizes are disregarded in general for X-ray analysis and for the 7 further
stages $20\times 20$ mm2 Si wafers are used as substrates. As Si wafers have
very low background contamination and very low surface roughness, they are
ideally suited for TXRF and GIXRF experiments. Measurements with good signal-
to-background ratio can be expected even though other substrates might be more
suitable for the collection of PM. Indeed, the collection efficiency depends
not only on the design of the airflow, where losses of particles should be
minimized 40, but also on the substrate surface. In order to preserve the
capabilities offered by TXRF, no pre-treatment of the Si wafers was used.
Sample sets selected for the present study were collected from two campaigns
at two cities, Budapest, Hungary, 24-31 May 2018; and Cassino, Central Italy,
20-27 September 2018, with sampling duration ranging from 20 min to 5 h. In
total 19 Si substrates collected on the 3 stages with the finest particle
distributions were used in this survey. Among these 19 samples 6 samples will
be discussed in more details and further information on these samples is
provided in Table 1. These 6 samples are representative for the range of
deposited particulate mass quantified for all 19 samples. A summary of the
quantification results for all 19 samples for the different excitation regimes
and elements in Fig. S1.

The deposition area from the May-type cascade impactor corresponds to a stripe
of 20 mm length and, depending on the stage, of 0.1 to 1 mm width (fine to
coarser PM). The width is determined by the width of the slits used as nozzles
for the different stages. This type of deposition pattern presents the
advantage of being highly suitable for investigation by means of TXRF and
GIXRF once the stripe is aligned along the incidence direction. Thus, the May-
type cascade impactor is ideal to demonstrate the capability offered by the
combination of cascade impactors and TXRF, respectively GIXRF analysis to
provide element, size- and time-resolved information on the PM collected.

Table 1: Description of the 6 selected Si substrates with aerosol particles collected in Budapest and at Cassino which are discussed in more details in Figs. 2 and 4. Samples are listed in the order of deposited particulate mass and cover the full range of elemental mass depositions quantified on the total of 19 samples investigated. Sample | Duration | Stage | Diameter range / nm
---|---|---|---
A | 30 min | 7 | 300 - 600
B | 20 min | 9 | 70 - 180
C | 20 min | 8 | 180 - 300
D | 1 h | 7 | 300 - 600
E | 5 h | 9 | 70 - 180
F | 5 h | 7 | 300 - 600

##  4 Experimental

The reference-free GIXRF measurements 17, 41, 42 for quantification of
elemental mass depositions were realized at the plane grating monochromator
(PGM) 43 beamline in the PTB laboratory at the BESSY II electron storage ring.
The experiments were conducted at an incident photon energy of 1620 eV which
is below the Si K-edge in order to suppress the contribution of the Si K XRF
lines. An ultrahigh-vacuum chamber equipped with a 9-axis manipulator was used
44. The use of an ultrahigh vacuum may induce loss of volatile material, for
example organic compounds, but this was not further considered in this work
where the focus lies on the evaluation of analytical techniques. The
instrument allows for precisely tuning the incident angle $\theta$ between the
incidence direction of the synchrotron radiation and the sample surface (Fig.
1). The fluorescence radiation emitted from the sample was detected by means
of a silicon drift detector (SDD) calibrated in terms of response function 45
and detection efficiency 46, which is placed in the polarization plane and
perpendicular to the propagation direction of the linearly polarized incident
X-ray beam in order to minimize scattered radiation. The SDD allows for an
energy-dispersive detection of the XRF emitted from the sample such that the
information from different elements can be discriminated and processed in
parallel during quantification. The incident photon flux is determined by
using a calibrated photodiode. The spectra were deconvoluted using the known
detector response functions for the relevant fluorescence lines and background
contributions, which was mainly resonant Raman scattering (RRS) from the Si K
shell 47 and to a lower extent Bremsstrahlung from L shell electrons from the
Si substrate. The resulting count rate $I$ for each fluorescence line of
interest is normalized with respect to the sine of the incident angle
$\theta$, the incident photon flux $I_{0}$, the effective solid angle of
detection $\frac{\Omega(\theta)}{4\pi}$ and the energy dependent detection
efficiency $\epsilon(E)$ of the SDD for the respective fluorescence photons in
order to derive the emitted fluorescence intensity. It has to be emphasized
that the calculation of the incident angle dependent solid angle of detection
requires an accurate knowledge of the detection geometry but also of the
incident beam profile.

Figure 1: Illustration of the experimental setup with the Si wafer and the
collected airborne PM on the top of it, a diode for measuring the reflectivity
and a calibrated SDD for recording the XRF emitted for different incidence
angles $\theta$ (top panel). Typical XRF spectra recorded for an incidence
angle beneath and above the critical angle of total external reflection
illustrate the lower background contributions from the Si wafer at the smaller
incidence angle (bottom panels).

From the absolute XRF intensity the elemental mass deposition $m_{\text{A}}$,
defined as mass per unit area, can be extracted for each position of the GIXRF
measurements where the incident angle $\theta$ was varied in variable steps
from 0∘ to 10∘ (Fig. 2). Hence, the excitation conditions on each sample were
gradually modified from total reflection conditions, where an XSW needs to be
considered, to shallow incidence angle conditions, where no XSW is present.
Nevertheless shallow incidence angles still provide efficient excitation
conditions by dispersing the incident X-ray radiation over a larger sample
area and increasing the incidence path length through the PM collected along
the stripe-like deposition pattern. The calculation of the XSW requires the
knowledge of the optical properties of the substrate, including possible
surface oxidation, for the incident photon energy used during the experiment.
However, even if the incident photon energy dependent optical properties are
measured beforehand from a blank Si substrate to not rely on tabulated data,
the presence of the PM on the top of the Si wafer will impact the reflectivity
to a certain extent as the contrast in electronic density at the interface
separating the bulk Si from the vacuum or PM is changing 48. Therefore, the
reflectivity $R(\theta)$ for each wafer was measured by means of a photodiode
positioned in a $\theta$ \- $2\theta$ configuration during the GIXRF
measurement. This approach is novel and allows for a reliable direct
calculation of the incident angle dependent XSW for each sample under the
actual measurement conditions. Any deviation from optimal alignment of the
sample, increased surface roughness of the substrate or impact of the
collected airborne PM on the XSW will be directly considered in this
experimental approach.

Figure 2: GIXRF data for the 6 selected different samples (labelled A to F and
described in Table 1). The changes in the angular intensity profiles for each
element indicate differences in the excitation of the XRF signal. The typical
particle-like signature of the main elements detected in the GIXRF measurement
gradually vanishes which is a clear indicator that the XSW on the top of the
substrates significantly differs between the samples. It can be noted as well
that for sample A, the angular evolution of O contains both particle- and
layer-like signatures. The latter contribution arises from the surface oxide
of the Si wafers used, but the relative contribution vanishes with increasing
mass of collected airborne PM.

##  5 Reference-free GIXRF Quantification

In a TXRF measurement the XRF intensity is usually recorded at a single
incidence angle corresponding to $\frac{1}{\sqrt{2}}$ ($\approx 70\%$) of the
critical angle for total external reflection $\theta_{c}$, which depends on
the incident photon energy and the substrate density 26 and which was about 1∘
for the Si wafers used. The relative intensity distribution within the XSW is
given by 26,

$XSW(\theta,z)=1+R(\theta)+2\sqrt{R(\theta)}\,\cos\left(\arccos(2\frac{\theta^{2}}{\theta_{c}^{2}}-1)-4\pi\sin\theta\,\frac{z\,E_{0}}{hc}\right)$
(1)

with $E_{0}$ the energy of incident photons, $z$ the height above the
reflecting substrate and $R(\theta)$ the measured reflectivity. Since the
projection of the incident beam has to be below the sample dimension to
discard scattering contributions to the signal, the reflectivity could only be
accurately measured for incidence angles above 0.6∘. For smaller angles a
geometrical correction needs to be considered with the consequence of a larger
error in the calculation and hence the quantification result. As already
noted, the use of experimental reflectivity data allows for more reliable
calculations of the XSW without requiring assumptions in how far the airborne
PM collected will impact the XSW created on the top of the substrate. In Fig.
3 it is shown that the reflectivity at a typical angle used for TXRF
measurements drops significantly with increasing mass deposition of airborne
PM. This observation means that the XSW is significantly different for each
sample and different compared to the case of a blank substrate (Fig. S2).

Figure 3: The reflectivity from the Si substrate for the same samples than
displayed in Figs. 2 and 4 indicates the growing impact on the attenuation of
X-rays within the collected PM, resulting in significant differences in the
$\overline{XSW}(\theta)$ between the different samples. The vertical bar in
the left panel indicates the position typically selected for a TXRF
measurement for a Si substrate and the incident photon energy used. This
position was used for the calculation of $\overline{XSW}(\theta)$ (Fig. S3).

In the following the mean intensity of the XSW, labeled
$\overline{XSW}(\theta)$, over the direction vertical $z$ (Fig. 1) to the
substrate surface is considered. The assumptions of a laterally and vertically
homogeneous chemical composition of the collected PM and of PM dimensions
extending over several periods $\frac{hc}{2\,E_{0}\,\sin\theta}$ of the XSW
are made thereby. The averaging of the XSW by integration is further backed up
by the fact that different particles sizes and compositions are intermixed on
each stage and that the deposition pattern is homogeneous along the direction
of the incident radiation (Fig. S3). A more intricate calculation would
require knowledge on the particle size and relative particle size distribution
49, as well as on the surface coverage 48. Under TXRF conditions, the mass
deposition $m_{\text{A},k}$ of element $k$ for each incidence angle $\theta$
can then be determined from the respective measured XRF count rate 17

$m_{\text{A},k}=\frac{-1}{\mu_{eff}(E_{0},E_{k})}\ln\left({1-\frac{I_{k}(\theta)\sin\theta\,\mu_{eff}(E_{0},E_{k})}{\frac{\Omega}{4\pi}\,I_{0}\,\overline{XSW}(\theta)\,\omega_{k}\,\tau_{k}(E_{0})\,\epsilon(E_{k})}}\right)$
(2)

where $\omega_{k}$ corresponds to the fluorescence yield and $\tau_{k}(E_{0})$
to the photoionization cross-section of the element (index $k$) being
quantified. The values of atomic fundamental parameters can be found in
literature databases 50 or selected parameters are determined in dedicated
experiments as for the fluorescence yield for C 51 or O 52. The factor
$\mu_{eff}(E_{0},E_{k})$ accounts for the effective absorption cross-section
of incident and emitted X-ray photons labelled $\mu_{in}(E_{0})$ and
$\mu_{out}(E_{k})$ respectively, within the PM investigated

$\mu_{eff}(E_{0},E_{k})=\sum_{j}c_{j}\left(\frac{\mu_{in,j}(E_{0})}{\sin\theta}+\frac{\mu_{out,j}(E_{k})}{\sin\frac{\pi}{2}-\theta}\right)$
(3)

and requires hence knowledge on the mass deposition of the different elements
present in order to take correctly into account the relative contributions via
the factor
$c_{k}=\frac{\overline{m_{\text{A},k}}}{\sum_{j}\overline{m_{\text{A},j}}}$
with $\overline{m_{\text{A},k}}$ being the mean quantified mass deposition at
incidence angles above the critical angle for total external reflection (more
precisely from 6∘ to 10∘) where no XSW is present ($\overline{XSW}(\theta)=1$
for $\theta>3\,\theta_{c}$). In this angular regime it can also be shown by
means of a first order Taylor expansion of Eq. 2 that the quantification
result will correspond to the one from a standard XRF quantification as used
in Ref. 53.

A consideration which is usually made at larger incidence angles during the
quantification is the correction for absorption of X-rays on the incidence and
emission paths

$M_{k}(E_{0},E_{k})=\frac{\sum_{j}\overline{m_{\text{A},j}}(\frac{\mu_{in,j}(E_{0})}{\sin\theta}+\frac{\mu_{out,j}(E_{k})}{\sin\frac{\pi}{2}-\theta})}{1-\exp({-\sum_{j}\overline{m_{\text{A},j}}(\frac{\mu_{in,j}(E_{0})}{\sin\theta}+\frac{\mu_{out,j}(E_{k})}{\sin\frac{\pi}{2}-\theta})})}$
(4)

It was found that for incidence angles in the range from 6$\circ$ to 10$\circ$
this factor accounts for at most a few percent only (less than 5$\%$) for most
of the samples. Only for samples with very high mass depositions a relative
correction of 25$\%$ to 30$\%$ was introduced in this iterative correction
scheme. For a most accurate correction factor and quantification a complete
knowledge of the matrix composition is required. Furthermore, secondary
fluorescence due to photoelectrons or fluorescence is neglected. This
introduces only a minor error for low mass depositions but, depending on the
matrix composition and the incident photon energy, should not be disregarded
for high mass depositions where errors of up to 20$\%$-30$\%$ can be
introduced 54.

Finally, the GIXRF measurement allows to compare the quantification results of
the elemental mass deposition $m_{\text{A},k}$ between TXRF conditions and XRF
conditions under shallow incidence angles. The uncertainty made in the
quantification depends on the uncertainties on the incident flux ($1\%$), the
XSW factor ($5\%$ in the angular range where it needs to be taken into
account), the atomic fundamental parameters (fluorescence yield, $10\%$ for
light elements, and photoionization cross-section, $7.5\%$), the detector
efficiency and spectral deconvolution ($2.5\%$), the counting statistics and
the solid angle of detection (about $15\%$ for the smallest incidence angles
to about $4\%$ for the largest incidence angles used for quantification)17.
Thus, the systematic errors, which disregard any sample effects, usually
amount to about $20\%$ in in the TXRF regime and to about $12\%$ or better for
the largest incidence angles used in the GIXRF measurements.

Note, that the mass deposition in terms of mass (or likewise number of atoms)
for each element per unit area is quantified. A conversion to mass, which is a
more commonly used metric in the aerosol community, can be straightforwardly
realized if the area on which the airborne PM is collected and its lateral
distribution are known.

##  6 Results & Discussion

Given the uniform distribution of the PM, quantification results by means of
Eq. 2 can be expected to be constant for each incidence angle covered
throughout a GIXRF measurement. Indeed, the different excitation conditions
realized at each incidence angle should only affect the lowest limit of
detection achievable due to a changing and even vanishing XSW and an
increasing penetration depth into the bulk volume of the substrate but not
impact the quantification result.

However, a consistent quantification result where comparable elemental mass
depositions are quantified for all incidence angles used is not achieved for
all samples. More explicitly, only for a part of the samples present a good
agreement under TXRF conditions, $\theta\approx 0.7^{\circ}$, and under XRF
conditions, $\theta>3.0^{\circ}$ (Fig. 4). For the lowest mass depositions
used, the quantification results are independent of the incidence angle and
agree reasonably well with each other (Fig. 4, upper panels). In this
situation, the quantification of the mass deposition does not depend on the
excitation conditions used such that the GIXRF measurement is useful to
validate the results from the TXRF measurement, which is usually performed at
a single position. This observation is congruent with the proven reliability
of TXRF for quantifying trace level contamination, as it is routinely done for
semiconductor applications. Nonetheless, the GIXRF data allows providing more
robust quantification data as it allows to inherently validate results
obtained by means of TXRF. A specificity can be observed for samples A and B
where an imperfect deconvolution of the XRF spectra recorded at larger
incidence angles affects the quantification results because of the underlying
Si-RRS which is not perfectly described by the model used. In particular for
Al, whose main characteristic line is close to the high energy cut-off of the
Si RRS at 1520 eV, but partially also for Mg this results as well in a larger
scattering of the quantification results at larger incidence angles (Fig. 1).
This issue illustrates perfectly the main benefit of TXRF for low mass
depositions since it allows suppressing background contributions from the
substrate (Fig. 1).

For higher mass loadings, a discrepancy between the quantification results
appears (Fig. 4, lower panels) in the sense that under TXRF conditions mass
depositions are underestimated. For these samples, the quantification results
are not consistent throughout the monitored angular range such that the GIXRF
measurement falsifies the TXRF results. These samples indicate that it is
necessary to be aware of the range of validity of TXRF quantification results
and to validate findings on unknown samples by a GIXRF measurement where the
excitation conditions are varied in a controlled manner by varying the
incidence angle. While the comparison of quantified mass depositions on each
sample allows assessing whether the robustness of the quantification results
obtained under TXRF conditions, GIXRF allows extending the dynamic range of
mass depositions which can be quantified. Indeed, the variation from shallow
to larger incidence angles still provides a considerable variation of the path
length of the incident photons through the airborne PM such that absorption
effects, which may affect the validity of quantification results, are probed
while the premise of an XSW which is not or only to a little extent perturbed
by the presence of airborne PM on the substrate is not given. The influence of
the airborne PM on the XSW is also qualitatively visible in the GIXRF
measurement: for the samples with the lowest mass deposition an enhanced XRF
rate was detected in the vicinity of the critical angle of total external
reflection compared to larger incidence angles, while for the highest mass the
opposite was the case. Hence, in the latter case no XSW was created and by
this way the GIXRF measurement by itself indicates already that quantification
under TXRF conditions is compromised despite the fact that the measured
reflectivity is used in the quantification (Fig. 3).

If it was not taken into account how increasing mass deposition affect the
contrast in optical density at the interface defined by the surface of the
substrate the discrepancy between quantification results for the different
incidence angles used would even be more important (Eq. 2). This insight
emphasizes the benefit of monitoring in parallel to a TXRF or GIXRF
measurement the reflectivity from the sample. The dependence of the XSW on the
surface coverage must be taken into account when quantifying the mass
deposition, a statement which is not only valid when using the reference-free
quantification approach but also when applying external standards. For the
extreme case of the two samples with the highest PM loads (panels E and F in
Figs. 2 and 4) the variation of the quantified mass deposition with the
incident angle indicates that an accurate quantification is tedious since here
the attenuation of the incident radiation within the PM collected would need
to be considered. This aspect introduces considerable uncertainties in the
final result. Hence, a GIXRF measurement allows to discard these types of
samples from further use in analytical campaigns.

Figure 4: Quantified mass deposition for the different incidence angles
covered when varying the excitation conditions during the GIXRF measurement
from the TXRF regime to the XRF regime under shallow incidence angles. The
vertical bar indicates the position typically selected for a TXRF measurement
for a Si substrate and the incident photon energy used, while the horizontal
bar indicate the mass deposition quantified at the largest incidence angles
for each element. For low mass depositions (upper 3 panels) a satisfyingly
good agreement can be observed, but for increasing mass deposition a growing
discrepancy appears for all the elements. This indicates that not all physical
effects due to attenuation of X-rays in the collected airborne PM are
accounted for in the quantification scheme. Under shallow incidence angles
attenuation is less important and has therefore a lesser impact on the
quantification scheme as can be seen from the results approaching a constant
value.

The relationship between the mass deposition and the XSW is also noteworthy
(Eq. 2) with regard to the need for using representative specimen when
applying external standards for quantification purposes. In case of internal
standards, reliable results are only obtained if the collected mass deposition
of the airborne PM is within the range of mass depositions covered by the
standard under the premise that a homogeneous intermixing is realized. In
other words, the dynamic range within which the calibration is valid needs to
be considered. Finally, in the reference-based approaches the XSW needs to be
comparable between the calibration material and investigated sample material,
be it locally when using an internal standard or between samples when using an
external standard, in order to avoid a calibration bias. An upper limit for
reliable TXRF quantification is discussed in literature in terms of critical
thickness 55 and saturation effect 56.

A further reason for the deviation of TXRF quantification results as compared
to the results obtained at the largest incidence angles in the GIXRF scan is
that the full volume of the PM collected is not illuminated homogeneously in
its depth direction because of the attenuation of the incident and reflected
X-ray radiation. For increasing incidence angles and high surface coverage the
effective path length is reduced as $\frac{1}{\sin\theta}$ such that the X-ray
attenuation within the PM volume becomes less pronounced. For samples with a
high surface density of airborne PM, this argument becomes even more crucial
under conditions where an XSW is expected since then the effective path
lengths of the incident and reflected X-rays need to be considered. This
insight impacts directly the reliability of quantification by means of TXRF
and it becomes necessary to indicate an upper limit for the range of validity
of TXRF quantification results. A challenge in air quality monitoring
campaigns is that not all stages from a sampling interval will be affected the
same as important variations between the different stages can be expected due
to inhomogeneous particle size distribution of airborne PM. In case all stages
are affected such a situation is the result of exceedingly long collection
times or high pollution levels.

Depending on the amount of collected airborne PM, the agreement respectively
discrepancy in the quantification results obtained for the different
excitation conditions achieved in a GIXRF measurement becomes even more
obvious when considering the distribution of the quantified mass deposition
for each element of a sample (Fig. 5). The results for the further samples are
included in Figs. S4 and S5. In Fig. 5 the relative range between the lower
and upper 5${}^{\text{th}}$ percentile are indicated. For sample C the results
for each element show a very good consistency with each other, mostly within a
range of several percent only as can also be recognized from the tabulated
values. This agreement can be considered as acceptable for air quality
monitoring campaigns. For the sample with the highest mass depositions (sample
F), the relative difference between the lower and upper 5${}^{\text{th}}$
percentile relative to the mean value amounts to about a factor of 3 to 4
depending on the sample considered. Given that the measurement were not
equidistant in the incidence angles, the results at the lower incidence angles
have a higher impact on the percentile intervals and the discrepancy between
the quantification results can even be larger than indicated. This aspect is
highlighted by the dashed circles in Fig. 5 which indicate the quantification
results obtained at the largest incidence angles which clearly indicate the
importance of the deviation obtained from the quantification performed in the
TXRF regime. This discrepancy makes the need for self-consistent validation by
means of GIXRF measurements evident: the more exhaustive data obtained by
quantifying the mass deposition for different incidence angles and excitation
conditions reveals immediately any discrepancies in the quantification under
TXRF conditions when for the sample considered deviations in the
quantification results are observed. In view of the demands on quantitative
techniques for regulatory purposes this critical assessment of the validity of
the results is mandatory.

Element |  $m_{\text{TXRF}}$ / $\frac{\text{g}}{\text{cm}^{2}}$ |  $\sigma_{\text{TXRF}}$ / $\frac{\text{g}}{\text{cm}^{2}}$ |  $m_{\text{GIXRF}}$ / $\frac{\text{g}}{\text{cm}^{2}}$ |  $\sigma_{\text{GIXRF}}$ / $\frac{\text{g}}{\text{cm}^{2}}$
---|---|---|---|---
C | 0.7 E-5 | 0.5 E-5 | 2.6 E-5 | 1.0 E-5
N | 0.3 E-6 | 0.1 E-6 | 1.0 E-6 | 0.4 E-6
O | 2.4 E-6 | 1.0 E-6 | 8.0 E-6 | 3.2 E-6
Na | 0.9 E-7 | 0.5 E-7 | 3.6 E-7 | 1.9 E-7
Mg | 2.3 E-8 | 1.2 E-9 | 9.2 E-8 | 4.8 E-8
Al | 0.3 E-7 | 0.2 E-7 | 1.1 E-7 | 0.6 E-7
Element |  $m_{\text{TXRF}}$ / $\frac{\text{g}}{\text{cm}^{2}}$ |  $\sigma_{\text{TXRF}}$ / $\frac{\text{g}}{\text{cm}^{2}}$ |  $m_{\text{GIXRF}}$ / $\frac{\text{g}}{\text{cm}^{2}}$ |  $\sigma_{\text{GIXRF}}$ / $\frac{\text{g}}{\text{cm}^{2}}$
---|---|---|---|---
C | 1.9 E-6 | 0.2 E-6 | 2.3 E-6 | 0.2 E-6
N | 1.5 E-6 | 0.1 E-6 | 1.8 E-6 | 0.2 E-6
O | 4.6 E-6 | 0.4 E-6 | 4.5 E-6 | 0.4 E-6
Na | 1.4 E-8 | 0.2 E-8 | 1.7 E-8 | 0.2 E-8
Mg | 1.6 E-9 | 0.2 E-9 | 2.0 E-9 | 0.3 E-9
Al | 0.8 E-8 | 0.1 E-8 | 1.1 E-8 | 0.2 E-8

Figure 5: Histogram of the GIXRF quantification results for samples C and F.
Indicated above the vertical bars is the difference between the $5\%$ and
$95\%$ percentile relative to the $5\%$ value. For sample C an agreement
throughout the full angular range, i.e., with and without XSW, is obtained,
while sample F TXRF underestimates the elemental mass depositions. This
becomes evident by a comparison to the quantification results obtained under
the XRF regime under shallow incidence angles (dashed circles), where no XSW
needs to be considered ($\theta>3\,\theta_{c}$). The tables include the
quantification results under TXRF conditions and XRF conditions using shallow
incidence angles. The $\sigma$ value does not represent the systematic
quantification error, but the standard deviation of the results obtained in
the TXRF regime (incidence angles from 0.6∘ to 0.8∘) and XRF conditions using
shallow incidence angles (incidence angles from 6∘ to 10∘). A ratio of up to a
factor 4 between the different results can be observed.

As a consequence, we propose to expand TXRF based quantitative analysis of
airborne PM to GIXRF. From an experimental point of view this approach allows
combining the benefit of TXRF (low limits of detection) with the ones of GIXRF
(higher dynamic range of mass depositions that can be covered, reliable
quantification for higher mass depositions) since total reflection excitation
conditions are necessarily covered during a GIXRF measurement. By this means,
unknown amounts of airborne PM can optimally be handled since low mass
depositions can be quantified in the angular regime below and high deposition
samples can in the angular regime above the critical angle of total external
reflection. In both cases the GIXRF measurements allow for robust and reliable
quantification with internal validation by assessing whether the
quantification results are in reasonable agreement. It can be noted that for
this purpose a sparser set of incidence angles can be monitored in order to
arrange for more time-efficient measurements as required for example during
field campaigns where high-throughput quantification of airborne PM is aimed
at. From an instrumental point of view this strategy is nowadays a solution
which can be readily implemented, as more and more successful examples for
laboratory GIXRF instruments and even commercially available GIXRF instruments
exist. In case only TXRF measurements can be performed, the consequence is
that only a restricted range of samples or mass depositions can be
investigated. Indeed, for reliable quantification under TXRF conditions the
attenuation of X-rays within the collected airborne PM and the differences in
the XSW created should remain below a threshold value. This criteria cannot be
specified generically since it depends on the incident photon energy and the
absolute elemental composition of the PM.

Furthermore, simple control cross-check measurements can be used to identify
possible issues with the quantification performed, regardless if standards are
used or a reference-free quantification scheme is being applied. One indicator
is the reflectivity from the substrate as compared to a blank substrate of the
same type. This cross-check can even be applied when samples are investigated
under TXRF conditions solely (Fig. 4). An other possibility is to consider the
angular intensity profile of the XRF originating from the bulk volume of the
substrate and to compare it against the one of a blank substrate (Fig. S6).
For increasing mass deposition of airborne PM and larger incidence angles
significant attenuation compared to a blank Si substrate can be observed. In
both cases the comparison to a blank substrate allows elucidating whether,
besides possible sample alignment issues, quantification by TXRF alone is
compromised due to the amount of airborne PM collected exceeding the range
within which TXRF can be validly applied.

##  7 Conclusion & Outlook

It was shown that GIXRF by means of a controlled variation of the excitation
conditions allows for applying a robust quantification scheme and, hence, for
assessing in a self-consistent manner the validity of quantification under
TXRF conditions over a wide range of mass depositions. Self-consistent means
that throughout the different excitation conditions covered during a GIXRF
measurement the quantification results are in agreement with each other for
each incidence angle in case of low mass depositions and for incidence angles
far above the critical angle of total reflection for higher mass depositions.
Varying pollution levels and particle size repartition make such verification
of the validity of results necessary since reliable quantification in the TXRF
regime can only be realized for mass depositions on the level of lowest amount
of matter where a linear calibration between count rate and mass deposition
can be guaranteed. This range depends on the matrix composition, the airborne
PM size and the incident photon energy, but cannot be assessed from TXRF
measurements alone. While increasing mass depositions result in nonlinear
effects in terms of different XSW created on the top of the substrates and
pronounced attenuation of the incident and reflected X-rays within the
airborne PM collected, such that quantification by means of TXRF alone is
compromised. Therefore criteria or thresholds for reliable quantification by
means of TXRF need to be established in the future. This pitfall cannot be
readily circumvented, let alone be identified by means of calibration samples
or procedures since, both, the range of elemental mass deposition and the
matrix composition on the substrates are unknown beforehand of any measurement
in the field. As a consequence a robust quantification scheme requires GIXRF
measurements, i.e., measurements covering two fundamentally different
excitation regimes. Only then the quantified elemental mass depositions can be
considered as reliable and validated, respectively the range of validity for
quantification results obtained by means of TXRF can be assessed for a
selected class of samples and experimental conditions. Indeed, it has to be
noted that the presented experiment with its emphasis on light elements was
realized in the soft X-ray regime but the conclusions made can also be applied
for higher X-ray energies.

While advanced instrumentation was applied for allowing for a physically
traceable quantification which does not rely on the use of standards, it can
be emphasized that this approach is transferable to laboratory instrumentation
57. Thus, possibilities are offered to transfer the approach presented to
instrumentation used for high-throughput measurements in the field or in the
laboratory. The additional implementation of a diode to measure the
reflectivity would not only allow for more accurate determination of the XSW
but would also provide a rough but straightforward indication whether the
quantity of airborne PM collected presents an issue for the quantification by
means of TXRF.

## Acknowledgments

Parts of this research was performed within the EMPIR project 19ENV08 AEROMET
II. This project has received funding from the EMPIR programme co-financed by
the Participating States and from the European Union’s Horizon 2020 research
and innovation programme. The support of the European Structural and
Investment Funds jointly financed by the European Commission and the Hungarian
Government through grant no. VEKOP-2.3.2-16-2016-00011 (on behalf of J. Osán)
is also appreciated.

## References

  * Apte et al. 2015 Apte, J. S.; Marshall, J. D.; Cohen, A. J.; Brauer, M. _Environ. Sci. Technol._ 2015, _49_ , 8057–8066
  * Lelieveld et al. 2015 Lelieveld, J.; Evans, J. S.; Fnais, M.; Giannadaki, D.; Pozzer, A. _Nature_ 2015, _525_ , 367–371
  * Leni et al. 2020 Leni, Z.; Cassagnes, L. E.; Daellenbach, K. R.; El Haddad, I.; Vlachou, A.; Uzu, G.; Prévôt, A. S. H.; Jaffrezo, J.-L.; Baumlin, N.; Salathe, M.; Baltensperger, U.; Dommen, J.; Geiser, M. _PLOS ONE_ 2020, _15_ , 1–17
  * Mellouki et al. 2015 Mellouki, A.; Wallington, T. J.; Chen, J. _Chem. Rev._ 2015, _115_ , 3984–4014
  * McNeill 2015 McNeill, V. F. _Environ. Sci. Technol._ 2015, _49_ , 1237–1244
  * Zhang et al. 2015 Zhang, R.; Wang, G.; Guo, S.; Zamora, M. L.; Ying, Q.; Lin, Y.; Wang, W.; Hu, M.; Wang, Y. _Chem. Rev._ 2015, _115_ , 3803 – 3855
  * Prather et al. 2008 Prather, K. A.; Hatch, C. D.; Grassian, V. H. _Annu. Rev. Anal. Chem._ 2008, _1_ , 485–514
  * Møller et al. 2008 Møller, P.; Folkmann, J. K.; Forchhammer, L.; Bräuner, E. V.; Danielsen, P. H.; Risom, L.; Loft, S. _Cancer Lett._ 2008, _266_ , 84 – 97
  * Stone et al. 2007 Stone, V.; Johnston, H.; Clift, M. J. D. _IEEE Transactions on NanoBioscience_ 2007, _6_ , 331–340
  * Valavanidis et al. 2008 Valavanidis, A.; Fiotakis, K.; Vlachogianni, T. _J. Environ. Sci. Health Pt. C_ 2008, _26_ , 339–362
  * Brook et al. 2010 Brook, R. D.; Rajagopalan, S.; Pope, C. A.; Brook, J. R.; Bhatnagar, A.; Diez-Roux, A. V.; Holguin, F.; Hong, Y.; Luepker, R. V.; Mittleman, M. A.; Peters, A.; Siscovick, D.; Smith, S. C.; Whitsel, L.; Kaufman, J. D. _Circulation_ 2010, _121_ , 2331–2378
  * Rönkkö and Timonen 2019 Rönkkö, T.; Timonen, H. _J Alzheimers Dis._ 2019, _72_ , 15–28
  * Daellenbach et al. 2020 Daellenbach, K. R. et al. _Nature_ 2020, _587_ , 414–419
  * Bulska and Ruszczyńska 2017 Bulska, E.; Ruszczyńska, A. _Phys. Sci. Rev._ 2017, _2_ , 20178002
  * Ault and Axson 2017 Ault, A. P.; Axson, J. L. _Anal. Chem._ 2017, _89_ , 430–452, PMID: 28105816
  * Furger et al. 2017 Furger, M.; Minguillón, M. C.; Yadav, V.; Slowik, J. G.; Hüglin, C.; Fröhlich, R.; Petterson, K.; Baltensperger, U.; Prévôt, A. S. H. _Atmos. Meas. Tech._ 2017, _10_ , 2061–2076
  * Beckhoff et al. 2007 Beckhoff, B.; Fliegauf, R.; Kolbe, M.; Müller, M.; Weser, J.; Ulm, G. _Anal. Chem._ 2007, _79_ , 7873–7882
  * Streli et al. 2008 Streli, C.; Wobrauschek, P.; Meirer, F.; Pepponi, G. _J. Anal. At. Spectrom._ 2008, _23_ , 792–798
  * Bukowiecki et al. 2008 Bukowiecki, N.; Lienemann, P.; Zwicky, C. N.; Furger, M.; Richard, A.; Falkenberg, G.; Rickers, K.; Grolimund, D.; Borca, C.; Hill, M.; Gehrig, R.; Baltensperger, U. _Spectrochim. Acta B_ 2008, _63_ , 929 – 938
  * Richard et al. 2010 Richard, A.; Bukowiecki, N.; Lienemann, P.; Furger, M.; Fierz, M.; Minguillón, M. C.; Weideli, B.; Figi, R.; Flechsig, U.; Appel, K.; Prévôt, A. S. H.; Baltensperger, U. _Atmos. Meas. Tech._ 2010, _3_ , 1473–1485
  * Bontempi et al. 2010 Bontempi, E.; Zacco, A.; Benedetti, D.; Borgese, L.; Colombi, P.; Stosnach, H.; Finzi, G.; Apostoli, P.; Buttini, P.; Depero, L. _Environ. Technol._ 2010, _31_ , 467–477
  * Prost et al. 2017 Prost, J.; Wobrauschek, P.; Streli, C. _X-Ray Spectrom._ 2017, _46_ , 454–460
  * Osán et al. 2020 Osán, J.; Börcsök, E.; Czömpöly, O.; Dian, C.; Groma, V.; Stabile, L.; Török, S. _Spectrochim. Acta B_ 2020, _167_ , 105852
  * Borgese et al. 2020 Borgese, L.; Bilo, F.; Zacco, A.; Federici, S.; Mutahi, A. W.; Bontempi, E.; Trzepla, K.; Hyslop, N.; Yatkin, S.; Wobrauschek, P.; Prost, J.; Ingerle, D.; Depero, L. E. _Spectrochim. Acta B_ 2020, _167_ , 105840
  * Fomba et al. 2020 Fomba, K. W.; Deabji, N.; Barcha, S. E. I.; Ouchen, I.; Elbaramoussi, E. M.; El Moursli, R. C.; Harnafi, M.; El Hajjaji, S.; Mellouki, A.; Herrmann, H. _Atmos. Meas. Tech._ 2020, _13_ , 4773–4790
  * Klockenkämper and von Bohlen 2014 Klockenkämper, R.; von Bohlen, A. _Total Reflection X-ray Fluorescence Analysis and Related Methods_ ; John Wiley & Sons, Ltd, 2014
  * Leland et al. 1987 Leland, D. J.; Bilbrey, D. B.; Leyden, D. E.; Wobrauschek, P.; Aiginger, H.; Puxbaum, H. _Analytical Chemistry_ 1987, _59_ , 1911–1914
  * Schmeling and Klockow 1997 Schmeling, M.; Klockow, D. _Anal. Chim. Acta_ 1997, _346_ , 121 – 126
  * Wagner and Mages 2010 Wagner, A.; Mages, M. _Spectrochim. Acta B_ 2010, _65_ , 471 – 477
  * Natali et al. 2016 Natali, M.; Zanella, A.; Rankovic, A.; Banas, D.; Cantaluppi, C.; Abbadie, L.; Lata, J. C. _Environ. Sci. Pollut. Res._ 2016, _23_ , 23496 – 23510
  * Bilo et al. 2017 Bilo, F.; Borgese, L.; Dalipi, R.; Zacco, A.; Federici, S.; Masperi, M.; Leonesio, P.; Bontempi, E.; Depero, L. E. _Chemosphere_ 2017, _178_ , 504 – 512
  * Schneider 1989 Schneider, B. _Spectrochim. Acta By_ 1989, _44_ , 519 – 523
  * Esaka et al. 2003 Esaka, F.; Watanabe, K.; Onodera, T.; Taguchi, T.; Magara, M.; Usuda, S. _Spectrochim. Acta B_ 2003, _58_ , 2145 – 2155
  * Injuk and Van Grieken 1995 Injuk, J.; Van Grieken, R. _Spectrochim. Acta B_ 1995, _50_ , 1787 – 1803
  * Prost et al. 2018 Prost, J.; Zinkl, A.; Ingerle, D.; Wobrauschek, P.; Streli, C. _Spectrochim. Acta B_ 2018, _147_ , 13 – 20
  * Hönicke et al. 2018 Hönicke, P.; Krämer, M.; Lühl, L.; Andrianov, K.; Beckhoff, B.; Dietsch, R.; Holz, T.; Kanngießer, B.; Weißbach, D.; Wilhein, T. _Spectrochim. Acta B_ 2018, _145_ , 36 – 42
  * Horntrich et al. 2012 Horntrich, C.; Kregsamer, P.; Prost, J.; Stadlbauer, F.; Wobrauschek, P.; Streli, C. _Spectrochim. Acta B_ 2012, _77_ , 31 – 34
  * Böttger et al. 2018 Böttger, S.; Tyssebotn, I. M. B.; Jansen, W.; Fittschen, U. E. _Spectrochim. Acta B_ 2018, _147_ , 93 – 99
  * May 1975 May, K. _J. Aerosol Sci._ 1975, _6_ , 413 – 419
  * Marple 2004 Marple, V. A. _Aerosol Sci. Technol._ 2004, _38_ , 247–292
  * Müller et al. 2014 Müller, M.; Hönicke, P.; Detlefs, B.; Fleischmann, C. _Materials_ 2014, _7_ , 3147–3159
  * Hönicke et al. 2019 Hönicke, P.; Detlefs, B.; Nolot, E.; Kayser, Y.; Mühle, U.; Pollakowski, B.; Beckhoff, B. _Journal of Vacuum Science & Technology A_ 2019, _37_ , 041502
  * Senf et al. 1998 Senf, F.; Flechsig, U.; Eggenstein, F.; Gudat, W.; Klein, R.; Rabus, H.; Ulm, G. _Journal of Synchrotron Radiation_ 1998, _5_ , 780–782
  * Lubeck et al. 2013 Lubeck, J.; Beckhoff, B.; Fliegauf, R.; Holfelder, I.; Hönicke, P.; Müller, M.; Pollakowski, B.; Reinhardt, F.; Weser, J. _Rev. Sci. Instrum._ 2013, _84_ , 045106
  * Scholze and Procop 2009 Scholze, F.; Procop, M. _X-Ray Spectrom._ 2009, _38_ , 312–321
  * Scholze and Procop 2005 Scholze, F.; Procop, M. _X-Ray Spectrom._ 2005, _34_ , 473–476
  * Müller et al. 2006 Müller, M.; Beckhoff, B.; Ulm, G.; Kanngießer, B. _Phys. Rev. A_ 2006, _74_ , 012702
  * Unterumsberger et al. 2020 Unterumsberger, R.; Hönicke, P.; Kayser, Y.; Pollakowski-Herrmann, B.; Gholhaki, S.; Guo, Q.; Palmer, R. E.; Beckhoff, B. _J. Anal. At. Spectrom._ 2020, _35_ , 1022–1033
  * Kayser et al. 2015 Kayser, Y.; Sá, J.; Szlachetko, J. _Nanoscale_ 2015, _7_ , 9320–9330
  * Schoonjans et al. 2011 Schoonjans, T.; Brunetti, A.; Golosio, B.; Sanchez del Rio, M.; Solé, V. A.; Ferrero, C.; Vincze, L. _Spectrochim. Acta B_ 2011, _66_ , 776 – 784
  * Beckhoff and Ulm 2001 Beckhoff, B.; Ulm, G. _Adv. X-Ray Anal._ 2001, _44_ , 349–354
  * Hönicke et al. 2016 Hönicke, P.; Kolbe, M.; Krumrey, M.; Unterumsberger, R.; Beckhoff, B. _Spectrochim. Acta B_ 2016, _124_ , 94 – 98
  * Beckhoff 2008 Beckhoff, B. _J. Anal. At. Spectrom._ 2008, _23_ , 845–853
  * Wählisch et al. 2020 Wählisch, A.; Streeck, C.; Hönicke, P.; Beckhoff, B. _J. Anal. At. Spectrom._ 2020, _35_ , 1664–1670
  * Klockenkämper and von Bohlen 1989 Klockenkämper, R.; von Bohlen, A. _Spectrochim. Acta B_ 1989, _44_ , 461 – 469
  * Hellin et al. 2004 Hellin, D.; Fyen, W.; Rip, J.; Delande, T.; Mertens, P. W.; De Gendt, S.; Vinckier, C. _J. Anal. At. Spectrom._ 2004, _19_ , 1517–1523
  * Hönicke et al. 2020 Hönicke, P.; Waldschläger, U.; Wiesner, T.; Krämer, M.; Beckhoff, B. _Spectrochim. Acta B_ 2020, _174_ , 106009


# Are pre-trained text representations useful for multilingual and multi-
dimensional language proficiency modeling?

Taraka Rama
University of North Texas, USA
<EMAIL_ADDRESS>
Sowmya Vajjala
National Research Council, Canada
<EMAIL_ADDRESS>

###### Abstract

Development of language proficiency models for non-native learners has been an
active area of interest in NLP research for the past few years. Although
language proficiency is multidimensional in nature, existing research
typically considers a single “overall proficiency” while building models.
Further, existing approaches also considers only one language at a time. This
paper describes our experiments and observations about the role of pre-trained
and fine-tuned multilingual embeddings in performing multi-dimensional,
multilingual language proficiency classification. We report experiments with
three languages – German, Italian, and Czech – and model seven dimensions of
proficiency ranging from vocabulary control to sociolinguistic
appropriateness. Our results indicate that while fine-tuned embeddings are
useful for multilingual proficiency modeling, none of the features achieve
consistently best performance for all dimensions of language proficiency111All
code, data and related supplementary material can be found at:
https://github.com/nishkalavallabhi/MultidimCEFRScoring.

##  1 Introduction

Automated Essay Scoring (AES) is the task of grading test taker writing using
computer programs. It has been an active area of research in NLP for the past
15 years. Although most of the existing research focused on English, recent
years saw the development of AES models for second language proficiency
assessment for non-English languages, typically modeled using the Common
European Framework of Reference (CEFR) reference scale of Europe (2002) in
Europe.

Most of the past research focused on monolingual AES models. However, the
notion of language proficiency is not limited to any one language. As a matter
of fact, CEFR of Europe (2002) provides language agnostic guidelines to
describe different levels of language proficiency, from A1 (beginner) to C2
(advanced). Hence, a universal, multilingual language proficiency model is an
interesting possibility to explore. From an application perspective, it will
be useful to know if one can achieve cross-lingual transfer and build an AES
system for a new language without or with little training data. Vajjala and
Rama (2018) explored these ideas with basic features such as n-grams and POS
tag ratios. The usefulness of large, pre-trained multilingual models (with or
without fine-tuning) from recent NLP research has not been studied for this
task, especially for non-English languages.

Further, AES research generally considers language proficiency as a single
construct. However, proficiency encompasses multiple dimensions such as
vocabulary richness, grammatical accuracy, coherence/cohesion, usage of idioms
etc. Attali and Burstein (2004). CEFR guidelines also provide language
proficiency rubrics for individual dimensions along with overall proficiency
for A1–C2. Modeling multiple dimensions instead of a single “overall
proficiency” could result in a more fine-grained assessment for offering
specific feedback.

Given this background, we explore the usefulness of multilingual pre-trained
embeddings for training multi-dimensional language proficiency scoring models
for three languages – German, Czech and Italian. The main contributions of our
paper are listed below:

  * •

We address the problem of multi-dimensional modeling of language proficiency
for three (non-English) languages.

  * •

We explore whether large pre-trained, multilingual embeddings are useful as
feature representations for this task with and without fine-tuning.

  * •

We investigate the possibility of a universal multilingual language
proficiency model and zero-shot cross lingual transfer using embedding
representations.

The paper is organized as follows. Section 2 briefly surveys the related work.
Section 3 describes our corpus, features, and experimental settings. Section 4
discusses our results in detail. Section 5 concludes the paper with pointers
to future work.

##  2 Related Work

Automated Essay Scoring (AES) is a well-researched problem in NLP and has been
applied to real-world language assessment scenarios for English Attali and
Burstein (2004). A wide range of features such as document length,
lexical/syntactic n-grams, and features capturing linguistic aspects such as
vocabulary, syntax and discourse are commonly used Klebanov and Flor (2013);
Phandi et al. (2015); Zesch et al. (2015). In the recent past, different forms
of text embeddings and pre-trained language models have also been explored
Alikaniotis et al. (2016); Dong and Zhang (2016); Mayfield and Black (2020)
along with approaches to combine linguistic features with neural networks Shin
(2018); Liu et al. (2019). Ke and Ng (2019) and Klebanov and Madnani (2020)
present the most recent surveys on the state of the art in AES (focusing on
English).

In terms of modeling, AES has been modeled as a classification, regression,
and ranking problem, with approaches ranging from linear regression to deep
learning models. Some of the recent work explored the usefulness of multi-task
learning Cummins and Rei (2018); Berggren et al. (2019) and transfer learning
Jin et al. (2018); Ballier et al. (2020). Going beyond approaches that work
for a single language, Vajjala and Rama (2018) reported on developing methods
for multi- and cross-lingual AES.

Much of the existing AES research has been focused on English, but there is a
growing body of research on other European languages: German Hancke and
Meurers (2013), Estonian Vajjala and Lõo (2014), Swedish Pilán et al. (2016),
Norwegian Berggren et al. (2019) which explored both language specific (e.g.,
case markers in Estonian) as well as language agnostic (e.g., POS n-grams)
features Vajjala and Rama (2018) for this task. However, to our knowledge, the
use of large pre-trained language models such as BERT Devlin et al. (2018) has
not been explored yet for AES in non-English languages.

Further, most of the approaches focused on modeling language proficiency as a
single variable. Although there is some research focusing on multiple
dimensions of language proficiency Lee et al. (2009); Attali and Sinharay
(2015); Agejev and Šnajder (2017); Mathias and Bhattacharyya (2020), none of
them focused on non-English languages or used recent multilingual pre-trained
models such as BERT. In this paper, we focus on this problem of multi-
dimensional modeling of language proficiency for three languages—German,
Italian, and Czech—and explore whether recent research on multilingual
embeddings can be useful for non-English AES.

##  3 Datasets and Methods

In this section, we describe the corpus, features, models, and implementation
details. We modeled the task as a classification problem and trained
individual models for each of the 7 dimensions of language proficiency. The
rest of this section describes the different steps involved in our approach in
detail.

###  3.1 Corpus

In this paper, we employed the publicly available MERLIN corpus Boyd et al.
(2014), which was also used in the experiments reported in some past research
Hancke (2013); Vajjala and Rama (2018) and in the recently conducted REPROLANG
challenge Branco et al. (2020). The MERLIN corpus222Available here for
download: https://merlin-platform.eu/C_download.php contains CEFR scale based
language proficiency annotations for texts produced by non-native learners in
three languages: German, Czech, and Italian in seven dimensions which are
described below:

  1. 1.

Overall proficiency is the generic label expected to summarize the language
proficiency across different dimensions.

  2. 2.

Grammatical accuracy refers to the usage and control over the language’s
grammar.

  3. 3.

Orthographic control refers to the aspects of language connected with writing
such as punctuation, spelling mistakes etc.

  4. 4.

Vocabulary range refers to the breadth of vocabulary use, including phrases,
idiomatic expressions, colloquialisms etc.

  5. 5.

Vocabulary control refers to the correct and appropriate use of vocabulary.

  6. 6.

Coherence and Cohesion refers to the ability to connect different parts of the
text using appropriate vocabulary (e.g., connecting words) and creating a
smoothly flowing text.

  7. 7.

Sociolinguistic appropriateness refers to the awareness of language use in
different social contexts. For example, using proper form of introduction,
ability to express both in formal as well as informal language, understanding
the sociocultural aspects of language use etc.

Detailed description of a dimension at each CEFR level is provided in the
structured overview of CEFR scales document of Europe (2002). In the MERLIN
corpus, these annotations were prepared by human graders who were trained on
these well defined rubrics. More details on the examination setup, grade
annotation guidelines, rating procedure, inter-rater reliability and
reliability of rating measures can be found in the project documentation
Bärenfänger (2013). We used the texts and their universal dependency parsed
versions – shared by Vajjala and Rama (2018) – consisting of 2266 documents in
total (1029 German, 803 Italian, 434 for Czech).

The German corpus had A1–C1, Italian corpus had A2–B1, and Czech had A2–B2
levels for the overall proficiency category. More CEFR levels were represented
in the corpus for other proficiency dimensions.333More details on the corpus
distribution can be found in MERLIN documentation, and the result files we
shared as supplementary material contain CEFR level distributions for all the
classification scenarios, for all languages. In this paper, we treat the
annotated labels in the corpus as the gold standard labels.

#### Missing labels

Less than 10 documents had an annotation of $1$ instead of the CEFR scale
(A1–C2) for some of the dimensions. The documentation did not provide any
reason behind this label assignment and we removed them from our experiments.
In the case of German and Italian, for less than ten documents, some
individual dimensions had a score of “0” while the overall rating was A1. For
these documents, we treated “0” score as A1 rating for that dimension. In the
case of Czech, about half of the documents ($247$) for the sociolinguistic
appropriateness dimension had a score of “0”. The corpus manual does not
provide any explanation for the missing annotation, therefore, we excluded
this dimension from all experiments involving Czech data.

#### Inter-dimensional correlations

Bärenfänger (2013)’s analyses on MERLIN corpus show that correlations among
the different dimensions (including overall proficiency) range from $0.2-0.8$
in different languages.444For details: Refer to Table 4 for Czech, Table 11
for German, Table 17 for Italian in Bärenfänger (2013). In general,
correlations between any two dimensions and specifically with overall
proficiency dimension are higher for German and Italian than for Czech. There
is no consistent high correlation of overall proficiency with any single
dimension. The variations show that these individual dimensions as are indeed
different from each other as well as overall proficiency dimension, and we
could expect that a model trained on one dimension need not necessarily
reflect on the language proficiency of the test taker in another dimension.
This further motivates our decision to explore a multi dimensional proficiency
assessment approach in this paper.

###  3.2 Features

One of the goals of the paper is to examine if text representations computed
from large, pre-trained, multilingual models such as mBERT Devlin et al.
(2018) and LASER Artetxe and Schwenk (2019) are useful for the AES task. We
trained classifiers based on these two pre-trained models and compare them
with two previously used features—document length baseline and the n-gram
features used in Vajjala and Rama (2018). All the features are described
below:

  * •

Baseline: Document length (number of tokens) is a standard feature in all AES
approaches Attali and Burstein (2004).

  * •

Lexical and syntactic features: n-grams ($1\leq n\leq 5$) of Word, Universal
POS tags (UPOS) from the Universal Dependencies project Nivre et al. (2016),
dependency triplets consisting of the head POS label, dependent POS label, and
the dependency label extracted using UDPipe Straka et al. (2016). While word
n-grams are useful only for monolingual setting, the syntactic level n-grams
were used in multi/cross-lingual scenarios as well, as they are all derived
from the same tagset/dependency relations.

  * •

LASER embeddings map a sentence in a source language to a fixed dimension
($1024$) vector in a common cross-lingual space allowing us to map the vectors
from different languages into a single space. Since the number of sentences in
an essay is variable, we map each sentence in the segmented text to a vector
and then compute the average of the vectors to yield a $1024$ dimension
representation as our feature vector.

  * •

mBERT: We apply the 12-layer pre-trained multilingual BERT (trained on
Wikipedias of 104 languages with shared word-piece vocabulary) for mapping an
essay (truncated to $400$ tokens which is the upper bound of the length for
93% of the documents) into a $768$ dimension vector. Specifically, we use the
vector for the CLS token from the final layer as the feature vector for non-
finetuned classification experiments. We used the MERLIN corpus texts to do
task specific fine-tuning of mBERT.

It is possible to use other representations such as using average of the
tokens’ embeddings of the last layer instead of using CLS token for mBERT, or
explore other recent pre-trained mono-/multilingual representations. Our goal
is not to explore the best representation but rather test if a representative
approach could be used for this problem. To our knowledge, only Mayfield and
Black (2020) studied the application of BERT for AES in English, and its
utility in the context of non-English and multilingual AES models has not been
explored.

Although, it is possible to use the “domain” features such as spelling/grammar
errors which are commonly seen in AES systems, our goal in this paper is to
explore how far we can go with the representations without any language
specific resources for this task. Considering that such representations are
expected to capture different aspects of language Jawahar et al. (2019);
Edmiston (2020), we could hypothesize that some of the domain specific
features are already captured by them.

###  3.3 Models and Evaluation

As discussed in Section 1, our motivation in this paper is to evaluate whether
pre-trained multilingual embedding representations are useful for performing
multidimensional AES, whether they can be used to achieve a universal
representation for this task (multilingual), as well as transfer from one
language to another (cross-lingual) and if the pre-trained embedding
representations can be transferred to the AES task (fine-tuning). To explore
this, we trained mono/multi/cross lingual classification models using each of
the features described in Section 3.2, for each of the 7 dimensions.

All the traditional classification models based on n-grams, LASER and mBERT
were tested using traditional classification algorithms: Logistic Regression,
Random Forests, and Linear SVM. The mBERT fine-tuned model consists of a
softmax classification layer on top of the CLS token’s embedding. We used the
MERLIN corpus texts to fine-tune mBERT for this task in all the three
classification scenarios.

We evaluate the classifiers in monolingual and multilingual scenarios through
stratified five-fold validation where the distribution of the labels is
preserved across the folds. Owing to the nature of the corpus and the presence
of unbalanced classes in all the languages and dimensions in the dataset, we
used weighted F1 score to compare model performance, as was done in the
REPROLANG challenge Branco et al. (2020). In the cross-lingual scenario, we
trained on the German dataset and tested separately on Czech and Italian
languages.

###  3.4 Implementation

All the POS and dependency n-gram features were computed using UDPipe Straka
et al. (2016). All the traditional classification models were implemented
using the Python library scikit-learn Pedregosa et al. (2011) with the default
settings. LASER embeddings were extracted using the python package
laserembeddings.555https://pypi.org/project/laserembeddings/ The extraction of
mBERT embeddings and fine-tuning was performed using the Hugging Face library
and
PyTorch.666https://huggingface.co/transformers/v2.2.0/model_doc/bert.html#bertforsequenceclassification
The code, processed dataset, and detailed result files are uploaded as
supplementary material with this paper.

##  4 Experiments and Results

Figure 1: German monolingual five-fold validation results. All POS and Dep
n-grams are based on Universal Dependencies framework. Figure 2: Italian
monolingual five-fold validation results Figure 3: Czech monolingual five-fold
validation results

As mentioned earlier, we trained monolingual, multilingual and cross-lingual
classification models for all the seven proficiency dimensions. We report
results with logistic regression, which performed the best in most of the
cases. The results for the other classifiers such as Random Forest and Linear
SVM are provided in the supplementary material.

###  4.1 Monolingual Classification

Figures 1, 2 and 3 show the results of monolingual classification for German,
Italian and Czech respectively for all the feature sets and proficiency
dimensions.

#### German

The fine-tuned BERT model performs the best (from Figure 1) for the Overall
CEFR proficiency prediction dimension closely followed by POS n-grams. Except
for the Vocabulary Range dimension, none of the other dimensions seem to
perform on par with Overall proficiency in terms of absolute numbers, though.
Fine-tuned mBERT performs the best for Orthographic control dimension, where
the rest of the feature sets performed rather worse. Overall, these results
seem to indicate that all our features only capture the ‘Overall proficiency’
dimension well, and to some extent the ‘Vocabulary Range’ dimension. All
features perform rather poorly at the task of prediction of orthographic
control.

#### Italian

The word n-grams perform the best for Overall Proficiency closely followed by
POS n-grams and fine-tuned mBERT model. There is not much variation among the
features, with little improvement over the strong document length baseline for
any feature group. Further, the performance on other dimensions seems far
worse than Overall Proficiency, compared to German. Orthographic control is
the worst performing dimension even for Italian. Word n-grams are the best
feature representation across all dimensions for Italian. Although mBERT fine-
tuning improved the performance over non fine-tuned version, both LASER and
mBERT based models don’t perform better than word or POS n-grams in any
dimension. Thus, while there are some similarities between German and Italian
classification, we also observe some differences.

#### Czech

Across all the dimensions, the results (Figure 3) for Czech are different from
German and Italian. The performance of the different systems on
Coherence/Cohesion dimension is much better than the Overall Proficiency.
Orthographic Control, which seemed to be the worst modeled dimension for
German and Italian, does better than grammatical accuracy and vocabulary
control. There is a larger difference between the baseline performance and the
best performance for most of the dimensions, than it was for German and
Italian.

The main conclusions from the monolingual classification experiments are as
follows:

  * •

The feature groups don’t capture multiple dimensions of proficiency well and
there is no single feature group that works equally well across all languages.

  * •

Pre-trained and fine-tuned text representations seem to perform comparably to
traditional n-gram features in several language-dimension combinations.

One possible reason for the variation across dimensions could be that the
corpus consists of texts written by language learners, coming from various
native language backgrounds. It is possible that there are no consistent
n-gram patterns in various dimensions to capture due to this characteristic.
Further, models such as LASER and mBERT are pre-trained on well formed texts,
and may not be able to capture the potentially erroroneous language patterns
in MERLIN texts. We can hypothesize that the overall proficiency label
potentially captures some percent of each dimension, and is probably easier to
model than others. However, even this hypothesis does not hold for the case of
Czech, where Coherence/Cohesion dimension perhaps much better than the overall
proficiency. Clearly, more analysis and experiments are needed to understand
these aspects. The current set of experiments indicate that it is a worthwhile
future direction to pursue.

###  4.2 Multilingual Classification

In multilingual classification, we work with a single dataset formed by
combining the essays from all the three languages. We trained and tested
classifiers for all combinations of feature sets and dimensions on the single
large dataset. Since CEFR guidelines for language proficiency are not specific
to any one language, we would expect multilingual models to perform on par
with individual monolingual models. The results of our multilingual
experiments are given in Figure 4.

Figure 4: Multi-dimensional, Multilingual language proficiency classification.
The doclen baseline for Orthography domain is $0.3956$ which is less than the
minimum threshold of $0.4$.

Our results show that the fine-tuned mBERT model performs the best on most of
the dimensions, closely followed by the UPOS n-grams features. To understand
the relation between the multilingual model and its constituents, we looked at
how each language fared in this model. For overall proficiency dimension, for
example, the best result is achieved with fine-tuned classifier based on mBERT
($0.745$), which is closer to the average of the results from the three
monolingual models. While German ($0.693$ in monolingual vs $0.683$ in
multilingual) and Italian ($0.829$ vs $0.826$) saw a slight dip in the
multilingual setup, Czech ($0.669$ vs $0.718$) saw a 5 point increase due to
multilingual classification.

Clearly, multilingual classification is a beneficial setup for languages with
lower monolingual performance or less data, without compromising on those
languages that had better performance. However, there is still a lot of
performance variation in terms of absolute numbers across dimensions. As with
the case of monolingual models, we can potentially attribute this to the fact
that we are dealing with a relatively smaller sized dataset in MERLIN, with
texts written by people with diverse native language backgrounds, although
more experiments are needed in this direction to confirm this.

###  4.3 Crosslingual Classification

Here, we train different classification models on German, and test them on
Italian and Czech. We chose German for the training side since German has the
largest number of essays in MERLIN corpus. In the case of mBERT, we performed
fine-tuning on German part of the corpus, and tested the models on Italian and
Czech texts respectively. The goal of this experiment is to test if there are
any universal patterns in proficiency across the languages and understand if
zero-shot cross-lingual transfer is possible for this task.

UPOS n-grams consistently performed better than other features for most of the
dimensions, in both the cross lingual setups. There is more performance
variation among the different dimensions for Italian compared to Czech. In the
case of Czech, similar to the monolingual case, the Coherence/cohesion
dimension achieved superior performance than others, even with the baseline
document length feature. This is a result worth considering further
qualitative analysis in future. More details on the results of this experiment
can be found in the Figures folder in the supplementary material. Our cross-
lingual experiments seem to indicate that the embedding representations we
chose are not useful for zero-shot learning, and that UPOS n-grams may serve
as a strong baseline for building AES systems with new languages.

###  4.4 Error Analysis

We observed substantial performance differences across
features/dimensions/languages in various experimental settings. While we don’t
have a procedure to understand the exact reasons for this yet, examining the
confusion matrices (provided in the supplementary material) may give us some
insights into the nature of some of these differences. Therefore, we manually
inspected a few confusion matrices, posing ourselves three questions:

  1. 1.

How does a given feature set perform across different dimensions for a given
language?

  2. 2.

How do different features perform for a single dimension for a given language?

  3. 3.

How does a given feature set perform for a given dimension among the three
languages?

In all these cases, we did not notice any major differences, and the confusion
matrices followed the expected trend (observed in previous research) –
immediately proximal levels such as A2/B1 or A1/A2 are harder to distinguish
accurately as compared to distant levels such as A1/B2 along with the expected
observation that levels with larger representation have better results. It is
neither possible to cover all possible combinations nor is it sufficient to
gain more insights into the models just by looking at confusion matrices
alone. Carefully planned interpretable analyses should be conducted in future
to understand these differences further.

##  5 Summary and Conclusions

In this paper, we reported several experiments exploring multi-dimensional
CEFR scale based language proficiency classification for three languages. Our
main conclusions from these experiments can be summarized as follows:

  1. 1.

UPOS n-gram features perform consistently well for all languages in
monolingual classification scenarios for modeling “overall proficiency”,
closely followed by embedding features in most language-dimension
combinations.

  2. 2.

Fine-tuned large pre-trained models such as mBERT are useful language
representations for multilingual classification, and languages with low
monolingual performance benefit from a multilingual setup.

  3. 3.

UPOS features seem to provide a strong baseline for zero-shot cross lingual
transfer, and fine-tuning was very not useful in this case.

  4. 4.

None of the feature groups consistently perform well across all
dimensions/languages/classification setups.

The first conclusion is similar to Mayfield and Black (2020)’s conclusion on
using BERT for English AES. However, these results need not be interpreted as
a “no” to pre-trained models. Considering that they are closely behind n-grams
in many cases and were slightly better than them for German, we believe they
are useful to this task and more research needs to be done in this direction
exploring other language models/fine-tuning options.

Pre-trained and fine-tuned models are clearly useful in a multilingual
classification setup, and it would be an interesting new direction to pursue
for this task. As a continuation of these experiments, one can look for a
larger CEFR annotated corpus for a language such as English, and explore
multilingual learning for languages with lesser data.

The results from the experiments presented in this paper highlight the
inherent difficulty in capturing multiple dimensions of language proficiency
through existing methods, and the need for more future research in this
direction. An important direction for future work is to develop better feature
representations that capture specific dimensions of language proficiency,
which can potentially work for many languages. Considering that all the
dimensions share some commonalities and differences with each other, multi-
task learning is another useful direction to explore.

## References

  * Agejev and Šnajder (2017) Tamara Sladoljev Agejev and Jan Šnajder. 2017.  Using analytic scoring rubrics in the automatic assessment of college-level summary writing tasks in l2.  In _Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_ , pages 181–186.
  * Alikaniotis et al. (2016) Dimitrios Alikaniotis, Helen Yannakoudakis, and Marek Rei. 2016.  Automatic text scoring using neural networks.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 715–725, Berlin, Germany. Association for Computational Linguistics.
  * Artetxe and Schwenk (2019) Mikel Artetxe and Holger Schwenk. 2019.  Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond.  _Transactions of the Association for Computational Linguistics_ , 7:597–610.
  * Attali and Burstein (2004) Yigal Attali and Jill Burstein. 2004.  Automated essay scoring with e-rater® v. 2.0.  _ETS Research Report Series_ , 2004(2).
  * Attali and Sinharay (2015) Yigal Attali and Sandip Sinharay. 2015.  Automated trait scores for toefl® writing tasks.  _ETS Research Report Series_ , 2015(1):1–14.
  * Ballier et al. (2020) Nicolas Ballier, Stéphane Canu, Caroline Petitjean, Gilles Gasso, Carlos Balhana, Theodora Alexopoulou, and Thomas Gaillat. 2020.  Machine learning for learner english: A plea for creating learner data challenges.  _International Journal of Learner Corpus Research_ , 6(1):72–103.
  * Bärenfänger (2013) Olaf Bärenfänger. 2013.  Assessing the reliability and scale functionality of the merlin written speech sample ratings.  Technical report, European Academy, Bolzano, Italy.
  * Berggren et al. (2019) Stig Johan Berggren, Taraka Rama, and Lilja Øvrelid. 2019.  Regression or classification? automated essay scoring for norwegian.  In _Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications_ , pages 92–102.
  * Boyd et al. (2014) Adriane Boyd, Jirka Hana, Lionel Nicolas, Detmar Meurers, Katrin Wisniewski, Andrea Abel, Karin Schöne, Barbora Stindlová, and Chiara Vettori. 2014\.  The merlin corpus: Learner language and the cefr.  In _LREC_ , pages 1281–1288.
  * Branco et al. (2020) António Branco, Nicoletta Calzolari, Piek Vossen, Gertjan van Noord, Dieter van Uytvanck, João Silva, Luís Gomes, André Moreira, and Willem Elbers. 2020.  A shared task of a new, collaborative type to foster reproducibility: A first exercise in the area of language science and technology with reprolang2020.  In _Proceedings of The 12th Language Resources and Evaluation Conference_ , pages 5539–5545.
  * Cummins and Rei (2018) Ronan Cummins and Marek Rei. 2018.  Neural multi-task learning in automated assessment.  _arXiv preprint arXiv:1801.06830_.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_.
  * Dong and Zhang (2016) Fei Dong and Yue Zhang. 2016.  Automatic features for essay scoring–an empirical study.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 1072–1077.
  * Edmiston (2020) Daniel Edmiston. 2020.  A systematic analysis of morphological content in bert models for multiple languages.  _arXiv preprint arXiv:2004.03032_.
  * of Europe (2002) Council of Europe. 2002.  Common european framework of reference for languages: Learning, teaching, assessment. structured overview of all cefr scales.
  * Hancke (2013) Julia Hancke. 2013.  Automatic prediction of cefr proficiency levels based on linguistic features of learner language.  _Master’s thesis, University of Tübingen_.
  * Hancke and Meurers (2013) Julia Hancke and Detmar Meurers. 2013.  Exploring cefr classification for german based on rich linguistic modeling.  pages 54–56.
  * Jawahar et al. (2019) Ganesh Jawahar, Benoît Sagot, and Djamé Seddah. 2019.  What does bert learn about the structure of language?  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 3651–3657.
  * Jin et al. (2018) Cancan Jin, Ben He, Kai Hui, and Le Sun. 2018.  Tdnn: a two-stage deep neural network for prompt-independent automated essay scoring.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1088–1097.
  * Ke and Ng (2019) Zixuan Ke and Vincent Ng. 2019.  Automated essay scoring: a survey of the state of the art.  In _Proceedings of the 28th International Joint Conference on Artificial Intelligence_ , pages 6300–6308. AAAI Press.
  * Klebanov and Flor (2013) Beata Beigman Klebanov and Michael Flor. 2013.  Word association profiles and their use for automated scoring of essays.  In _Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , volume 1, pages 1148–1158.
  * Klebanov and Madnani (2020) Beata Beigman Klebanov and Nitin Madnani. 2020.  Automated evaluation of writing – 50 years and counting.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7796–7810, Online. Association for Computational Linguistics.
  * Lee et al. (2009) Yong-Won Lee, Claudia Gentile, and Robert Kantor. 2009.  Toward automated multi-trait scoring of essays: Investigating links among holistic, analytic, and text feature scores.  _Applied Linguistics_ , 31(3):391–417.
  * Liu et al. (2019) Jiawei Liu, Yang Xu, and Lingzhe Zhao. 2019.  Automated essay scoring based on two-stage learning.  _CoRR_ , abs/1901.07744.
  * Mathias and Bhattacharyya (2020) Sandeep Mathias and Pushpak Bhattacharyya. 2020.  Can neural networks automatically score essay traits? In _Proceedings of the Fifteenth Workshop on Innovative Use of NLP for Building Educational Applications_ , pages 85–91, Seattle, WA, USA → Online. Association for Computational Linguistics.
  * Mayfield and Black (2020) Elijah Mayfield and Alan W Black. 2020.  Should you fine-tune BERT for automated essay scoring? In _Proceedings of the Fifteenth Workshop on Innovative Use of NLP for Building Educational Applications_ , pages 151–162, Seattle, WA, USA → Online. Association for Computational Linguistics.
  * Nivre et al. (2016) Joakim Nivre, Marie-Catherine de Marneffe, Filip Ginter, Yoav Goldberg, Jan Hajic, Christopher D Manning, Ryan T McDonald, Slav Petrov, Sampo Pyysalo, Natalia Silveira, Reut Tsarfaty, and Daniel Zeman. 2016.  Universal Dependencies v1: A Multilingual Treebank Collection.  In _Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016)_ , pages 1659–1666.
  * Pedregosa et al. (2011) F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011.  Scikit-learn: Machine learning in Python.  _Journal of Machine Learning Research_ , 12:2825–2830.
  * Phandi et al. (2015) Peter Phandi, Kian Ming A Chai, and Hwee Tou Ng. 2015.  Flexible domain adaptation for automated essay scoring using correlated linear regression.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_ , pages 431–439.
  * Pilán et al. (2016) Ildikó Pilán, David Alfter, and Elena Volodina. 2016.  Coursebook texts as a helping hand for classifying linguistic complexity in language learners’ writings.  _CL4LC 2016_ , page 120.
  * Shin (2018) Eunjin Shin. 2018.  A neural network approach to automated essay scoring: A comparison with the method of integrating deep language features using coh-metrix.
  * Straka et al. (2016) Milan Straka, Jan Hajic, and Jana Straková. 2016.  UDPipe: Trainable Pipeline for Processing CoNLL-U Files Performing Tokenization, Morphological Analysis, POS Tagging and Parsing.  In _LREC_.
  * Vajjala and Lõo (2014) Sowmya Vajjala and Kaidi Lõo. 2014.  Automatic cefr level prediction for estonian learner text.  In _Proceedings of the third workshop on NLP for computer-assisted language learning at SLTC 2014, Uppsala University_ , 107. Linköping University Electronic Press.
  * Vajjala and Rama (2018) Sowmya Vajjala and Taraka Rama. 2018.  Experiments with universal cefr classification.  In _Proceedings of the Thirteenth Workshop on Innovative Use of NLP for Building Educational Applications_ , pages 147–153, New Orleans, Louisiana. Association for Computational Linguistics.
  * Zesch et al. (2015) Torsten Zesch, Michael Wojatzki, and Dirk Scholten-Akoun. 2015.  Task-independent features for automated essay grading.  In _BEA@ NAACL-HLT_ , pages 224–232.

##  Appendix A Supplemental Material

The code and data for these experiments can be found at:
https://github.com/nishkalavallabhi/MultidimCEFRScoring