# Gaps in Topological Magnon Spectra: Intrinsic vs. Extrinsic Effects

Seung-Hwan Do  Materials Science and Technology Division, Oak Ridge National
Laboratory, Oak Ridge, Tennessee 37831, USA  Joseph A. M. Paddison  Materials
Science and Technology Division, Oak Ridge National Laboratory, Oak Ridge,
Tennessee 37831, USA  Gabriele Sala  Second Target Station, Oak Ridge National
Laboratory, Oak Ridge, Tennessee 37831, USA  Travis J. Williams  Neutron
Scattering Division, Oak Ridge National Laboratory, Oak Ridge, Tennessee
37831, USA  Koji Kaneko  Materials Sciences Research Center, Japan Atomic
Energy Agency, Tokai, Ibaraki 319-1195, Japan  Advanced Science Research
Center, Japan Atomic Energy Agency, Tokai, Ibaraki 319-1195, Japan  Keitaro
Kuwahara  Institute of Quantum Beam Science, Ibaraki University, Mito, Ibaraki
310-8512, Japan  A. F. May  Materials Science and Technology Division, Oak
Ridge National Laboratory, Oak Ridge, Tennessee 37831, USA  Jiaqiang Yan
Materials Science and Technology Division, Oak Ridge National Laboratory, Oak
Ridge, Tennessee 37831, USA  Michael A. McGuire  Materials Science and
Technology Division, Oak Ridge National Laboratory, Oak Ridge, Tennessee
37831, USA  Matthew B. Stone  Neutron Scattering Division, Oak Ridge National
Laboratory, Oak Ridge, Tennessee 37831, USA  Mark D. Lumsden  Neutron
Scattering Division, Oak Ridge National Laboratory, Oak Ridge, Tennessee
37831, USA  Andrew D. Christianson  Materials Science and Technology Division,
Oak Ridge National Laboratory, Oak Ridge, Tennessee 37831, USA

###### Abstract

For topological magnon spectra, determining and explaining the presence of a
gap at a magnon crossing point is a key to characterize the topological
properties of the system. An inelastic neutron scattering study of a single
crystal is a powerful experimental technique that is widely employed to probe
the magnetic excitation spectra of topological materials. Here, we show that
when the scattering intensity rapidly disperses in the vicinity of a crossing
point, such as a Dirac point, the apparent topological gap size is extremely
sensitive to experimental conditions including sample mosaic, resolution, and
momentum integration range. We demonstrate these effects using comprehensive
neutron-scattering measurements of CrCl3. Our measurements confirm the gapless
nature of the Dirac magnon in CrCl3, but also reveal an artificial, i.e.
extrinsic, magnon gap unless the momentum integration range is carefully
controlled. Our study provides an explanation of the apparent discrepancies
between spectroscopic and first-principles estimates of Dirac magnon gap
sizes, and provides guidelines for accurate spectroscopic measurement of
topological magnon gaps.

††preprint: APS/123-QED

This manuscript has been authored by UT-Battelle, LLC under Contract No. DE-
AC05-00OR22725 with the U.S. Department of Energy. The United States
Government retains and the publisher, by accepting the article for
publication, acknowledges that the United States Government retains a non-
exclusive, paid-up, irrevocable, world-wide license to publish or reproduce
the published form of this manuscript, or allow others to do so, for United
States Government purposes. The Department of Energy will provide public
access to these results of federally sponsored research in accordance with the
DOE Public Access Plan (http://energy.gov/downloads/doe-public-access-plan).

Magnons in a honeycomb-lattice ferromagnet have an analogous description to
the single-orbital tight-binding model for electrons in graphene. As a result,
the magnons in a honeycomb ferromagnet exhibit a linear energy-momentum
dispersion near a magnon crossing point, often termed a Dirac magnon [Fig. 1]
[1, 2, 3]. Observing the Dirac magnon is of intense current interest as a
means of assessing potential topological properties. In particular, if spin-
orbit coupling (SOC) accompanying a broken inversion symmetry generates a
nonzero Dzyalloshinskii–Moriya interaction (DMI), a gap opens at the Dirac
crossing point, in close analogy with the spin-orbit-induced semiconducting
gap in graphene. This can yield a topologically-protected magnon edge-state
with low-dissipation, which results in a non-trivial topological magnon
insulator [2, 3, 4, 5]. Consequently, experimentally identifying materials
with a topological magnon gap (TMG) is a crucial goal, with wide-ranging
implications for spin-transport-based technologies such as magnon spintronics
[6, 7, 8].

Figure 1:  Demonstration of how data processing parameters (Q-range of
integration) and sample mosaicity lead to the appearance of artificial gaps in
a gapless Dirac magnon spectrum. (a) Gapless Dirac magnon dispersion for spins
$S$ on a honeycomb lattice with ferromagnetic nearest-neighbor interaction
$J_{1}$. (b) The Q-E contour map is obtained by integrating over data within
the finite orthogonal Q-range, 2$\delta$ shown in (a). Due to rapid dispersion
away from the crossing point, an artificial gap ($\Delta_{\text{K}}$) appears
in the spectrum with an apparent size ($\Delta_{\text{K}}$, defined as the
distance between the two peak centers) that depends on the Q-integration
range. (c) Sample mosaicity generates a broad distribution of off-centered
Dirac cones in Q, and the superposed magnon spectra produces an artificial gap
at the K-point. (d) Calculated Dirac magnon spectra for a 10∘ sample mosaicity
with a $\delta$=0.01 r.l.u.. Data within the yellow bar are integrated to
obtain the energy scan at the K-point (far right panels).  Figure 2:  (a) INS
spectra of CrCl3 measured at $T=5$ K ($<T_{\text{N}}$) along high-symmetry
directions as indicated in the $HK$-reciprocal space map shown in the inset.
Data were obtained by integrating over a thickness of $Q=$0.024 Å-1 (0.02
r.l.u. along $[H,0,0]$) in the $HK$-plane and $-2.5\leq[0,0,L]\leq 2.5$ along
the out-of-plane direction. (b) Corresponding spin-wave calculations convolved
with the energy resolution function of the instrument. Constant energy slices
for (c) $E=6.0\pm 0.15$ meV and (d) $E=2.5\pm 0.15$ meV are compared with the
corresponding calculations.

Several chromium-based (Cr3+, $S$=3/2) honeycomb ferromagnetic materials have
been proposed to host gapped Dirac magnons, namely CrBr3, CrI3, and Cr$M$Te3
($M$=Si, Ge) [9, 10, 11, 12]. In all of these materials, the key experimental
evidence for a TMG has been obtained from single-crystal neutron spectroscopy
experiments, which report gaps $\sim 3$ to $5$ meV. However, the origin of the
gap opening remains unclear, with proposals including DMI, Kitaev
interactions, and magnon-phonon coupling [9, 13, 10, 14, 15, 5]. Furthermore,
the estimated gaps based upon the DMI and Kitaev interaction strengths are
much bigger than the predicted gap size of first-principle calculations for
the small SOC of half-filled $t_{2g}$ orbitals of Cr3+ [13, 16]. In this
context, accurate neutron spectroscopy experiments can be crucial, because
they provide a measurement of the magnon dispersion in $4$-dimensional
momentum-energy (Q-$E$) space that can, in principle, be quantitatively
compared with calculations. In practice, however, the weakness of the neutron-
sample interaction often requires compromises to increase the signal strength,
such as the use of many co-aligned single crystals with increased mosaicity,
the relaxation of Q and E resolution to gain more neutron flux, and the
integration of measured data over a significant range of Q and E to improve
counting statistics of the measurement. These approaches to enhance signal can
result in spurious findings when the neutron intensity rapidly disperses or
has a singularity in Q-E space as is the case for a Dirac magnon, and this may
in turn cause a significant impact on the observation and estimates of the
TMG.

In this letter, we present a detailed analysis of extrinsic effects on
spectroscopic estimates of TMG sizes. Our key finding is that a dominant
_extrinsic_ contribution to the apparent magnon gap occurs if “typical”
Q-integration ranges are used, which can cause the intrinsic TMG to be
substantially overestimated or misdiagnosed. We identify this effect using a
simple model. We then present high-resolution neutron spectroscopy
measurements on a single crystal of CrCl3. Our results establish CrCl3 as an
ideal quasi-2D Heisenberg ferromagnet with a gapless Dirac magnon, and provide
a comprehensive map of the excitation spectrum near the Dirac point. However,
we also find that an extrinsic magnon gap appears unless the Q-integration
range is carefully controlled. Finally, we reinterpret published spectroscopic
studies of CrBr3 [11] and CrSiTe3 [12], and show that the apparent magnon gaps
in these materials likely contain substantial extrinsic contributions. Our
results resolve apparent discrepancies between spectroscopic and first-
principles estimates of TMG sizes, and provide guidelines for accurate
spectroscopic measurement of topological magnon gaps.

Figure 1 shows calculated magnon spectra for a honeycomb-lattice Heisenberg
ferromagnet with nearest-neighbor (n.n.) interaction $J_{1}$. This model
preserves effective time-reversal magnon symmetry and exhibits a gapless Dirac
magnon. As shown in Fig. 1(a), two cone-shaped dispersions touch at a single
location in three-dimensional Q-E space. The linear dispersion near the Dirac
point is revealed by taking a slice along the $[-K,K,0]$ direction [Fig.
1(b)]. In practice, it is necessary to integrate the intensity over a finite
Q-range (denoted $2\delta$) orthogonal to the slice direction. The gapless
Dirac magnon is faithfully reproduced only in the limit $\delta\rightarrow 0$
[Fig. 1(a)]. In contrast, taking $\delta=0.03$ r.l.u. yields an apparent gap
of size $\sim$ $0.5SJ_{1}$, which is $8$% of the full bandwidth of $6SJ_{1}$
[Fig. 1(b)]. Here, our choice of $\delta$ is informed by previous experimental
studies, in which “typical” values of $\delta$ are between $0.03$ and $0.20$
r.l.u. [17, 11, 9, 10]. This extrinsic gap occurs because the intensity is
averaged over the Dirac cone away from the Dirac point, and is substantial
even for small $\delta$ because the intensity varies rapidly with Q. This
Q-integration effect is also distinct from the effect of sample mosaicity such
as considered for recent study of CrI3[10], which generates a superposition of
off-centered Dirac cones [Fig. 1(c)], and causes an additional artificial gap
at the Dirac point [Fig. 1(d)].

We investigate extrinsic contributions to the magnon gap using spectroscopic
measurements of the van-der-Waals ferromagnet CrCl3, which contains
undistorted Cr3+ honeycomb layers below 150 K (space group $R\bar{3}$) [18,
19]. CrCl3 has a ferromagnetic spin alignment in the plane in each honeycomb
layer below $T_{\text{N}}=14$ K [18]. Compared to CrBr3 and CrI3, the
relatively light Cl ligand atom of CrCl3 is likely to give a small SOC and
hence anisotropic exchange interactions should also be correspondingly weak
and host a gapless Dirac magnon, as suggested by recent neutron-scattering
measurements on polycrystalline CrCl3 samples [20, 21]. Furthermore, unlike
other Cr-halides such as CrI3, CrCl3 can be grown in as a large single crystal
with a small sample mosaic. Consequently, CrCl3 is an ideal model system to
investigate the impact of extrinsic effects and data treatment on the spin-
wave spectrum near the Dirac point.

Figure 3:  (a) Q-$E$ slices along [$H$, $H$, 0]-directions at K$=-0.05$, $0$,
$0.05$, $0.1$, and $0.15$ for $[-K,K,0]$. Data were obtained by integrating
over $-0.01\leq K\leq 0.01$ and $-2.5\leq L\leq 2.5$. The dashed lines connect
ends of acoustic and optical magnon bonds varying with $[-K,K,0]$, which shows
a crossing of two magnon bands at a Dirac point,
$\mathrm{K}=(\frac{1}{3},\frac{1}{3},0)$. (b) Constant energy slices of the
INS spectra. The dashed lines indicate energy evolution of acoustic and
optical magnons along the transverse Q-direction near the Dirac point. (c-d)
Opening an artificial gap at crossing point, $(\frac{2}{3},-\frac{1}{3},0)$,
with increasing integration range, d${Q}^{\text{rad}}$. The spectra were
collected using high resolution mode for $E_{i}=11$ meV with the SEQUOIA
spectrometer, and obtained by integrating over (c) d${Q}^{\text{rad}}=\pm
0.01$ r.l.u., (d) d${Q}^{\text{rad}}=\pm 0.03$ r.l.u., and (e)
d${Q}^{\text{rad}}=\pm 0.06$ r.l.u. ($0.02$, $0.06$, $0.13$Å-1, respectively).
The energy-resolution convolved spin-wave spectra are compared in the second
and third rows, and are calculated for the identical Q-integration range as
the experimental data. The constant Q-scans at the Dirac point were compared
by integrating over d${Q}^{\text{trans}}=\pm 0.02$ r.l.u. (a yellow bar in the
contour map). The blue bar is the instrumental resolution at $E^{Dirac}=4.4$
meV. The peak positions and full-width-at-half maximum are quantified by
Gaussian fits and exhibited as points with error-bars, respectively, as a
function of d${Q}^{\text{rad}}$ in (f). The distance between the positions
indicate the apparent gap size.

A CrCl3 single crystal with a mosaic $<0.68^{\circ}$ was grown by re-
crystallization of commercial CrCl3 powder [22]. The INS data was collected
with the SEQUOIA time-of-flight spectrometer [23] at the Spallation Neutron
Source located at Oak Ridge National Laboratory. The Fermi choppers were
phased for the high-resolution mode of SEQUOIA for incident energies
$E_{\text{i}}=4$, $11$, and $25$ meV [24]. Figure 2(a) shows the measured
magnon spectra at $5$ K ($T<T_{\text{N}}$). The spectra are composed of one
acoustic and one optical magnon branch with an overall magnon bandwidth of $8$
meV ($W_{HK}$) in the $HK$-plane. The acoustic magnon emanates from the
$\Gamma$ point $(0,0,0)$, and meets the optical magnon at the K-points,
exhibiting typical spin-waves for a ferromagnetic honeycomb lattice [2].

To model the observed magnon spectra, we used linear spin-wave theory (LSWT)
using the SpinW package [25]. The spin Hamiltonian was modeled with ${\cal
H}=J_{n}\sum_{i,j}S_{i}S_{j}-D^{z}\sum_{i}(S_{i}^{z})^{2}$, for $S=3/2$ of
Cr3+. We consider Heisenberg ($J_{n}$) exchanges up to the third n.n.
interactions in the honeycomb plane and an easy-plane single-ion anisotropy
($D^{z}>0$). We exclude bond-dependent exchange interactions (Kitaev and
$\Gamma$ terms) because these would yield different mode energies at different
K-points coupled with the aligned spin direction [26], in disagreement with
our experimental data. We also note that the symmetry-allowed second n.n. DMI
term cannot generate a magnon gap at the K-point for the in-plane magnetic
structure with the symmetry operations of exchange matrix for $R\bar{3}$
structure [17]. The inter-layer coupling was assumed to be negligible due to
the highly two-dimensional spin-wave dispersion (see [24] for a sensitivity of
interlayer coupling in spin-wave analysis). The calculated magnon dispersions
from the model Hamiltonian were fitted to the experimental dispersion points
extracted by fitting a Gaussian function to the INS spectra. The best fit was
for $J_{1}=-0.934(5)$ meV, $J_{2}=-0.0302(2)$ meV, $J_{3}=0.0488(1)$ meV, and
$D^{z}=0.0100(1)$ meV [24]. The dominant n.n. ferromagnetic interaction
$J_{1}$ is responsible for the formation of the ferromagnetic honeycomb spin-
lattice in CrCl3. The single-ion anisotropy was found to be nearly zero,
indicating a nearly isotropic Cr3+ spin. Fig. 2(b-d) show that the calculated
spectra reproduce the measured spectra with high fidelity, demonstrating a
quasi-two-dimensional Heisenberg ferromagnetic honeycomb spin-lattice for
CrCl3. The determined Hamiltonian preserves time-reversal symmetry of magnons,
resulting in a gapless Dirac magnon at the K-point.

Details of the magnon spectra near the Dirac point are shown in Fig. 3. In the
figure, the magnon spectra are displayed along the radial Q-direction
(parallel to $[H,H,0]$) with varying transverse Q-component ($[-K,K,0]$). The
separate acoustic and optical modes intersect at the single Q-position of the
Dirac point $(\frac{1}{3},\frac{1}{3},0)$. This sharp band touching is viewed
as a nodal point at $E^{Dirac}=4.4$ meV in the horizontal Q-$E$ slices of the
spectra [Fig. 3(b)], representing the shape of a Dirac cone dispersion.
Deviating from $E^{\text{Dirac}}$, the Dirac point evolves into triangular
scattering patterns around K [Fig.2(c,d)] with modulating intensity across the
Dirac point in energy. The variation in intensity is associated with the
isospin locked with offset momentum winding around the Dirac point [26, 27].
As the result, the two conical dispersions having anti-phased winding patterns
of intensity meet at the K-point [refer to Fig 1(a)]. When Q is perpendicular
to the radial-direction, the two bands have an identical magnon structure
factor, and the Dirac magnon reveals a clear band crossing, as shown in Fig.
3(c), evidencing a gapless Dirac magnon in CrCl3.

As described above, a focal issue in the search for topological properties of
the magnon spectrum is identifying the presence of an intrinsic gap (TMG) at
the K-point. Here we investigate the impact of Q-integration range on the
spectrum near the Dirac point in CrCl3. Fig. 3(c)-(e) show the results of the
differently histogrammed spectra in the vicinity of the K-point while varying
the orthogonal momentum-integration range, d${Q}^{\text{rad}}$. (see Fig. 3(b)
for d${Q}^{\text{rad}}$). For d${Q}^{\text{rad}}=\pm 0.01$, the experimental
spectrum shows a gapless feature [Fig.3(c)]. However, a slightly wider
integration range with d${Q}^{\text{rad}}=\pm 0.03$ opens an apparent gap with
$\Delta_{\text{K}}\sim 0.7$ meV ($\sim 0.09$ $W_{HK}$) [Fig.3(d)] and the
apparent gap size increases with $\Delta_{\text{K}}\sim 1.3$ meV ($0.16$
$W_{HK}$) for d${Q}^{\text{rad}}=\pm 0.06$ [Fig.3(e)] (see Fig. 3(f) for
d${Q}^{\text{rad}}$ dependence of the gap size). The measured spectra are
directly compared to the spin-wave calculation from the obtained Hamiltonian
parameters, reflecting the resolution function of the SEQUOIA spectrometer for
the identical momentum integration ranges. The good agreement between
calculations and experiment demonstrates that the apparent gap is fully
explained by the choice of Q-integration range. This artificial gap
consistently appears in the radial Q scan for d${Q}^{\text{trans}}$ range,
reflecting the conical shape of dispersion. It is worth noting that this
apparent gap exhibits a finite spectral intensity within the gap, where the
instrumental resolution is smaller than the apparent gap size, which is in
contrast to the TMG case showing zero intensity between the peak splitting
[24]. Therefore, to clarify TMG, careful comparison of the measured data to
the energy-resolution convolved spin-wave calculation including momentum
integration range should be performed.

Figure 4:  Calculations of INS spectra for topological magnon candidates,
considering extrinsic effects. (a) Calculated spectra for time-of-flight
spectrometer experiment for CrBr3 [11]. Spin-waves were calculated for only
the Heisenberg Hamiltonian parameters considering the integration range
d${Q}^{\text{trans}}=0.2$ r.l.u. ($0.23$Å-1). (b) Calculated energy scans at
K1 and K2. (c) Calculated spectra for a triple-axis spectrometer experiment
for CrSiTe3 [12]. The spin-wave spectrum was calculated for only the
Heisenberg parameters, and convolved with the instrumental energy resolution
function for the experimental conditions presented in Ref. [12]. The simulated
resolution ellipse functions at the K-points at $E^{\text{Dirac}}=10$ meV are
indicated with gray ellipse. The K-points are divided into ‘focus’ and
‘defocus’ regions according to the coupling status of the dispersion and
resolution ellipse, and the energy scans simulated at
K${}_{1}^{\text{focus}}$, K${}_{1}^{\text{defocus}}$, and
K${}_{2}^{\text{defocus}}$ are compared in (d).

The discussion above has important implications for the ongoing debate
regarding the observations of the Dirac magnon gap in other chromium van der
Waals honeycomb ferromagnets. The previously used momentum integration range
and a large sample mosaic for CrBr3 (d${Q}^{\text{trans}}=\pm 0.2$) and CrI3
(sample mosaic$=8\sim 17^{\circ}$), respectively, are likely to cause a large
extrinsic gap contribution [11, 9, 10], thus the TMGs are possibly
overestimated. Figure 4(a) shows the calculated INS spectrum of the spin-wave
spectra for CrBr3 [11], including the Q-integration range and instrumental
resolution used in Ref. [11]. This calculation assumes only Heisenberg
interactions ($J_{1}=-1.48$ meV, $J_{2}=-0.08$ meV, $J_{3}=0.11$, and
$D^{z}=-0.02$ meV, from Ref. [11]) with zero DMI. For simplicity, the sample
mosaic was assumed to be zero. The resulting spectrum shows the upper and
lower magnon spectra having decreased intensities near the Dirac points (K1,
K2), which reproduces the observed magnon dispersion in Ref. [11]. Noticeably,
most of the observed gap, corresponding to the peak splittings $\sim 4$ meV,
in the energy scans at K1 and K2 is explained by the orthogonal momentum
integration range effect on a pure Heisenberg model, without DMI. As a
consequence, the size of the gap at the Dirac point is likely overestimated.

In contrast to time-of-flight spectroscopy measurement on Cr trihalides,
topological magnon candidtates Cr$M$Te3 ($M$=Si,Ge) were measured using
triple-axis spectrometers [12]. For triple-axis measurements, integration
ranges are not a concern but resolution effects can be significant,
particularly when horizontal focusing is used. Therefore, the observed spectra
are strongly coupled with the shape of the resolution function (gray ellipse
in Fig. 4(c)). Figure 4(c) exhibits newly calculated spin-wave spectra for
CrSiTe3 [12], including the resolution calculation for the experiment using
Reslib [28]. For the spin-wave calculation, we used only Heisenberg spin
Hamiltonian parameters ($J_{1}=-1.49$ meV, $J_{2}=-0.15$ meV, $J_{c1}=-0.07$,
$J_{c2}=-0.06$, and $D^{z}=-0.01$ meV, from Ref. [12]) with zero DMI. As can
be seen, while the gapless Dirac dispersion parallel to the the ellipse
(focused) at K${}_{1}^{\text{focus}}$ leads to a single peak (gapless), the
Dirac dispersions anti-parallel to the ellipse (defocused) at
K${}_{1}^{\text{defocus}}$ and K${}_{2}^{\text{defocus}}$ lead to an apparent
gaps in the constant Q-scans [Fig. 4(d)]. In this case, the defocused
resolution condition plays similar role to the orthogonal Q-integration range
in Fig. 3. Confirming the TMG in Cr$M$Te3 ($M$=Si,Ge) requires quantitative
comparison of the data and the fully resolution-convolved simulation. Assuming
that the data in Ref. [12] (Fig.2H) was measured at the focused K1
$(\frac{1}{3},\frac{1}{3},3)$ and defocused K2 $(\frac{2}{3},\frac{2}{3},3)$,
our new resolution-convolved spin-wave simulations suggest a DMI$\sim 0.06$
meV that is a half of the value reported in Ref. [12] (see [24] for details).

In conclusion, revealing and understanding topological magnons are important
steps towards the realization of magnon-based electronic devices as well as
for the fundamental goal of discovering a topological magnon insulator. To
meet these challenges, accurately clarifying and estimating an intrinsic Dirac
magnon gap is a critical issue in defining the topological properties, and INS
experiments play a key role in this endeavor. In particular, our study
provides important guidance for spectroscopic measurements of systems having a
singularity or a band crossing where the spectrum rapidly disperses in
momentum space. We have shown that instrumental and data-processing effects
can introduce an artificial gap, so that accurate estimation of the
topological gap size requires careful data histograms and comparison with
resolution-convolved calculations. Our results are relevant not only to Dirac
magnon gaps, but also to similar Weyl magnon, determining avoided crossing of
rattler modes to acoustic phonons [29], and quantifying the life-time of
rapidly decaying phonon spectra [30], where similar effects may be
anticipated.

###### Acknowledgements.

We thank Yixi Su for providing their experimental conditions and Michael E.
Manley for useful discussions. This research was supported by the U.S.
Department of Energy, Office of Science, Basic Energy Sciences, Materials
Science and Engineering Division. Work at the Oak Ridge National Laboratory
Spallation Neutron Source was supported by U.S. DOE, Office of Science, BES,
Scientific User Facilities Division.

## References

  * Goerbig and Montambaux [2017] M. Goerbig and G. Montambaux, “Dirac fermions in condensed matter and beyond,” in  _Dirac Matter_, edited by B. Duplantier, V. Rivasseau, and J.-N. Fuchs (Springer International Publishing, Cham, 2017) pp. 25–53.
  * Owerre [2016] S. A. Owerre, Journal of Physics: Condensed Matter 28, 386001 (2016).
  * Pershoguba  _et al._ [2018] S. S. Pershoguba, S. Banerjee, J. C. Lashley, J. Park, H. Ågren, G. Aeppli, and A. V. Balatsky, Phys. Rev. X 8, 011010 (2018).
  * Chisnell  _et al._ [2015] R. Chisnell, J. S. Helton, D. E. Freedman, D. K. Singh, R. I. Bewley, D. G. Nocera, and Y. S. Lee, Phys. Rev. Lett. 115, 147201 (2015).
  * Aguilera  _et al._ [2020] E. Aguilera, R. Jaeschke-Ubiergo, N. Vidal-Silva, L. E. F. F. Torres, and A. S. Nunez, Phys. Rev. B 102, 024409 (2020).
  * Li  _et al._ [2021] Z.-X. Li, Y. Cao, and P. Yan, Physics Reports 915, 1 (2021), topological insulators and semimetals in classical magnetic systems.
  * Malz  _et al._ [2019] D. Malz, J. Knolle, and A. Nunnenkamp, Nature communications 10, 1 (2019).
  * Rückriegel  _et al._ [2018] A. Rückriegel, A. Brataas, and R. A. Duine, Phys. Rev. B 97, 081106 (2018).
  * Chen  _et al._ [2018] L. Chen, J.-H. Chung, B. Gao, T. Chen, M. B. Stone, A. I. Kolesnikov, Q. Huang, and P. Dai, Phys. Rev. X 8, 041028 (2018).
  * Chen  _et al._ [2021a] L. Chen, J.-H. Chung, M. B. Stone, A. I. Kolesnikov, B. Winn, V. O. Garlea, D. L. Abernathy, B. Gao, M. Augustin, E. J. G. Santos, and P. Dai, Phys. Rev. X 11, 031047 (2021a).
  * Cai  _et al._ [2021] Z. Cai, S. Bao, Z.-L. Gu, Y.-P. Gao, Z. Ma, Y. Shangguan, W. Si, Z.-Y. Dong, W. Wang, Y. Wu, D. Lin, J. Wang, K. Ran, S. Li, D. Adroja, X. Xi, S.-L. Yu, X. Wu, J.-X. Li, and J. Wen, Phys. Rev. B 104, L020402 (2021).
  * Zhu  _et al._ [2021] F. Zhu, L. Zhang, X. Wang, F. J. dos Santos, J. Song, T. Mueller, K. Schmalzl, W. F. Schmidt, A. Ivanov, J. T. Park, J. Xu, J. Ma, S. Lounis, S. Blügel, Y. Mokrousov, Y. Su, and T. Brückel, Science Advances 7, eabi7532 (2021).
  * Kvashnin  _et al._ [2020] Y. O. Kvashnin, A. Bergman, A. I. Lichtenstein, and M. I. Katsnelson, Phys. Rev. B 102, 115162 (2020).
  * Delugas  _et al._ [2021] P. Delugas, O. Baseggio, I. Timrov, S. Baroni, and T. Gorni, arXiv preprint arXiv:2105.04531 (2021).
  * Lee  _et al._ [2020] I. Lee, F. G. Utermohlen, D. Weber, K. Hwang, C. Zhang, J. van Tol, J. E. Goldberger, N. Trivedi, and P. C. Hammel, Phys. Rev. Lett. 124, 017201 (2020).
  * Stavropoulos  _et al._ [2021] P. P. Stavropoulos, X. Liu, and H.-Y. Kee, Phys. Rev. Research 3, 013216 (2021).
  * Yuan  _et al._ [2020] B. Yuan, I. Khait, G.-J. Shu, F. C. Chou, M. B. Stone, J. P. Clancy, A. Paramekanti, and Y.-J. Kim, Phys. Rev. X 10, 011062 (2020).
  * McGuire  _et al._ [2017] M. A. McGuire, G. Clark, S. KC, W. M. Chance, G. E. Jellison, V. R. Cooper, X. Xu, and B. C. Sales, Phys. Rev. Materials 1, 014001 (2017).
  * Glamazda  _et al._ [2017] A. Glamazda, P. Lemmens, S.-H. Do, Y. S. Kwon, and K.-Y. Choi, Phys. Rev. B 95, 174429 (2017).
  * Chen  _et al._ [2021b] L. Chen, M. B. Stone, A. I. Kolesnikov, B. Winn, W. Shon, P. Dai, and J.-H. Chung, 2D Materials 9, 015006 (2021b).
  * Schneeloch  _et al._ [2021] J. A. Schneeloch, Y. Tao, Y. Cheng, L. Daemen, G. Xu, Q. Zhang, and D. Louca, arXiv preprint arXiv:2110.10771 (2021).
  * May  _et al._ [2020] A. F. May, J. Yan, and M. A. McGuire, Journal of Applied Physics 128, 051101 (2020).
  * Granroth  _et al._ [2010] G. E. Granroth, A. I. Kolesnikov, T. E. Sherline, J. P. Clancy, K. A. Ross, J. P. C. Ruff, B. D. Gaulin, and S. E. Nagler, Journal of Physics: Conference Series 251, 012058 (2010).
  * [24] “See supplemental material for additional data and analysis.” .
  * Toth and Lake [2015] S. Toth and B. Lake, Journal of Physics: Condensed Matter 27, 166002 (2015).
  * Elliot  _et al._ [2021] M. Elliot, P. A. McClarty, D. Prabhakaran, R. Johnson, H. Walker, P. Manuel, and R. Coldea, Nature Communications 12, 3936 (2021).
  * Shivam  _et al._ [2017] S. Shivam, R. Coldea, R. Moessner, and P. McClarty, arXiv preprint arXiv:1712.08535 (2017).
  * Farhi  _et al._ [2014] E. Farhi, Y. Debab, and P. Willendrup, Journal of Neutron Research 17, 5 (2014).
  * Christensen  _et al._ [2008] M. Christensen, A. B. Abrahamsen, N. B. Christensen, F. Juranyi, N. H. Andersen, K. Lefmann, J. Andreasson, C. R. Bahl, and B. B. Iversen, Nature materials 7, 811 (2008).
  * Xiao  _et al._ [2022] E. Xiao, H. Ma, M. S. Bryan, L. Fu, J. M. Mann, B. Winn, D. L. Abernathy, R. P. Hermann, A. R. Khanolkar, C. A. Dennett, _et al._ , arXiv preprint arXiv:2202.11041 (2022).


# CamemBERT-bio: Leveraging Continual Pre-training for Cost-Effective Models
on French Biomedical Data

###### Abstract

Clinical data in hospitals are increasingly accessible for research through
clinical data warehouses. However these documents are unstructured and it is
therefore necessary to extract information from medical reports to conduct
clinical studies. Transfer learning with BERT-like models such as CamemBERT
has allowed major advances for French, especially for named entity
recognition. However, these models are trained for plain language and are less
efficient on biomedical data. Addressing this gap, we introduce CamemBERT-bio,
a dedicated French biomedical model derived from a new public French
biomedical dataset. Through continual pre-training of the original CamemBERT,
CamemBERT-bio achieves an improvement of 2.54 points of F1-score on average
across various biomedical named entity recognition tasks, reinforcing the
potential of continual pre-training as an equally proficient yet less
computationally intensive alternative to training from scratch. Additionally,
we highlight the importance of using a standard evaluation protocol that
provides a clear view of the current state-of-the-art for French biomedical
models.

Keywords: EHR, clinical NLP, CamemBERT, information extraction, biomedical,
named entity recognition

CamemBERT-bio: Leveraging Continual Pre-training for Cost-Effective Models on
French Biomedical Data

Rian Touchent, Laurent Romary, Eric de La Clergerie
---
Inria, Sorbonne Université
2 rue Simone IFF 75012 Paris, 21 rue de l’école de médecine 75006 Paris
<EMAIL_ADDRESS>

Abstract content

## 1\. Introduction

In recent years, there has been a development of clinical data warehouses
(CDWs) in hospitals. These are clinical databases aimed at being more
accessible for research purposes. These documents represent an opportunity for
massive clinical studies using real data. They can take various forms within
Electronic Health Records (EHR), such as reports, medical imaging, or
prescriptions. However, most of the information is found in clinical reports.
It is estimated that up to 80% of entities are missing from other modalities
Raghavan et al. (2014). Although these data are highly valuable, they are
unstructured, which requires preprocessing before they can be used in a
clinical study.

BERT-based models (Devlin et al., 2019) consistently demonstrate state-of-the-
art results for a wide range of natural language processing tasks. The
adaptation of BERT to the French language, particularly with the CamemBERT
model (Martin et al., 2020), has replicated these performances in French
natural language processing. CamemBERT is based on RoBERTa (Liu et al., 2019),
which is a more efficient version of BERT. It is trained on a French corpus
extracted from the web called OSCAR (Ortiz Suárez et al., 2019).

To extract information from medical reports, it is necessary to have high-
performing language models trained on French clinical data, particularly for
named entity recognition. It is possible to simply use CamemBERT; however, the
results of this model on biomedical data are disappointing Cardon et al.
(2020), as it exhibits lower performance compared to heuristic models on
certain evaluation datasets. These results are predictable because CamemBERT
is trained on plain language, often sourced from web pages such as forums.
However, biomedical data, especially clinical data, are significantly
different. They contain technical terms that are very rare or absent in
everyday language, and they have a radically distinct style, often
telegraphic, rarely consisting of complete sentences, with varying
abbreviations.

One of the major challenges with healthcare data warehouses is data
confidentiality. These data are regulated and subject to strong regulations by
the CNIL (French Data Protection Authority). As a result, adaptations of
CamemBERT to the biomedical domain conducted within hospital infrastructures
Dura et al. (2022) cannot be publicly released. Their training datasets are
subject to publication constraints. These constraints also apply to the
resulting models. Therefore, it is not possible to exchange these models
between different healthcare institutions. A publicly available model would
not have these constraints and could be used in various institutions.

Using continual-pretraining on a new French biomedical corpus, we introduce a
new model named CamemBERT-bio, which shows a 2.54 points improvement in
F-score on several French biomedical named entity recognition tasks.
Furthermore, we engage in a discussion on the evaluation of French clinical
models, emphasizing the importance of adhering to established standard
practices. By following this methodology, we successfully showcase the
effectiveness of continual-pretraining in the context of a French model, which
contrast with recent suggestions pertaining to the same domain and language
(Labrak et al., 2023).

In this article, we present three main contributions :

  * •

The creation of a new public French dataset specialized in the biomedical
domain.

  * •

The introduction of a publicly available adaptation of CamemBERT for the
biomedical domain, which demonstrates improved performance on named entity
recognition tasks.

  * •

The demonstration that continual-pretraining on a French model is successful,
necessitating a reevaluation of previous work due to the impact of evaluation
methodology on result interpretation.

## 2\. Related Works

Research on adapting language models to new domains is extensive. Gururangan
et al. (2020) demonstrate that a second phase of pre-training on a target
domain can improve performance on various tasks, even when the target domain
corpus is small in size. In the biomedical domain, it has been observed that
there can be up to a 3-point increase in F-measure compared to the same model
without the second phase of pre-training.

This study by Gururangan et al. (2020) has inspired the creation of new models
based on BERT, utilizing a second phase of pre-training on various specialized
domains. Lee et al. (2019) introduced BioBERT, a BERT-based model specialized
for biomedical text in English. BioBERT demonstrates improved performance on
various biomedical NLP tasks, including a $0.62\%$ F-measure improvement on
named entity recognition, a $2.80\%$ F-measure improvement on relation
extraction, and a $12.24\%$ MMR improvement on question-answering. The second
phase of pre-training is conducted on a corpus extracted from PubMed and PMC,
consisting of approximately 18 billion words from biomedical scientific
articles. While the corpus is substantial and solely composed of scientific-
style text, performance gains are observed across all text styles. The
presence of medical vocabulary in the corpus likely contributes to significant
improvement compared to general language models.

Training new models from scratch is also a viable approach. This is explored
in SciBERT (Beltagy et al., 2019) and PubMedBERT (Gu et al., 2022), two models
specialized in biomedical scientific articles. PubMedBERT demonstrates that
this method yields better performance than models trained with a second phase
of specialization. However, the performance gains are relatively modest, and
this approach is more computationally expensive. Starting from scratch
requires longer training times and a larger corpus to achieve comparable
performance.

For the French language, the reference models are CamemBERT (Martin et al.,
2020) and FlauBERT (Le et al., 2020). Several works have attempted to adapt
CamemBERT to the biomedical domain.

Copara et al. (2020) explored a second phase of pre-training on 31,000 French
biomedical scientific articles. However, they did not observe a significant
improvement on a clinical named entity recognition task using the large
version of CamemBERT. This could be explained by the combination of a
relatively small corpus (31k documents compared to the 18 billion words in
BioBERT) and the large version of the CamemBERT model.

Le Clercq de Lannoy et al. (2022) also adapted CamemBERT to the biomedical
domain. They aggregated documents from various sources, including PubMed,
Cochrane, ISTEX, and Wikipedia, forming a larger partially public corpus of
approximately 136 million words. They observed a 2-point improvement in
F-measure on a named entity recognition evaluation set composed of drug
notices (EMEA), but no significant improvement on a set composed of scientific
article titles (MEDLINE).

Dura et al. (2022) continued the pre-training of CamemBERT on 21 million
clinical documents from the APHP (Assistance Publique - Hôpitaux de Paris)
clinical data warehouse. They observed a significant 3% improvement on APMed,
a private clinical named entity recognition dataset owned by APHP. They also
achieved similar scores to CamemBERT on EMEA and MEDLINE. Their new model
performs better on clinical data, yet it obtains scores similar to CamemBERT
in other biomedical domains.

Labrak et al. (2023) introduced a public French biomedical model named DrBERT.
Through their experiments, they explored both continual-pretraining and from-
scratch training strategies. Their findings indicated a superior performance
when training from scratch, suggesting that continual-pretraining with
CamemBERT for French biomedical data may not be as effective. However, when
applied to PubMedBERT, continual-pretraining yielded results nearly on par
with the from-scratch approach.

Finally, Berhe et al. (2023) introduced AliBERT, which was trained on a French
biomedical corpus primarily comprising articles from ScienceDirect and theses
collected through Sudoc. The model leverages a new regularized Unigram-based
tokenizer and underwent extensive training on 48 GPUs for a total duration of
20 hours. Their pretrained model outperforms notable French non-domain-
specific models, such as CamemBERT and FlauBERT, in two biomedical downstream
tasks. Unfortunately, the model is not currently available.

## 3\. CamemBERT-bio

Corpus | Details |  Size
---|---|---
ISTEX | Scientific literature |  276 M
CLEAR | Drug leaflets |  73 M
E3C | Clinical cases and leaflets |  64 M
Total |  |  413 M
Table 1: Composition of the biomed-fr corpus (in millions of words)

### 3.1. Corpus : biomed-fr

First, we built a French biomedical corpus composed exclusively of public
documents to minimize the usage constraints mentioned earlier. The documents
come from three different sources (see Table.1), the main one being ISTEX.
This new corpus, named biomed-fr, consists of 413 million words, equivalent to
2.7 GB of data. Martin et al. (2020) have shown that with only 4 GB of data,
it is possible to achieve performance almost comparable to the model trained
with the 138 GB OSCAR dataset (Ortiz Suárez et al., 2019). For an adaptation
of CamemBERT to the biomedical domain, this amount of data can be considered
sufficient.

|  |  |  | CamemBERT-bio
---|---|---|---|---
Style | Dataset | Score | CamemBERT | biomed-fr-small | biomed-fr
Clinical | CAS1 | F1 |  70.50 $\pm$ 1.75 | 72.94 $\pm$ 1.12 | 73.03 $\pm$ 1.29
P |  70.12 $\pm$ 1.93 | 72.97 $\pm$ 0.84 | 71.71 $\pm$ 1.61
R |  70.89 $\pm$ 1.78 | 72.92 $\pm$ 1.39 | 74.42 $\pm$ 1.49
CAS2 | F1 |  79.02 $\pm$ 0.92 | 80.00 $\pm$ 0.32 | 81.66 $\pm$ 0.59
P |  77.3 $\pm$ 1.36 | 78.29 $\pm$ 0.91 | 80.96 $\pm$ 0.91
R |  80.83 $\pm$ 0.96 | 81.80 $\pm$ 0.48 | 82.37 $\pm$ 0.69
E3C | F1 |  67.63 $\pm$ 1.45 | 67.96 $\pm$ 1.85 | 69.85 $\pm$ 1.58
P | 78.19 $\pm$ 0.72 |  77.41 $\pm$ 1.01 | 79.11 $\pm$ 0.42
R |  59.61 $\pm$ 2.25 | 60.57 $\pm$ 2.32 | 62.56 $\pm$ 2.50
Leaflets | EMEA | F1 |  74.14 $\pm$ 1.95 | 75.93 $\pm$ 2.42 | 76.71 $\pm$ 1.50
P |  74.62 $\pm$ 1.97 | 76.23 $\pm$ 2.27 | 76.92 $\pm$ 1.96
R |  73.68 $\pm$ 2.22 | 75.63 $\pm$ 2.61 | 76.52 $\pm$ 1.62
Scientific | MEDLINE | F1 | 65.73 $\pm$ 0.40 |  65.48 $\pm$ 0.31 | 68.47 $\pm$ 0.54
P | 64.94 $\pm$ 0.82 |  64.43 $\pm$ 0.50 | 67.77 $\pm$ 0.88
R | 66.56 $\pm$ 0.56 | 66.56 $\pm$ 0.16 | 69.21 $\pm$ 1.32
Table 2: F-scores on different biomedical named entity recognition tasks

#### ISTEX

The ISTEX database contains references to 27 million scientific publications.
We extracted 108,183 French documents published in a biology or medical
journal since 1990. Articles published before this date often contain numerous
typographical errors, as they are often scanned articles that require optical
character recognition algorithms, resulting in a certain number of errors.
Such errors are found to a lesser extent in articles published after 1990.
Some documents, although in French, contain passages in English. Therefore,
there is an indeterminate amount of English in this corpus. However, it is
unlikely that this will significantly impact pre-training. Typographical
errors and the presence of other languages are aspects that can be addressed
in future versions of biomed-fr.

#### CLEAR

The CLEAR corpus (Grabar and Cardon, 2018) consists of encyclopedia articles,
drug leaflets, and abstracts of scientific articles. Each document is
available in two versions: one in technical language and the other in
simplified language. We retrieved all of these documents in both versions.
Regarding the drug leaflets, we removed redundant sentences at the beginning
and end of each document, such as the website navigation bar from which the
documents were extracted or information about the company selling the
documents.

#### E3C

This corpus (Magnini et al., 2020) is composed of three layers. The first two
layers are annotated or semi-annotated and will be used for evaluation. The
last layer is not annotated, and that is the one we retrieved. It consists of
medical specialty admission competitions, drug leaflets, and medical thesis
abstracts. There may be duplicates of some leaflets found in the CLEAR corpus.

#### biomed-fr-small

By randomly selecting 10% of content from biomed-fr, we created a smaller
corpus called biomed-fr-small. The corpus allows us to study the impact of
corpus size.

### 3.2. Pre-training Strategies

For the adaptation of CamemBERT to the biomedical domain, we conducted a
second phase of pre-training on both versions of the biomed-fr corpus,
starting from the weights and configuration of the camembert-base model. We
applied the Masked Language Modeling (MLM) task with whole-word masking,
following the method of Martin et al. (2020). We used the Adam optimizer
Kingma and Ba (2017) with $\beta_{1}=0.9$ and $\beta_{2}=0.98$, and a learning
rate of $5e-5$. We performed 50,000 steps over 39 hours using two Tesla V100
GPUs. A batch size of 8 per GPU and gradient accumulation over 16 steps were
used to achieve an effective batch size of 256.

### 3.3. Fine-tuning and Evaluation

Regarding model evaluation, we collected three named entity recognition
evaluation datasets. These datasets cover various styles, allowing us to
assess the model’s versatility across different subdomains of biomedicine.

#### QUAERO

The QUAERO corpus (Névéol et al., 2014) consists of two evaluation sets: EMEA,
containing drug leaflets, and MEDLINE, containing scientific article titles.
The entities are manually annotated following 10 semantic groups from the UMLS
(Lindberg et al., 1993). As some of these entities are nested, we kept only
the entities with the coarsest granularity. F-scores are calculated in the
same way.

#### E3C

For evaluation, unlike the biomed-fr corpus, we use layers 1 and 2. These
layers contain documents of different types, including clinical cases
extracted from scientific articles. Layer 2 is semi-annotated, and it is used
as the training set for fine-tuning, with 10% dedicated to the validation set.
We evaluate on layer 1, which is fully manually annotated. There is only one
class, and the objective is to find clinical entities in the text, regardless
of their type.

#### CAS

The CAS corpus (Grouin et al., 2019) also consists of clinical cases from
scientific articles. We focus on task 3 of DEFT 2020 Cardon et al. (2020),
which is an information extraction task based on CAS. It includes two
subtasks, and thus two sets of annotations. In the first subtask, two classes
need to be identified: pathology and signs or symptoms. The second subtask
concerns associated information, including anatomy, dose, examination, mode,
timing, substance, treatment, and value. These two tasks will be referred to
as CAS1 and CAS2, respectively.

#### Fine-tuning

For fine-tuning, we used Optuna Akiba et al. (2019) for hyperparameter
selection. We set the learning rate to $5e-5$, the warmup ratio to 0.224, and
the batch size to 16. We performed 2000 steps. Predictions were made using a
simple linear layer on top of the model. None of the CamemBERT layers were
frozen.

#### Evaluation

Scores are measured using the seqeval tool Nakayama (2018) in strict mode with
micro-average and the "IOB2" scheme. For each evaluation, the best fine-tuned
model on the validation set is selected to measure the final score on the test
set. We average the results over 10 evaluations with different seeds.

## 4\. Results and Discussion

|  | CAS1 | CAS2 | EMEA | MEDLINE
---|---|---|---|---|---
Evaluator | Authors | F1 | F1 | F1 | P | R | F1 | P | R
seqeval |  Dura et al. (2022)-fine-tuned | - | - | 72.90 | - | - | 59.70 | - | -
Dura et al. (2022)-from-scratch | - | - | 69.30 | - | - | 60.10 | - | -
ours | 73.03 | 81.66 | 76.71 | 76.92 | 76.52 | 68.47 | 67.77 | 69.21
BRATeval | Le Clercq de Lannoy et al. (2022) |  |  | 67.4 | 73.4 | 62.2 | 55.3 | 62.2 | 49.7
Mulligen et al. (2016) | - | - | 74.9 | 71.6 | 78.5 | 69.8 | 68 | 71.6
Copara et al. (2020) | 61.53 | 73.7 | - | - | - | - | - | -
ours | 84.97 | 83.25 | 77.80 | 79.77 | 75.93 | 56.16 | 75.33 | 44.82
Table 3: Comparison of CamemBERT-bio with different approaches on the 4 named
entity recognition tasks. In the first part of the table, scores are measured
with seqeval Nakayama (2018), and in the second part with BRATeval, which is
the evaluation tool provided for the CLEF eHealth Evaluation lab 2016 campaign
Névéol et al. (2016).

#### CamemBERT vs CamemBERT-bio

We observe a significant performance gain on all evaluation datasets with our
new model (see Table.2). On average, we achieve a 2.54-point improvement in
F-score. This gain is observed across all styles, demonstrating the model’s
versatility for both clinical and scientific domains.

#### biomed-fr-small vs biomed-fr

We observe a decrease in performance with the biomed-fr-small dataset, but
there is still a significant gain on certain datasets compared to CamemBERT.
This confirms that the size of the corpus positively influences the
performance, even in a specialized domain like biomedicine.

| Pre-training corpus
---|---
Authors | Origin |  Size11footnotemark: 1
Dura et al. (2022) | APHP | 21 MD
Le Clercq de Lannoy et al. (2022) | misc | 136 MW
Copara et al. (2020) | PubMed | 31 KD
ours | biomed-fr | 413 MW
Table 4: Pre-training corpora of related works (cf. Table.3)

#### Comparison with the state of the art

We compared the performance of CamemBERT-bio with various previously mentioned
approaches (see Table.3). CamemBERT-bio achieves the best results for almost
all evaluation datasets. Dura et al. (2022) did not observe improvement on
EMEA and MEDLINE compared to CamemBERT because their pre-training corpus (see
Table.4) consists of documents from APHP, making it a less diverse corpus.
However, they gain several points on their evaluation dataset, which is also
based on APHP documents. Mulligen et al. (2016) presents the highest score on
MEDLINE and the best recall on EMEA. Their approach is based on a knowledge-
based model, allowing them to achieve the best recall on both QUAERO
evaluation datasets. Furthermore, their approach is the only one in this table
capable of handling nested entities, giving them an advantage.

It is important to note that these different CamemBERT-based approaches have
various experimental setups. The presence of CRF layers instead of a simple
linear layer after CamemBERT, freezing CamemBERT layers, and variations in
hyperparameters are examples of differences observed in addition to the pre-
training corpus, which makes the comparison more challenging.

#### Tokenization analysis

CamemBERT-bio is a biomedical-adapted model based on CamemBERT. Unlike a new
model trained from scratch, it shares the same vocabulary. The vocabulary of
CamemBERT was constructed using SentencePiece (Kudo and Richardson, 2018) on
an OSCAR sample. Therefore, it is a general-purpose vocabulary designed for
everyday language. We can hypothesize that the tokenization of CamemBERT may
result in oversplitting of biomedical technical terms.

To investigate this possibility, we trained a specialized tokenizer on biomed-
fr-small and calculated the intersection of the two vocabularies (Table.5).

Terms | general | specialized
---|---|---
échocardiographie | écho-cardi-ographie | échocardiographi-e
transthoracique | trans-thorac-ique | trans-thoracique
glimépiride | g-lim-épi-ride | gli-m-épi-ride
cardiopathie | cardio-pathie | cardiopathie
diastoliques | dia-s-tol-iques | diastolique-s
Table 5: Comparison of tokenization between a general-purpose tokenizer and a
specialized tokenizer for some biomedical technical terms.

We find a 45% intersection between the two vocabularies, which is quite close
to the 42% intersection found by Beltagy et al. (2019) between the vocabulary
of BERT and SciBERT. Therefore, there is a significant difference in the most
frequent terms.

## 5\. Evaluating Models: Methodology and Discussion

In a recent work, Labrak et al. (2023) introduced a new French biomedical
language model named DrBERT. The authors contend that performing continual-
pretraining on biomedical data from CamemBERT leads to reduced performance.
They used a different methodology for the evaluation of named entity
recognition, prompting us to examine the implications of each approach on the
interpretation of the results.

Our evaluation approach is centered around micro F1, which is measured using
seqeval in strict mode with the IOB2 scheme. Their approch is based on
weighted F1 with independant token classification. Every token has a label and
the model is evaluated for each of them, whereas with seqeval, the model is
evaluated for each entity. Notably, the "O" token, representing non-entity, is
the most frequent token and thus holds significant weight in the evaluation
process. As all tokens are labeled, the number of predictions depends on the
tokenizer, thereby influencing the final score. In order to explore the impact
of token labeling variations on the evaluation, we conducted an additional
experiment on EMEA and MEDLINE where we reproduced the DrBERT methodolody that
we named token-with-O, and another one where we excluded the "O" token and
only labeled the first token of each entity, that will be refered as entity-
without-O. Furthermore, we included a seqeval strict score using the IOB2
scheme, although it is not directly comparable to our approach due to their
implementation of nested entities by concatenating their names to form new
entities, while we simply removed them.

|  | EMEA | MEDLINE
---|---|---|---
Methodology | Model | weighted-f1 | macro-f1 | micro-f1 | seqeval-f1 | weighted-f1 | macro-f1 | micro-f1 | seqeval-f1
token-with-O | DrBERT-7GB | 87.45 | 34.95 | - | - | 75.52 | 15.07 | - | -
CamemBERT-bio | 90.37 | 36.27 | - | - | 77.89 | 14.82 | - | -
CamemBERT | 88.33 | 47.45 | - | - | 76.2 | 11.92 | - | -
entity-without-O | DrBERT-7GB | 66.72 | 24.72 | 68.34 | 59.39 | 60.70 | 10.80 | 63.40 | 50.45
CamemBERT-bio | 73.53 | 24.15 | 75.05 | 67.58 | 62.04 | 8.695 | 65.44 | 52.9
CamemBERT | 71.85 | 22.71 | 72.93 | 64.23 | 60.95 | 9.413 | 63.47 | 51.75
Table 6: Performance comparison of CamemBERT, CamemBERT-bio, and DrBERT on
EMEA and MEDLINE using the evaluation methodology token-with-O, along with a
modified variant entity-without-O. The reported scores are averaged over 10
runs.

We observe significant differences in scores between the two methodologies
(Table.6). The methodology token-with-O consistently reports higher scores
across all tasks compared to the alternative method, entity-without-O.
However, the reported scores do not precisely align with those presented by
Labrak et al. (2023), where DrBERT exhibits improvement over CamemBERT for
EMEA, and match CamemBERT for MEDLINE. We posit that these discrepancies arise
from disparities in hyperparameters. Nevertheless, the comparison with entity-
without-O still provides insightful observations.

Notably, the best-performing model varies depending on the chosen methodology
for one metric. CamemBERT achieves the highest macro F1 score on the EMEA
dataset, whereas with entity-without-O, DrBERT-7GB emerges as the top model.
This observation underscores the influential role of the "O" class in
determining the best-performing model and prompts us to consider the aspect we
aim to evaluate.

In terms of macro F1, DrBERT consistently outperforms CamemBERT-bio when
evaluated using the entity-without-O methodology. On the other hand,
CamemBERT-bio exhibits better performance across all other metrics. This
indicates that DrBERT demonstrates a more balanced performance across
different classes. These findings suggest that these models may possess
complementary strengths, and it is important to focus on enhancing the
performance of CamemBERT-bio in this aspect.

It is worth noting that the evaluations conducted by Labrak et al. (2023)
encompass a wide range of tasks beyond EMEA and MEDLINE. This observation
underscores the importance of establishing a unified benchmark to facilitate
fair comparisons between models for the french biomedical domain. A noteworthy
tool for this purpose is BRATeval from the CLEF eHealth Evaluation lab 2016
campaign Névéol et al. (2016), as it evaluates based on exact match character
offsets, thus avoiding any influence from dataset pre-processing methods.

Furthermore, Labrak et al. (2023) suggested that continual-pretraining from a
French model on French biomedical data isn’t effective, as it results in an
F1-score loss of up to 20 points. However, considering our success with
continual-pretraining and the points we raised about the methodology, we
suggest a reassessment of these findings.

## 6\. Environmental Impact

Considering the environmental implications is crucial when discussing language
model training due to its potential compute intensity.

| Training time (hours) | Hardware type | Total GPU-hours |  | Estimation of
---
carbon emitted (kg CO2 eq.)
DrBERT | 20h | 128xV100 | 2560 | 26.11
AliBERT | 20h | 48xA100 | 960 | 8.16
CamemBERT-bio | 39h | 2xV100 | 78 | 0.8
Table 7: Carbon emitted estimation based on hardware and training time. We
used a rate of 34g CO2eq. per kWh, reflecting the average over the last 12
months in France starting from September 2022. This time frame and location
coincide with when and where all experiments were conducted.

Considering estimated carbon emissions 111Estimations were conducted using the
MachineLearning Impact calculator presented by Lacoste et al. (2019) during
training, AliBERT emits 10 times the amount of CamemBERT-bio, while DrBERT
releases 32 times more (Table.7). While a direct comparison with AliBERT
wasn’t possible, the minor performance variance with DrBERT, set against the
pronounced disparities in computational and environmental costs, leads us to
advocate for continual-pretraining as the preferred adaptation method for
biomedical language models.

## 7\. Limitations

It is important to discuss some limitations of our studies.

Firstly, our training corpus biomed-fr is limited in its diversity. This is
because we chose to only use publicly available materials, which tend to lean
towards scientific content. As a result, our analysis may not fully represent
performances on real private clinical data.

Furthermore, we didn’t explore any potential biases in our data and some part
of our dataset would benefit from further cleaning. These aspects could impact
the outputs of our model and should be investigated.

Additionally, our evaluation focused on one task which is Named Entity
Recognition. This might mean our understanding of how well our model performs
is restricted. Future studies should aim to assess our model across a wider
range of tasks and datasets to get a clearer picture of its strengths and
weaknesses.

This underscores the need for further research to address these constraints
and enhance our understanding of the subject matter.

## 8\. Conclusion and Perspectives

We have introduced a new French biomedical corpus called biomed-fr consisting
of 413 million words, composed of drug leaflets and documents from scientific
literature in medicine and biology. This new corpus has allowed us to adapt
CamemBERT to the biomedical domain through a second phase of pre-training. We
observe an improvement in performance on all our named entity recognition
evaluation datasets, with an average gain of 2.54 F-score points. Our model
establishes a new state-of-the-art on these French biomedical language
processing tasks. We have some directions for future versions of biomed-fr.
Firstly, we can further clean the data by removing passages within the
documents that are not in French or by excluding documents with a high number
of typographical errors. Secondly, we can increase the amount of data. This
could involve leveraging archived documents on HAL related to life sciences,
particularly those published by INSERM, or retrieving abstracts of French
articles from PubMed.

The analysis of tokenization prompts us to consider expanding the vocabulary
for CamemBERT-bio. The relatively modest performance gain of PubMedBERT
compared to BioBERT and the similar performance of DrBERT compared to
CamemBERT-bio despite their specialized vocabulary, the over-segmentation of
technical terms and the low intersection rate between the generalist
vocabulary and the specialized vocabulary demonstrate the value of the
experiment. However, taking into account the comparable performance levels and
significant difference in environmental impact, we advocate strongly for the
continual-pretraining method in adapting language models to the biomedical
domain.

Finally, in recent months, numerous generative models, often with billions of
parameters, have demonstrated remarkable performance on biomedical tasks,
sometimes surpassing specialized models like BioBERT Agrawal et al. (2022);
Singhal et al. (2023). This is a promising research direction for biomedical
information extraction. However, we have reasons to believe that BERT-type
models still have value Lehman et al. (2023). Firstly, in a clinical context,
models are often used within healthcare institution infrastructures, which
entails resource constraints. It is easier to deploy small specialized models
than large generalist models in such cases. Secondly, the use of these
generative models often requires accessing remote servers, typically through
APIs, which makes their utilization challenging considering the
confidentiality constraints imposed on clinical documents.

## 9\. Bibliographical References

##

  * Agrawal et al. (2022) Monica Agrawal, Stefan Hegselmann, Hunter Lang, Yoon Kim, and David Sontag. 2022\.  Large language models are few-shot clinical information extractors.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 1998–2022, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Akiba et al. (2019) Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. 2019\.  Optuna: A next-generation hyperparameter optimization framework.  In _Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, KDD ’19, page 2623–2631, New York, NY, USA. Association for Computing Machinery.
  * Cardon et al. (2020) Rémi Cardon, Natalia Grabar, Cyril Grouin, and Thierry Hamon. 2020.  Présentation de la campagne d’évaluation DEFT 2020 : similarité textuelle en domaine ouvert et extraction d’information précise dans des cas cliniques (presentation of the DEFT 2020 challenge : open domain textual similarity and precise information extraction from clinical cases ).  In _Actes de la 6e conférence conjointe Journées d’Études sur la Parole (JEP, 33e édition), Traitement Automatique des Langues Naturelles (TALN, 27e édition), Rencontre des Étudiants Chercheurs en Informatique pour le Traitement Automatique des Langues (RÉCITAL, 22e édition). Atelier DÉfi Fouille de Textes_ , pages 1–13, Nancy, France. ATALA et AFCP.
  * Copara et al. (2020) Jenny Copara, Julien Knafou, Nona Naderi, Claudia Moro, Patrick Ruch, and Douglas Teodoro. 2020.  Contextualized French Language Models for Biomedical Named Entity Recognition.  In _Traitement Automatique des Langues Naturelles (TALN, 27e édition). Atelier DÉfi Fouille de Textes_ , pages 36–48, Nancy, France. ATALA.
  * Dura et al. (2022) Basile Dura, Charline Jean, Xavier Tannier, Alice Calliger, Romain Bey, Antoine Neuraz, and Rémi Flicoteaux. 2022.  Learning structures of the French clinical language:development and validation of word embedding models using 21 million clinical reports from electronic health records.  Technical Report arXiv:2207.12940, arXiv.  ArXiv:2207.12940 [cs, stat] type: article.
  * Gururangan et al. (2020) Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020.  Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 8342–8360, Online. Association for Computational Linguistics.
  * Kingma and Ba (2017) Diederik P. Kingma and Jimmy Ba. 2017.  Adam: A Method for Stochastic Optimization.  ArXiv:1412.6980 [cs].
  * Lacoste et al. (2019) Alexandre Lacoste, Alexandra Luccioni, Victor Schmidt, and Thomas Dandres. 2019\.  Quantifying the carbon emissions of machine learning.  _arXiv preprint arXiv:1910.09700_.
  * Le Clercq de Lannoy et al. (2022) Tiphaine Le Clercq de Lannoy, Romaric Besançon, Olivier Ferret, Julien Tourille, Frédérique Brin-Henry, and Bianca Vieru. 2022.  Stratégies d’adaptation pour la reconnaissance d’entités médicales en français (Adaptation strategies for biomedical named entity recognition in French).  In _Actes de la 29e Conférence sur le Traitement Automatique des Langues Naturelles. Volume 1 : conférence principale_ , pages 215–225, Avignon, France. ATALA.
  * Lehman et al. (2023) Eric Lehman, Evan Hernandez, Diwakar Mahajan, Jonas Wulff, Micah J Smith, Zachary Ziegler, Daniel Nadler, Peter Szolovits, Alistair Johnson, and Emily Alsentzer. 2023.  Do we still need clinical language models? In _Proceedings of the Conference on Health, Inference, and Learning_ , volume 209 of _Proceedings of Machine Learning Research_ , pages 578–597. PMLR.
  * Mulligen et al. (2016) Erik M. van Mulligen, Zubair Afzal, Saber A. Akhondi, Dang Vo, and Jan A. Kors. 2016\.  Erasmus MC at CLEF eHealth 2016: Concept Recognition and Coding in French Texts.
  * Nakayama (2018) Hiroki Nakayama. 2018.  seqeval: A python framework for sequence labeling evaluation.  Software available from https://github.com/chakki-works/seqeval.
  * Névéol et al. (2016) Aurélie Névéol, K. Bretonnel Cohen, Cyril Grouin, Thierry Hamon, Thomas Lavergne, Liadh Kelly, Lorraine Goeuriot, Grégoire Rey, Aude Robert, Xavier Tannier, and Pierre Zweigenbaum. 2016.  Clinical Information Extraction at the CLEF eHealth Evaluation lab 2016.  _CEUR workshop proceedings_ , 1609:28–42.
  * Raghavan et al. (2014) Preethi Raghavan, James L. Chen, Eric Fosler-Lussier, and Albert M. Lai. 2014.  How essential are unstructured clinical narratives and information fusion to clinical trial recruitment? _AMIA Summits on Translational Science Proceedings_ , 2014:218–223.
  * Singhal et al. (2023) Karan Singhal, Shekoofeh Azizi, Tao Tu, S. Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, Perry Payne, Martin Seneviratne, Paul Gamble, Chris Kelly, Abubakr Babiker, Nathanael Schärli, Aakanksha Chowdhery, Philip Mansfield, Dina Demner-Fushman, Blaise Agüera y Arcas, Dale Webster, Greg S. Corrado, Yossi Matias, Katherine Chou, Juraj Gottweis, Nenad Tomasev, Yun Liu, Alvin Rajkomar, Joelle Barral, Christopher Semturs, Alan Karthikesalingam, and Vivek Natarajan. 2023.  Large language models encode clinical knowledge.  _Nature_ , 620(7972):172–180.

## 10\. Language Resource References

##

  * Beltagy et al. (2019) Beltagy, Iz and Lo, Kyle and Cohan, Arman. 2019.  _SciBERT: A Pretrained Language Model for Scientific Text_.  Association for Computational Linguistics.
  * Berhe et al. (2023) Berhe, Aman and Draznieks, Guillaume and Martenot, Vincent and Masdeu, Valentin and Davy, Lucas and Zucker, Jean-Daniel. 2023.  _ALIBERT: A PRETRAINED LANGUAGE MODEL FOR FRENCH BIOMEDICAL TEXT_.  Working paper or preprint.
  * Devlin et al. (2019) Devlin, Jacob and Chang, Ming-Wei and Lee, Kenton and Toutanova, Kristina. 2019\.  _BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding_.  Association for Computational Linguistics.
  * Grabar and Cardon (2018) Grabar, Natalia and Cardon, Rémi. 2018.  _CLEAR – Simple Corpus for Medical French_.  Association for Computational Linguistics.
  * Grouin et al. (2019) Grouin, Cyril and Grabar, Natalia and Claveau, Vincent and Hamon, Thierry. 2019\.  _Clinical Case Reports for NLP_.  Association for Computational Linguistics.
  * Gu et al. (2022) Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. 2022.  Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing.  _ACM Transactions on Computing for Healthcare_ , 3(1):1–23.  ArXiv:2007.15779 [cs].
  * Kudo and Richardson (2018) Kudo, Taku and Richardson, John. 2018.  _SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing_.  Association for Computational Linguistics.
  * Labrak et al. (2023) Labrak, Yanis and Bazoge, Adrien and Dufour, Richard and Rouvier, Mickael and Morin, Emmanuel and Daille, Béatrice and Gourraud, Pierre-Antoine. 2023.  _DrBERT: A Robust Pre-trained Model in French for Biomedical and Clinical domains_.  Association for Computational Linguistics.
  * Le et al. (2020) Le, Hang and Vial, Loïc and Frej, Jibril and Segonne, Vincent and Coavoux, Maximin and Lecouteux, Benjamin and Allauzen, Alexandre and Crabbé, Benoît and Besacier, Laurent and Schwab, Didier. 2020.  _FlauBERT: Unsupervised Language Model Pre-training for French_.  European Language Resources Association.
  * Lee et al. (2019) Lee, Jinhyuk and Yoon, Wonjin and Kim, Sungdong and Kim, Donghyeon and Kim, Sunkyu and So, Chan Ho and Kang, Jaewoo. 2019.  _BioBERT: a pre-trained biomedical language representation model for biomedical text mining_.
  * Lindberg et al. (1993) Lindberg, D. A. and Humphreys, B. L. and McCray, A. T. 1993.  _The Unified Medical Language System_.
  * Liu et al. (2019) Liu, Yinhan and Ott, Myle and Goyal, Naman and Du, Jingfei and Joshi, Mandar and Chen, Danqi and Levy, Omer and Lewis, Mike and Zettlemoyer, Luke and Stoyanov, Veselin. 2019.  _RoBERTa: A Robustly Optimized BERT Pretraining Approach_.  arXiv.  ArXiv:1907.11692 [cs].
  * Magnini et al. (2020) Bernardo Magnini and Begoña Altuna and Alberto Lavelli and Manuela Speranza and Roberto Zanoli. 2020.  _The E3C Project: Collection and Annotation of a Multilingual Corpus of Clinical Cases_.
  * Martin et al. (2020) Martin, Louis and Muller, Benjamin and Ortiz Suárez, Pedro Javier and Dupont, Yoann and Romary, Laurent and de la Clergerie, Éric and Seddah, Djamé and Sagot, Benoît. 2020.  _CamemBERT: a Tasty French Language Model_.  Association for Computational Linguistics.
  * Névéol et al. (2014) Névéol, Aurélie and Grouin, Cyril and Leixa, Jeremy and Rosset, Sophie and Zweigenbaum, Pierre. 2014.  _The QUAERO French Medical Corpus: A Ressource for Medical Entity Recognition and Normalization_.
  * Ortiz Suárez et al. (2019) Pedro Javier Ortiz Suárez and Benoît Sagot and Laurent Romary. 2019.  _Asynchronous pipelines for processing huge corpora on medium to low resource infrastructures_.  Leibniz-Institut für Deutsche Sprache, Proceedings of the Workshop on Challenges in the Management of Large Corpora (CMLC-7) 2019. Cardiff, 22nd July 2019.

|  |  |  | CamemBERT-bio
---|---|---|---|---
Style | Dataset | Score | CamemBERT | biomed-fr-small | biomed-fr
Clinical | CAS1 | F1 | 67.85 | 70.18 | 71.37
P | 73.13 | 74.11 | 75.22
R | 64.38 | 67.39 | 68.34
CAS2 | F1 | 72.40 | 74.69 | 74.32
P | 73.22 | 74.46 | 72.46
R | 72.03 | 75.23 | 77.56
Leaflets | EMEA | F1 | 50.74 | 53.1 | 55.69
P | 51.99 | 55.88 | 56.01
R | 50.67 | 51.78 | 56.09
Scientific | MEDLINE | F1 | 45.09 | 47.73 | 48.18
P | 46.57 | 48.04 | 49.16
R | 47.18 | 52.62 | 50.38
Table 8: F-scores on different biomedical named entity recognition tasks with
macro-average

## Appendices

### A. Macro-average

In our main results (Table.2), scores are measured using the seqeval tool
Nakayama (2018) in strict mode with micro-average and the "IOB2" scheme. We
also conducted the same evaluation using macro-average (Table.8).