* Zinchenko et al. (2016) Zinchenko I. A., Pilyugin L. S., Grebel E. K., Sánchez S. F., Vílchez J. M., 2016, MNRAS, 462, 2715
  * Zinchenko et al. (2019) Zinchenko I., Just A., Pilyugin L., Lara-Lopez M., 2019, Astronomy & Astrophysics, 623, A7
  * Zurita & Bresolin (2012) Zurita A., Bresolin F., 2012, MNRAS, 427, 1463
  * de Jong et al. (2019) de Jong R. S., et al., 2019, The Messenger, 175, 3
  * van den Hoek & Groenewegen (1997) van den Hoek L. B., Groenewegen M. A. T., 1997, A&AS, 123, 305

##  Appendix A Scaled radial profiles

Figure 15:  The $1\mathchar 45\relax\sigma$ host-to-host scatter of the scaled
radial gradients for different scale radii at $z=0$. We define $R_{25}$,
$R_{50}$, and $R_{90}$ for the gas and stars in Table 1. $R_{\rm disk}$ is the
exponential scale length of the stellar disk determined via a 2-component fit
to the surface density, and $R_{\rm phys}$ is the physical radial coordinates
of the disk, i.e. unscaled coordinates. For each scale radius, we measure the
gradient of all galaxies across an equal radial range that corresponds to
$4-12\,\textnormal{kpc}$ physical for the galaxy with the median scale length.
The $1\mathchar 45\relax\sigma$ host-to-host scatter is smallest when
measuring the gradients in physical space, which motivates our choice for our
analysis in this paper.

Fig. 15 compares the host-to-host scatter in radial gradients of [O/H] in gas
in our simulated galaxies when scaling these gradients to various galaxy scale
radii at $z=0$. We scale each galaxy’s profile using: $R_{25}$, $R_{50}$, and
$R_{90}$ for the gas and the stars, along with the exponential scale length,
$R_{\rm disk}$, from a 2-component (sérsic plus exponential) fit to the
surface density. Table 1 lists the values for each galaxy. We also compare
these scaled gradients to the gradients in physical radii (as measured in
Section. 3.2. We bin each profile equally in scaled radius, defining the bin
width such that the galaxy with the median scale length has a binwidth of
$0.25\,\textnormal{kpc}$ physical. We measure the radial gradient of each
galaxy across an equal radial range for each scale radius. We define this
radial range such that we measure the galaxy with the median scale length
across a physical range $4-12\,\textnormal{kpc}$. This range corresponds to:
$\approx 3.3-9.8R^{\rm star}_{25}$, $\approx 1.4-4.1R^{\rm star}_{50}$,
$\approx 0.4-1.1R^{\rm star}_{90}$, $\approx 0.5-1.6R^{\rm gas}_{25}$,
$\approx 0.3-1.0R^{\rm gas}_{50}$, $\approx 0.2-0.7R^{\rm gas}_{90}$, and
$\approx 1.0-3.1R_{\rm disk}$.

Measuring the gas abundance radial gradient (from $4-12\,\textnormal{kpc}$) in
physical space minimizes the host-to-host scatter, to $\sigma\approx
0.005\,\textnormal{dex}$. $R^{\rm star}_{25}$ has the next smallest
$1\mathchar 45\relax\sigma$ scatter with $\sigma\approx
0.009\,\textnormal{dex}$. The gradients are the least self-similar when scaled
by $R^{\rm gas}_{90}$, for which the $1\mathchar 45\relax\sigma$ scatter is
$\approx 0.1$. The self-similarity of the radial profiles in physical space
motivates our choice in this paper, because there is no compelling reason to
scale the profiles of our galaxies. We emphasize, though, that this may be a
result of the small mass range of our suite (halo masses are $M_{\rm
200m}=1-3\times 10^{12}\,\textnormal{M}_{\odot}$, stellar masses are in Table
1) and may not be generalizable to galaxies across a wider mass range.

##  Appendix B All gas versus star-forming gas

Figure 16:  A comparison of [O/H] measured in all gas versus only star-forming
gas, at $4<R<12\,\textnormal{kpc}$ and $|Z|<1$ kpc. For each host, we measure
the mean and standard deviation of its [O/H] (stacking 10 consecutive snapshot
across $\approx 200\,\textnormal{Myr}$ to boost the number of star-forming gas
elements), and we compute the galaxy-wide difference between star-forming gas
and all gas. Top panels show histograms of the difference in the mean [O/H],
while bottom panels show histograms of the difference in the standard
deviation of [O/H]. Left panels show $z=0$ and right panels show $z=1$
($t_{\rm lookback}=7.8\,\textnormal{Gyr}$). The solid vertical lines show the
mean of each difference. Star-forming gas is on average more metal rich than
all gas by $\approx 0.04\,\textnormal{dex}$ at $z=1$ and $\approx
0.01\,\textnormal{dex}$ at $z=0$. Furthermore, star-forming gas is slightly
better mixed (with less scatter), with $\sigma_{\rm[O/H]}\approx
0.05\,\textnormal{dex}$ smaller at $z=1$ and $\approx 0.008\,\textnormal{dex}$
smaller at $z=0$.

In this paper, we examine elemental abundances in all gas, as initial
conditions for chemical tagging of stars. We choose to measure all gas in part
because star-forming gas represents only a small fraction of all gas elements
at a given snapshot, leading to significant Poisson noise. In principal, we
could attempt to identify photo-ionized (HII) regions near young star
particles to compare with gas-phase measurements via nebular emission lines,
but doing this correctly requires generating synthetic observations via ray-
tracing, which is beyond the scope of our analysis. In future work (Bellardini
et al., in prep.) we will compare in detail the spatial variations in
abundance of star particles that form out of this gas to the gas itself. Here,
we explore the impact of measuring only star-forming gas instead of all gas.

Fig. 16 compares measuring [O/H] in star-forming versus all gas at $z=1$
($t_{\rm lookback}=7.8\,\textnormal{Gyr}$) and $z=0$. For each galaxy, we
select gas elements at $4<R<12\,\textnormal{kpc}$ and
$|Z|<1\,\textnormal{kpc}$, and we stack this measurement across 10 snapshots
($\approx 200\,\textnormal{Myr}$), because at any single snapshot there are
few star-forming gas elements. For reference, for these same simulations at
$z=0$, Benincasa et al. (2020) find typical GMC lifetimes, and hence lifetimes
of star-forming regions, of $5-7\,\textnormal{Myr}$. We first measure the
difference in the average abundance between star-forming and all gas for each
galaxy. Fig. 16 (top row) shows a histogram of this offset in the mean [O/H].
A positive value means star-forming gas has a higher [O/H] than all gas for
that galaxy. The black vertical line shows the mean value of the histogram. On
average, star-forming gas has modestly higher [O/H] than all gas by $\approx
0.04\,\textnormal{dex}$ at $z=1$ and $\approx 0.01\,\textnormal{dex}$ at
$z=0$. The difference in [O/H] is typically $\lesssim 0.02\,\textnormal{dex}$
for $z=0$ and always less than $0.03\,\textnormal{dex}$. The discrepancy is
larger at higher redshift, the difference is typically $\lesssim
0.04\,\textnormal{dex}$ and always less than $0.13\,\textnormal{dex}$. This is
likely because cosmic accretion and star-formation rates are higher at earlier
times, leading to less efficient small-scale mixing of gas. Of course, a
simple offset in the [O/H] normalization does not alone mean that spatial
variations are different.

Fig. 16 (bottom row) shows the difference in the standard deviation of star-
forming versus all gas. Again, the black line shows the mean value. On
average, [O/H] for star-forming gas has slightly smaller standard deviation
than for all gas. This difference is larger at $z=1$ than at $z=0$. However,
the difference is typically small, $\lesssim 0.05\,\textnormal{dex}$. This
suggests that the azimuthal variations of star-forming gas may be smaller than
that of all gas, especially if the scatter is driven primarily by radial
variations in abundance. Thus, chemically tagging the birth radii of stars may
may be complicated by azimuthal variations for redshifts higher than we show
in Sec. 3.5, which we will explore further in Bellardini et al. in prep.

We also explore the differences in the radial gradients for star-forming
versus all gas (not shown). At $z=0$ the radial gradients of star-forming and
all gas are consistent to within $\pm
0.005\,\textnormal{dex}\,\textnormal{kpc}^{-1}$, less than the host-to-host
scatter in Fig. 5. At $z=1$, for the galaxies with sufficient star-forming gas
to measure a reliable radial gradient, they agree with the gradients for all
gas to within $\pm 0.002\,\textnormal{dex}\,\textnormal{kpc}^{-1}$. We also
compare compare the radial profiles of [O/H] for newly formed stars (in age
bins of $200\,\textnormal{Myr}$) with that of all gas at the same snapshots.
For $z\lesssim 0.5$, these profiles overlap to within uncertainty. For
$z\gtrsim 0.5$, the stellar and gas abundance profiles start to diverge, such
that the young stars tend to have higher [O/H] and flatter gradients than all
gas, which we will explore further in Bellardini et al., in prep.

In summary, while analyzing all gas is a reasonable, if not perfect, proxy for
star-forming gas in our simulations. Given the short lifetimes of star-forming
gas clouds (Benincasa et al., 2020) and the strict conditions for particles to
be star-forming (Hopkins et al., 2018), only a small fraction of gas elements
are star-forming at a given snapshot ($\lesssim 1\%$ of gas particles at the
redshifts we observe), so analyzing all gas greatly reduces the statistical
uncertainty. In the future we will study the abundances of star particles that
form from this gas, to compare with these results in detail.

##  Appendix C Impact of diffusion coefficient

Figure 17:  Vertical (left), radial (middle), and azimuthal variations (right)
in [O/H] between our fiducial simulation of m12i, a version with no subgrid
metal diffusion, and a re-simulation increasing the diffusion coefficient by
$10$ times. The vertical profiles show no clear systematic variations at a
level important for our analysis. We normalize the radial profiles to
$4\,\textnormal{kpc}$ (the approximate edge of the bulge) for clarity in
comparison. The radial gradients (measured from $4-12\,\textnormal{kpc}$ for
$z=0$, $2-8\,\textnormal{kpc}$ for $z=1$) vary by no more than $\approx
0.005\,\textnormal{dex}\,\textnormal{kpc}^{-1}$ between our fiducial
simulation and the simulation with $10$ times higher metal diffusion, while
the simulation with no metal diffusion has a $\approx
0.13\,\textnormal{dex}\,\textnormal{kpc}^{-1}$ steepe gradient at $z=0$. In
the right panel, we scaled down the azimuthal scatter in the simulation with
no metal diffusion by a factor of $10$, for comparison. Thus, neglecting metal
diffusion/mixing leads to $10\times$ higher azimuthal scatter, and moreover,
this scatter does not depend much on azimuthal scale. The enhanced metal
diffusion re-simulation shows smaller azimuthal scatter at small azimuthal
scales, given the enhanced mixing rate on these small scales. However, that
simulation shows similar scatter at large azimuthal scales, indicating that
disk-wide azimuthal scatter is not sensitive to the detailed choice of
diffusion coefficient.

Our FIRE-2 simulations model the sub-grid diffusion/mixing of metals in gas
via unresolved turbulent eddies (Su et al., 2017; Escala et al., 2018; Hopkins
et al., 2018):

$\frac{\partial Z_{i}}{\partial t}+\nabla\cdot(D\nabla Z_{i})=0$ (2)

where $Z_{i}$ is the mass fraction of a metal in gas element $i$, and $D$ is
the diffusion coefficient. While there is some uncertainty in the exact value
to choose for this coefficient, our fiducial value is physically motivated
based on tests of the metal diffusion implementation in FIRE-2 on idealized,
converged turbulent box simulations by Colbrook et al. (2017) and other more
extensive studies by Rennehan et al. (2019). Here, we compare our key results
using our fiducial diffusion coefficient $D$ in m12i against a re-simulations
of m12i with all identical physics/parameters, except one has a diffusion
coefficient that is $10$ times higher (that is, faster mixing) and the other
includes no subgrid mixing.

Fig. 17 compares the vertical, radial, and azimuthal variations for m12i. The
left panel shows the vertical gradient in gas [O/H], similar to Fig. 6. At
$z=0$, we find no differences within $200\,\textnormal{pc}$ and at most $\sim
0.015\,\textnormal{dex}$ difference at $1\,\textnormal{kpc}$. The differences
are stronger at $z=1$ for $10\times$ higher diffusion and stronger at $z=0$
for the simulation with no diffusion, though again, not at a significant level
to change our interpretations, especially within $200\,\textnormal{pc}$.

Fig. 17 (center) shows the radial profile in gas [O/H], normalized to the
abundance at $R=4\,\textnormal{kpc}$ (given a strong upturn at smaller $R$).
The radial gradients, measured over our fiducial radial ranges, vary by
$\lesssim 0.005\,\textnormal{dex}\,\textnormal{kpc}^{-1}$ between the
$10\times$ diffusion simulation and the fiducial simulation. The gradients
vary by less than $0.014\,\textnormal{dex}$ between the fiducial simulation
and the one with no metal diffusion at $z=0$. The simulation with no subgrid
diffusion has a steeper gradient at $z=0$, potentially because the metals are
less efficient at spreading from a given radius, in the absence of subgrid
diffusion, once the disk has become rotationally dominated and radial
turbulence is no longer efficient at moving the gas particles. We thus
conclude that the radial gradients are reasonably robust to choices of the
strength of the diffusion coefficient, however, in the unphysical case of no
subgrid diffusion, the gradient can be (unphysically) steeper.

Fig. 17 (right) compares the azimuthal variations versus angular bin width, at
$R=8\,\textnormal{kpc}$. The simulation with no subgrid diffusion has
$10\times$ higher azimuthal scatter, so we scale down its values in Fig. 17 by
$10\times$ for visual comparison. Using no subgrid diffusion leads to scatter
that is largely independent of azimuthal scale at high $z$, but that increases
with azimuthal scale at low $z$. Without subgrid diffusion, a small number of
gas elements can absorb most of the metals, while a significant number of
(neighboring) elements can remain nearly un-enriched. This is a patently
unphysical scenario, and it yields azimuthal scatter that disagrees with
observations by an order of magnitude. At both redshifts, using a higher
diffusion coefficient leads to smaller azimuthal variations at small scales,
because diffusion smooths variations between nearby gas elements on scales
approaching the resolution (Escala et al., 2018). However, the azimuthal
variations are nearly unchanged on large azimuthal scales. Therefore, our
results on small azimuthal scales are likely sensitive to the exact choice of
diffusion coefficient, but the large-scale azimuthal variations are robust. An
important caveat to this comparison is that it is only one simulated galaxy,
and individual simulations with the same initial conditions and physics can
show non-trivial stochastic variations from random number generators,
floating-point roundoff, and chaotic behavior (e.g. Keller et al., 2019).
Indeed, we find minor fluctuations between these simulations, for example, in
the exact timing of mergers, which can affect all panels in Fig. 17. We
consider it likely that the differences in azimuthal variations on small
scales are robust, but any other differences in Fig. 17 are potentially
stochastic.

  *[MW]: Milky Way
  *[LG]: Local Group
  *[FIRE]: Feedback In Realistic Environments
  *[MFM]: Meshless Finite Mass
  *[DM]: Dark Matter
  *[ISM]: interstellar medium
  *[GMC]: giant molecular clouds
  *[AMR]: adaptive mesh refinement


# From FreEM to D’AlemBERT: a Large Corpus and a Language Model for Early
Modern French

###### Abstract

Language models for historical states of language are becoming increasingly
important to allow the optimal digitisation and analysis of old textual
sources. Because these historical states are at the same time more complex to
process and more scarce in the corpora available, specific efforts are
necessary to train natural language processing (NLP) tools adapted to the
data. In this paper, we present our efforts to develop NLP tools for Early
Modern French (historical French from the 16th to the 18th centuries). We
present the FreEM _max_ corpus of Early Modern French and D’AlemBERT, a
RoBERTa-based language model trained on FreEM _max_. We evaluate the
usefulness of D’AlemBERT by fine-tuning it on a part-of-speech tagging task,
outperforming previous work on the test set. Importantly, we find evidence for
the transfer learning capacity of the language model, since its performance on
lesser-resourced time periods appears to have been boosted by the more
resourced ones. We release D’AlemBERT and the open-sourced subpart of the
FreEM _max_ corpus.

Keywords: Digital humanities, Early Modern French, Language modelling, Neural
language representation models, Less-resourced languages, Corpus creation, POS
tagging

languageresourceLanguage Resources

From FreEM to D’AlemBERT: a Large Corpus and a Language Model for Early Modern
French

Simon Gabay3, Pedro Ortiz Suarez1,2, Alexandre Bartz2, Alix Chagué1
---
Rachel Bawden1, Philippe Gambette4, Benoît Sagot1
Inria1, Sorbonne Université2, Université de Genève3, LIGM, Université Gustage
Eiffel, CNRS4
2 rue Simone Iff, 75012 Paris (France)1, 21 rue de l’École de médecine, 75006
Paris (France)2,
rue du Général-Dufour 24, 1211 Genève (Switzerland)3,
5 boulevard Descartes, F-77454 Champs-sur-Marne (France)4
{pedro.ortiz, alix.chague, benoit.sagot<EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>

Abstract content

## 1\. Introduction

With the rise of digital humanities, it is becoming increasingly important to
develop high quality tools to automatically process old states of languages.
Libraries, archives and museums, among others, are digitising large numbers of
historical sources, from which high quality data must be extracted for further
study by specialists of human sciences following new approaches such as
“distant reading” [Moretti, 2013]. Many (sub)tasks such as automatic OCR post-
correction [Rijhwani et al., 2021] and linguistic annotation [Camps et al.,
2020] benefit from pretrained language models to improve their accuracy, and
this is what motivated us to develop a BERT-like [Devlin et al., 2019]
contextualised language model for Early Modern French.

Languages evolve over time on many different levels: from one century to
another, we see variations in spelling, syntax, the lexicon etc. However this
variation is not uniform: it tends, at least for “literate scriptors”
(literature, journalism, law, etc.), to converge towards a single norm over
time, and this has especially been the case for French because of the
prominent role of the Académie française and the _remarqueurs_ [Ayres-Bennett
and Seijido, 2011]. The result of this convergence is, for instance, that
spelling and word order within sentences have become more strict, where they
were less so in the past. From a computational perspective, historical states
of language are therefore not only different from the contemporary state, but,
from a computational perspective, are also more complex because they do not
follow a strict and explicit norm. In French, this explicit norm appeared in
the 17th c. and was slowly integrated throughout the 18th c.

On top of this first linguistic problem, a second issue appears: because the
production of textual sources has continued to grow exponentially, it is
easier to collect a corpus for contemporary French than for the 19th c.
French, which is itself easier than for the 18th c. French, etc. The further
we go back in time, the more scarce resources are, which creates the following
paradox: we have more data when the language is homogeneous and simple for the
computer to process, and less when it is heterogeneous and harder to process.

The following paper will address the development of D’AlemBERT a neural
language model in a complex setting, defined here as the state of language
with scarce heterogeneous resources. We will also present FreEM _max_ the data
used to train the model, discuss its conception, and evaluate its efficiency
with a classical natural language processing (NLP) task, part-of-speech (POS)
tagging, crucial for corpus linguistics and the digital humanities. We release
both the D’AlemBERT model and a subset of the FreEM _max_ dataset that we were
allowed to open source by the original authors.

## 2\. Related Work

Large datasets for historical states of languages or extinct languages do
exist. The _Corpus Middelnederlands_ for Medieval Dutch Reenen_corpus_1998 and
the _Base Geste_ for Medieval French camps_geste_2016 are freely available
online, encoded in TEI. It is also the case for other corpora for later states
of language, such as the Reference corpus of historical Slovene, covering
approximately three centuries of Slovene (1584–1899) erjavec_reference_2015,
and the “corpus noyau” of _Presto_ presto_corpus_2018. This last corpus, in
its extended version, uses other French corpora such as Espistemon for
Renaissance French demonet_epistemon_1998 and the University of Chicago’s
_American and French Research on the Treasury of the French Language_ (ARTFL)
morrissey_artfl_1981; or like Frantext atilf_frantext_1998, which is a
generalist French corpus, covering the different states of the French language
between the 11th and the 21st century. Although most of these text collections
are free, the two biggest ones, Frantext and ARTFL, are not freely available
or open-sourced.

Concerning language modelling in French, two main models are available for
contemporary French, CamemBERT [Martin et al., 2020] and FlauBERT [Le et al.,
2020]. CamemBERT was trained on a freely available, automatically web-crawled
corpus called OSCAR [Ortiz Suárez et al., 2019, Ortiz Suárez et al., 2020]
while FlauBERT was trained on a mix of web-crawled data and manually curated
(partly non freely available) contemporary French corpora. Neither of these
models was explicitly pre-trained for historical French.111Note however that
texts in Old, Middle and Modern French do exist in the internet, and might
have found their way to the training corpus of these two models. This is
especially the case for Modern French texts, which automatic language
classification tools can easily classify as Contemporary French. However
efficient language models have been trained for less-resourced or extinct
Languages such as Latin [Bamman and Burns, 2020], following the approach of ?)
for training language models with less data than was previously thought. There
have also been some recent projects that specifically target Early Modern
French such as that of Pie Extended [Clérice, 2020] that uses the hierarchical
encoding architecture originally proposed by ?) which itself is constructed by
stacking multiple Bi-LSTM-CRFs. ?) distributes pre-trained models for POS
tagging and lemmatisation.

## 3\. Corpora

Source |  Normalised |  Translation
---|---|---
Surquoy, SIRE, s’il plait à votre Maieté de e ouuenir des mieres de on Etat, dõt au moins ell’a tiré cét aduantage, qu’en vne grande ieunese ell’a acquis vne grande experiẽce, elle verra que tous les mal-heurs de sõ bas âge ont pris leur commencement en emblables occaions; |  Sur quoi, SIRE, s’il plaît à votre Majesté de se souvenir des misères de son état dont au moins elle a tiré cet avantage, qu’en une grande jeunesse elle a acquis une grande expérience, elle verra que tous les malheurs de son bas âge ont pris leur commencement en semblables occasions ; |  “Whereupon, SIR, if it pleases your Majesty to remember the miseries of her state, from which at least she has derived this advantage, that in great youth she has acquired great experience, she will see that all the misfortunes of her early life took their beginning on similar occasions;”

Table 1: Example of normalisation taken from the Lettres of ?).

For the past few years, we have been involved in the development of linguistic
resources for Early Modern French. The initiative, called FreEM (which stands
for FREnch Early Modern), aims to collect the corpora required for various NLP
tasks such as lemmatisation, POS tagging, linguistic normalisation and named
entity recognition. Two of these corpora are introduced here: FreEM _max_ (see
Section 3.2) and FreEM _LPM_ (see Section 3.3).

### 3.1. Early Modern French

Experiments are based on data of which the core comprises Early Modern French
literary texts. We loosely define Early Modern French as a state of language
following Middle French in 1500—following here the terminus ad quem used by
the Dictionnaire de Moyen Français [Martin, Robert (dir.), 2020]—and ending
with the French Revolution in 1789. It therefore encompasses three centuries
(16th, 17th and 18th c.), or two linguistic periods: the français préclassique
or “preclassical French”, 1500–1630 and the français classique or “classical
French”, 1630–1689; both periodisations are currently used in French
linguistics (e.g. by ?) and ?)).

A typical example of Early Modern French, taken from ?), is given in Table 3.
We note here the presence of several phenomena that have now disappeared in
contemporary French, such as the presence of abbreviations (dõt$\to$dont), the
long s (, see mieres), the use of v instead of u (vne for une), the
conservation of etymological letters (votre$<$Latin vŏster rather than votre)
and calligraphic letters (-y in Surquoy), the absence of welding (mal-heurs
and not malheurs) and the opposite (Surquoy and not Sur quoi).

For NLP tasks, which process raw sequences, such differences with respect to
contemporary French are not trivial, and they prevent the processing of
historical texts with tools trained on recent sources.

### 3.2. FreEM _max_

Usable historical documents are difficult to find because, as previously
mentioned, they are more rare than contemporary ones; editors tend to
normalise the language (i.e. use the spelling conventions of contemporary
French, see [Gabay, 2014]), transcriptions are not (always) distributed in a
digital format. FreEM _max_ FREEMmax is an attempt to solve this problem, and
the aim of this dataset is to group together the largest number or texts
possible written in Early Modern French. The texts have a variety of sources,
which can be grouped into three main types:

  * •

Two institutional datasets have been used and are non open-sourced:

    * –

Frantext intégral atilf_frantext_1998, the biggest database of French texts
(only the texts between 1500 and 1800), a very small portion of which is open
access: Frantext Démonstration atilf_frantext_1998-1;

    * –

Electronic Enlightenment bodleian_libraries_electronic_2008, an online
collection of edited correspondences of the Early Modern period;

  * •

Several come from research projects distributing transcriptions online:

    * –

The Antonomaz project, French mazarinades
(https://cahier.hypotheses.org/antonomaz);

    * –

The II.B section (in French) of the Actis Pacis Westphalicae, diplomatic
letters for the Peace of Westphalia (http://kaskade.dwds.de/dstar/apwcf/);

    * –

The Bibliothèques virtuelles humanistes, 16th c. French literature
(http://www.bvh.univ-tours.fr);

    * –

The Corpus électronique de la première modernité, 17th c. French literature
(http://www.cepm.paris-sorbonne.fr)

    * –

The Condé project, coutumiers normands (https://conde.hypotheses.org)

    * –

The Corpus Descartes, works of René Descartes
(https://www.unicaen.fr/puc/sources/prodescartes/);

    * –

The Bibliothèque dramatique of the CELLF, 17th c. French plays
(http://bibdramatique.huma-num.fr);

    * –

The Fabula numerica project, French fables (https://obvil.sorbonne-
universite.fr/projets/fabula-numerica);

    * –

The  Fonds Boissy, plays of Louis de Boissy (https://www.licorn-
research.fr/Boissy.html);

    * –

The Mercure Galant project, the famous French gazette and literary magazine
between 1672 and 1710 (https://obvil.sorbonne-universite.fr/corpus/mercure-
galant);

    * –

The Rousseau online project, works of Jean-Jacques Rousseau
(https://www.rousseauonline.ch);

    * –

The Sermo project, sermons of the 16th and 17th c. (http://sermo.unine.ch);

    * –

The Théâtre classique project, 17th and 18th c. French plays
(http://www.theatre-classique.fr);

  * •

Additional sources come from researchers who kindly accepted to offer their
personal transcriptions or data scrapped by our team:

    * –

Transcriptions of Anne-Élisabeth Spica (17th c. French novels);

    * –

Transcriptions found on Wikisource (https://fr.wikisource.org);

    * –

Transcriptions (ePub files) found on Gallica (https://gallica.bnf.fr);

    * –

Transcriptions found on various websites online.

Additional data for later states of the language, up to the 1920’s (mainly
from FRANTEXT intégral), are also provided for two main reasons: on the one
hand, it is common to normalise Early Modern French into Contemporary French
[Gabay, 2014] because of the linguistic proximity between these the two states
of the language, and on the other hand, it helps to collect (precious)
additional data to avoid ending up with with too small of a corpus for our
needs.

Origin | #Tokens
---|---
Spica corpus | 691,467
Antonomaz project | 119,194
Acta Pacis Westphlicae II B | 2,463,047
Bibliothèque Bleue | 776,838
BVH | 2,434,657
CEPM | 2,707,432
Condé project | 3,173,845
Descartes | 1,025,337
CELLF | 1,873,772
Electronic enlightenment | 6,568,047
Fabula project | 145,978
Frantext intégral ($>$1500, $<$1800) | 60,018,390
Frantext intégral ($>$1800) | 71,504,440
Frantext Démonstration | 1,255,454
Gallica | 5,212,333
Boissy project | 438,215
Mercure galant | 5,427,469
Rousseau Online project | 2,428,587
Scrapping | 1,936,835
Sermo project | 529,647
Théâtre classique project | 13,916,169
Wikisource | 996,329
TOTAL | 185,643,482
Table 2: Breakdown of the FreEM _max_ corpus by text origin.

The final result is far from being balanced or representative (see Figure 1).
16th c. French documents are under-represented, as well as 18th c. literature.
The 17th c. is clearly over-represented, especially its second half—probably
one of the most important of French literature, which could explain this
situation (on top of our personal interest for this specific period).

Figure 1: Distribution of the documents in the FreEM _max_ corpus per year

As some texts are still (partially) protected by restrictive licences, the
FreEM _max_ corpus exists in both open and non-open versions, only the open
one being distributed. In order to limit the impact of licences forbidding the
modification of files, we have designed a pipeline to distribute the data as
it was found and recreate it (see Figure 2).

Metadata is prepared manually in order to have the same categories for each
document, whatever its origin. As well as the author, the title and the date
(where relevant), we also provide the genre (“theatre”), sometimes a subgenre
(“tragedy”), the linguistic status (normalised or not) and the licence
attached to the transcription.

Figure 2: FreEM _max_ compilation pipeline. All files are kept in their
original format. Metadata is manually prepared in separate files in order to
automatically transform and clean (in blue) all the available documents into
XML TEI files following the same encoding. It allows us to distribute open
data (in green) but also data distributed with restrictions regarding the
modification of the original format (in orange). Non-open texts (in red) are
not distributed.

### 3.3. FreEM _LPM_

The FreEM _LPM_ (“Lemma, POS tags, Morphology”) has already been presented
[Gabay et al., 2020 10]. The POS-annotated data, is a mixture of two different
sources. On the one hand, there is the CornMol corpus [Camps et al., 2020],
made up of normalised 17th c. French comedies. On the other hand, there is a
gold subset of the Presto corpus [Blumenthal et al., 2017], made up of texts
of different genres written during the 16th, 17th and 18th c., which have
previously used to train annotation tools [Diwersy et al., 2017], and was
heavily corrected by us to match our annotation principles [Gabay et al.,
2020].

On top of traditional in-domain tests, an out-of-domain testing dataset was
prepared to control the capacity of the model to generalise to other genres
and periods. Centuries covered are the 16th, 17th, 18th, 19th and 20th. There
are two test sets for each century: one made up only of theatre, the other of
everything but theatre. Each test set comprises 10 short samples (c. 100
tokens), as representative as possible of the linguistic production of the
century (female and male authors, decade of publication, genre, etc.).

All the data from FreEM _LPM_ (but almost none of the out-of-domain) can be
found in FreEM _max_.

## 4\. D’AlemBERT: a neural language model for Early Modern French

In this section, we describe the pretraining data, architecture, training
objective and optimisation setup we use for D’AlemBERT, our new neural
language model for Early Modern French.

### 4.1. Pre-processing

Similar to RoBERTa [Liu et al., 2019] we segment the input text data into
subword units using Byte-Pair encoding (BPE) [Sennrich et al., 2016] in the
implementation proposed by [Radford et al., 2019] that uses bytes instead of
unicode characters as the base subword units. The BPE encoding does not
require pre-tokenisation (at the word or token level), thus removing the need
to develop a specific tokeniser for Early Modern French. We use a vocabulary
size of 32,768 subword tokens. These subwords are learned on the entire FreEM
_max_ dataset.

### 4.2. Language Modelling

Original
---
Model | 16 | 17 | 18 | 19 | 20 | Avg
 _Drama_
Pie Extended | _90.34_ | _94.47_ | _94.64_ | - | - | _93.15_
CamemBERT | 87.06 | 89.01 | 90.92 | - | - | 89.00
D’AlemBERT | 94.17 | 96.59 | 96.28 | - | - | 95.68
 _Varia_
Pie Extended | _89.85_ | _93.44_ | _95.98_ | - | - | _93.09_
CamemBERT | 86.90 | 88.85 | 92.85 | - | - | 89.53
D’AlemBERT | 93.86 | 95.73 | 96.95 | - | - | 95.51
 _Both_
Pie Extended | _90.08_ | _93.95_ | _95.33_ | - | - | _93.12_
CamemBERT | 86.98 | 88.93 | 91.89 | - | - | 89.27
D’AlemBERT | 94.02 | 96.16 | 96.62 | - | - | 95.60
Normalised or Contemporary
---
Model | 16 | 17 | 18 | 19 | 20 | Avg
 _Drama_
Pie Extended | _93.69_ | _95.75_ | _95.61_ | _95.03_ | _93.71_ | _94.76_
CamemBERT | 90.18 | 91.51 | 91.37 | 91.13 | 91.42 | 91.12
D’AlemBERT | 96.25 | 96.97 | 96.80 | 96.25 | 95.00 | 96.25
 _Varia_
Pie Extended | _92.52_ | _94.81_ | _95.98_ | _92.24_ | _94.03_ | _93.94_
CamemBERT | 89.79 | 90.69 | 93.06 | 90.54 | 89.78 | 93.94
D’AlemBERT | 94.52 | 96.64 | 96.88 | 94.90 | 95.30 | 95.65
 _Both_
Pie Extended | _93.08_ | _95.28_ | _95.80_ | _93.65_ | _93.87_ | _94.35_
CamemBERT | 89.99 | 91.10 | 92.22 | 90.84 | 90.60 | 92.53
D’AlemBERT | 95.39 | 96.81 | 96.84 | 95.58 | 95.15 | 95.95

Table 3: Comparison between D’AlemBERT, CamemBERT and Pie Extended performance
on FreEM _LPM_.

#### Transformer

D’AlemBERT uses the exact same architecture as RoBERTa, which is a multi-layer
bidirectional Transformer [Vaswani et al., 2017]. D’AlemBERT uses the original
_base_ architecture of RoBERTa (12 layers, 768 hidden dimensions, 12 attention
heads, 110M parameters).

#### Pretraining Objective

We train our model on the Masked Language Modelling (MLM) task as proposed by
RoBERTa’s authors [Liu et al., 2019]: given an input text sequence composed of
$N$ tokens $x_{1},...,x_{N}$, we select 15% of tokens for possible
replacement. Among those selected tokens, 80% are replaced with the special
<MASK> token, 10% are left unchanged and 10% are replaced by a random token.
The model is then trained to predict the masked tokens using cross-entropy
loss.

Again, following the RoBERTa approach, we dynamically mask tokens instead of
fixing them statically for the whole dataset during preprocessing. We also
choose not to use the next sentence prediction (NSP) task originally used in
BERT [Devlin et al., 2019], as it has been shown that it does not improve
downstream task performance [Conneau and Lample, 2019, Liu et al., 2019].

#### Optimisation

We optimise our model in the exact same way as [Liu et al., 2019] using Adam
[Kingma and Ba, 2015] ($\beta_{1}=0.9$, $\beta_{2}=0.98$) for 100k steps with
large batch sizes of 8,192 sequences, each sequence containing at most 512
tokens.

#### Pre-training

We use the RoBERTa implementation in the Zelda Rose
library,222https://github.com/LoicGrobol/zeldarose and again, in the same way
as ?) our learning rate is warmed up for 10k steps up to a peak value of
$0.0003$ instead of the original $0.0001$ used by the original implementation
of RoBERTa [Liu et al., 2019], as our model diverged with the $0.0001$ value.
We hypothesise that this is either due to the smaller size of FreEM _max_
(compared to the corpora used for RoBERTa or CamemBERT) or to our large batch
size. We train our model for 31k steps, which amounts to 41 epochs. The total
pre-training times, the details of the infrastructure we used and even the
carbon emissions of our model are reported in Appendix A.

## 5\. Evaluation and Discussion

In order to evaluate our D’AlemBERT model, we fine-tune it for POS tagging on
the FreEM _LPM_ corpus. We use the flair
framework333https://github.com/flairNLP/flair for sequence tagging [Akbik et
al., 2019]. To fine-tune D’AlemBERT for POS we follow the same approach as ?)
with some modifications: we append a linear layer of size 256 that takes as
input the last hidden representation of the <s> special token and the mean of
the last hidden representation of the subword units of each token (token as
defined for FreEM _LPM_), that is, we use a _“mean”_ subword pooling strategy.
We fine-tune D’AlemBERT with a learning rate of 0.000005 for a total of 10
epochs. We also fine-tune CamemBERT using the exact same hyperparameters as
that we use for D’AlemBERT.

FreEM _LPM_ provides a standard split (train, dev, test), however it also
proposes an evaluation on a _out-of-domain_ subcorpus that is not contained in
the standard split and that is separated by century (from the 16th to the 20th
century) and that also contains both the _Normalised_ and _Original_ versions
of the texts for the 16th, 17th and 18th centuries. The idea of this out-of-
domain evaluation corpus is to have a fine-grained evaluation of the models to
better assess their performance in all the different types of text that one
might encounter when working with Early Modern French data.

Following the approach of ?), we report the scores obtained on the out-of-
domain testing dataset of FreEM _LPM_ in Table 3. We use the scores previously
reported by ?) using _Pie Extended_ as our baseline as well as the fine-tuned
CamemBERT that serves as a second baseline as well as a rough estimation of
how much knowledge can D’AlemBERT transfer from the FreEM _max_ into this
task.

We can see that D’AlemBERT consistently outperforms Pie Extended and CamemBERT
in both the normalised and original versions of our out-of-domain testing data
and for all different periods by a considerable margin. We can also see that
on average the difference in score between D’AlemBERT and Pie Extended is
greater for the original split than the normalised one. This suggests that
D’AlemBERT can generalise more effectively to non-normalised data than the
more traditional architecture used by Pie Extended. Moreover we can also see
that the difference in scores is also greater for the 16th c. and 17th c.
data. This is interesting, especially for the 16th c, because, as we can see
in Figure 1, this is the least represented period in the FreEM _max_ corpus.
This result actually suggests that D’AlemBERT might be able to do effective
transfer learning from the 18th c., 19th c. and 20th c. data to the 16th c.
and 17th c. data.

As for CamemBERT, we can see that it consistently scores lower than both
D’AlemBERT and Pie Extended. Moreover, we can see that it struggles
particularly with the non-normalised data of the 16th c., 17th c. and 18th c..
This results clearly shows that CamemBERT cannot easily generalised to these
earlier states of languages, or at least not with the quantity of data found
in the training set of FreEM _LPM_. These results also show the impressive
capacity of D’AlemBERT of quickly generalising to diverse set of states of
language, as well as its capacity to transfer knowledge from the FreEM _max_
corpus into this task. The obtained results are also a testament to the
importance of the pre-training data, specially taking in account that the pre-
training set of CamemBERT is more than 100 times bigger than that of
D’AlemBERT.

## 6\. Conclusion

In this paper we presented the manually curated FreEM _max_ corpus of Early
Modern French as well as D’AlemBERT, a RoBERTa-based language model trained on
FreEM _max_. With D’AlemBERT, we showed that it is possible to successfully
train a transformer-based language model for historical French with even less
data than originally shown in previous works [Martin et al., 2020]. Moreover
with our POS tagging evaluation we were able to observe some form of transfer
learning and generalisation across multiple states of the language
corresponding to different periods of time. Both our corpus and our model will
be of use to digital humanists and linguists interested in Early Modern
French. For our future work, we hope that will be able to study the
application of our D’AlemBERT model to other NLP tasks such as text
normalisation, named entity recognition and even document structuring, where
we hope to more extensively study the transfer learning capabilities of our
approach.

## 7\. Acknowledgements

We would like to warmly thank Karine Abiven, Bertrand Gaiffe, Annette
Gerstenberg, Pierre Larrivée, Gaël Lejeune, Laurent Romary, Anne-Élisabeth
Spica and Martin Wynne for their help in gathering the data. This work was
also performed using HPC resources from GENCI-IDRIS (Grant
2021-AD011011330R1).

This work was also partly funded by Rachel Bawden’s and Benoît Sagot’s chairs
in the PRAIRIE institute funded by the French national agency ANR as part of
the “Investissements d’avenir” programme under the reference ANR-19-P3IA-0001,
as well as the BASNUM ANR project (ANR-18-CE38-0003).

## 8\. Bibliographical References

## References

  * Akbik et al., 2019 Akbik, A., Bergmann, T., Blythe, D., Rasul, K., Schweter, S., and Vollgraf, R.  (2019).  FLAIR: An easy-to-use framework for state-of-the-art NLP.  In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations), pages 54–59, Minneapolis, Minnesota, June. Association for Computational Linguistics.
  * Amatuzzi et al., 2019 Amatuzzi, A., Skupien Dekens, C., Ayres-Bennett, W., Gerstenberg, A., and Schoesler, L.  (2019).  Améliorer et appliquer les outils numériques. ressources et approches pour l’étude du changement linguistique en français préclassique et classique.  In Le français en Diachronie, Travaux de Linguistique Romane, pages 337–364. Editions de linguistique et de philologie.
  * Ayres-Bennett and Seijido, 2011 Wendy Ayres-Bennett et al., editors.  (2011).  Remarques et observations sur la langue française. Histoire et évolution d’un genre.  Number 1 in Histoire et évolution du français. Classiques Garnier.
  * Bamman and Burns, 2020 Bamman, D. and Burns, P. J.  (2020).  Latin BERT: A contextual language model for classical philology.  CoRR, abs/2009.10053.
  * Bender et al., 2021 Bender, E. M., Gebru, T., McMillan-Major, A., and Shmitchell, S.  (2021).  On the dangers of stochastic parrots: Can language models be too big?  In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAccT ’21, page 610–623, New York, NY, USA. Association for Computing Machinery.
  * Blumenthal et al., 2017 Blumenthal, P., Diwersy, S., Falaise, A., Lay, M.-H., Souvay, G., and Vigier, D.  (2017).  Presto, un corpus diachronique pour le français des XVIe-XXe siècles.  In Actes de la 24ème conférence sur le Traitement Automatique des Langues Naturelles - TALN’17. Association pour le traitement automatique des langues.
  * Camps et al., 2020 Camps, J.-B., Gabay, S., Fièvre, P., Clérice, T., and Cafiero, F.  (2020).  Corpus and Models for Lemmatisation and POS-tagging of Classical French Theatre.  Journal of Data Mining & Digital Humanities.
  * Clérice, 2020 Clérice, T.  (2020).  Pie extended, an extension for pie with pre-processing and post-processing, June.
  * Conneau and Lample, 2019 Conneau, A. and Lample, G.  (2019).  Cross-lingual language model pretraining.  In H. Wallach, et al., editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc.
  * Desrochers et al., 2016 Desrochers, S., Paradis, C., and Weaver, V. M.  (2016).  A validation of dram rapl power measurements.  In Proceedings of the Second International Symposium on Memory Systems, MEMSYS ’16, page 455–470, New York, NY, USA. Association for Computing Machinery.
  * Devlin et al., 2019 Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K.  (2019).  BERT: Pre-training of deep bidirectional transformers for language understanding.  In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota, June. Association for Computational Linguistics.
  * Diwersy et al., 2017 Diwersy, S., Falaise, A., Lay, M.-H., and Souvay, G.  (2017).  Ressources et méthodes pour l’analyse diachronique.  Langages, N° 206(2):21–44.
  * Gabay et al., 2020 Gabay, S., Camps, J.-B., and Clérice, T.  (2020).  Manuel d’annotation linguistique pour le français moderne (XVIe -XVIIIe siècles).
  * Gabay et al., 2020 10 Gabay, S., Clérice, T., Camps, J.-B., Tanguy, J.-B., and Gille-Levenson, M.  (2020-10).  Standardizing linguistic data: method and tools for annotating (pre-orthographic) French.  In Proceedings of the 2nd International Digital Tools & Uses Congress (DTUC ’20), Hammamet, Tunisia.
  * Gabay, 2014 Gabay, S.  (2014).  Pourquoi moderniser l’orthographe? principes d’ecdotique et littérature du XVIIe siècle.  Vox Romanica, 73(1):27–42.
  * Guez de Balzac, 1624 Guez de Balzac, J.-L.  (1624).  Lettres du sieur de Balzac.  T. Du Bray.
  * Kingma and Ba, 2015 Kingma, D. P. and Ba, J.  (2015).  Adam: A method for stochastic optimization.  In Yoshua Bengio et al., editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
  * Le et al., 2020 Le, H., Vial, L., Frej, J., Segonne, V., Coavoux, M., Lecouteux, B., Allauzen, A., Crabbé, B., Besacier, L., and Schwab, D.  (2020).  FlauBERT : des modèles de langue contextualisés pré-entraînés pour le français (FlauBERT : Unsupervised language model pre-training for French).  In Actes de la 6e conférence conjointe Journées d’Études sur la Parole (JEP, 33e édition), Traitement Automatique des Langues Naturelles (TALN, 27e édition), Rencontre des Étudiants Chercheurs en Informatique pour le Traitement Automatique des Langues (RÉCITAL, 22e édition). Volume 2 : Traitement Automatique des Langues Naturelles, pages 268–278, Nancy, France, 6. ATALA et AFCP.
  * Liu et al., 2019 Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V.  (2019).  RoBERTa: A Robustly Optimized BERT Pretraining Approach.  arXiv e-prints, page arXiv:1907.11692, July.
  * Manjavacas et al., 2019 Manjavacas, E., Kádár, Á., and Kestemont, M.  (2019).  Improving lemmatization of non-standard languages with joint learning.  In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1493–1503, Minneapolis, Minnesota, June. Association for Computational Linguistics.
  * Martin et al., 2020 Martin, L., Muller, B., Ortiz Suárez, P. J., Dupont, Y., Romary, L., de la Clergerie, É., Seddah, D., and Sagot, B.  (2020).  CamemBERT: a tasty French language model.  In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7203–7219, Online, July. Association for Computational Linguistics.
  * Martin, Robert (dir.), 2020 Martin, Robert (dir.).  (2020).  Dictionnaire du Moyen Français.  ATILF - CNRS & Université de Lorraine.
  * Moretti, 2013 Moretti, F.  (2013).  Distant reading.  Verso.
  * Ortiz Suárez et al., 2019 Ortiz Suárez, P. J., Sagot, B., and Romary, L.  (2019).  Asynchronous pipelines for processing huge corpora on medium to low resource infrastructures.  In Piotr Bański, et al., editors, Proceedings of the Workshop on Challenges in the Management of Large Corpora (CMLC-7) 2019. Cardiff, 22nd July 2019, pages 9–16, Mannheim. Leibniz-Institut für Deutsche Sprache.
  * Ortiz Suárez et al., 2020 Ortiz Suárez, P. J., Romary, L., and Sagot, B.  (2020).  A monolingual approach to contextualized word embeddings for mid-resource languages.  In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1703–1714, Online, July. Association for Computational Linguistics.
  * Radford et al., 2019 Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I.  (2019).  Language models are unsupervised multitask learners.  OpenAI Blog, 1:8.
  * Rijhwani et al., 2021 Rijhwani, S., Rosenblum, D., Anastasopoulos, A., and Neubig, G.  (2021).  Lexically aware semi-supervised learning for OCR post-correction.  Transactions of the Association for Computational Linguistics, 9:1285–1302.
  * Schwartz et al., 2020 Schwartz, R., Dodge, J., Smith, N. A., and Etzioni, O.  (2020).  Green ai.  Commun. ACM, 63(12):54–63.
  * Schweter and Akbik, 2020 Schweter, S. and Akbik, A.  (2020).  FLERT: Document-Level Features for Named Entity Recognition.  arXiv e-prints, page arXiv:2011.06993, November.
  * Sennrich et al., 2016 Sennrich, R., Haddow, B., and Birch, A.  (2016).  Neural machine translation of rare words with subword units.  In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715–1725, Berlin, Germany, August. Association for Computational Linguistics.
  * Strubell et al., 2019 Strubell, E., Ganesh, A., and McCallum, A.  (2019).  Energy and policy considerations for deep learning in NLP.  In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3645–3650, Florence, Italy, July. Association for Computational Linguistics.
  * Vachon, 2010 Vachon, C. H.  (2010).  Le Changement linguistique au XVIe siècle: une étude basée sur des textes littéraires français.  ELiPhi, Éditions de linguistique et de philologie.
  * Vaswani et al., 2017 Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L. u., and Polosukhin, I.  (2017).  Attention is all you need.  In I. Guyon, et al., editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.

## 9\. Language Resource References

lrec2022-bib languageresource.bib

##  Appendix A Carbon Footprint

Model | Power (W) | Time (h) | (PUE$\cdotp$kWh) | CO2e (kg)
---|---|---|---|---
Pre-train | 48640 | 20 | 1537.02 | 46.11
Evaluation | 589 | 1 | 0.93 | 0.03
Total CO2e |  |  |  | 46.14

Table 4: Average power draw, number of models trained, training times in
hours, mean power consumption including power usage effectiveness (PUE), and
CO2 emissions; for each setting.

In light of recent interest concerning the energy consumption and carbon
emission of machine learning models and specifically of those of language
models [Schwartz et al., 2020, Bender et al., 2021], we have decided to report
the power consumption and carbon footprint of all our experiments following
the approach of ?). We report the energy consumption and carbon emissions of
both the pre-training of D’AlemBERT and its evaluation.

#### Pre-training:

We use a cluster of 32 machines, each one having 4 GPU Nvidia Tesla V100 SXM2
32GiB, 192GiB of RAM, and two Intel Xeon Gold 6248 processors. One Nvidia
Tesla V100 card is rated at around 300W,444 Nvidia Tesla V100 specification
while the Xeon Gold 6248 processor is rated at 150W.555Intel Xeon Gold 6248
specification For the DRAM we can use the work of ?) to estimate the total
power draw of 192GiB of RAM at around 20W. Thus, the total power draw of the
pre-training adds up to around 48640W.

#### Evaluation:

We use a single machine with a single GPU Nvidia Tesla V100 SXM2 32GiB, 384GiB
of RAM and two Intel Xeon Gold 6226 processors. The Xeon Gold 6226 processor
is rated at 125 W,666Intel Xeon Gold 6226 specification and the DRAM total
power draw can be estimated at around 39W. Therefore, the total power draw of
the evaluation adds up to around 589W.

With this information, we use the formula proposed by ?) to compute the total
power required for each setting:

$p_{t}=\frac{1.58t(cp_{c}+p_{r}+gp_{g})}{1000}$

Where $c$ and $g$ are the number of CPUs and GPUs respectively, $p_{c}$ is the
average power draw (in W) from all CPU sockets, $p_{r}$ the average power draw
from all DRAM sockets and $p_{g}$ the average power draw of a single GPU. We
estimate the total power consumption by adding GPU, CPU and DRAM consumption,
and then multiplying by the _Power Usage Effectiveness_ (PUE), which accounts
for the additional energy required to support the compute infrastructure. We
use a PUE coefficient of 1.58, the 2018 global average for data centres
[Strubell et al., 2019]. In Table 4 we report the training times in hours, as
well as the total power draw (in Watts) of the system used to train the
models. We use this information to compute the total power consumption of each
setting, also reported in Table 4.

We can further estimate the CO2 emissions in kilograms of each single model by
multiplying the total power consumption by the average CO2 emissions per kWh
in our region, which were around 30g/kWh between the 30th and the 31st of
December,777Rte - éCO2mix. when the models were trained. Thus the total CO2
emissions in kg for one single model can be computed as:

$\text{CO}_{2}\text{e}=0.030p_{t}$

All emissions are also reported in Table 4.


# Cold Quasar Investigation: Comparing Star Formation Rates to Black Hole
Growth

Sasha Mintz  Department of Physics, Virginia Polytechnic Institute and State
University, 850 West Campus Drive, Blacksburg, VA 24061  Brandon Coleman
Allison Kirkpatrick  Department of Physics & Astronomy, University of Kansas,
1251 Wescoe Hall Dr., Lawrence, KS 66045

###### Abstract

Cold quasars are a rare population of luminous, unobscured quasars associated
with host galaxies that have a high star formation rate. We aimed to study the
host galaxies of 64 of these cold quasars in order to probe how the
supermassive black holes and host galaxies were co-evolving. We compiled data
from the XMM-XXL survey and cross-matched with the VHS, WISE, and HerMES
surveys to obtain multiwavelength photometry spanning the X-ray to the
infrared and including optical spectroscopy. From the data, we calculated the
supermassive black hole’s mass using broad emission from the MgII and Hbeta
lines. We compared this with the stellar mass of the entire galaxy and find
that the black holes are significantly more massive than would be predicted by
local relations, indicating that the majority of black hole growth precedes
the bulk of the the stellar mass formation. In addition to this, we created a
spectral energy distribution for each galaxy to calculate the star formation
rate. We compared the star formation rate with the black hole accretion rate
and find that the stellar mass is rapidly increasing at a relative rate faster
than the black hole growth, supporting the picture where the black hole grows
first.

active galactic nuclei – galactic evolution – starburst galaxies – star
formation – quasars – supermassive black holes

##  1 Introduction

Likely every massive galaxy hosts a supermassive black hole (SMBH) at its
center. The most active period of supermassive black hole accretion and
feedback is predicted by theoretical models to occur after the galaxy has
formed most of its stars (Hopkins, 2007; Dekel, 2014; Byrne et al., 2023). In
these models, star formation is triggered by major mergers of two massive gas
and dust-rich galaxies, followed by sudden rapid accretion onto the
supermassive black hole (Sanders & Mirabel, 1996; Petric et al., 2011). The
end of this phase is signified by the active galactic nucleus (AGN) launching
extremely powerful winds that push away any obscuring dust (Glikman et al.,
2015; Murray et al., 2005). This merger scenario is predicted to be the
leading cause of extremely luminous quasars (Hopkins, 2007).

The co-evolution of supermassive black holes and their host galaxies is a
vibrant area of astronomical research, with many open questions. As discussed
in the simple merger scenario above, feedback emanating from the AGN can
launch large scale winds that propagate through the host galaxy (Pontzen &
Governato, 2012; Fabian, 2012). These large scale winds may be responsible for
quenching star formation (“negative feedback”) through the removal of gas, if
the mass loading factor is significant (Carniani et al., 2017; Chen et al.,
2022; Revalski et al., 2022) On the other hand, theoretical and observational
studies also show that these winds may trigger star formation (“positive
feedback”) by compressing molecular clouds (Ciotti & Ostriker, 2007).

AGN and quasars (we define a “quasar” as an AGN with $L_{X}>10^{44}$erg/s)
whose host galaxies are gas-rich and still forming stars at a rapid pace are
also observed in the literature (Xie et al., 2021; Scholtz et al., 2021;
Molina et al., 2023; Perna et al., 2018; Kirkpatrick et al., 2019).
Kirkpatrick (2020) argued that the most extreme of these quasars should form a
new category, called “Cold Quasars”. In these galaxies, there is a significant
presence of cold gas in addition to an extremely high star formation rate
(SFR$>200\,M_{\odot}$/yr). These galaxies represent a rare population where
unobscured quasars are found to reside in starbursting galaxies (Coleman et
al., 2022; Cooke et al., 2020).

Directly measuring the impact of postive or negative feedback through winds is
only possible with spectroscopy, but co-evolution can be inferred by looking
at global relations in large populations. The most well-known relation is the
tight correlation that is observed between the mass of the supermassive black
hole ($M_{\bullet}$) and the mass of stars ($M_{\ast}$) in the host galaxy
(Kormendy & Ho, 2013). This correlation persists over several orders of
magnitude in the local universe, and is observed at higher redshift, although
with some evolution of the normalization (Somerville, 2009; Sarria et al.,
2010; Schulze & Wisotzki, 2011). This tight correlation has been used to infer
a regulatory process between black hole growth and star formation (Silk &
Mamon, 2012; Martín-Navarro et al., 2020), although proving causation
observationally within galaxies remains elusive.

In unosbscured quasars, identifying a correlation between the black hole and
host galaxy masses and mass growth rates (through SFR or black hole accretion
rate, BHAR) suffers from the difficulty of separating AGN from host emission.
Unobscured quasars are extremely luminous, and in the mid-infrared and
optical, the quasars outshine their hosts, so that typical SFR tracers (e.g.,
H$\alpha$) cannot be used (Baldwin, 2018; Kewley, 1999; Kauffmann et al.,
2003). Thus, the best wavelength to study the host galaxy SFR is in the far
infrared, as unobscured AGN within star forming galaxies do not heat the dust
substantially beyond $\lambda$ $>$ 100 $\mu$m (Mullaney et al., 2011;
Kirkpatrick et al., 2012; Kirkpatrick, 2020; Sokol et al., 2023).

In this paper, we trace the growth of the black holes in Cold Quasars through
$M_{\bullet}$ and BHAR. In Section 2, we discuss selecting Cold Quasars from
the XMM-XXL survey. In Section 3, we detail how measurements were made. We
compare the black hole properties with host $M_{\ast}$ and SFR. We present our
conclusions in Section 4. In this work, we assume a standard flat cosmology
with $H_{0}=70\,$Mpc/km/s, $\Omega_{M}=0.3$, and $\Lambda=0.7$.

##  2 Data

A significant section of the sky must be surveyed in order to locate rare,
star-forming quasars. The XXL survey was chosen as the primary data source for
this project due to its large area and public multiwavelength data set. The
northern XXL field spans $\sim 18\,$deg2 and was surveyed with the X-ray
Multi-Mirror Mission (XMM-Newton) down to a depth of
$F_{\rm,0.5-10\,keV}=10^{-15}\,$erg/s/cm2 (Pierre, 2016). The public XXL
catalog contains more than 8000 X-ray sources (Menzel et al., 2016; Liu et
al., 2016). The survey was designed with the specific goal of finding high
luminosity X-ray emission, which are characteristic of AGNs. The XXL data
release also includes spectroscopic redshifts from the Sloan Digital Sky
Survey (SDSS; Menzel et al., 2016).

In order to enhance the dataset, we cross-matched between the XXL survey and
several additional astronomical surveys, including the Wide-field Infrared
Survey Explorer (WISE), Herschel Multi-tiered Extragalactic Survey (HerMES),
and the VISTA Hemisphere Survey (VHS) (Wright, 2010; Oliver et al., 2012;
McMahon et al., 2021). Full details of the cross-matching can be found in
Coleman et al. 2023 (in prep). We briefly summarize. X-ray sources are point-
like, which can make identifying the correct association challenging. The
maximum likelihood estimator (MLE) method considers three factors to determine
correct counterparts to the X-ray sources: 1) the density of background
sources in the field as a function of magnitude, 2) the density of sources
around the X-ray source per unit magnitude after subtracting the background
distribution, and 3) the positional errors associated with each catalog
(Ananna et al., 2017). We found that the entirety of the cold quasar candidate
subsample examined in this work had unambiguous matches.

Herschel/SPIRE is critical for identifying cold quasars. The HERMES survey
covers 9.3 deg2 of the XXL field (Oliver et al., 2012). The $5\sigma$ depth of
the SPIRE 250 $\mu$m image is 25.8 mJy.

To identify potential cold quasars, we began with 8,445 XXL X-ray sources.
There were a total of 3,042 sources that had a unique optical counterpart from
SDSS, and 2,525 (83%) had a spectroscopic redshift attached to them. Out of
that, there were 1,772 sources that had multiwavelength counterparts in VHS
and WISE.

We identified a total of 293 galaxies to be cold quasar candidates. These were
determined by the following criteria set forth in Kirkpatrick (2020):

  1. 1.

X-ray luminosity greater than $10^{44}$ erg/s

  2. 2.

Detected in all three Herschel/SPIRE bands

  3. 3.

Absolute B-band magnitude ($M_{B}$) $<$ -23

To transform between the SDSS filters and the B band, we used the equation

$m_{B}=m_{g}+0.17(m_{u}-m_{g})+0.11$ (1)

(Jester et al., 2005). We refer to this sample of 293 galaxies as cold quasar
candidates. We explore their possible starbursting nature in Section 3.4.

Figure 1: The comparison of supermassive black hole mass, $M_{\bullet}$, to
stellar mass, $M_{\ast}$, for our final sample of 64 cold quasar candidates.
Points have been shaded by redshift, corresponding to the colorbar on the
right. The black linear line represents the $M_{\rm bulge}-M_{\bullet}$
relationship measured from local galaxies (Kormendy & Ho, 2013). Nearly all of
the cold quasar candidates lie above the local relation, regardless of
redshift. This indicates that the stellar mass must increase more,
proportionally, for the galaxies to reach the local relation by $z=0$.

##  3 Results and Discussion

###  3.1 $M_{\bullet}$ Calculation

In order to determine the masses of the supermassive black holes within the
cold quasar candidates, we fit the optical broad lines with PyQSOFit (Guo et
al., 2018). PyQSOFit fits each optical spectrum with a model of continuum
emission due to both an accretion disk and a possible stellar population (Yip
et al., 2004a, b), Feii templates (Boroson & Green, 1992; Vestergaard &
Wilkes, 2001), a Balmer continuum (Dietrich et al., 2002), and a series of
broad lines. The program returns values such as emission line strength, full
width at half maximum (FWHM), and luminosity values at varying wavelengths
(the luminosities at 3000 Å and 5100 Å, $L_{3000}$ and $L_{5000}$ are relevant
for this work).We used PyQSOFit to measure the luminosity and FWHM of the Mgii
and H$\beta$ emission lines. If the spectrum contained neither, due to
insufficient wavelength coverage, it was removed from the sample. From the
original cold quasar candidate sample, we were able to fit Mgii for 245
galaxies and H$\beta$ for seven galaxies. For the Mgii candidates, we used the
following equation from Kollmeier (2006) to calculate the mass of the
supermassive black hole:

$[H]M_{\bullet}=2.04\left(\frac{L_{3000}}{10^{44}\,{\rm
erg\,s^{-1}}}\right)^{0.88}\left(\frac{\rm FWHM_{\rm Mg{\sc ii}}}{\rm
km\,s^{-1}}\right)^{2}$ (2)

There are multiple calibrations in the literature for calculating masses. We
also calculate Mgii masses following Equation 2 in Trakhtenbrot & Netzer
(2012) and find them to be consistent within the uncertainties.

For the H$\beta$ candidates, we calculated the mass with:

$M_{\bullet}=8.13\left(\frac{L_{5100}}{10^{44}\,\rm
erg\,s^{-1}}\right)^{0.5}\left(\frac{\rm FWHM_{\rm H\beta}}{\rm
km\,s^{-1}}\right)^{2}$ (3)

from Kollmeier (2006) as well. The uncertainties on $M_{\bullet}$ are
dominated by the uncertainties on the FWHM of the broad lines. To estimate the
$M_{\bullet}$ uncertainty, we ran a Monte Carlo simulation where we resampled
the FWHM of the broad line, assuming a normal distribution of the associated
uncertainty. We then recalculated $M_{\bullet}$ with the resampled FWHM. We
ran this simulation 10,000 times. The final uncertainty is the standard
deviation of the $M_{\bullet}$ distribution. After calculating the
uncertainties on each sample, those that had higher uncertainty than the mass
itself were removed from the dataset. This left a total of 64 galaxies.

###  3.2 The $M_{\bullet}$-$M_{\ast}$ Relation for Cold Quasars

We calculate $M_{\ast}$ for the final 64 galaxies using the fitting code
SED3Fit (Berta et al., 2013). SED3Fit fits the unattenuated stellar emission,
attenuated stellar emission, and dust emission from star formation using an
energy balance technique. The code uses stellar libraries from Bruzual &
Charlot (2003) and IR-dust libraries from da Cunha et al. (2008). SED3Fit also
takes into account the possible contribution from an AGN accretion disk and
torus using the library from Feltre et al. (2012).

We show the $M_{\bullet}$ versus $M_{\ast}$ in Figure 1 as a function of
redshift. To understand how $M_{\bullet}$ compares to other galaxies, we also
overplot the ratio of black hole mass to bulge mass in local galaxies from
Kormendy & Ho (2013). It is notable that the majority of our sample lie above
this relation. The highest redshift sources in general host the most massive
black holes and lie furthest away from the local relation. The measurement of
stellar mass can be uncertain if the AGN dominates the optical portion of the
SED. It is notable, however, that most galaxies still lie above the local
relation even given the large uncertainties on $M_{\ast}$. To reach the local
relation by $z=0$, the stellar mass in cold quasar candidates needs to
dramatically increase. $M_{\bullet}$, on the other hand, is already on par
with some of the most massive black holes in the local universe.

Table 1: Properties of Cold Quasar Candidates Galaxy ID | RA | Dec | $z$ | FWHM | $M_{\bullet}$ | $M_{\ast}$ | SFR | BHAR
---|---|---|---|---|---|---|---|---
| (J2000) | (J2000) |  | (km/s) | ($10^{8}$$M_{\odot}$) | ($10^{11}$$M_{\odot}$) | ($M_{\odot}$/yr) |  $10^{44}$erg/sec
N_4_19 | 02:27:40.9 | -3:52:50.40 | 1.931 | 2922.9 | $(4.03\pm 1.3)$ | $(2.03\pm 1.0)$ | $(893\pm 364.8)$ | $(1.70\pm 0.08)$
N_9_19 | 02:27:32.3 | -3:27:36.60 | 1.783 | 2426.4 | $(11.2\pm 2.8)$ | $(15.4\pm 10)$ | $(387.3\pm 162.9)$ | $(3.20\pm 0.15)$
N_11_7 | 02:24:28.8 | -3:15:33.32 | 1.353 | 4915.5 | $(3.75\pm 2.6)$ | $(0.59\pm 0.1)$ | $(394.3\pm 152.3)$ | $(0.66\pm 0.01)$
N_11_22 | 02:24:22.2 | -3:10:52.63 | 1.225 | 3093.3 | $(12.3\pm 9.6)$ | $(8.00\pm 1.0)$ | $(252.8\pm 95.5)$ | $(2.88\pm 0.13)$
N_12_35 | 02:25:36.5 | -3:28:33.15 | 2.205 | 6744.6 | $(37.8\pm 11.2)$ | $(4.55\pm 1.0)$ | $(933.3\pm 387.6)$ | $(1.72\pm 0.10)$
N_12_40 | 02:25:14.5 | -3:10:59.27 | 2.246 | 4359.7 | $(12.5\pm 1.7)$ | $(0.62\pm 0.1)$ | $(477.5\pm 212.0)$ | $(1.30\pm 0.08)$
N_18_41 | 02:22:34.7 | -4:04:17.69 | 2.043 | 3921.7 | $(7.85\pm 5.89)$ | $(0.86\pm 0.1)$ | $(1326.6\pm 542.5)$ | $(5.06\pm 0.16)$
N_27_14 | 02:22:54.9 | -4:35:8.03 | 1.221 | 13953.5 | $(15.3\pm 7.3)$ | $(0.88\pm 0.1)$ | $(187.7\pm 70.1)$ | $(1.18\pm 0.32)$
N_28_45 | 02:23:19.0 | -4:46:14.24 | 1.979 | 10996.9 | $(29.3\pm 19.2)$ | $(0.17\pm 0.1)$ | $(163.7\pm 72.1)$ | $(1.06\pm 0.06)$
N_30_35 | 02:25:57.6 | -4:50:05.46 | 2.275 | 6345.5 | $(58.1\pm 13.2)$ | $(1.94\pm 1.0)$ | $(382.2\pm 159.8)$ | $(1.23\pm 0.14)$
N_31_30 | 02:24:9.8 | -4:47:18.15 | 1.819 | 6194.2 | $(25.4\pm 1.4)$ | $(0.75\pm 0.1)$ | $(922.6\pm 374.1)$ | $(1.42\pm 0.09)$
N_35_40 | 02:24:24.6 | -5:04:14.59 | 2.048 | 11867.1 | $(30\pm 2.03)$ | $(0.83\pm 0.1)$ | $(518.8\pm 213.9)$ | $(4.98\pm 0.23)$
N_36_31 | 02:23:12.4 | -5:06:25.09 | 2.205 | 7747.5 | $(37.7\pm 1.0)$ | $(0.69\pm 0.1)$ | $(411.0\pm 173.7)$ | $(5.47\pm 0.37)$
N_37_12 | 02:19:34.7 | -5:06:28.04 | 2.293 | 9738.6 | $(43.1\pm 15.4)$ | $(1.94\pm 1.0)$ | $(588.3\pm 242.0)$ | $(15.4\pm 1.3)$
N_37_57 | 02:19:08.3 | -4:55:01.41 | 1.843 | 12959.7 | $(45.7\pm 5.1)$ | $(0.20\pm 0.1)$ | $(154.1\pm 66.3)$ | $(4.53\pm 0.17)$
N_38_79 | 02:18:30.5 | -4:56:22.9 | 1.397 | 4501.9 | $(26.5\pm 18.9)$ | $(7.81\pm 1.0)$ | $(302.2\pm 116.7)$ | $(1.31\pm 0.08)$
N_39_5 | 02:19:26.7 | -4:45:05.17 | 1.467 | 5057.7 | $(56.2\pm 7.9)$ | $(0.23\pm 0.1)$ | $(600.0\pm 233.9)$ | $(2.26\pm 0.06)$
N_39_19 | 02:19:03.5 | -4.:39:35.06 | 1.843 | 11151.0 | $(52.0\pm 18.4)$ | $(1.45\pm 1.0)$ | $(432.0\pm 176.6)$ | $(1.82\pm 0.07)$
N_41_24 | 02:16:43.7 | -5:22:36.36 | 1.851 | 10073.2 | $(49.6\pm 2.1)$ | $(2.36\pm 1.0)$ | $(433.8\pm 177.4)$ | $(1.74\pm 0.16)$
N_42_6 | 02:18:34.5 | -5:25:52.49 | 1.537 | 5372.5 | $(3.26\pm 1.9)$ | $(0.39\pm 0.1)$ | $(617.5\pm 243.5)$ | $(2.24\pm 0.10)$
N_44_59 | 02:26:44.2 | -4:07:20.29 | 1.478 | 5953.4 | $(16.4\pm 4.6)$ | $(2.36\pm 1.0)$ | $(396.8\pm 155.3)$ | $(2.11\pm 0.10)$
N_44_71 | 02:27:23.7 | -4:02:15.01 | 1.822 | 7513.2 | $(7.08\pm 5.84)$ | $(0.74\pm 0.1)$ | $(712.0\pm 289.2)$ | $3.48\pm 0.16)$
N_47_22 | 02:27:29.2 | -4:32:27.86 | 2.261 | 8938.9 | $(79.7\pm 26.1)$ | $(1.24\pm 1.0)$ | $(1273.6\pm 519.0)$ | $(2.88\pm 0.11)$
N_47_34 | 02:28:39.4 | -4:29:12.43 | 1.735 | 11564.7 | $(21.2\pm 5.3)$ | $(0.22\pm 0.1)$ | $(203.8\pm 84.4)$ | $(2.45\pm 0.16)$
N_47_50 | 02:27:33.9 | -4:25:23.51 | 1.585 | 5383.8 | $(8.25\pm 6.01)$ | $(0.59\pm 0.1)$ | $(332.6\pm 132.7)$ | $(2.32\pm 0.09)$
N_48_27 | 02:23:50.7 | -4:31:58.30 | 1.504 | 3863.1 | $(7.20\pm 2.48)$ | $(1.71\pm 1.0)$ | $(189.2\pm 75.3)$ | $(4.18\pm 0.11)$
N_50_22 | 02:25:25.6 | -4:35:09.65 | 2.165 | 10687.6 | $(27.1\pm 3.3)$ | $(0.59\pm 0.1)$ | $(339.0\pm 142.1)$ | $(1.15\pm 0.06)$
N_51_9 | 02:26:09.2 | -4:36:50.88 | 1.813 | 12965.0 | $(29.9\pm 8.0)$ | $(1.05\pm 0.1)$ | $(277.9\pm 114.4)$ | $(2.69\pm 0.08)$
N_54_30 | 02:21:24.5 | -5:02:05.40 | 1.411 | 4195.8 | $(14.3\pm 09.7)$ | $(2.68\pm 1.0)$ | $(210.8\pm 82.8)$ | $(2.15\pm 0.11)$
N_56_24 | 02:21:58.8 | -5:29:32.38 | 1.353 | 7126.1 | $(19.1\pm 2.9)$ | $(0.22\pm 0.1)$ | $(174.5\pm 68.1)$ | $(4.94\pm 0.15)$
N_62_16 | 02:25:42.3 | -5:29:50.79 | 1.286 | 2559.0 | $(28.9\pm 1.0)$ | $(1.24\pm 1.0)$ | $(234.1\pm 88.9)$ | $(1.78\pm 0.05$
N_63_24 | 02:24:31.5 | -5:28:18.91 | 2.085 | 7066.5 | $(276\pm 11.0)$ | $(40.5\pm 10.0)$ | $(1302.1\pm 533.3)$ | $(1.61\pm 0.08)$
N_66_41 | 02:23:05.0 | -5:42:50.94 | 1.068 | 3754.7 | $(4.40\pm 2.04)$ | $(0.59\pm 0.1)$ | $(151.0\pm 54.7)$ | $(2.39\pm 0.10)$
N_71_35 | 02:20:33.4 | -6:00:22.16 | 1.695 | 10143.8 | $(93.6\pm 0.70)$ | $(0.89\pm 0.1)$ | $(456.1\pm 187.1)$ | $(5.27\pm 0.23)$
N_75_4 | 02:17:55.0 | -5:39:21.38 | 1.882 | 5595.1 | $(18.6\pm 1.8)$ | $(0.44\pm 0.1)$ | $(247.2\pm 103.1)$ | $(2.15\pm 0.08)$
N_77_4 | 02:20:01.6 | -5:22:17.08 | 2.221 | 8760.7 | $(77.3\pm 4.4)$ | $(0.49\pm 0.1)$ | $(426.5\pm 176.8)$ | $(1.70\pm 0.07)$
N_77_15 | 02:19:38.8 | -5:18:05.57 | 2.223 | 8227.1 | $(42.2\pm 12.4$ | $(2.50\pm 1.0)$ | $(200.0\pm 88.5)$ | $(1.08\pm 0.06)$
N_78_16 | 02:15:32.0 | -5:15:19.11 | 1.743 | 6930.9 | $(32.1\pm 10.1)$ | $(3.49\pm 1.0)$ | $(148.8\pm 63.1)$ | $(14.2\pm 1.2)$
N_83_23 | 02:14:19.7 | -4:25:29.83 | 2.174 | 12947.8 | $(18.5\pm 12.3)$ | $(1.49\pm 1.0)$ | $(802.6\pm 335.6)$ | $(6.55\pm 0.21)$
N_86_2 | 02:16:30.8 | -4:20:51.53 | 1.529 | 5484.7 | $(12.0\pm 8.3)$ | $(0.50\pm 0.1)$ | $(379.7\pm 150.0)$ | $(8.01\pm 0.46)$
N_90_4 | 02:17:14.5 | -4:00:20.97 | 1.402 | 4041.9 | $(2.95\pm 0.87)$ | $(0.37\pm 0.1)$ | $(235.1\pm 91.8)$ | $(3.60\pm 0.12)$
N_96_11 | 02:13:24.7 | -3:35:11.44 | 1.141 | 6280.8 | $(14.4\pm 1.42)$ | $(1.04\pm 1.0)$ | $(347.3\pm 127.1)$ | $(3.18\pm 0.10)$
N_97_1 | 02:23:20.6 | -5:41:24.97 | 1.753 | 8445.0 | $(25.5\pm 2.1)$ | $(1.93\pm 1.0)$ | $(301.5\pm 125.5)$ | $(4.36\pm 0.22)$
N_99_3 | 02:14:31.7 | -5:28:00.14 | 1.643 | 4479.0 | $(7.74\pm 6.9)$ | $(1.81\pm 1.0)$ | $(272.2\pm 110.7)$ | $(4.33\pm 0.27)$
Table 2: Continuation of Properties of Cold Quasar Candidates

Galaxy ID | RA | Dec | $z$ | FWHM | $M_{\bullet}$ | $M_{\ast}$ | SFR | BHAR
---|---|---|---|---|---|---|---|---
| (J2000) | (J2000) |  | (km/s) | ($10^{11}$$M_{\odot}$) | ($10^{11}$$M_{\odot}$) | ($M_{\odot}$/yr) | erg/s
N_100_18 | 02:21:32.67 | -03:44:00.8 | 1.605 | 7099.4 | $(15.2\pm 1.5)$ | $(1.24\pm 1.0)$ | $(640.9\pm 256.4)$ | $(14.9\pm 1.9)$
N_101_4 | 02:13:39.1 | -5:42:31.35 | 2.063 | 12645.0 | $(29.1\pm 23.1)$ | $(3.44\pm 1.0)$ | $(726.1\pm 303.0)$ | $(2.11\pm 0.10)$
N_101_11 | 02:14:16.4 | -5:39:44.94 | 1.644 | 6820.4 | $(6.11\pm 6.0)$ | $(0.361\pm 0.1)$ | $(1056.1\pm 423.1)$ | $(3.16\pm 0.17)$
N_101_25 | 02:14:04.1 | -5:33:31.61 | 2.428 | 8855.3 | $(27.0\pm 18.0)$ | $(1.87\pm 1.0)$ | $(1129.3\pm 466.5)$ | $(2.99\pm 0.14)$
N_101_28 | 02:14:37.3 | -5:30:10.65 | 1.854 | 5688.4 | $(24.7\pm 5.1)$ | $(1.17\pm 1.0)$ | $(440.4\pm 181.6)$ | $(3.97\pm 0.09)$
N_102_8 | 02:28:04.6 | -4:15:59.28 | 2.254 | 8218.5 | $(16.9\pm 9.2)$ | $(0.13\pm 0.1)$ | $(459.9\pm 190.8)$ | $(1.23\pm 0.06)$
N_102_22 | 02:29:01.3 | -4:08:35.41 | 1.524 | 6923.23 | $(11.0\pm 4.1)$ | $(0.42\pm 0.1)$ | $(554.1\pm 218.1)$ | $(9.66\pm 0.71)$
N_102_36 | 02:28:28.1 | -4:00:44.5 | 1.903 | 4114.0 | $(48.1\pm 6.3)$ | $(22.5\pm 10.0)$ | $(240.4\pm 101.5)$ | $(2.58\pm 0.18)$
N_103_5 | 02:28:49.1 | -4:36:26.98 | 1.935 | 19012.9 | $(129.0\pm 33.0)$ | $(0.33\pm 0.10)$ | $(829.4\pm 339.0)$ | $(1.09\pm 0.03)$
N_103_9 | 02:28:45.5 | -4:33:50.39 | 1.871 | 5224.3 | $(156\pm 5.0)$ | $(80.8\pm 10.0)$ | $(748.3\pm 305.0)$ | $(1.10\pm 0.10)$
N_105_18 | 02:30:00.9 | -4:28:35.40 | 1.486 | 3402.4 | $(6.31\pm 4.07)$ | $(0.96\pm 0.10)$ | $(355.6\pm 140.6)$ | $(8.89\pm 0.26)$
N_122_31 | 02:13:54.9 | -4:05:46.89 | 1.930 | 5462.5 | $(3.33\pm 1.0)$ | $(0.51\pm 0.10)$ | $(371.4\pm 160.5)$ | $(3.53\pm 0.03)$
N_131_26 | 02:13:12.7 | -4:30:46.38 | 1.801 | 4758.9 | $(8.32\pm 2.64)$ | $(3.91\pm 1.0)$ | $(514.3\pm 212.7)$ | $(4.56\pm 0.21)$
N_135_19 | 02:11:29.8 | -4:33:24.00 | 2.430 | 12178.8 | $(121.0\pm 4)$ | $(1.90\pm 1.0)$ | $(429.9\pm 228.2)$ | $(2.15\pm 0.07)$
N_141_23 | 02:12:47.4 | -5.:06:11.13 | 1.811 | 11782.8 | $(28.8\pm 14.3)$ | $(0.96\pm 0.1)$ | $(386.6\pm 164.0)$ | $(3.30\pm 0.14)$
N_142_7 | 02:12:16.3 | -5:14:00.43 | 1.230 | 7812.9 | $(6.57\pm 4.25)$ | $(1.80\pm 1.0)$ | $(247.7\pm 94.1)$ | $(1.73\pm 0.05)$
N_152_6 | 02:15:51.8 | -6:17:33.71 | 1.225 | 4869.9 | $(16.5\pm 4.2)$ | $(0.48\pm 0.10)$ | $(389.9\pm 145.8)$ | $(22.8\pm 7.3)$
N_152_47 | 02:15:29.2 | -6:00:46.40 | 1.658 | 7317.9 | $(27.0\pm 4.4)$ | $(0.98\pm 0.1)$ | $(239.1\pm 102.7)$ | $(4.42\pm 0.07)$
N_153_21 | 02:13:35.3 | -6:01:33.49 | 1.372 | 3787.1 | $(2.59\pm 1.94)$ | $(0.67\pm 0.1)$ | $(312.2\pm 125.2)$ | $(18.2\pm 3.4)$

††footnotetext: FWHM is measured from the MgII or H$\beta$ line using
PyQSOFit.

Figure 2: The comparison of normalized black hole accretion rate vs. specific
star formation rate for our sample. A one-to-one correlation is plotted as the
dashed line. Our cold quasar candidates are proportionally growing their
stellar mass much faster than their black hole mass.

###  3.3 Measuring Star Formation Rate and Black Hole Accretion Rate

We calculate the star formation rates (SFRs) for the cold quasar candidates
using the 250 $\mu$m photometry. We opt not to use the SFR output by SED3Fit,
as SED3Fit underpredicts the far-IR emission for many of these galaxies, due
to their extreme dust content. The spectral energy distribution (SED) for cold
quasars is dominated by a power-law in the mid-infrared, but the quasar does
not contribute to the the heating of the dust beyond $\lambda>50\,\mu$m, even
assuming the most generous far-IR AGN heating models (Kirkpatrick, 2020).
Therefore, the 250 $\mu$m emission can be attributed to the star-forming host.
Quasars that significantly contribute to the energy budget in the far-IR in
very dusty galaxies are heavily obscured, which these quasars are not (Sajina
et al., 2012; Ricci et al., 2017). We calculate $L_{\rm 250}$ in the rest-
frame for each galaxy and then scale a star-forming template from Kirkpatrick
et al. (2012) to this luminosity. The star-forming template represents the
empirical emission of dusty star-forming galaxies ($L_{\rm
IR}>10^{11}\,L_{\odot}$) at $z\sim 1$. We integrate the scaled template from
$8-1000\,\mu$m to obtain $L_{\rm IR}$. We convert $L_{\rm IR}$ to a SFR using

${\rm SFR}\,[M_{\odot}/{\rm yr}]=1.59\times 10^{-10}*L_{\rm IR}\,[L_{\odot}]$
(4)

which assumes a Kroupa IMF (Murphy et al., 2011; Kroupa, 2001). The SFRs for
each of these galaxies was extremely high, ranging from
$200-1000\,M_{\odot}$/yr (see Table LABEL:tab:CQ_Table), which is similar to
the original cold quasar sample in Stripe82X (Kirkpatrick, 2020; Coleman et
al., 2022).

We determine the black hole accretion rate (BHAR) by applying a K-correction
(Equation 7; Lusso, 2012) to the X-ray luminosity in order to determine the
bolometric luminosity, $L_{\rm Bol}$. We then converted $L_{\rm Bol}$ to BHAR
assuming a 10% radiative efficiency.

In Figure 2, we compare BHAR/$M_{\bullet}$ to SFR/$M_{\ast}$ for our sample.
We choose to “normalize” BHAR and SFR in this manner, rather than directly
comparing the two values, because we are probing a range of masses. Larger
galaxies naturally have more gas, driving up BHAR and SFR. Any correlation
between the two parameters is then likely a product of the galaxy mass.

Interestingly, Figure 2 shows no correlation between black hole growth and
stellar growth as parameterized by BHAR/$M_{\bullet}$ and SFR/$M_{\ast}$. In
most galaxies, the mass of the stellar population is increasing proportionally
faster than the mass of the supermassive black hole (indicated by most
galaxies lying below the dashed one-to-one correlation line).

Figure 3: The location of these galaxies on the main sequence relation (black
line and grey shaded region) from (Elbaz et al., 2011). Galaxies have been
shaded by their black hole mass. There is a significant distribution of the
galaxies above the main sequence line.

###  3.4 Backward Growth?

Our sample was selected on the basis of having a detection in all three
Herschel/SPIRE bands, but with the deeper Herschel observations in the XXL
field, many of these candidates may in fact be main sequence galaxies rather
than starbursts.

The original sample of Stripe82X cold quasars were found to all have elevated
SFRs, placing them in the starburst region (Kirkpatrick, 2020; Cooke et al.,
2020; Coleman et al., 2022). Figure 3 investigates the main sequence location
of our cold quasar candidates. The main sequence is a tight correlation
between SFR and $M_{\ast}$ observed for the majority of star-forming galaxies
(Noeske et al., 2007; Elbaz et al., 2007). We use the main sequence relation
parameterized in Elbaz et al. (2011) as that was calibrated on Herschel-
detected galaxies out to $z\sim 2$. Our cold quasar candidates primarily lie
in the starburst regime, above the main sequence line. Interestingly, a few
galaxies lie well below the main sequence, despite being detected by Herschel.
According to the criteria in (Coleman et al., 2022), only the quasars lying
above the main sequence are truly cold quasars.

We shade points according to $M_{\bullet}$, but the only trend is that the
more massive black holes lie at higher redshift. This is driven by the
difficulty of detecting smaller black holes with increasing redshift due to
survey flux limits.

Finally, we explore how the high SFRs correlate with high $M_{\bullet}$ in
Figure 4, which shows the distance from the main sequence as a function of the
distance from the local $M_{\bullet}-M_{\ast}$ relation. We have used each
galaxy’s redshift to predict the sSFR it would have on the main sequence, and
we have used each galaxy’s $M_{\ast}$ to predict the $M_{\bullet}$ it should
have if it were on the local relation.

Most of our sample have an overmassive black hole compared with the local
relation, indicating that the stellar population needs to increase in size, in
some cases by two orders of magnitude. This increase may be through conversion
of the available gas to stars, or it may occur through minor galaxy mergers
which increase stellar mass, without significantly affecting the black hole
growth (Jahnke & Macciò, 2011; Rodriguez-Gomez et al., 2016). Similar to
Figures 2 and 3, Figure 4 does not show any trend between the star forming and
black hole properties of a galaxy. Except for a few outliers, all of our cold
quasar candidates have overmassive black holes and are starbursting.

Taken together with Figure 2, cold quasars seem to be experiencing an
evolutionary sequence in which the majority of the supermassive black hole
growth predates the majority of the stellar growth. This sort of backward
evolution is not common, especially in local galaxies where mergers are
driving high SFRs but weaker black hole growth that is heavily obscured
(Sanders & Mirabel, 1996; Petric et al., 2011). Merger driven starbursts were
the observational impetus for the theoretical models that predict quasar
growth occurs after star formation has abated (Hopkins, 2007, e.g.). Cold
quasars, on the other hand, are still strongly star forming, but the black
hole growth rate seems to be declining. Over time, each of these galaxy’s
black holes will cease to accrete further mass, but continue to grow their
stellar population, possibly becoming the massive elliptical galaxies we see
today.

Figure 4: The distance of each galaxy’s sSFR from the main sequence versus the
distance of each galaxy’s supermassive black hole from the local relation
(Kormendy & Ho, 2013). The main sequence and local relation are shown as the
dashed lines. Most of our cold quasar candidates have overmassive black holes
given their stellar mass, and they are starbursting, lying in the upper right
quadrant. This points to a picture whereby the black hole growth precedes the
bulk of the stellar growth.

##  4 Conclusions

We present a total of 64 galaxies that are to be included within the cold
quasar dataset. For each of these 64 galaxies, we measured its supermassive
black hole mass using the MgII and H$\beta$ emission from PyQSOFit, using
spectra from SDSS. In addition, we measured their stellar mass using SED3Fits.
We calculated the star formation rates of each of these galaxies by creating
spectral energy distributions and scaling a template of star formation rate to
the 250$\mu$m point. Additionally, we measured the black hole accretion rate
of each galaxy using K-corrections of their X-ray luminosity, assuming a 10%
radiation efficiency. We conclude the following:

  1. 1.

From Figure 1, the supermassive black holes are concluded to be extremely
overmassive as compared to local relations. This signifies that the black
holes have grown first in these galaxies, and are now completing their
evolutionary track.

  2. 2.

Using Figure 2, it can be seen that these galaxies’ star formation rates are
growing proportionally faster than their black holes, implying that their
stellar masses need to increase (in some cases double) in order to reach the
local relation. The combination of overmassive black holes and increased SFR
is indicative of an increase in galaxy growth as well.

  3. 3.

Figure 3 portrays an interesting conclusion, showing that there are some cold
quasars that are starbursting, and a select few are quenching or along the
main sequence.

  4. 4.

Figure 4 exhibits the conclusion that the majority of the cold quasar
candidates have overmassive black holes and are starbursting, with the
majority of black hole growth slowing and SFR increasing.

The overview of cold quasars from this study is that they are extremely
complex objects. Seeing that there are a select few that are quenching, some
that have star formation rates of 1000 solar masses per year, and most contain
much larger black holes than predicted exhibits the multi-faceted approach
that must be taken in order to begin to understand these quasars. These
objects are worth examining in more detail in order to truly understand how
these may have come into existence, and how they may be evolving in the
distant future.

##  5 Acknowledgements

SM thanks Meg Urry and Benny Trakhtenbrot for helpful conversations. SM would
also like to thank Derek Sikorski for insightful discussions. We would like
the thank the NSF REU Award Number 2149897 for funding SM’s research with the
University of Kansas. This research made use of Astropy, a core Python package
for Astronomy (Astropy Collaboration et al., 2013). This project also made use
of the Python packages SciPy (Virtanen et al., 2020), Numpy (van der Walt et
al., 2011), and Matplotlib (Hunter, 2007).

##  6 Data Availability

The data used in this paper are available in Menzel et al. (2016). The
datasets were derived from sources in the public domain: XMM archive
(http://nxsa.esac.esa.int/nxsa-web/#home), and cross-matched with HerMES
(http://hedam.lam.fr/HerMES/index/download), WISE
(https://irsa.ipac.caltech.edu/cgi-bin/Gator/nph-dd), SDSS
(https://www.sdss.org/dr14/data_access/), and VHS
(http://horus.roe.ac.uk:8080/vdfs/VImageList_form.jsp).

## References

  * Ananna et al. (2017) Ananna, T. T., Salvato, M., LaMassa, S., et al. 2017, ApJ, 850, 66, doi: 10.3847/1538-4357/aa937d
  * Astropy Collaboration et al. (2013) Astropy Collaboration, Robitaille, T. P., Tollerud, E. J., et al. 2013, A&A, 558, A33, doi: 10.1051/0004-6361/201322068
  * Baldwin (2018) Baldwin, C, e. a. 2018, Monthly Notices of the Royal Astronomical Society, 473, 4698
  * Berta et al. (2013) Berta, S., Lutz, D., Santini, P., et al. 2013, A&A, 551, A100, doi: 10.1051/0004-6361/201220859
  * Boroson & Green (1992) Boroson, T. A., & Green, R. F. 1992, ApJS, 80, 109, doi: 10.1086/191661
  * Bruzual & Charlot (2003) Bruzual, G., & Charlot, S. 2003, MNRAS, 344, 1000, doi: 10.1046/j.1365-8711.2003.06897.x
  * Byrne et al. (2023) Byrne, L., Faucher-Giguère, C.-A., Stern, J., et al. 2023, MNRAS, 520, 722, doi: 10.1093/mnras/stad171
  * Carniani et al. (2017) Carniani, S., Marconi, A., Maiolino, R., et al. 2017, A&A, 605, A105, doi: 10.1051/0004-6361/201730672
  * Chen et al. (2022) Chen, Z., He, Z., Ho, L. C., et al. 2022, Nature Astronomy, 6, 339, doi: 10.1038/s41550-021-01561-3
  * Ciotti & Ostriker (2007) Ciotti, L., & Ostriker, J. P. 2007, The Astronomical Journal, 665, 1038
  * Coleman et al. (2022) Coleman, B., Kirkpatrick, A., Cooke, K. C., et al. 2022, MNRAS, 515, 82, doi: 10.1093/mnras/stac1679
  * Cooke et al. (2020) Cooke, K. C., Kirkpatrick, A., Estrada, M., et al. 2020, ApJ, 903, 106, doi: 10.3847/1538-4357/abb94a
  * da Cunha et al. (2008) da Cunha, E., Charlot, S., & Elbaz, D. 2008, MNRAS, 388, 1595, doi: 10.1111/j.1365-2966.2008.13535.x
  * Dekel (2014) Dekel, A. 2014, Monthly Notices of the Royal Astronomical Society, 444, 2071
  * Dietrich et al. (2002) Dietrich, M., Hamann, F., Shields, J. C., et al. 2002, ApJ, 581, 912, doi: 10.1086/344410
  * Elbaz et al. (2007) Elbaz, D., Daddi, E., Le Borgne, D., et al. 2007, A&A, 468, 33, doi: 10.1051/0004-6361:20077525
  * Elbaz et al. (2011) Elbaz, D., Dickinson, M., Hwang, H. S., et al. 2011, A&A, 533, A119, doi: 10.1051/0004-6361/201117239
  * Fabian (2012) Fabian, A. C. 2012, ARA&A, 50, 455, doi: 10.1146/annurev-astro-081811-125521
  * Feltre et al. (2012) Feltre, A., Hatziminaoglou, E., Fritz, J., & Franceschini, A. 2012, MNRAS, 426, 120, doi: 10.1111/j.1365-2966.2012.21695.x
  * Glikman et al. (2015) Glikman, E., Simmons, B., Mailly, M., et al. 2015, ApJ, 806, 218, doi: 10.1088/0004-637X/806/2/218
  * Guo et al. (2018) Guo, H., Shen, Y., & Wang, S. 2018, PyQSOFit: Python code to fit the spectrum of quasars, Astrophysics Source Code Library, record ascl:1809.008
  * Hopkins (2007) Hopkins, A. M. 2007, in Astronomical Society of the Pacific Conference Series, Vol. 380, Deepest Astronomical Surveys, ed. J. Afonso, H. C. Ferguson, B. Mobasher, & R. Norris, 423, doi: 10.48550/arXiv.astro-ph/0611283
  * Hunter (2007) Hunter, J. D. 2007, Computing in Science and Engineering, 9, 90, doi: 10.1109/MCSE.2007.55
  * Jahnke & Macciò (2011) Jahnke, K., & Macciò, A. V. 2011, ApJ, 734, 92, doi: 10.1088/0004-637X/734/2/92
  * Jester et al. (2005) Jester, S., Schneider, D. P., Richards, G. T., et al. 2005, AJ, 130, 873, doi: 10.1086/432466
  * Kauffmann et al. (2003) Kauffmann, G., Heckman, T. M., Tremonti, C., et al. 2003, MNRAS, 346, 1055, doi: 10.1111/j.1365-2966.2003.07154.x
  * Kewley (1999) Kewley, L, e. a. 1999, Astrophysics and Space Science, 266, 131
  * Kirkpatrick et al. (2019) Kirkpatrick, A., Sharon, C., Keller, E., & Pope, A. 2019, ApJ, 879, 41, doi: 10.3847/1538-4357/ab223a
  * Kirkpatrick et al. (2012) Kirkpatrick, A., Pope, A., Alexander, D. M., et al. 2012, ApJ, 759, 139, doi: 10.1088/0004-637X/759/2/139
  * Kirkpatrick (2020) Kirkpatrick, A. e. a. 2020, The Astrophysical Journal, 900
  * Kollmeier (2006) Kollmeier, J. A. 2006, The Astrophysical Journal, 648, 128
  * Kormendy & Ho (2013) Kormendy, J., & Ho, L. C. 2013, arXiv e-prints, arXiv:1308.6483, doi: 10.48550/arXiv.1308.6483
  * Kroupa (2001) Kroupa, P. 2001, MNRAS, 322, 231, doi: 10.1046/j.1365-8711.2001.04022.x
  * Liu et al. (2016) Liu, Z., Merloni, A., Georgakakis, A., et al. 2016, MNRAS, 459, 1602, doi: 10.1093/mnras/stw753
  * Lusso (2012) Lusso, E. e. a. 2012, Monthly Notices of the Royal Astronomical Society, 425, 623
  * Martín-Navarro et al. (2020) Martín-Navarro, I., Burchett, J. N., & Mezcua, M. 2020, MNRAS, 491, 1311, doi: 10.1093/mnras/stz3073
  * McMahon et al. (2021) McMahon, R. G., Banerji, M., Gonzalez, E., et al. 2021, VizieR Online Data Catalog, II/367
  * Menzel et al. (2016) Menzel, M. L., Merloni, A., Georgakakis, A., et al. 2016, MNRAS, 457, 110, doi: 10.1093/mnras/stv2749
  * Molina et al. (2023) Molina, J., Ho, L. C., Wang, R., et al. 2023, ApJ, 944, 30, doi: 10.3847/1538-4357/acaa9b
  * Mullaney et al. (2011) Mullaney, J. R., Alexander, D. M., Goulding, A. D., & Hickox, R. C. 2011, MNRAS, 414, 1082, doi: 10.1111/j.1365-2966.2011.18448.x
  * Murphy et al. (2011) Murphy, E. J., Condon, J. J., Schinnerer, E., et al. 2011, ApJ, 737, 67, doi: 10.1088/0004-637X/737/2/67
  * Murray et al. (2005) Murray, N., Quataert, E., & Thompson, T. A. 2005, ApJ, 618, 569, doi: 10.1086/426067
  * Noeske et al. (2007) Noeske, K. G., Weiner, B. J., Faber, S. M., et al. 2007, ApJ, 660, L43, doi: 10.1086/517926
  * Oliver et al. (2012) Oliver, S. J., Bock, J., Altieri, B., et al. 2012, MNRAS, 424, 1614, doi: 10.1111/j.1365-2966.2012.20912.x
  * Perna et al. (2018) Perna, M., Sargent, M. T., Brusa, M., et al. 2018, A&A, 619, A90, doi: 10.1051/0004-6361/201833040
  * Petric et al. (2011) Petric, A. O., Armus, L., Howell, J., et al. 2011, ApJ, 730, 28, doi: 10.1088/0004-637X/730/1/28
  * Pierre (2016) Pierre, M, e. a. 2016, A&A, 592, 16
  * Pontzen & Governato (2012) Pontzen, A., & Governato, F. 2012, Monthly Notices of the Royal Astronomical Society, 421, 3464
  * Revalski et al. (2022) Revalski, M., Crenshaw, D. M., Rafelski, M., et al. 2022, ApJ, 930, 14, doi: 10.3847/1538-4357/ac5f3d
  * Ricci et al. (2017) Ricci, C., Assef, R. J., Stern, D., et al. 2017, ApJ, 835, 105, doi: 10.3847/1538-4357/835/1/105
  * Rodriguez-Gomez et al. (2016) Rodriguez-Gomez, V., Pillepich, A., Sales, L. V., et al. 2016, MNRAS, 458, 2371, doi: 10.1093/mnras/stw456
  * Sajina et al. (2012) Sajina, A., Yan, L., Fadda, D., Dasyra, K., & Huynh, M. 2012, ApJ, 757, 13, doi: 10.1088/0004-637X/757/1/13
  * Sanders & Mirabel (1996) Sanders, D. B., & Mirabel, I. F. 1996, ARA&A, 34, 749, doi: 10.1146/annurev.astro.34.1.749
  * Sarria et al. (2010) Sarria, J. E., Maiolino, R., La Franca, F., et al. 2010, A&A, 522, L3, doi: 10.1051/0004-6361/201015696
  * Scholtz et al. (2021) Scholtz, J., Harrison, C. M., Rosario, D. J., et al. 2021, MNRAS, 505, 5469, doi: 10.1093/mnras/stab1631
  * Schulze & Wisotzki (2011) Schulze, A., & Wisotzki, L. 2011, A&A, 535, A87, doi: 10.1051/0004-6361/201117564
  * Silk & Mamon (2012) Silk, J., & Mamon, G. A. 2012, Research in Astronomy and Astrophysics, 12, 917, doi: 10.1088/1674-4527/12/8/004
  * Sokol et al. (2023) Sokol, A. D., Yun, M., Pope, A., Kirkpatrick, A., & Cooke, K. 2023, MNRAS, 521, 818, doi: 10.1093/mnras/stad589
  * Somerville (2009) Somerville, R. S. 2009, MNRAS, 399, 1988, doi: 10.1111/j.1365-2966.2009.15325.x
  * Trakhtenbrot & Netzer (2012) Trakhtenbrot, B., & Netzer, H. 2012, MNRAS, 427, 3081, doi: 10.1111/j.1365-2966.2012.22056.x
  * van der Walt et al. (2011) van der Walt, S., Colbert, S. C., & Varoquaux, G. 2011, Computing in Science and Engineering, 13, 22, doi: 10.1109/MCSE.2011.37
  * Vestergaard & Wilkes (2001) Vestergaard, M., & Wilkes, B. J. 2001, ApJS, 134, 1, doi: 10.1086/320357
  * Virtanen et al. (2020) Virtanen, P., Gommers, R., Oliphant, T. E., et al. 2020, Nature Methods, 17, 261, doi: 10.1038/s41592-019-0686-2
  * Wright (2010) Wright, E, e. a. 2010, The Astronomical Journal, 140, 1868
  * Xie et al. (2021) Xie, Y., Ho, L. C., Zhuang, M.-Y., & Shangguan, J. 2021, ApJ, 910, 124, doi: 10.3847/1538-4357/abe404
  * Yip et al. (2004a) Yip, C. W., Connolly, A. J., Szalay, A. S., et al. 2004a, AJ, 128, 585, doi: 10.1086/422429
  * Yip et al. (2004b) Yip, C. W., Connolly, A. J., Vanden Berk, D. E., et al. 2004b, AJ, 128, 2603, doi: 10.1086/425626