11institutetext: 1Astronomical Observatory, Volgina 7, 11060 Belgrade, Serbia
11email<EMAIL_ADDRESS>
# The isolated dark matter-poor galaxy that ran away

An example from IllustrisTNG

Ana Mitrašinović 11 Majda Smole 11 Miroslav Micic 11

(Received xxx; accepted xxx)

Since the discovery of dark matter-deficient galaxies, numerous studies have
shown that these exotic galaxies naturally occur in the $\Lambda$CDM model due
to stronger tidal interactions. They are typically satellites, with stellar
masses in the $10^{8}-10^{9}\;\mathrm{M}_{\odot}$ range, of more massive
galaxies. The recent discovery of a massive galaxy lacking dark matter and
also lacking a more massive neighbor is puzzling. Two possible scenarios have
been suggested in the literature: either the galaxy lost its dark matter early
or it had been lacking ab initio. As a proof of concept for the former
assumption, we present an example from IllustrisTNG300. At present, the galaxy
has a stellar mass of $M_{\star}\simeq 6.8\cdot 10^{9}\;\mathrm{M}_{\odot}$,
with no gas, $M_{\mathrm{DM}}/M_{\mathrm{B}}\simeq 1.31$, and a stellar half-
mass radius of $R_{0.5,\star}=2.45\;\mathrm{kpc}$. It lost the majority of its
dark matter early, between $z=2.32$ and $z=1.53$. Since then, it has continued
to dwell in the cluster environment, interacting with the cluster members
without merging, while accelerating on its orbit. Eventually, it left the
cluster and it has spent the last $\sim 2\;\mathrm{Gyr}$ in isolation,
residing just outside the most massive cluster in the simulation. Thus, the
galaxy represents the first example found in simulations of both an isolated
dark matter-poor galaxy that lost its extended envelope early and a fairly
compact stellar system that has managed to escape.

###### Key Words.:

galaxies: evolution – galaxies: interactions – galaxies: clusters: general –
dark matter

##  1 Introduction

The discovery of galaxies lacking dark matter (van Dokkum et al., 2018b, 2019)
has sparked a debate on recent challenges to the standard cosmological model
($\Lambda$CDM). Yang et al. (2020) argued that self-interacting dark matter is
a favorable scenario for explaining the formation and existence of such
galaxies. Questioning the uncertainties of the data, modified Newtonian
dynamics (MOND) has also been proposed (e.g., Famaey et al., 2018; Haghi et
al., 2019). Also, Martin et al. (2018) questioned the validity of the data,
arguing that the discovered galaxy is a typical dwarf, but this claim was
rejected (e.g., van Dokkum et al., 2018a; Wasserman et al., 2018; Keim et al.,
2022). Furthermore, Saulder et al. (2020) warned that isolated dark matter-
deficient galaxies in simulations could represent a numerical artifact
(arising from problems with the algorithm identifying bound structures and the
periodic boundary conditions of the simulation boxes).

Numerous studies have since shown that these exotic galaxies are not in
tension with the $\Lambda$CDM model and that they may naturally occur due to
stronger tidal interactions (e.g., Ogiya, 2018; Montes et al., 2020; Shin et
al., 2020; Jackson et al., 2021; Macciò et al., 2021; Ogiya et al., 2022;
Trujillo-Gomez et al., 2022). In particular, Moreno et al. (2022) estimated
that $\sim 30\%$ massive central galaxies have at least one dark matter-
deficient satellite with stellar masses in the
$10^{8}-10^{9}\;\mathrm{M}_{\odot}$ range. Exploring the evolution of
satellite galaxies in IllustrisTNG simulation boxes, Montero-Dorta et al.
(2022) also considered dark matter-deficient systems and reinforced the notion
that the tidal stripping is responsible for the lack of dark matter.

In this context, the recent discovery (Comerón et al., 2023) of a massive NGC
1277 galaxy lacking dark matter, while also lacking a more massive companion,
was puzzling. These authors considered multiple possible scenarios to explain
this phenomenon: either the galaxy lost its dark matter early or it had been
lacking ab initio. To test the viability of the former assumption, we set our
aims on the search for candidates in IllustrisTNG. From the theoretical point
of view, the existence of backsplash galaxies (Borrow et al., 2023, and
references within), namely, ones that had resided inside a cluster and have
since migrated back outside, would favor the possibility that galaxies may, in
fact, ”run away” from dense environments. Moreover, Chilingarian & Zolotukhin
(2015) argued that the observed isolated compact elliptical and isolated
quiescent dwarf galaxies are tidally stripped systems that had run away from
their host clusters.

Here, we present one such example from IllustrisTNG300: an isolated dark
matter-poor galaxy that lost its extended dark matter envelope early. Since
then, it has dwelled in the cluster environment, interacting with the cluster
members without merging, while accelerating on its orbit; eventually, it left
the cluster, spending the last $\sim 2$ Gyr in isolation. This paper is
organized as follows. We describe our candidate selection criteria in Section
2. In Section 3, we present the evolutionary path of the galaxy separated into
stages. We discuss our results, along with the limitations and caveats of the
simulation in Section 4. We summarize our work and give our concluding remarks
in Section 5.

##  2 Candidate selection

We utilized the IllustrisTNG cosmological simulation suite111Publicly
available at https://www.tng-project.org/data/. (Weinberger et al., 2017;
Pillepich et al., 2018b; Nelson et al., 2019). Choosing an appropriate
simulation box was not a straightforward task. The largest simulation box,
TNG300 (Marinacci et al., 2018; Naiman et al., 2018; Nelson et al., 2018;
Pillepich et al., 2018a; Springel et al., 2018), has the poorest particle mass
resolution. However, due to its size, it allows us to explore galaxies
experiencing vastly different environments. Knowing that dark, matter-poor
galaxies are not always low-mass objects (e.g., Comerón et al., 2023), the
resolution limit should not pose a challenge. Moreover, the TNG300 box offers
the largest sample of galaxies, which would help identify extremely rare
present-day isolated dark, matter-poor galaxies.

We describe our selection criteria in the following. First, we considered only
the subhalos of cosmological origin (that have formed due to the process of
structure formation and collapse, and have $\texttt{SubhaloFlag}=1$222Meaning
that a subhalo should be considered as a galaxy. Otherwise, subhalos with
$\texttt{SubhaloFlag}=0$ are, at their formation time, satellites, tidal
debris, or kinematically distinct substructures of already formed galaxies. As
such, they typically have little to no dark matter and would contaminate our
search.). Then, at $z=0$, we extracted the ones that are isolated (i.e., they
are the only subhalo within their parent halo) and that have the dark-matter-
to-baryon-mass ratio of $M_{\mathrm{DM}}/M_{\mathrm{B}}<2$. While the filter
yielded 448 subhalos, the majority of them are poorly resolved. We chose the
best-resolved, most massive subhalo, ID3254912333SubfindID at $z=0$. (herein,
”the galaxy”). The galaxy has, at $z=0$, a total mass of $M\simeq 1.56\cdot
10^{10}\;\mathrm{M}_{\odot}$, with the mass of stellar component of
$M_{\star}\simeq 6.8\cdot 10^{9}\;\mathrm{M}_{\odot}$, no gas, and
$M_{\mathrm{DM}}/M_{\mathrm{B}}\simeq 1.31$. The respective half-mass radii
are $R_{0.5,\mathrm{DM}}=2.95\;\mathrm{kpc}$ and
$R_{0.5,\star}=2.45\;\mathrm{kpc}$444Throughout this work, all given units are
physical, for convenience..

##  3 Results

While the galaxy still contains a non-negligible amount of dark matter at
present, the total mass of the dark matter component is comparable to its
stellar counterpart, indicating it is a dark matter-poor galaxy. Additionally,
about half of the total dark matter mass is confined within the stellar half-
mass radius, suggesting a history of substantial stripping. Due to the
outside-in nature of the tidal stripping (e.g., Diemand et al., 2007; Choi et
al., 2009), the particles residing in the core could not be stripped without a
significant disruption of the stellar component and its mass-loss (e.g., Smith
et al., 2016; Łokas, 2020). In the following, we present and discuss the
galaxy’s cosmological evolution according to its subsequent stages.

###  3.1 Mergers and early history

The galaxy formed at $z=10$, with $M\simeq 2.6\cdot
10^{9}\;\mathrm{M}_{\odot}$ and $M_{\mathrm{DM}}/M_{\mathrm{B}}\simeq 5.88$.
Until $z=7.01$, just before its merger phase, it acquired a substantial amount
of dark matter through accretion, resulting in $M\simeq 9.7\cdot
10^{9}\;\mathrm{M}_{\odot}$ and $M_{\mathrm{DM}}/M_{\mathrm{B}}\simeq 10.81$.

According to the SubLink (Rodriguez-Gomez et al., 2015) data, this galaxy
merged with 16 subhalos. The merger phase spanned over 17 early snapshots and
lasted for $1.84\;\mathrm{Gyr}$. The galaxy had only one major merger, with a
mass ratio of $q\simeq 0.599$ at $z=3.71$. However, given that subsequent
mergers have mass ratios significantly below $q=0.1$, these may have been
misclassified. They are likely to be accreted non-cosmological subhalos (i.e.,
clumps). Checking the merger history catalog (Rodriguez-Gomez et al., 2017;
Eisert et al., 2023) confirms this assumption. The galaxy has had two mergers
in total: one major and one minor, which were concluded in nearly subsequent
snapshots. Considering this and typical merger timescales, these are not
likely to be two separate events (i.e., binary mergers) but, rather, a single
multiple merger.

After this phase, at $z=2.44$ (just before the first infall), the galaxy
reached $M\simeq 1.17\cdot 10^{11}\;\mathrm{M}_{\odot}$, with
$M_{\mathrm{DM}}/M_{\mathrm{B}}\simeq 6.9$ and with the total masses of sub-
components (dark matter, gas, and stars) of: $M_{\mathrm{DM}}\simeq 1.03\cdot
10^{11}\;\mathrm{M}_{\odot}$, $M_{\mathrm{G}}\simeq 1.43\cdot
10^{10}\;\mathrm{M}_{\odot}$, and $M_{\star}\simeq 5.94\cdot
10^{8}\;\mathrm{M}_{\odot}$. The respective half-mass radii are:
$R_{0.5,\mathrm{DM}}=19.71\;\mathrm{kpc}$,
$R_{0.5,\mathrm{G}}=20.82\;\mathrm{kpc}$ and
$R_{0.5,\star}=3.36\;\mathrm{kpc}$. The galaxy has also acquired a black hole
particle, with $M_{\bullet}\simeq 1.82\cdot 10^{7}\;\mathrm{M}_{\odot}$.

###  3.2 First infall and intense phase

The galaxy starts its first infall into a small group with $M\simeq 1.92\cdot
10^{12}\;\mathrm{M}_{\odot}$, containing a little over 20 subhalos (herein,
”members,” when referring to subhalos within groups and clusters) at $z=2.32$.
As the infall starts, the galaxy immediately interacts with a group of already
interacting galaxies, one of which is the most massive galaxy of the group. In
this process, the galaxy abruptly gets stripped of about half its total mass
from the outskirts, both dark matter and gas.

The evolution of the relevant parameters are shown in Figure 1. Since the
galaxy is almost constantly tidally disturbed during this intense phase, as a
measure of its center, we used the position of a particle with the lowest
potential energy, instead of a calculated center of mass. This provides a more
accurate estimate, as the difference between the two can become significant
due to tidal structures (e.g., Mitrašinović, 2022). Aside from being tidally
disturbed, during this period, the galaxy occasionally and briefly becomes a
satellite of more massive cluster members as it interacts with them. This can
be roughly estimated from the irregularities in its orbit and relative
velocity curve.

During its first pericentric passage, the galaxy accretes some dark matter and
gaseous particles, and the star formation rate (SFR) increases, producing
stars whose total mass consequentially increases. At $z=1.6$ (snapshot 38 and
lookback time of $\simeq 9.77\;\mathrm{Gyr}$), the group merges with a nearby
cluster of $M\simeq 1.61\cdot 10^{13}\;\mathrm{M}_{\odot}$ containing a little
over $100$ members. Upon this event, the galaxy loses the remaining majority
of its dark matter (at $z=1.53$), while accreting a considerable amount of gas
on its way to the first pericenter ($z=1.3$, snapshot 43, and lookback time of
$\simeq 8.99\;\mathrm{Gyr}$). This triggers a burst of star formation, sharply
increasing the stellar mass to essentially present-day value. During this
period, the dark matter to baryon mass ratio $M_{\mathrm{DM}}/M_{\mathrm{B}}$
drops as low as $0.52$. While a fraction of accreted gas is transformed into
stars, its majority gets stripped between the second and third pericentric
passage, and the galaxy becomes gas-free. As a consequence,
$M_{\mathrm{DM}}/M_{\mathrm{B}}$ fluctuates around $1$ until the present
(i.e., $z=0$).

Figure 1: Total mass of galactic components (dark matter, stars, gas) shown
from top to bottom, with their half-mass radii, $R_{0.5}$ (different colors
represent different components, as per the legend); star formation rate (SFR);
relative distance from the center of the group or cluster (blue, left
$y$-axis), and the galaxy’s relative velocity (orange, right $y$-axis). The
dashed vertical line represents the moment when the group merges with a nearby
cluster.

It is worth mentioning that the galaxy loses its acquired black hole particle
near the first pericenter in the cluster. However, this is not a physical
process but stems from the black hole re-centering scheme employed in the
simulation. The problem was highlighted by Borrow et al. (2023) and it is
known to be quite common among backsplash galaxies.

###  3.3 Cluster dwelling and the isolated phase

We go on to consider the next phase when the host cluster merges with a nearby
massive cluster at $z=0.58$ (snapshot 64 and lookback time of $\simeq
5.72\;\mathrm{Gyr}$). The resulting cluster has $M\simeq 8.25\cdot
10^{14}\;\mathrm{M}_{\odot}$ and over $4000$ members. In Figure 2, we show the
relevant evolution parameters in this later phase. During its time spent in
this cluster, the galaxy blazes through, reaching the first and only
pericenter around $z=0.31$ (snapshot 77 and lookback time $\simeq
3.62\;\mathrm{Gyr}$) and its relative velocity peaking at
$v_{\mathrm{S}}-v_{\mathrm{G}}\simeq 3016.95\;\mathrm{km}\;\mathrm{s^{-1}}$.
Around the pericenter, the galaxy loses a small fraction of its stellar mass
and almost all its dark matter particles at the outskirts.

Figure 2: Same format and notation as in Figure 1, with the exclusion of gas
(and by extent SFR) and the inclusion of a shaded area, representing the time
galaxy spends in isolation.

On its way to the apocenter, it accretes back some stellar and dark matter
particles until it engages in an interaction with multiple cluster members
near the cluster outskirts at $d/R_{200}\simeq 0.84$. Upon that event, the
galaxy suffers mild to moderate tidal stripping, accelerates, diverts its
orbit, and gets on its way out of the cluster. This interaction lasts for no
more than four consecutive snapshots, making it almost impossible to examine
its characteristics in detail. To isolate the interaction’s contribution from
global cluster effects, we approximate the cluster contribution with smooth
NFW potential (Navarro et al., 1997), which corresponds to the calculated
cluster density profile. We calculated the galaxy’s orbit in this idealized
smooth cluster potential and the results are shown in Figure 3. The orbit is
diverted during the interaction, and after it ends, the galaxy gets kicked
outside the cluster, as the difference between the realistic and expected
distances becomes considerably high. At the same time, the galaxy accelerates,
and its real relative velocity compared to the expected values is about twice
as high at all times, peaking at $1371\;\mathrm{km}\;\mathrm{s^{-1}}$ when the
expected relative velocity is $640\;\mathrm{km}\;\mathrm{s^{-1}}$. This
process remarkably resembles the gravitational slingshot mechanism.

While it is still gravitationally bound to the host, the cluster merges with
the most massive other cluster in the simulation ($M\simeq 2.64\cdot
10^{15}\;\mathrm{M}_{\odot}$ and a little over $12000$ members) at $z=0.2$
(snapshot 84 and lookback time $=2.48\;\mathrm{Gyr}$). The galaxy is located
on the opposite side (from where the cluster merger occurs), at a distance of
$d/R_{200}\simeq 3.06$ from the new host. Since the cluster merger is not an
instantaneous process, in a few subsequent snapshots, we can see that the
galaxy is still gravitationally bound to its previous host, until it gets
detected as isolated at $z=0.15$ (snapshot 87 and lookback time $\simeq
2\;\mathrm{Gyr}$).

Figure 3: Relative distance from the cluster’s center (top) and the galaxy’s
relative velocity (bottom). Solid blue line (data) represents the real values,
as seen in Figure 2. Dashed black line (extrapolated) represents the data
obtained by calculating galaxy’s orbit in an idealized smooth cluster
potential.

Since for $\sim 2\;\mathrm{Gyr}$, it is dwelling in isolation, just outside
the most massive cluster in the simulation box, with essentially no changes in
its mass and with its stellar half-mass radius slightly stretching. Chaotic
variations in the dark matter half-mass radius, $R_{0.5,\mathrm{DM}}$, are due
to particle resolution. The number of dark matter particles fluctuates around
$140$, as the galaxy constantly captures and loses single particles at its
outskirts. The present-day stellar photometric data for SDSS bands, in
particular $g$ and $r$, are taken from Nelson et al. (2018). With
$(g-r)=(-23.153+24.002)\simeq 0.85$, the galaxy is quite red and almost
perfectly spherical, according to the stellar mass tensor obtained from the
supplementary catalog (Genel et al., 2015). An independent calculation of the
stellar mass tensor, going back to the time when the stellar component becomes
well-resolved for such task, confirms that the galaxy remains almost perfectly
spherical at all times. The result is expected, as the stars form at the very
core of a tidally stripped dark matter halo.

##  4 Discussion

The galaxy presented in this work has had quite an interesting evolution,
which cannot be considered typical in any way. With
$M_{\mathrm{DM}}/M_{\mathrm{B}}\simeq 1.31$, it certainly is dark matter-poor,
but the dark matter still accounts for a little over $50\%$ of the total mass
(with a large fraction of it confined within the stellar half-mass radius).
This discrepancy with the observed dark matter-deficient galaxies could be
because simulated galaxies typically have higher central dark matter fractions
(e.g., Lovell et al., 2018), which are not easily stripped. Particle
resolution can also play a role and this mass ratio should thus be considered
an upper limit.

While its present-day color, shape, and extent make it easy to argue that the
galaxy should be classified as a spherical dwarf (dSph), its stellar component
is overly massive for a dwarf galaxy (Revaz & Jablonka, 2018). Hence, we may
claim the galaxy is a massive compact galaxy (MCG), but this would not satisfy
the compact observational criterion given by Barro et al. (2013) or a more
liberal one given by Damjanov et al. (2015), nor is it massive enough overall
(e.g., Lohmann et al., 2023). Given that almost the entirety of its stellar
component is $\sim 9\;\mathrm{Gyr}$ old and it has largely remained unchanged
since its formation, the galaxy can represent an example of a so-called relic
galaxy (Ferré-Mateu et al., 2015). Observationally confirmed relics (also
known as red nuggets) are typically more massive than our example galaxy
(Lisiecki et al., 2023), but this assumption may be due to bias.

Interestingly, since the galaxy lost the majority of its dark matter very
early, it does not represent a tidally stripped remnant of a massive galaxy.
Instead, its present-day stellar component formed in a single, brief starburst
episode at the core of an already stripped dark matter halo. Even though the
galaxy does not satisfy the criteria for compact-size galaxies, we argue that
the early loss of an extended dark matter and the gaseous envelope has made it
compact enough to avoid mergers in higher-density environments. Moreover,
since the stripping happened very early on, massive galaxy clusters had not
yet formed. This, in addition to its fairly compact size, has helped the
galaxy avoid mergers and limited its interactions to members of proto-clusters
and later clusters through non-merger events, as it has continued to
accelerate on its orbit. After one such interaction that appears to have
played out as essentially a gravitational slingshot, the galaxy left the most
massive cluster in the simulation. This is the first example found in
simulations of both an isolated dark matter-poor galaxy that lost its extended
envelope early and a fairly compact stellar system that managed to run away.
As the loss of the extended envelope is related to the tidal stripping and the
galaxy never crossed the boundaries of the simulation box, the example is
physically grounded and does not represent a numerical artifact (Saulder et
al., 2020).

##  5 Summary and conclusions

After the discovery of dark matter-deficient galaxies and the numerous studies
that followed, a consensus formed around the notion that these exotic objects
could naturally occur in the $\Lambda$CDM model due to stronger tidal
interactions through the stripping process. They are typically satellites,
with stellar masses in the $10^{8}-10^{9}\;\mathrm{M}_{\odot}$ range, of more
massive galaxies. Thus, the recent discovery of a massive galaxy lacking dark
matter, without a more massive neighbor, was puzzling. The authors responsible
for the discovery discussed two possible scenarios for this to happen, one of
which was that the galaxy lost its dark matter early. To test the viability of
this scenario, we searched for examples in IllustrisTNG300 cosmological
simulation. In this work, we present the evolutionary path of one such
example. The first dark matter-poor galaxy in an isolated environment found in
cosmological simulations, which is physically grounded and does not represent
a numerical artifact.

The galaxy has, at present, a stellar mass of $M_{\star}\simeq 6.8\cdot
10^{9}\;\mathrm{M}_{\odot}$, no gas, $M_{\mathrm{DM}}/M_{\mathrm{B}}\simeq
1.31$, and stellar half-mass radius of $R_{0.5,\star}=2.45\;\mathrm{kpc}$.
While its characteristics do not match those of the NGC 1277 galaxy, its
evolutionary path is a perfect example of how a galaxy that loses its extended
dark matter and gaseous envelope early, becoming dark matter-poor, can manage
to run away from cluster environment and massive neighbors.

The galaxy lost the majority of its dark matter early (starting with a multi-
body non-merger encounter), between $z=2.32$ and $z=1.53$ (about $9.6$ Gyr
ago), even before it formed its present-day stellar component. On its way to
the first pericentric passage in the first proto-cluster, around $8.99$ Gyr
ago, it accreted a considerable amount of gas, triggering a burst of star
formation and sharply increasing the stellar mass to essentially present-day
value. While a fraction of accreted gas was then transformed into stars, the
majority was stripped between the second and third pericentric passage and the
galaxy eventually became gas-free ($\sim 6.67$ Gyr ago). As a consequence,
$M_{\mathrm{DM}}/M_{\mathrm{B}}$ fluctuates around $1$ until present (i.e.,
$z=0$).

After the host protocluster merged with a nearby massive galaxy cluster $\sim
5.72\;\mathrm{Gyr}$ ago, the galaxy blazed through the new cluster, seemingly
unnoticed, with a single pericentric passage. On its way to the apocenter, it
had a non-merger interaction with multiple cluster members near the outskirts,
then accelerating, diverting its orbit, and eventually leaving the cluster.
Since then, for the last $\sim 2\;\mathrm{Gyr}$, it has remained largely
unchanged and isolated, residing just outside the most massive cluster in the
simulation.

For this scenario to be viable, a galaxy has to lose its extended dark matter
envelope very early, that is, before massive galaxy clusters formed. In this
way, a galaxy has become compact enough early on (although not necessarily
compact by standard definitions and constraints), which has allowed it to
avoid mergers in dense environments and to interact with cluster members only
through non-merger events. During these interactions, it has been able to
accelerate and divert its orbit, eventually ending up leaving the cluster.

The uniqueness of the galaxy presented in this work and its peculiar evolution
suggest that the whole evolutionary path, particularly in the later phase,
represents a series of particular circumstances. However, circumstantial
events, despite being rare, are physically grounded. The upcoming public
release of the MillenniumTNG simulations555https://www.mtng-project.org/ will
primarily expand our knowledge and understanding of large-scale clustering as
well as the formation and evolution of galaxy clusters (e.g., Bose et al.,
2023; Hernández-Aguayo et al., 2023; Pakmor et al., 2023). As an appealing
consequence, it will also provide fertile ground for the study of intercluster
dynamics and the evolution of peculiar galaxy classes and events, as well as
of environmental effects in general, with much better statistics than have
been previously available.

###### Acknowledgements.

This research was supported by the Ministry of Science, Technological
Development and Innovation of the Republic of Serbia (MSTDIRS) through the
contract no. 451-03-47/2023-01/200002 made with Astronomical Observatory
(Belgrade, Serbia).

## References

  * Barro et al. (2013) Barro, G., Faber, S. M., Pérez-González, P. G., et al. 2013, ApJ, 765, 104
  * Borrow et al. (2023) Borrow, J., Vogelsberger, M., O’Neil, S., McDonald, M. A., & Smith, A. 2023, MNRAS, 520, 649
  * Bose et al. (2023) Bose, S., Hadzhiyska, B., Barrera, M., et al. 2023, MNRAS, 524, 2579
  * Chilingarian & Zolotukhin (2015) Chilingarian, I. & Zolotukhin, I. 2015, Science, 348, 418
  * Choi et al. (2009) Choi, J.-H., Weinberg, M. D., & Katz, N. 2009, MNRAS, 400, 1247
  * Comerón et al. (2023) Comerón, S., Trujillo, I., Cappellari, M., et al. 2023, A&A, 675, A143
  * Damjanov et al. (2015) Damjanov, I., Geller, M. J., Zahid, H. J., & Hwang, H. S. 2015, ApJ, 806, 158
  * Diemand et al. (2007) Diemand, J., Kuhlen, M., & Madau, P. 2007, ApJ, 667, 859
  * Eisert et al. (2023) Eisert, L., Pillepich, A., Nelson, D., et al. 2023, MNRAS, 519, 2199
  * Famaey et al. (2018) Famaey, B., McGaugh, S., & Milgrom, M. 2018, MNRAS, 480, 473
  * Ferré-Mateu et al. (2015) Ferré-Mateu, A., Mezcua, M., Trujillo, I., Balcells, M., & van den Bosch, R. C. E. 2015, ApJ, 808, 79
  * Genel et al. (2015) Genel, S., Fall, S. M., Hernquist, L., et al. 2015, ApJ, 804, L40
  * Haghi et al. (2019) Haghi, H., Kroupa, P., Banik, I., et al. 2019, MNRAS, 487, 2441
  * Hernández-Aguayo et al. (2023) Hernández-Aguayo, C., Springel, V., Pakmor, R., et al. 2023, MNRAS, 524, 2556
  * Jackson et al. (2021) Jackson, R. A., Kaviraj, S., Martin, G., et al. 2021, MNRAS, 502, 1785
  * Keim et al. (2022) Keim, M. A., van Dokkum, P., Danieli, S., et al. 2022, ApJ, 935, 160
  * Lisiecki et al. (2023) Lisiecki, K., Małek, K., Siudek, M., et al. 2023, A&A, 669, A95
  * Lohmann et al. (2023) Lohmann, F. S., Schnorr-Müller, A., Trevisan, M., Ricci, T. V., & Clerici, K. S. 2023, MNRAS, 524, 5266
  * Łokas (2020) Łokas, E. L. 2020, A&A, 638, A133
  * Lovell et al. (2018) Lovell, M. R., Pillepich, A., Genel, S., et al. 2018, MNRAS, 481, 1950
  * Macciò et al. (2021) Macciò, A. V., Prats, D. H., Dixon, K. L., et al. 2021, MNRAS, 501, 693
  * Marinacci et al. (2018) Marinacci, F., Vogelsberger, M., Pakmor, R., et al. 2018, MNRAS, 480, 5113
  * Martin et al. (2018) Martin, N. F., Collins, M. L. M., Longeard, N., & Tollerud, E. 2018, ApJ, 859, L5
  * Mitrašinović (2022) Mitrašinović, A. 2022, Serbian Astronomical Journal, 204, 39
  * Montero-Dorta et al. (2022) Montero-Dorta, A. D., Rodriguez, F., Artale, M. C., Smith, R., & Chaves-Montero, J. 2022, arXiv e-prints, arXiv:2212.12090
  * Montes et al. (2020) Montes, M., Infante-Sainz, R., Madrigal-Aguado, A., et al. 2020, ApJ, 904, 114
  * Moreno et al. (2022) Moreno, J., Danieli, S., Bullock, J. S., et al. 2022, Nature Astronomy [arXiv:2202.05836]
  * Naiman et al. (2018) Naiman, J. P., Pillepich, A., Springel, V., et al. 2018, MNRAS, 477, 1206
  * Navarro et al. (1997) Navarro, J. F., Frenk, C. S., & White, S. D. M. 1997, ApJ, 490, 493
  * Nelson et al. (2018) Nelson, D., Pillepich, A., Springel, V., et al. 2018, MNRAS, 475, 624
  * Nelson et al. (2019) Nelson, D., Springel, V., Pillepich, A., et al. 2019, Computational Astrophysics and Cosmology, 6, 2
  * Ogiya (2018) Ogiya, G. 2018, MNRAS, 480, L106
  * Ogiya et al. (2022) Ogiya, G., van den Bosch, F. C., & Burkert, A. 2022, MNRAS, 510, 2724
  * Pakmor et al. (2023) Pakmor, R., Springel, V., Coles, J. P., et al. 2023, MNRAS, 524, 2539
  * Pillepich et al. (2018a) Pillepich, A., Nelson, D., Hernquist, L., et al. 2018a, MNRAS, 475, 648
  * Pillepich et al. (2018b) Pillepich, A., Springel, V., Nelson, D., et al. 2018b, MNRAS, 473, 4077
  * Revaz & Jablonka (2018) Revaz, Y. & Jablonka, P. 2018, A&A, 616, A96
  * Rodriguez-Gomez et al. (2015) Rodriguez-Gomez, V., Genel, S., Vogelsberger, M., et al. 2015, MNRAS, 449, 49
  * Rodriguez-Gomez et al. (2017) Rodriguez-Gomez, V., Sales, L. V., Genel, S., et al. 2017, MNRAS, 467, 3083
  * Saulder et al. (2020) Saulder, C., Snaith, O., Park, C., & Laigle, C. 2020, MNRAS, 491, 1278
  * Shin et al. (2020) Shin, E.-j., Jung, M., Kwon, G., et al. 2020, ApJ, 899, 25
  * Smith et al. (2016) Smith, R., Choi, H., Lee, J., et al. 2016, ApJ, 833, 109
  * Springel et al. (2018) Springel, V., Pakmor, R., Pillepich, A., et al. 2018, MNRAS, 475, 676
  * Trujillo-Gomez et al. (2022) Trujillo-Gomez, S., Kruijssen, J. M. D., & Reina-Campos, M. 2022, MNRAS, 510, 3356
  * van Dokkum et al. (2018a) van Dokkum, P., Cohen, Y., Danieli, S., et al. 2018a, Research Notes of the American Astronomical Society, 2, 54
  * van Dokkum et al. (2019) van Dokkum, P., Danieli, S., Abraham, R., Conroy, C., & Romanowsky, A. J. 2019, ApJ, 874, L5
  * van Dokkum et al. (2018b) van Dokkum, P., Danieli, S., Cohen, Y., et al. 2018b, Nature, 555, 629
  * Wasserman et al. (2018) Wasserman, A., Romanowsky, A. J., Brodie, J., et al. 2018, ApJ, 863, L15
  * Weinberger et al. (2017) Weinberger, R., Springel, V., Hernquist, L., et al. 2017, MNRAS, 465, 3291
  * Yang et al. (2020) Yang, D., Yu, H.-B., & An, H. 2020, Phys. Rev. Lett., 125, 111105

##  Appendix A The first infall: A visual example

Figure 4 was generated using the IllustrisTNG visualization tool666Available
at https://www.tng-project.org/data/vis/. and shows the galaxy in two
consecutive moments before and after infall begins.

Figure 4:  Galaxy at $z=2.44$ (snapshot 29) just before the first infall (top)
and $z=2.32$ (snapshot 30) when the infall starts (bottom). Left column shows
the stellar density, while the right column shows the gas column density.


# Lessons from the Bible on Modern Topics:
Low-Resource Multilingual Topic Model Evaluation

Shudong Hao
Computer Science
University of Colorado
Boulder, CO
<EMAIL_ADDRESS>&Jordan Boyd-Graber
Computer Science, iSchool,
lsc, and umiacs,
University of Maryland
College Park, MD
<EMAIL_ADDRESS>&Michael J. Paul
Information Science
University of Colorado
Boulder, CO
<EMAIL_ADDRESS>
###### Abstract

Multilingual topic models enable document analysis across languages through
coherent multilingual summaries of the data. However, there is no standard and
effective metric to evaluate the quality of multilingual topics. We introduce
a new intrinsic evaluation of multilingual topic models that correlates well
with human judgments of multilingual topic coherence as well as performance in
downstream applications. Importantly, we also study evaluation for low-
resource languages. Because standard metrics fail to accurately measure topic
quality when robust external resources are unavailable, we propose an
adaptation model that improves the accuracy and reliability of these metrics
in low-resource settings.

##  1 Introduction

Topic models provide a high-level view of the main themes of a document
collection Boyd-Graber et al. (2017). Document collections, however, are often
not in a single language, driving the development of multilingual topic
models. These models discover topics that are consistent across languages,
providing useful tools for multilingual text analysis Vulić et al. (2015),
such as detecting cultural differences Gutiérrez et al. (2016) and bilingual
dictionary extraction Liu et al. (2015).

Monolingual topic models can be evaluated through likelihood Wallach et al.
(2009b) or coherence Newman et al. (2010), but topic model evaluation is not
well understood in multilingual settings. Our contributions are two-fold. We
introduce an improved intrinsic evaluation metric for multilingual topic
models, called Crosslingual Normalized Pointwise Mutual Information (cnpmi,
Section 2). We explore the behaviors of cnpmi at both the model and topic
levels with six language pairs and varying model specifications. This metric
correlates well with human judgments and crosslingual classification results
(Sections 5 and 6).

We also focus on evaluation in low-resource languages, which lack large
parallel corpora, dictionaries, and other tools that are often used in
learning and evaluating topic models. To adapt cnpmi to these settings, we
create a coherence estimator (Section 3) that extrapolates statistics derived
from antiquated, specialized texts like the Bible: often the only resource
available for many languages.

##  2 Evaluating Multilingual Coherence

A multilingual topic contains one topic for each language. For a multilingual
topic to be meaningful to humans (Figure 1), the meanings should be consistent
across the languages, in addition to coherent within each language (i.e., all
words in a topic are related).

This section describes our approach to evaluating the quality of multilingual
topics. After defining the multilingual topic model, we describe topic model
evaluation extending standard monolingual approaches to multilingual settings.

###  2.1 Multilingual Topic Modeling

Probabilistic topic models associate each document in a corpus with a
distribution over latent topics, while each topic is associated with a
distribution over words in the vocabulary. The most widely used topic model,
latent Dirichlet allocation (Blei et al., 2003, lda), can be extended to
connect languages. These extensions require additional knowledge to link
languages together.

One common encoding of multilingual knowledge is document links (indicators
that documents are parallel or comparable), used in polylingual topic models
Mimno et al. (2009); Ni et al. (2009). In these models, each document $d$
indexes a tuple of parallel/comparable language-specific documents,
$d^{(\ell)}$, and the language-specific “views” of a document share the
document-topic distribution $\theta_{d}$. The generative story for the
document-links model is:

1for  _each topic $k$ and each language $\ell$_ do

2 Draw a distribution over words $\phi_{\ell k}\sim\textrm{Dirichlet}(\beta)$;

3

4for  _each document tuple $d=\left(d^{(1)},\ldots,d^{(L)}\right)$_ do

5 Draw a distribution over topics $\theta_{d}\sim\textrm{Dirichlet}(\alpha)$;

6 for  _each language $\ell=1,\ldots,L$_ do

7 for  _each token $t\in d^{(\ell)}$_ do

8 Draw a topic $z_{n}\sim\theta_{d}$;

9 Draw a word $w_{n}\sim\phi_{\ell z}$;

10

Alternatively, word translations Jagarlamudi and Daumé III (2010), concept
links Gutiérrez et al. (2016); Yang et al. (2017), and multi-level priors
Krstovski et al. (2016) can also provide multilingual knowledges. Since the
polylingual topic model is the most common approach for building multilingual
topic models Vulić et al. (2013, 2015); Liu et al. (2015); Krstovski and Smith
(2016), our study will focus on this model.

###  2.2 Monolingual Evaluation

Most automatic topic model evaluation metrics use co-occurrence statistics of
word pairs from a reference corpus to evaluate topic coherence, assuming that
coherent topics contain words that often appear together Newman et al. (2010).
The most successful Lau et al. (2014) is normalized pointwise mutual
information (Bouma, 2009, npmi). npmi compares the joint probability of words
appearing together $\Pr(w_{i},w_{j})$ to their probability assuming
independence $\Pr(w_{i})\Pr(w_{j})$, normalized by the joint probability:

$\displaystyle\textsc{npmi}{}(w_{i},w_{j})=\frac{\log\frac{\Pr(w_{i},w_{j})}{\Pr(w_{i})\Pr(w_{j})}}{\log\Pr(w_{i},w_{j})}.$
(1)

The word probabilities are calculated from a reference corpus, $\mathcal{R}$,
typically a large corpus such as Wikipedia that can provide meaningful co-
occurrence patterns that are independent of the target dataset.

The quality of topic $k$ is the average npmi of all word pairs $(w_{i},w_{j})$
in the topic:

$\displaystyle\textsc{npmi}{}_{k}=\frac{-1}{\binom{C}{2}}\sum_{i\in\mathcal{W}(k,C)}\sum_{j\neq
i}\textsc{npmi}{}(w_{i},w_{j}),$ (2)

where $\mathcal{W}(k,C)$ are the $C$ most probable words in the topic-word
distribution $\phi_{k}$ (the number of words is the topic’s cardinality).
Higher $\textsc{npmi}{}_{k}$ means the topic’s top words are more coupled.

###  2.3 Existing Multilingual Evaluations

While automatic evaluation has been well-studied for monolingual topic models,
there are no robust evaluations for multilingual topic models. We first
consider two straightforward metrics that could be used for multilingual
evaluation, both with limitations. We then propose an extension of npmi that
addresses these limitations.

#### Internal Coherence.

A simple adaptation of npmi is to calculate the monolingual npmi score for
each language independently and take the average. We refer this as internal
npmi (inpmi) as it evaluates coherence within a language. However, this metric
does not consider whether the topic is coherent across languages—that is,
whether a language-specific word distribution $\phi_{\ell_{1}k}$ is related to
the corresponding distribution in another language, $\phi_{\ell_{2}k}$.

#### Crosslingual Consistency.

Another straightforward measurement is Matching Translation Accuracy (Boyd-
Graber and Blei, 2009, mta), which counts the number of word translations in a
topic between two languages using a bilingual dictionary. This metric can
measure whether a topic is well-aligned across languages literally, but cannot
capture non-literal more holistic similarities across languages.

Figure 1: Topic 5 is multilingually coherent: both the English and Swedish
topics are about technology. Topic 6 is about biology in English but food in
Romanian, so it is low quality although coherent monolingually. Topic 7 is
monolingually incoherent, so it is a low quality topic even if it contains
word translations.

###  2.4 New Metric: Crosslingual npmi

We extend npmi to multilingual models, with a metric we call crosslingual
normalized pointwise mutual information (cnpmi). This metric will be the focus
of our experiments.

A multilingually coherent topic means that if $w_{i,\ell_{1}}$ in language
$\ell_{1}$ and $w_{j,\ell_{2}}$ in language $\ell_{2}$ are in the same topic,
they should appear in similar contexts in comparable or parallel corpora
$\mathcal{R}^{(\ell_{1},\ell_{2})}$. Our adaptation of npmi is based on the
same principles as the monolingual version, but focuses on the co-occurrences
of bilingual word pairs. Given a bilingual word pair
$\left(w_{i,\ell_{1}},w_{j,\ell_{2}}\right)$ the co-occurrence of this word
pair is the event where word $w_{i,\ell_{1}}$ appears in a document in
language $\ell_{1}$ and the word $w_{j,\ell_{2}}$ appears in a comparable or
parallel document in language $\ell_{2}$.

The co-occurrence probability of each bilingual word pair is:

$\displaystyle\begin{split}&\Pr\left(w_{i,\ell_{1}},w_{j,\ell_{2}}\right)\\\
&\triangleq\frac{\left|\left\\{{\mathbf{d}}:w_{i,\ell_{1}}\in{d}^{(\ell_{1})},w_{j,\ell_{2}}\in{d}^{(\ell_{2})}\right\\}\right|}{\left|\mathcal{R}^{(\ell_{1},\ell_{2})}\right|},\end{split}$
(3)

where $\mathbf{d}=\left(d^{(\ell_{1})},d^{(\ell_{2})}\right)$ is a pair of
parallel/comparable documents in the reference corpus
$\mathcal{R}^{(\ell_{1},\ell_{2})}$. When one or both words in a bilingual
pair do not appear in the reference corpus, the co-occurrence score is zero.

Similar to monolingual settings, cnpmi for a bilingual topic $k$ is the
average of the npmi scores of all $C^{2}$ bilingual word pairs,

$\displaystyle\textsc{cnpmi}{}(\ell_{1},\ell_{2},k)=\frac{\sum_{i,j}^{C}\textsc{npmi}{}\left(w_{i,\ell_{1}},w_{j,\ell_{2}}\right)}{C^{2}}.$
(4)

It is straightforward to generalize cnpmi from a language pair to multiple
languages by averaging cnpmi$(\ell_{i},\ell_{j},k)$ over all language pairs
$(\ell_{i},\ell_{j})$.

Figure 2: The coherence estimator takes multilingual topics and features from
them then outputs an estimated topic coherence.

##  3 Adapting to Low-Resource Languages

cnpmi needs a reference corpus for co-occurrence statistics. Wikipedia, which
has good coverage of topics and vocabularies is a common choice Lau and
Baldwin (2016). Unfortunately, Wikipedia is often unavailable or not large
enough for low-resource languages. It only covers $282$
languages,111https://meta.wikimedia.org/wiki/List_of_Wikipedias and only $249$
languages have more than $1{,}000$ pages: many of pages are short or unlinked
to a high-resource language. Since cnpmi requires comparable documents, the
usable reference corpus is defined by _paired_ documents.

Another option for a parallel reference corpus is the Bible Resnik et al.
(1999), which is available in most world languages;222The Bible is available
in $2{,}530$ languages. however, it is small and archaic. It is good at
evaluating topics such as family and religion, but not “modern” topics like
biology and Internet. Without reference co-occurrence statistics relevant to
these topics, cnpmi will fail to judge topic coherence—it must give the
ambiguous answer of zero. Such a score could mean a totally incoherent topic
where each word pair never appears together (Topics 6 in Figure 1), or an
unjudgeable topic (Topic 5).

Figure 3: As the estimator adds additional features, the estimated topic
coherence scores (solid lines) approach to Wikipedia cnpmi (dashed lines).

Our goal is to obtain a reliable estimation of topic coherence for low-
resource languages when the Bible is the only reference. We propose a model
that can correct the drawbacks of a Bible-derived cnpmi. While we assume
bilingual topics paired with English, our approach can be applied to any high-
resource/low-resource language pair.

We take Wikipedia’s cnpmi from high-resource languages as accurate
estimations. We then build a coherence estimator on topics from high-resource
languages, with the Wikipedia cnpmi as the target output. We use linear
regression using the below features. Given a topic in low-resource language,
the estimator produces an estimated coherence (Figure 2).

###  3.1 Estimator Features

The key to the estimator is to find features that capture whether we should
trust the Bible. For generality, we focus on features independent of the
available resources other than the Bible. This section describes the features,
which we split into four groups.

#### Base Features (base)

Our base features include information we can collect from the Bible and the
topic model: cardinality $C$, cnpmi and inpmi, mta, and topic word coverage
(twc), which counts the percentage of topic words in a topic that appear in a
reference corpus.

#### Crosslingual Gap (gap)

A low cnpmi score could indicate a topic pair where each language has a
monolingually coherent topic but that are not about the same theme (Topic 6 in
Figure 1). Thus, we add two features to capture this information using the
Bible: mismatch coefficients (mc) and internal comparison coefficients (icc):

$\displaystyle\textsc{mc}{}({\ell_{1};\ell_{2},k})$
$\displaystyle=\frac{\textsc{cnpmi}{}(\ell_{1},\ell_{2},k)}{\textsc{inpmi}{}(\ell_{1},k)+\alpha},$
(5) $\displaystyle\textsc{icc}{}(\ell_{1},\ell_{2},k)$
$\displaystyle=\frac{\textsc{inpmi}{}(\ell_{1},k)+\alpha}{\textsc{inpmi}{}(\ell_{2},k)+\alpha},$
(6)

where $\alpha$ is a smoothing factor ($\alpha=0.001$ in our experiments). mc
recognizes the gap between crosslingual and monolingual coherence, so a higher
mc score indicates a gap between coherence within and across languages.
Similarly, icc compares monolingual coherence to tell if both languages are
coherent: the closer to $1$ the icc is, the more comparable internal coherence
both languages have.

#### Word Era (era)

Because the Bible’s vocabulary is unable to evaluate modern topics, we must
tell the model what the modern words are. The word era features are the
earliest usage year 333https://oxforddictionaries.com/ for each word in a
topic. We use both the mean and standard deviation as features.

#### Meaning Drift (drift).

The meaning of a word can expand and drift over time. For example, in the
Bible, “web” appears in Isaiah 59:5:

> They hatch cockatrice’ eggs, and weave the spider’s web.

The word “web” could be evaluated correctly in an animal topic. For modern
topics, however, Bible fails to capture modern meanings of “web”, as in Topic
5 (Figure 1).

To address this meaning drift, we use a method similar to Hamilton et al.
(2016). For each English word, we calculate the context vector from Bible and
from Wikipedia with a window size of five and calculate the cosine similarity
between them as word similarity. Similar context vectors mean that the usage
in the Bible is consistent with Wikipedia. We calculate word similarities for
all the English topic words in a topic and use the average and standard
deviation as features.

###  3.2 Example

In Figure 3, Topic 1 is coherent while Topic 8 is not. From left to right, we
incrementally add new feature sets, and show how the estimated topic coherence
scores (dashed lines) approach the ideal cnpmi (dotted lines). When only using
the base features, the estimator gives a higher prediction to Topic 8 than to
Topic 1. Their low mta and twc prevent accurate evaluations. Adding gap does
not help much. However, $\textsc{icc}{}(\textsc{en},\textsc{am},k=1)$ is much
smaller, which might indicate a large gap of internal coherence between the
two languages.

Adding era makes the estimated scores flip between the two topics. Topic 1 has
word era of $1823$, much older than Topic 8’s word era of $1923$, indicating
that Topic 8 includes modern words the Bible lacks (e.g., “computer”). Using
all the features, the estimator gives more accurate topic coherence
evaluations.

Pair | Training | Reference
---|---|---
|  | Wikipedia | The Bible | Wiktionary
en-ro | $1{,}272$ | $8{,}126$ | $1{,}189$ | $29{,}836$
en-sv | $3{,}378$ | $9{,}067$ | $1{,}189$ | $42{,}953$
en-am | $421$ | $1{,}581$ | $1{,}189$ | $1{,}091$
en-tl | $542$ | $4{,}166$ | $1{,}189$ | $10{,}970$
en-tr | $874$ | $5{,}524$ | $1{,}189$ | $16{,}853$
en-zh | $874$ | $10{,}000$ | $1{,}189$ | $22{,}946$
Table 1: Number of document pairs in the training and reference datasets and
number of dictionary entries for each language pair.

##  4 Experiments: Bible to Wikipedia

We experiment on six languages (Table 1) from three corpora: Romanian (ro) and
Swedish (sv) from EuroParl as representative of well-studied and rich-resource
languages Koehn (2005); Amharic (am) and Tagalog (tl) from collected news, as
low-resource languages Huang et al. (2002a, b); and Chinese (zh) and Turkish
(tr) from TED Talks 2013 Tiedemann (2012), adding language variety to our
experiments. Each language is paired with English as a bilingual corpus.

Typical preprocessing methods (stemming, stop word removal, etc.) are often
unavailable for low-resource languages. For a meaningful comparison across
languages, we do not apply any stemming or lemmatization strategies, including
English, except removing digit numbers and symbols. However, we remove words
that appear in more than $30\%$ of documents for each language.

Each language pair is separately trained using the MALLET McCallum (2002)
implementation of the polylingual topic model. Each experiment runs five Gibbs
sampling chains with $1{,}000$ iterations per chain with twenty topics. The
hyperparameters are set to the default values ($\alpha=0.1$, $\beta=0.01$),
and are optimized every $50$ iterations in MALLET using slice sampling Wallach
et al. (2009a).

###  4.1 Evaluating Multilingual Topics

We use Wikipedia and the Bible as reference corpora for calculating co-
occurrence statistics. Different numbers of Wikipedia articles are available
for each language pair (Table 1), while the Bible contains a complete set of
$1{,}189$ chapters for all of its translations Christodoulopoulos and Steedman
(2015). We use Wiktionary as the dictionary to calculate mta.

###  4.2 Training the Estimator

In addition to experimenting on Wikipedia-based cnpmi, we also re-evaluate the
topics’ Bible coherence using our estimator. In the following experiments, we
use an AdaBoost regressor with linear regression as the coherence estimator
Friedman (2002); Collins et al. (2000). The estimator takes a topic and low-
quality cnpmi score as input and outputs (hopefully) an improved cnpmi score.

To make our testing scenario more realistic, we treat one language as our
estimator’s test language and train on multilingual topics from the other
languages. We use three-fold cross-validation over languages to select the
best hyperparameters, including the learning rate and loss function in
AdaBoost.R2 Drucker (1997).

Figure 4: The interface for topic quality judgments. Users read the topic
first, and make a judgment on whether the words in this pair are talking about
the same thing. The translations are here for illustration; they are not shown
to the users.

##  5 Topic-Level Evaluation

We first study cnpmi at the topic level: does a particular topic make sense?
An effective evaluation should be consistent with human judgment of the topics
Chang et al. (2009). In this section, we measure gold-standard human
interpretability of multilingual topics to establish which automatic measures
of topic interpretability work best.

| Wikipedia | The Bible | mta
---|---|---|---
| cnpmi | inpmi | cnpmi | inpmi
en-ro | $0.490$ | $0.118$ | $-0.096$ | $0.031$ | $\bm{0.592}$
en-sv | $\bm{0.453}$ | $-0.295$ | $0.164$ | $-0.351$ | $0.248$
en-am | $0.110$ | $0.019$ | $\bm{0.289}$ | $0.249$ | $0.172$
en-tl | $\bm{0.512}$ | $0.277$ | $0.166$ | $0.002$ | $0.289$
en-tr | $0.664$ | $0.243$ | $0.209$ | $-0.246$ | $\bm{0.677}$
en-zh | $\bm{0.436}$ | $0.297$ | $0.274$ | $0.157$ | $0.411$
Table 2: Pearson correlations between human judgments and cnpmi are higher
than inpmi, while mta correlations are comparable to cnpmi.

###  5.1 Task Design

Following monolingual coherence evaluations Lau et al. (2014), we present
topic pairs to bilingual CrowdFlower users. Each task is a topic pair with the
top ten topic words ($C=10$) for each language. We ask if both languages’ top
words in a multilingual topic are talking about the same concept (Figure 4),
and make a judgment on a three-point scale—coherent (2 points), somewhat
coherent (1 point), and incoherent (0 points). To ensure the users have
adequate language competency, we insert several topics that are easily
identifiable as incoherent as a qualification test.

We randomly select sixty topics from each language pair ($360$ topics total),
and each topic is judged by five users. We take the average of the judgment
points and calculate Pearson correlations with the proposed evaluation metrics
(Table 2). npmi-based scores are separately calculated from each reference
corpus.

Test | Bible | Train
---|---|---
|  | ro+sv | zh+tr | ro+sv+zh+tr
am | $-0.015$ | $0.332$ | $0.315$ | $0.333$
tl | $-0.309$ | $0.767$ | $0.631$ | $0.705$
|  | am+tl | zh+tr | am+tl+zh+tr
ro | $-0.269$ | $0.736$ | $0.681$ | $0.713$
sv | $0.000$ | $0.787$ | $0.645$ | $0.683$
|  | ro+sv | am+tl | ro+sv+am+tl
zh | $0.217$ | $0.751$ | $0.732$ | $0.741$
tr | $0.113$ | $0.680$ | $0.642$ | $0.666$
Table 3: Correlations between the Wikipedia-based cnpmi and the Bible-based
cnpmi, before and after using the coherence estimator, at the topic level.
Strong correlations indicate that the estimator improves cnpmi estimates.

###  5.2 Agreement with Human Judgments

cnpmi (the extended metric) has higher correlations with human judgments than
inpmi (the naive adaptation of monolingual npmi), while mta (matching
translation accuracy) correlations are comparable to cnpmi.

Unsurprisingly, when using Wikipedia as the reference, the correlations are
usually higher than when using the Bible. The Bible’s archaic content limits
its ability to estimate human judgments in modern corpora (Section 3).

Next, we compare cnpmi to two baselines: inpmi and mta. As expected, cnpmi
outperforms inpmi regardless of reference corpus overall, because inpmi only
considers monolingual coherence. mta has higher correlations than cnpmi scores
from the Bible, because the Bible fails to give accurate estimates due to
limited topic coverage. mta, on the other hand, only depends on dictionaries,
which are more comprehensive than the Bible. It is also possible that users
are judging coherence based on translations across a topic pair, rather than
the overall coherence, which would closely correlate with mta.

###  5.3 Re-Estimating Topic-Level Coherence

The Bible—by itself—produces cnpmi values that do not correlate well with
human judgments (Table 2). After training an estimator (Section 4.2), we
calculate Pearson’s correlation between Wikipedia’s cnpmi and the estimated
topic coherence score (Table 3). A higher correlation with Wikipedia’s cnpmi
means more accurate coherence.

As a baseline, the correlation of Bible-based cnpmi without adaptation has
negative and near-zero correlations with Wikipedia;444Normally one would not
estimate cnpmi on rich-resource languages using low-resource languages. For
completeness, however, we also include these situations. it does not capture
coherence. After training the estimator, the correlations become stronger,
indicating the estimated scores are closer to Wikipedia’s cnpmi.

Figure 5: mta fails to capture semantically related words (Topic 1) and only
looks at translation pairs regardless of internal coherence (Topic 2).

###  5.4 When mta Falls Short

We analyze mta from two aspects—the inability to capture semantically-related
non-translation topic words, and insensitivity to cardinality—to show why mta
is not an ideal measurement, even though it correlates well with human
judgments.

#### Semantics

We take two examples with en-zh (Topic 1) and en-tl (Topic 2) in Figure 5.
Topic 1 has fewer translation pairs than Topic 2, which leads to a lower mta
score for Topic 1. However, all words in Topic 1 talk about art, while it is
hard to interpret Topic 2. Wikipedia cnpmi scores reveals Topic 1 is more
coherent. Because our experiments are on datasets with little divergence
between the themes discussed across languages, this is uncommon for us but
could appear in noisier datasets.

#### Cardinality

Increasing cardinality diminishes a topic’s coherence Lau and Baldwin (2016).
We vary the cardinality of topics from ten to fifty at intervals of ten
(Figure 6). As cardinality increases, more low-probability and irrelevant
words appear the topic, which lowers cnpmi scores. However, mta stays stable
or increases with increasing cardinality. Thus, mta fails to fulfill a
critical property of topic model evaluation.

Figure 6: Increasing cardinality of topic pairs makes it harder to judge the
coherence. Decreasing cnpmi scores reflect the diminished interpretability of
topics, while mta scores do not.

Finally, mta requires a comprehensive multilingual dictionary, which may be
unavailable for low-resource languages. Additionally, most languages often
only have one dictionary, which makes it problematic to use the same resource
(a language’s single multilingual dictionary) for training and evaluating
models that use a dictionary to build multilingual topics Hu et al. (2014).
Given these concerns, we continue the paper’s focus on cnpmi as a data-driven
alternative to mta. However, for many applications mta may suffice as a
simple, adequate evaluation metric.

Figure 7: Adding more document links to the model produces more multilingually
coherent topics. cnpmi captures this improvement.

##  6 Model-Level Evaluation

While the previous section looked at individual topics, we also care about how
well cnpmi characterizes the quality of _models_ through an average of a
model’s constituent topics.

###  6.1 Training Knowledge

Adding more knowledge to multilingual topic models improves topics Hu et al.
(2014), so an effective evaluation should reflect this improvement as knowlege
is added to the model. For polylingual topic models, this knowledge takes the
form of the _number_ of linked documents.

We start by experimenting with no multilingual knowledge: no document pairs
share a topic distribution $\theta_{d}$ (but the documents are in the
collection as unlinked documents). We then increase the number of document
pairs that share $\theta_{d}$ from $20\%$ of the corpus to $100\%$. Fixing the
topic cardinality at ten, cnpmi captures the improvements in models (Figure 7)
through a higher coherence score.

Test | Bible | Train
---|---|---
|  | ro+sv | zh+tr | ro+sv+zh+tr
am |  $0.607$ | $0.677$ | $0.707$ | $0.694$
tl | $0.796$ | $0.875$ | $0.924$ | $0.918$
|  | am+tl | zh+tr | am+tl+zh+tr
ro | $0.631$ | $0.912$ | $0.919$ | $0.931$
sv | $0.797$ | $0.959$ | $0.848$ | $0.878$
|  | ro+sv | am+tl | ro+sv+am+tl
zh | $0.907$ | $0.918$ | $0.951$ | $0.939$
tr | $0.911$ | $0.862$ | $0.898$ | $0.887$
Table 4: At the model level, the estimator improves correlations between cnpmi
and downstream classification for all languages except for Turkish.

###  6.2 Agreement with Machines

Topic models are often used as a feature extraction technique for downstream
machine learning applications, and topic model evaluations should reflect
whether these features are useful Ramage et al. (2009). For each model, we
apply a document classifier trained on the model parameters to test whether
cnpmi is consistent with classification accuracy.

Specifically, we want our classifier to transfer information from training on
one language to testing on another Smet et al. (2011); Heyman et al. (2016).
We train a classifier on one language’s documents, where each document’s
feature vector is the document-topic distribution $\theta_{d}$. We apply this
to ted Talks, where each document is labeled with multiple categories. We
choose the most frequent seven categories across the corpus as
labels,555design, global issues, art, science, technology, business, and
culture and only have labeled documents in one side of a bilingual topic
model. cnpmi has very strong correlations with classification results, though
using the Bible as the reference corpus gives slightly lower correlation—with
higher variance—than Wikipedia (Figure 8).

Figure 8: Pearson correlation between classification F1 scores and cnpmi: both
cnpmi data sources predict whether a classifier using topic features will work
well, but Wikipedia has slightly higher correlation with lower variance.

###  6.3 Re-Estimating Model-Level Coherence

In Section 5.3, we improve Bible-based cnpmi scores for individual topics.
Here, we show the estimator also improves model-level coherence. We apply the
estimator on the models created in Section 6.2 and calculate the correlation
between estimated scores and Wikipedia’s cnpmi (Table 4).

The coherence estimator substantially improves scores except for Turkish: the
correlation is better _before_ applying the estimator ($0.911$). We suspect a
lack of overlap between topics between Turkish and languages other than
Chinese is to blame (Figure 9); the features used by the estimator do not
generalize well to other kinds of features; training on many languages pairs
would hopefully solve this issue. Turkish is also morphologically rich, and
our preprocessing completely ignores morphology.

Figure 9: The overlap of topics and domain: only one out of nine Turkish and
Chinese topics have domain overlap with Tagalog and Amharic topics. This
hinders the Turkish estimator from capturing model-level properties.

###  6.4 Reference Size

One challenge with low-resource languages is that even if Wikipedia is
available, it may have too few documents to accurately calculate coherence. As
a final analysis, we examine how the reliability of cnpmi degrades with a
smaller reference corpus.

We randomly sample $20\%$ to $100\%$ of document pairs from the reference
corpora and evaluate the polylingual topic model with all document links
(Figure 10), again fixing the cardinality as $10$.

cnpmi is stable across different amounts of reference documents, as long as
the number of reference documents is sufficiently large. If there are too few
reference documents (for example, $20\%$ of Amharic Wikipedia is only $316$
documents), then cnpmi degrades.

Figure 10: cnpmi is stable once the number of reference documents is large
enough (around five thousand documents).

##  7 Related Work

#### Topic Coherence

Many coherence metrics based on co-occurrence statistics have been proposed
besides npmi. Similar metrics—such as asymmetrical word pair metrics Mimno et
al. (2011) and combinations of existing measurements Lau et al. (2014); Röder
et al. (2015)—correlate well with human judgments. npmi has been the current
gold standard for evaluation and improvements of monolingual topic models
Pecina (2010); Newman et al. (2011).

#### External Tasks

Another approach is to use a model for predictive tasks: the better the
results are on external tasks, the better a topic model is assumed to be. A
common task is held-out likelihood Wallach et al. (2009b); Jagarlamudi and
Daumé III (2010); Fukumasu et al. (2012), but as Chang et al. (2009) show,
this does not always reflect human interpretability. Other specific tasks have
also been used, such as bilingual dictionary extraction Liu et al. (2015); Ma
and Nasukawa (2017), cultural difference deteciton Gutiérrez et al. (2016),
and crosslingual document clustering Vulić et al. (2015).

#### Representation Learning

Topic models are one example of a broad class of techniques of learning
representations of documents Bengio et al. (2013). Other approaches learn
respresentations at the word Klementiev et al. (2012); Vyas and Carpuat
(2016), paragraph Mogadala and Rettinger (2016), or corpus level Søgaard et
al. (2015). However, neural representation learning approaches are often data
hungry and not adaptable to low-resource languages. The approaches here could
help improve the evaluation of all multilingual representation learning
algorithms Schnabel et al. (2015).

##  8 Conclusion

We have provided a comprehensive analysis of topic model evaluation in
multilingual settings, including for low-resource languages. While evaluation
is an important area of topic model research, no previous work has studied
evaluation of multilingual topic models. Our work provided two primary
contributions to this area, including a new intrinsic evaluation metric,
cnpmi, as well as a model for adapting this metric to low-resource languages
without large reference corpora.

As the first study on evaluation for multilingual topic models, there is still
room for improvement and further applications. For example, human judgment is
more difficult to measure than in monolingual settings, and it is still an
open question on how to design a reliable and accurate survey for multilingual
quality judgments. As a measurement of multilingual coherence, we plan to
extend cnpmi to high-dimensional representations, e.g., multilingual word
embeddings, particularly in low-resource languages Ruder et al. (2017).

## Acknowledgement

We thank the anonymous reviewers for their insightful and constructive
comments. Hao has been supported under subcontract to Raytheon bbn
Technologies, by darpa award hr0011-15-c-0113. Boyd-Graber and Paul were
supported by nsf grant iis-1564275. Any opinions, findings, conclusions, or
recommendations expressed here are those of the authors and do not necessarily
reflect the view of the sponsors.

## References

  * Bengio et al. (2013) Yoshua Bengio, Aaron Courville, and Pascal Vincent. 2013.  Representation learning: A review and new perspectives.  IEEE Trans. Pattern Anal. Mach. Intell. 35(8):1798–1828.  https://doi.org/10.1109/TPAMI.2013.50.
  * Blei et al. (2003) David M. Blei, Andrew Ng, and Michael I. Jordan. 2003.  Latent Dirichlet Allocation.  Journal of Machine Learning Research 3:993–1022.  http://www.jmlr.org/papers/v3/blei03a.html.
  * Bouma (2009) Gerlof Bouma. 2009.  Normalized (Pointwise) Mutual Information in Collocation Extraction.  In Proceedings of the Biennial GSCL Conference. pages 31–40.
  * Boyd-Graber and Blei (2009) Jordan Boyd-Graber and David M. Blei. 2009.  Multilingual Topic Models for Unaligned Text.  In UAI 2009, Proceedings of the Twenty-Fifth Conference on Uncertainty in Artificial Intelligence, Montreal, QC, Canada, June 18-21, 2009. pages 75–82.  https://dslpitt.org/uai/displayArticleDetails.jsp?mmnu=1&smnu=2&article_id=1644&proceeding_id=25.
  * Boyd-Graber et al. (2017) Jordan Boyd-Graber, Yuening Hu, and David Mimno. 2017.  Applications of Topic Models, volume 11 of Foundations and Trends in Information Retrieval.  NOW Publishers.  http://www.nowpublishers.com/article/Details/INR-030.
  * Chang et al. (2009) Jonathan Chang, Jordan Boyd-Graber, Sean Gerrish, Chong Wang, and David M. Blei. 2009.  Reading Tea Leaves: How Humans Interpret Topic Models.  In Advances in Neural Information Processing Systems 22: 23rd Annual Conference on Neural Information Processing Systems 2009. Proceedings of a meeting held 7-10 December 2009, Vancouver, British Columbia, Canada.. pages 288–296.  http://papers.nips.cc/paper/3700-reading-tea-leaves-how-humans-interpret-topic-models.
  * Christodoulopoulos and Steedman (2015) Christos Christodoulopoulos and Mark Steedman. 2015.  A Massively Parallel Corpus: the Bible in 100 Languages.  Language Resources and Evaluation 49(2):375–395.  https://doi.org/10.1007/s10579-014-9287-y.
  * Collins et al. (2000) Michael Collins, Robert E. Schapire, and Yoram Singer. 2000.  Logistic Regression, AdaBoost and Bregman Distances.  In Proceedings of the Thirteenth Annual Conference on Computational Learning Theory (COLT 2000), June 28 - July 1, 2000, Palo Alto, California. pages 158–169.
  * Drucker (1997) Harris Drucker. 1997.  Improving Regressors Using Boosting Techniques.  In Proceedings of the Fourteenth International Conference on Machine Learning (ICML 1997), Nashville, Tennessee, USA, July 8-12, 1997. pages 107–115.
  * Friedman (2002) Jerome H Friedman. 2002.  Stochastic Gradient Boosting.  Computational Statistics & Data Analysis 38(4):367–378.
  * Fukumasu et al. (2012) Kosuke Fukumasu, Koji Eguchi, and Eric P. Xing. 2012.  Symmetric Correspondence Topic Models for Multilingual Text Analysis.  In Advances in Neural Information Processing Systems 25: 26th Annual Conference on Neural Information Processing Systems 2012. Proceedings of a meeting held December 3-6, 2012, Lake Tahoe, Nevada, United States.. pages 1295–1303.  http://papers.nips.cc/paper/4583-symmetric-correspondence-topic-models-for-multilingual-text-analysis.
  * Gutiérrez et al. (2016) E. Dario Gutiérrez, Ekaterina Shutova, Patricia Lichtenstein, Gerard de Melo, and Luca Gilardi. 2016.  Detecting Cross-Cultural Differences Using a Multilingual Topic Model.  Transactions of the Association for Computational Linguistics 4:47–60.  https://tacl2013.cs.columbia.edu/ojs/index.php/tacl/article/view/755.
  * Hamilton et al. (2016) William L. Hamilton, Jure Leskovec, and Dan Jurafsky. 2016.  Cultural Shift or Linguistic Drift? Comparing Two Computational Measures of Semantic Change.  In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA, November 1-4, 2016. pages 2116–2121.  http://aclweb.org/anthology/D/D16/D16-1229.pdf.
  * Heyman et al. (2016) Geert Heyman, Ivan Vulic, and Marie-Francine Moens. 2016.  C-BiLDA Extracting Cross-lingual Topics from Non-parallel Texts by Distinguishing Shared from Unshared Content.  Data Mining and Knowledge Discovery 30(5):1299–1323.  https://doi.org/10.1007/s10618-015-0442-x.
  * Hu et al. (2014) Yuening Hu, Ke Zhai, Vladimir Eidelman, and Jordan L. Boyd-Graber. 2014.  Polylingual Tree-Based Topic Models for Translation Domain Adaptation.  In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, ACL 2014, June 22-27, 2014, Baltimore, MD, USA, Volume 1: Long Papers. pages 1166–1176.  http://aclweb.org/anthology/P/P14/P14-1110.pdf.
  * Huang et al. (2002a) Shudong Huang, David Graff, and George Doddington. 2002a.  NMSU Amharic Language Pack from REFLEX V1.1.  Web download file. Philadelphia: Linguistic Data Consortium.
  * Huang et al. (2002b) Shudong Huang, David Graff, and George Doddington. 2002b.  Tagalog Language Pack from reflex V1.1.  Web download file. Philadelphia: Linguistic Data Consortium.
  * Jagarlamudi and Daumé III (2010) Jagadeesh Jagarlamudi and Hal Daumé III. 2010.  Extracting Multilingual Topics from Unaligned Comparable Corpora.  In Advances in Information Retrieval, 32nd European Conference on IR Research, ECIR. pages 444–456.
  * Klementiev et al. (2012) Alexandre Klementiev, Ivan Titov, and Binod Bhattarai. 2012.  Inducing Crosslingual Distributed Representations of Words.  In COLING 2012, 24th International Conference on Computational Linguistics, Proceedings of the Conference: Technical Papers, 8-15 December 2012, Mumbai, India. pages 1459–1474.  http://aclweb.org/anthology/C/C12/C12-1089.pdf.
  * Koehn (2005) Philipp Koehn. 2005.  Europarl: A Parallel Corpus for Statistical Machine Translation.  MT Summit .
  * Krstovski and Smith (2016) Kriste Krstovski and David A. Smith. 2016.  Bootstrapping Translation Detection and Sentence Extraction from Comparable Corpora.  In NAACL HLT 2016, The 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, San Diego California, USA, June 12-17, 2016. pages 1127–1132.  http://aclweb.org/anthology/N/N16/N16-1132.pdf.
  * Krstovski et al. (2016) Kriste Krstovski, David A. Smith, and Michael J. Kurtz. 2016.  Online Multilingual Topic Models with Multi-Level Hyperpriors.  In NAACL HLT 2016, The 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, San Diego California, USA, June 12-17, 2016. pages 454–459.  http://aclweb.org/anthology/N/N16/N16-1053.pdf.
  * Lau and Baldwin (2016) Jey Han Lau and Timothy Baldwin. 2016.  The Sensitivity of Topic Coherence Evaluation to Topic Cardinality.  In NAACL HLT 2016, The 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, San Diego California, USA, June 12-17, 2016. pages 483–487.  http://aclweb.org/anthology/N/N16/N16-1057.pdf.
  * Lau et al. (2014) Jey Han Lau, David Newman, and Timothy Baldwin. 2014.  Machine Reading Tea Leaves: Automatically Evaluating Topic Coherence and Topic Model Quality.  In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2014, April 26-30, 2014, Gothenburg, Sweden. pages 530–539.  http://aclweb.org/anthology/E/E14/E14-1056.pdf.
  * Liu et al. (2015) Xiaodong Liu, Kevin Duh, and Yuji Matsumoto. 2015.  Multilingual Topic Models for Bilingual Dictionary Extraction.  ACM Transactions on Asian & Low-Resource Language Information Processing 14(3):11:1–11:22.  https://doi.org/10.1145/2699939.
  * Ma and Nasukawa (2017) Tengfei Ma and Tetsuya Nasukawa. 2017.  Inverted Bilingual Topic Models for Lexicon Extraction from Non-parallel Data.  In Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI 2017, Melbourne, Australia, August 19-25, 2017. pages 4075–4081.  https://doi.org/10.24963/ijcai.2017/569.
  * McCallum (2002) Andrew Kachites McCallum. 2002.  MALLET: A Machine Learning for Language Toolkit.  http://mallet.cs.umass.edu.
  * Mimno et al. (2009) David M. Mimno, Hanna M. Wallach, Jason Naradowsky, David A. Smith, and Andrew McCallum. 2009.  Polylingual Topic Models.  In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, EMNLP 2009, 6-7 August 2009, Singapore, A meeting of SIGDAT, a Special Interest Group of the ACL. pages 880–889.  http://www.aclweb.org/anthology/D09-1092.
  * Mimno et al. (2011) David M. Mimno, Hanna M. Wallach, Edmund M. Talley, Miriam Leenders, and Andrew McCallum. 2011.  Optimizing Semantic Coherence in Topic Models.  In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, EMNLP 2011, 27-31 July 2011, John McIntyre Conference Centre, Edinburgh, UK, A meeting of SIGDAT, a Special Interest Group of the ACL. pages 262–272.  http://www.aclweb.org/anthology/D11-1024.
  * Mogadala and Rettinger (2016) Aditya Mogadala and Achim Rettinger. 2016.  Bilingual Word Embeddings from Parallel and Non-parallel Corpora for Cross-Language Text Classification.  In NAACL HLT 2016, The 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, San Diego California, USA, June 12-17, 2016. pages 692–702.  http://aclweb.org/anthology/N/N16/N16-1083.pdf.
  * Newman et al. (2011) David Newman, Edwin V. Bonilla, and Wray L. Buntine. 2011.  Improving Topic Coherence with Regularized Topic Models.  In Proceedings of Advances in Neural Information Processing Systems. pages 496–504.
  * Newman et al. (2010) David Newman, Jey Han Lau, Karl Grieser, and Timothy Baldwin. 2010.  Automatic Evaluation of Topic Coherence.  In Human Language Technologies: Conference of the North American Chapter of the Association of Computational Linguistics, Proceedings, June 2-4, 2010, Los Angeles, California, USA. pages 100–108.  http://www.aclweb.org/anthology/N10-1012.
  * Ni et al. (2009) Xiaochuan Ni, Jian-Tao Sun, Jian Hu, and Zheng Chen. 2009.  Mining Multilingual Topics from Wikipedia.  In Proceedings of the 18th International Conference on World Wide Web, WWW 2009, Madrid, Spain, April 20-24, 2009. pages 1155–1156.  https://doi.org/10.1145/1526709.1526904.
  * Pecina (2010) Pavel Pecina. 2010.  Lexical Association Measures and Collocation Extraction.  Language Resources and Evaluation 44(1-2):137–158.  https://doi.org/10.1007/s10579-009-9101-4.
  * Ramage et al. (2009) Daniel Ramage, David Leo Wright Hall, Ramesh Nallapati, and Christopher D. Manning. 2009.  Labeled LDA: A Supervised Topic Model for Credit Attribution in Multi-labeled Corpora.  In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, EMNLP 2009, 6-7 August 2009, Singapore, A meeting of SIGDAT, a Special Interest Group of the ACL. pages 248–256.  http://www.aclweb.org/anthology/D09-1026.
  * Resnik et al. (1999) Philip Resnik, Mari Broman Olsen, and Mona T. Diab. 1999.  The Bible as a Parallel Corpus: Annotating the ’Book of 2000 Tongues’.  Computers and the Humanities 33(1-2):129–153.  https://doi.org/10.1023/A:1001798929185.
  * Röder et al. (2015) Michael Röder, Andreas Both, and Alexander Hinneburg. 2015.  Exploring the Space of Topic Coherence Measures.  In Proceedings of the Eighth ACM International Conference on Web Search and Data Mining, WSDM 2015, Shanghai, China, February 2-6, 2015. pages 399–408.  https://doi.org/10.1145/2684822.2685324.
  * Ruder et al. (2017) Sebastian Ruder, Ivan Vulić, and Anders Søgaard. 2017.  A Survey of Cross-lingual Word Embedding Models.  CoRR abs/1706.04902.
  * Schnabel et al. (2015) Tobias Schnabel, Igor Labutov, David M. Mimno, and Thorsten Joachims. 2015.  Evaluation Methods for Unsupervised Word Embeddings.  In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, EMNLP 2015, Lisbon, Portugal, September 17-21, 2015. pages 298–307.  http://aclweb.org/anthology/D/D15/D15-1036.pdf.
  * Smet et al. (2011) Wim De Smet, Jie Tang, and Marie-Francine Moens. 2011.  Knowledge Transfer across Multilingual Corpora via Latent Topics.  In Advances in Knowledge Discovery and Data Mining - 15th Pacific-Asia Conference, PAKDD 2011, Shenzhen, China, May 24-27, 2011, Proceedings, Part I. pages 549–560.  https://doi.org/10.1007/978-3-642-20841-6_45.
  * Søgaard et al. (2015) Anders Søgaard, Zeljko Agic, Héctor Martínez Alonso, Barbara Plank, Bernd Bohnet, and Anders Johannsen. 2015.  Inverted Indexing for Cross-lingual NLP.  In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing of the Asian Federation of Natural Language Processing, ACL 2015, July 26-31, 2015, Beijing, China, Volume 1: Long Papers. pages 1713–1722.  http://aclweb.org/anthology/P/P15/P15-1165.pdf.
  * Tiedemann (2012) Jörg Tiedemann. 2012.  Parallel Data, Tools and Interfaces in OPUS.  In Proceedings of the Eighth International Conference on Language Resources and Evaluation, LREC 2012, Istanbul, Turkey, May 23-25, 2012. pages 2214–2218.  http://www.lrec-conf.org/proceedings/lrec2012/summaries/463.html.
  * Vulić et al. (2013) Ivan Vulić, Wim De Smet, and Marie-Francine Moens. 2013.  Cross-language Information Retrieval Models Based on Latent Topic Models Trained with Document-aligned Comparable Corpora.  Information Retrieval 16(3):331–368.  https://doi.org/10.1007/s10791-012-9200-5.
  * Vulić et al. (2015) Ivan Vulić, Wim De Smet, Jie Tang, and Marie-Francine Moens. 2015.  Probabilistic Topic Modeling in Multilingual Settings: An Overview of Its Methodology and Applications.  Information Processing & Management 51(1):111–147.  https://doi.org/10.1016/j.ipm.2014.08.003.
  * Vyas and Carpuat (2016) Yogarshi Vyas and Marine Carpuat. 2016.  Sparse Bilingual Word Representations for Cross-lingual Lexical Entailment.  In NAACL HLT 2016, The 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, San Diego California, USA, June 12-17, 2016. pages 1187–1197.  http://aclweb.org/anthology/N/N16/N16-1142.pdf.
  * Wallach et al. (2009a) Hanna Wallach, David Mimno, and Andrew McCallum. 2009a.  Rethinking LDA: Why Priors Matter.  In Advances in Neural Information Processing Systems 22: 23rd Annual Conference on Neural Information Processing Systems 2009. Proceedings of a meeting held 7-10 December 2009, Vancouver, British Columbia, Canada.. pages 1973–1981.  http://papers.nips.cc/paper/3854-rethinking-lda-why-priors-matter.
  * Wallach et al. (2009b) Hanna M. Wallach, Iain Murray, Ruslan Salakhutdinov, and David M. Mimno. 2009b.  Evaluation Methods for Topic Models.  In Proceedings of the 26th Annual International Conference on Machine Learning, ICML 2009, Montreal, Quebec, Canada, June 14-18, 2009. pages 1105–1112.  https://doi.org/10.1145/1553374.1553515.
  * Yang et al. (2017) Weiwei Yang, Jordan L. Boyd-Graber, and Philip Resnik. 2017.  Adapting Topic Models Using Lexical Associations with Tree Priors.  In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, Denmark, September 9-11, 2017. pages 1901–1906.  https://aclanthology.info/papers/D17-1203/d17-1203.