# Exploring hidden sectors with two-particle angular correlations at future
$e^{+}e^{-}$ colliders

E. Musumeci  Instituto de Física Corpuscular, CSIC – University of Valencia,
46980 Paterna (Valencia), Spain  A. Irles  Instituto de Física Corpuscular,
CSIC – University of Valencia, 46980 Paterna (Valencia), Spain  R. Perez-Ramos
DRII-IPSA, 63 bis Boulevard de Brandebourg, 94200 Ivry-sur-Seine, France
Laboratoire de Physique Théorique et Hautes Energies (LPTHE), UMR 7589,
Sorbonne Université et CNRS, 4 place Jussieu, 75252 Paris Cedex 05, France  I.
Corredoira  Instituto Galego de Física de Altas Enerxìas (IGFAE, USC), Spain
E. Sarkisyan-Grinbaum  Experimental Physics Department, CERN, 1211 Geneva 23,
Switzerland  Department of Physics, The University of Texas at Arlington,
Arlington, TX 76019, USA  V. A. Mitsou  Instituto de Física Corpuscular, CSIC
– University of Valencia, 46980 Paterna (Valencia), Spain  Physics Division,
School of Applied Mathematical and Physical Sciences, National Technical
University of Athens, 15780 Zografou Campus, Athens, Greece  M. A. Sanchis-
Lozano  Instituto de Física Corpuscular, CSIC – University of Valencia, 46980
Paterna (Valencia), Spain

###### Abstract

Future $e^{+}e^{-}$ colliders are called to play a fundamental role in
measuring Standard Model (SM) parameters with unprecedented precision and the
discovery of physics beyond the SM (BSM). This study focus on a QCD-like
Hidden Valley (HV) scenario, with relatively light v-quarks ($\lesssim 100$
GeV), perturbing the QCD partonic cascade and increasing azimuthal and
(pseudo)rapidity correlations of final-state SM hadrons. Using Pythia8 and
detector fast simulation tools we study ridge-like structures arising in the
two-particle angular correlation function, including selection cuts and
detector effects.

$e^{+}e^{-}$ colliders, angular correlations, ridge phenomenon, New Physics,
Hidden Valley

††preprint: IFIC-2021-038, PSI-PR-21-24, UWThPh 2021-20

##  I Introduction

Historically, angular correlations have been a fundamental tool for the study
of hadronic dynamics and new phenomena in different collision systems, namely,
$pp$, $pA$ and $AA$ CMS Collaboration (2016); ALICE Collaboration (2013);
ATLAS Collaboration (2013). In particular, the discovery of long-range near-
side ridges showing up in two-particle correlations still lacks a satisfactory
explanation. On the other hand, no clear ridge-like signal was found in the
$e^{+}e^{-}$; data an- alyzed by ALEPH Badea  _et al._ (2019) and Belle Belle
Collaboration (2022) experiments except for a recent claim in Chen  _et al._
(2023).

In this work, we focus on two-particle angular correlations as a way to look
for BSM physics at future high-energy $e^{+}e^{-}$ colliders. The theoretical
framework is based on the so-called Hidden Valley (HV) scenario, which stands
for a wide class of models with one or more hidden sectors beyond the SM, with
new (valley) particles whose mass can vary from zero to one hundred GeV.

##  II Two-particle angular correlations

The clean environment of $e^{+}e^{-}$ collisions, in contrast to hadronic
collisions, is especially suitable to define a reference frame whose $z$-axis
lies along the direction of the back-to-back jets in most events. The thrust
reference frame has been used throughout this work, so the rapidity $y$ of a
particle always refers to the thrust or $z$-axis. The azimuthal angle $\phi$
is defined as usual on the transverse plane to the thrust axis, event by
event. Note that only rapidity and azimuthal differences of two final-state SM
particles labelled as ‘1’ and ‘2’, $\Delta y\equiv y_{1}-y_{2}$,
$\Delta\phi\equiv\phi_{1}-\phi_{2}$, are used in our study.

The two-particle correlation function is defined as

$C^{(2)}(\Delta y,\;\Delta\phi)=\frac{S(\Delta y,\;\Delta\phi)}{B(\Delta
y,\;\Delta\phi)},$ (1)

where $S(\Delta y,\Delta\phi)$ stands for the density of particle pairs within
the same event

$S(\Delta
y,\;\Delta\phi)=\frac{1}{N_{\text{pairs}}}\frac{d^{2}N^{\text{same}}}{d\Delta
yd\Delta\phi},$ (2)

while $B(\Delta y,\Delta\phi)$ represents the density of mixed pairs from
distinct back-back $q\bar{q}$ events.

$B(\Delta
y,\;\Delta\phi)=\frac{1}{N_{\text{mix}}}\frac{d^{2}N^{\text{mix}}}{d\Delta
yd\Delta\phi}.$ (3)

The so-called azimuthal Yield,$Y(\Delta\phi)$, is of particular interest,
being defined by integration over a $\Delta y$ range as:

$Y(\Delta\phi)=\frac{\int_{y_{\text{inf}}\leq|\Delta y|\leq
y_{\text{sup}}}S(\Delta y,\;\Delta\phi)dy}{\int_{y_{\text{inf}}\leq|\Delta
y|\leq y_{\text{sup}}}B(\Delta y,\;\Delta\phi)dy},$ (4)

where $y_{\rm inf/sup}$ defines the lower/limit integration limit for
different rapidity intervals of interest.

##  III HV Phenomenology

In most HV models Strassler and Zurek (2007), the SM gauge group sector
$G_{\text{SM}}$ is extended by (at least) a new gauge group $G_{\text{V}}$
under which all SM particles are neutral. Hence, a new category of v-particles
emerges charged under $G_{\text{V}}$ but neutral under $G_{\text{SM}}$.
Moreover, “communicators”, charged under both $G_{\text{SM}}$ and
$G_{\text{V}}$, are introduced to the theory allowing interactions between SM
and HV particles: either a intermediate (very massive) vector $Z_{v}$ boson,
or the hidden partners of SM quarks or leptons, generically denoted as
$F_{v}$, assumed to be pair produced. In this study, $F_{v}$ are fermions
(spin = 1/2) and $q_{v}$ scalars.

Different mechanisms can connect the hidden and SM sectors through
communicators, e.g., via the tree-level channel: $e^{+}e^{-}\to Z_{v}\to
q_{v}\bar{q}_{v}\to$ hadrons. Alternatively, communicators can be pair
produced via SM $\gamma^{\ast}/Z$ coupling to a $F_{v}\bar{F}_{v}$ pair,
yielding both visible and invisible cascades. We have checked that the latter
channel significantly dominates over the former within the range of energies
under study.

For some values of the HV parameter space, communicators can promptly decay
into a particle $f$ of the visible sector (its SM partner) and a $q_{v}$ of
the hidden sector according to the splitting: $F_{v}\to fq_{v}$. Let us note
that the $q_{v}$ mass may strongly influence on the kinematics of the visible
cascade leading to SM particles. This mass is currently unconstrained and can
range from zero to close to the $F_{v}$ mass Pérez-Ramos  _et al._ (2022).

We are especially interested in the influence of the invisible (i.e., HV)
sector on the partonic cascade into final-state SM particles, leading to
potentially observable effects. This can be understood, e.g., as both visible
and hidden cascades have to share the event’s total energy, thereby modifying
the respective available phase spaces. For concreteness, we restrict our study
to a QCD-like HV scenario with $q_{v}$-quarks, $g_{v}$-gluons, and an
equivalent “strong” coupling constant $\alpha_{v}$. For simplicity,
$\alpha_{v}$ is assumed not running with energy but fixed to $\alpha_{v}=0.1$,
as no significant differences are found in our analysis by varying this value.

(a) Hidden valley (b) SM light quarks

Figure 1: Leading diagrams of production in $e^{+}e^{-}$ collisions for
LABEL:sub@fig:hv HV, and LABEL:sub@fig:sm SM light quarks including bottom.

##  IV ANALYSIS AT DETECTOR LEVEL

To evaluate the feasibility of detecting the HV signal in $e^{+}e^{-}$
colliders, we relied on the Pythia8 Monte Carlo event generator Sjöstrand  _et
al._ (2015). Fast detector simulations were performed using the SGV tool
Berggren (2012) and the geometry and acceptances corresponding to the large
model of the ILD concept for the International Linear Collider (ILC) as
described in ILD Concept Group (2020). The simulated events are provided in
the same event model as used by the ILD concept group, and the tools available
in the ILCSoft package are used for the event reconstruction and analysis. The
ILD reconstruction is based on the so-called Particle Flow approach, which
aims to reconstruct all individual particles produced in the final state via
pattern recognition algorithms. The reconstructed candidates of single
particles are called Particle Flow Objects (PFOs). In our study, including
detector effects, we first set the center-of-mass energy $\sqrt{s}$ of
$e^{+}e^{-}$ collisions equal to $250\;\mathrm{GeV}$ to coincide with the
planned first commissioning of the collider as a Higgs boson factory. The
integrated luminosity considered is $\mathcal{L}=2$ ab-1 as foreseen for the
ILC at $\sqrt{s}=250~{}{\mathrm{Ge\kern-1.00006ptV}}$ for unpolarised beams.
As a further outlook, we extended our study only at particle level up to
$\sqrt{s}=500$ GeV and 1 TeV. The case of the longitudinally polarised beams
will be addressed in future work once this option becomes available.

The HV signal proceeds via the process $e^{+}e^{-}\to
D_{v}\bar{D}_{v}\to\text{hadrons}$ (where $D_{V}$ denotes the lightest
communicator, being the hidden partner of the d-quark) as depicted in Fig. 1a.
In our benchmark scenarios we set $\alpha_{v}=0.1$,
$m_{D_{v}}=125~{}{\mathrm{Ge\kern-1.00006ptV}}$, with four different v-quark
mass values: $m_{q_{v}}=0.1,10,50$ and 100 GeV. At higher $\sqrt{s}$, we also
included $T_{v}\bar{T}_{v}$ pair production provided that it becomes
kinematically possible.

In our analysis at $\sqrt{s}=250~{}{\mathrm{Ge\kern-1.00006ptV}}$, the
$e^{+}e^{-}\to q\bar{q}$ background comes out from the inclusive production of
all the SM quark species except the top flavor since its production lies below
threshold (see Fig. 1).

Table 1:  Cross-sections for $e^{+}e^{-}\to D_{v}\bar{D}_{v}$ processes and different $m_{q_{v}}$ masses, for $e^{+}e^{-}\to q\bar{q}$ and $WW$ fusion at $\sqrt{s}=250~{}{\mathrm{Ge\kern-1.00006ptV}}$. The efficiencies of the selection criteria described in the text, and the average charged-track multiplicity and its $RMS$ are shown.  Process | $\sigma_{\textsc{Pythia8}}$ | Efficiency | $<N_{\text{ch}}>$
---|---|---|---
[pb] | [%]
$e^{+}e^{-}\to D_{v}\bar{D}_{v}$ |  |  |
$m_{q_{v}}=0.1\;\mathrm{GeV}$ | $0.13$ | 36 | $12.4\pm 3.7$
$m_{q_{v}}=10\;\mathrm{GeV}$ | $0.12$ | 36 | $12.4\pm 3.7$
$m_{q_{v}}=50\;\mathrm{GeV}$ | $0.12$ | 42 | $11.4\pm 3.5$
$m_{q_{v}}=100\;\mathrm{GeV}$ | $0.12$ | 42 |  1$6.5\pm 2.1$
$e^{+}e^{-}\to q\bar{q}$ with ISR | $48$ | $\lesssim 0.01$ |  1$9.9\pm 3.4$
$WW$ fusion | $7.4$ | $\lesssim 0.001$ | -
Figure 2: Di-jet invariant mass distributions after selection cuts to SM
background and HV signal for different $m_{q_{v}}$ values.

In a preceding investigation Musumeci  _et al._ (2023), we examined the
aforementioned signal and background at particle level, without including
Initial State Radiation (ISR), which plays a fundamental role in our analysis,
nor Vector Boson Fusion (VBF). In this Letter we include both, extending our
study to detector level. The cross-sections for the HV and SM processes have
been estimated with Pythia8 and are reported in Table 1.

For the reconstruction of $S$, and due to the significant difference in cross-
sections, specific cuts to maximize background removal were implemented.
Events with displaced vertices due to the production of heavy-flavored
decaying hadrons have been excluded too. Constraints have been set on the
number of neutral PFOs and charged ones ($\leq 22$ and $\leq 15$,
respectively). Other cuts were applied to the emission angle
($|\cos{\theta_{\gamma_{\text{ISR}}}}|<0.5$) and energy
($E_{\gamma_{\text{ISR}}}<$ 40 GeV) of reconstructed ISR photons. Constraints
on the di-jet invariant mass $m_{jj}$ have also been defined, considering
values below 130 GeV. Furthermore, a threshold has been implemented for the
maximum energy of the leading jet, restricting it to be less than 80 GeV.
Figure 2 illustrates the di-jet invariant mass distribution following the
application of the above-mentioned cuts. The selection cut efficiency,
reported in Table 1, shows a drastic reduction of the SM background while the
HV signal is affected to a much lesser extent. For the reconstruction of $B$,
a thrust value larger than 0.95 was required, keeping the same requirements on
PFO multiplicities as for $S$.

(a) Detector level distributions with no selection applied for the $S(\Delta
y,\Delta\phi)$ reconstruction.

(b) Detector level distributions after the signal enriching experimental cuts
applied for the $S(\Delta y,\Delta\phi)$ reconstruction.

Figure 3: Plot of the two-particle angular correlation function,
$C^{(2)}(\Delta y,\Delta\phi)$ at detector level separately for the HV signal
and the SM background in $e^{+}e^{-}$ collisions at
$\sqrt{s}=250~{}{\mathrm{Ge\kern-1.00006ptV}}$. In the HV case,
$m_{q_{v}}=100~{}{\mathrm{Ge\kern-1.00006ptV}}$,
$m_{D_{v}}=125~{}{\mathrm{Ge\kern-1.00006ptV}}$ and $\alpha_{v}=0.1$. For the
SM case, light quarks include the bottom flavor.

(a) Detector-level yield distributions with no selection applied for the
$S(\Delta y,\Delta\phi)$ reconstruction.

(b) Detector-level yield distributions after the experimental selection
applied to enrich the $S(\Delta y,\Delta\phi)$ sample with HV signal events.

Figure 4: Yield $Y(\Delta\phi)$ for both HV signal (different blue lines types
depending on the $q_{v}$ mass) and SM background (shaded) for the $0<|\Delta
y|<1.6$ and $1.6<|\Delta y|<5$ intervals. Notice the different shapes of the
HV signal and SM background providing a valuable signature of NP using angular
correlations.

(a) Ratio of detector level yield distributions for $0<|\Delta y|<1.6$. (b)
The same as in case (a) for the rapidity interval $1.6<|\Delta y|<5$. Notice
the lowering of the HV correlations with respect to the background, in
contrast to the enhancement seen in the above plot.

Figure 5: Ratios of yield distributions $Y(\Delta\phi)$ after all experimental
cuts for different $q_{v}$ mass assignments. The red band along unity shows
the statistical uncertainties of the sample assuming 2 ab-1 of integrated
luminosity and no polarization.

Figure 3 shows three-dimensional plots of the two-particle correlation
function $C^{(2)}(\Delta y,\Delta\phi)$ separately for HV and for SM
scenarios, before (Fig. 3a) and after (Fig. 3b) cuts. As an example, in the HV
case, we set $m_{q_{v}}=100\;\mathrm{GeV}$,
$m_{D_{v}}=125~{}{\mathrm{Ge\kern-1.00006ptV}}$ and $\alpha_{v}=0.1$ and the
products of the decay $D_{v}\to d+q_{v}$ initiating a partonic (visible and
invisible) shower. As expected, a near-side peak shows up at ($\Delta y\simeq
0$, $\Delta\phi\simeq 0$), receiving contributions mainly from track pairs
within the same jet. On the other hand, an away-side correlation ridge,
extending over rapidity around $\Delta\phi\simeq\pi$, results from back-to-
back momentum balance, in principle unrelated to New Physics (NP). After cuts,
a near-side ridge (with two pronounced and symmetric bumps) shows up for
$1.6<|\Delta y|<3$ at $\Delta\phi\simeq 0$ for the SM case, similar to the
pure HV scenario. This structure arises from the ISR effect, since the
effective center-of-mass energy approaches the $Z$ mass and thus resonant
production enhances the production cross-section.

Nonetheless, to examine in more detail the possibility of discriminating the
HV signal from the SM background, we depict in Fig. 4 the yield
$Y(\Delta\phi)$ (as defined in Eq. (4)) for both $0<|\Delta y|\leq 1.6$ and
$1.6<|\Delta y|<5$ ranges, before (Fig. 4a) and after(Fig. 4b) cuts. In this
context, the HV signal (for $m_{q_{v}}$ = 0.1, 10, 50, and 100 GeV) is
considered together with the SM background, while a standalone analysis of the
SM background is also presented. Fig. 5 shows the ratio signal-to-background
of the Yield at the detector level and all experimental cuts applied for (a)
$0<|\Delta y|<1.6$ (Fig. 5a) and (b) $1.6<|\Delta y|<5$ (Fig 5b). A clear
difference becomes apparent for $0<|\Delta y|\leq 1.6$: a sizeable peak at
$\Delta\phi\sim\pi$ characterizes the HV scenario, unlike the pure SM case.
This remarkable discrepancy would potentially serve as a valuable signature of
a hidden sector.We claim that such a different distribution shape could be
interpreted as a hint of NP, highlighting angular correlations as
complementary to more conventional BSM searches.

##  V Outlook at higher-energies

In this section, we explore higher energies at particle level. Specifically,
we consider $\sqrt{s}$=0.5 and 1 TeV. As the HV signal $e^{+}e^{-}\to
Q_{v}\bar{Q}_{v}\to\text{hadrons}$, we focus on the the lightest communicator
$D_{v}$ and the heaviest one $T_{v}$ which decays into $q_{v}t$. Besides the
$q\bar{q}$ and VBF backgrounds, we also consider the production of $t\bar{t}$,
which becomes relevant at these energies. Table 2 presents the cross-sections
obtained from Pythia8. We set the mass of $D_{v}$ and $T_{v}$ equal to
$\sqrt{s}/2$, and results for different $m_{q_{v}}$ are shown. No large
variations are obtained around this mass assignment to the communicator.
According to expectations, the contribution from the SM backgrounds decreases
with energy. For the HV signal, a reduction of the cross-section by two orders
of magnitude is obtained at $\sqrt{s}=1$ TeV.

Table 2:  Cross-sections of the HV signal and SM background at $\sqrt{s}=500$ GeV and $\sqrt{s}=1$ TeV. Cross-section predictions do not depend on the $m_{q_{v}}$ value unless it is too large to make the process kinematically inaccessible. The signal corresponds to $D_{v}$ and $T_{v}$ pair production with the $D_{v}$/$T_{v}$ mass set equal to $\sqrt{s}/2$. Whenever kinematically allowed, $t\bar{t}$ production has been included as a SM background source in addition to lighter flavors considered at $\sqrt{s}=250$ GeV. Process |  $\sigma_{\sqrt{s}=500\text{GeV}}$ [pb] |  $\sigma_{\sqrt{s}=1\text{TeV}}$ [pb]
---|---|---
| $m_{D_{v}}=250~{}{\mathrm{Ge\kern-1.00006ptV}}$ | $m_{D_{v}}=500~{}{\mathrm{Ge\kern-1.00006ptV}}$
$e^{+}e^{-}\to D_{v}\bar{D}_{v}$ | $2.4\times 10^{-2}$ | $4.4\times 10^{-3}$
|  $m_{T_{v}}=250$ GeV |  $m_{T_{v}}=500$ GeV
$e^{+}e^{-}\to T_{v}\bar{T}_{v}$ | $9.5\times 10^{-2}$ | $1.8\times 10^{-2}$
$e^{+}e^{-}\to q\bar{q}$ with ISR | 11 | 2.9
$e^{+}e^{-}\to t\bar{t}$ | $0.59$ | $0.19$
$WW$ fusion | 3.4 | 1.3

##  VI Conclusions

The analysis of angular particle correlations in high-energy colliders can
provide valuable insights into matter under extreme temperature and density
conditions, somehow reproducing the early universe conditions when quarks and
gluons had not yet bound to form hadrons. On the other hand, this kind of
analysis, based on rather diffuse signals, can become a complementary tool to
other more conventional searches to uncover the existence of hidden sectors
BSM as postulated in Refs. Sanchis-Lozano (2009); Sanchis-Lozano and
Sarkisyan-Grinbaum (2018) and analyzed in this work.

This study, at detector level, explores the physics potential for HV discovery
using two-particle azimuthal correlations at future $e^{+}e^{-}$ colliders.
Specifically, we consider a QCD-like hidden sector containing not-too-heavy
$v$-quarks and $v$-gluons, which can interact with the SM sector via
communicators of mass typically $\lesssim 1~{}{\mathrm{Te\kern-1.00006ptV}}$.
In particular, we have focused on $D_{v}\bar{D}_{v}$ pair production at
$\sqrt{s}=250$ GeV, the expected lightest communicator in this scenario. We
also have shortly extended our study to higher energies at particle level
pointing at promising prospects too.

To conclude, our results show that two-particle azimuthal correlations in a
$e^{+}e^{-}$ factory might become a useful tool to discover NP if
kinematically possible.

###### Acknowledgements.

Acknowledgements: AI, EM and VAM acknowledge support by Spanish MICIU/ AEI and
European Union / FEDER via the grant PID2021-122134NB-C21. EM and VAM
acknowledge support by Generalitat Valenciana (GV) via the Excellence Grant
CIPROM/2021/073. AI acknowledges support by GV under the grant number
CIDEGENT/2020/21, the financial support from the MCIN with funding from the
European Union NextGenerationEU and by GV via the Programa de Planes
Complementarios de I+D+i (PRTR 2022) Project Si4HiggsFactories, reference
ASFAE$/2022/015$. VAM acknowledges support by Spanish MICIU via the mobility
grant PRX22/00633. EM acknowledges support by CSIC via the mobility grant
iMOVE23097. IC acknowledges support by the Xunta de Galicia (CIGUS Network of
Research Centers). MASL acknowledges support by the Spanish Agencia Estatal de
Investigacion under grant PID2020-113334GB-I00 / AEI / 10.13039/501100011033,
and by GV under grant CIPROM/2022/36.

## References

  * CMS Collaboration (2016) CMS Collaboration, Phys. Rev. Lett. 116, 172302 (2016), arXiv:1510.03068 [nucl-ex] .
  * ALICE Collaboration (2013) ALICE Collaboration, Physics Letters B 719, 29 (2013).
  * ATLAS Collaboration (2013) ATLAS Collaboration, Phys. Rev. Lett. 110, 182302 (2013).
  * Badea  _et al._ (2019) A. Badea, A. Baty, P. Chang, G. M. Innocenti, M. Maggi, C. McGinn, M. Peters, T.-A. Sheng, J. Thaler, and Y.-J. Lee, Phys. Rev. Lett. 123, 212002 (2019), arXiv:1906.00489 [hep-ex] .
  * Belle Collaboration (2022) Belle Collaboration, Phys. Rev. Lett. 128, 142005 (2022), arXiv:2201.01694 [hep-ex] .
  * Chen  _et al._ (2023) Y.-C. Chen _et al._ , (2023), arXiv:2312.05084 [hep-ex] .
  * Strassler and Zurek (2007) M. J. Strassler and K. M. Zurek, Phys. Lett. B 651, 374 (2007), arXiv:hep-ph/0604261 .
  * Pérez-Ramos  _et al._ (2022) R. Pérez-Ramos, M.-A. Sanchis-Lozano, and E. K. Sarkisyan-Grinbaum, Phys. Rev. D 105, 053001 (2022), arXiv:2110.05900 [hep-ph] .
  * Sjöstrand  _et al._ (2015) T. Sjöstrand, S. Ask, J. R. Christiansen, R. Corke, N. Desai, P. Ilten, S. Mrenna, S. Prestel, C. O. Rasmussen, and P. Z. Skands, Comput. Phys. Commun. 191, 159 (2015), arXiv:1410.3012 [hep-ph] .
  * Berggren (2012) M. Berggren, “Sgv 3.0 - a fast detector simulation,” (2012), arXiv:1203.0217 [physics.ins-det] .
  * ILD Concept Group (2020) ILD Concept Group, (2020), arXiv:2003.01116 [physics.ins-det] .
  * Musumeci  _et al._ (2023) E. Musumeci, R. Perez-Ramos, A. Irles, I. Corredoira, V. A. Mitsou, E. Sarkisyan-Grinbaum, and M. A. Sanchis-Lozano, in  _International Workshop on Future Linear Colliders_ (2023) arXiv:2307.14734 [hep-ph] .
  * Sanchis-Lozano (2009) M.-A. Sanchis-Lozano, Int. J. Mod. Phys. A 24, 4529 (2009), arXiv:0812.2397 [hep-ph] .
  * Sanchis-Lozano and Sarkisyan-Grinbaum (2018) M.-A. Sanchis-Lozano and E. K. Sarkisyan-Grinbaum, Phys. Lett. B 781, 505 (2018), arXiv:1802.06703 [hep-ph] .


# Topic Modeling with Contextualized Word Representation Clusters

Laure Thompson
University of Massachusetts Amherst
<EMAIL_ADDRESS>
David Mimno
Cornell University
<EMAIL_ADDRESS>

###### Abstract

Clustering token-level contextualized word representations produces output
that shares many similarities with topic models for English text collections.
Unlike clusterings of vocabulary-level word embeddings, the resulting models
more naturally capture polysemy and can be used as a way of organizing
documents. We evaluate token clusterings trained from several different output
layers of popular contextualized language models. We find that BERT and GPT-2
produce high quality clusterings, but RoBERTa does not. These cluster models
are simple, reliable, and can perform as well as, if not better than, LDA
topic models, maintaining high topic quality even when the number of topics is
large relative to the size of the local collection.

##  1 Introduction

Contextualized word representations such as those produced by BERT Devlin et
al. (2019) have revolutionized natural language processing for a number of
structured prediction problems. Recent work has shown that these
contextualized representations can support type-level semantic clusters Sia et
al. (2020). In this work we show that token-level clustering provides
contextualized semantic information equivalent to that recovered by
statistical topic models Blei et al. (2003). From the perspective of
contextualized word representations, this result suggests new directions for
semantic analysis using both existing models and new architectures more
specifically suited for such analysis. From the perspective of topic modeling,
this result implies that transfer learning through contextualized word
representations can fill gaps in probabilistic modeling (especially for short
documents and small collections) but also suggests new approaches for latent
semantic analysis that are more closely tied to mainstream transformer
architectures.

Topic modeling is often associated with probabilistic generative models in the
machine learning literature, but from the perspective of most actual
applications the core benefit of such models is that they provide an
interpretable latent space that is grounded in the text of a specific
collection. Standard topic modeling algorithms operate by estimating the
assignment of individual tokens to topics, either through a Gibbs sampling
state or through parameters of variational distributions. These token-level
assignments can then provide disambiguation of tokens based on context, a
broad overview of the themes of a corpus, and visualizations of the location
of those themes within the corpus Boyd-Graber et al. (2017).

A related but distinct objective is vocabulary clustering. These methods
operate at the level of distinct word types, but have no inherent connection
to words in context (e.g. Brown et al., 1992; Arora et al., 2013;
Lancichinetti et al., 2015). Recently, there has also been considerable
interest in continuous type-level embeddings such as GloVe Pennington et al.
(2014) and word2vec Mikolov et al. (2013a, b), which can be clustered to form
interpretable semantic groups. Although it has not been widely used, the
original word2vec distribution includes code for $k$-means clustering of
vectors. Sia et al. (2020) extends this behavior to contextualized embeddings,
but does not take advantage of the contextual, token-based nature of such
embeddings.

In this work, we demonstrate a new property of contextualized word
representations: if you run a simple $k$-means algorithm on token-level
embeddings, the resulting word clusters share similar properties to the output
of an LDA model. Traditional topic modeling can be viewed as token clustering.
Indeed, a clustering of tokens based on BERT vectors is functionally
indistinguishable from a Gibbs sampling state for LDA, which assigns each
token to exactly one topic. For topic modeling, clustering is based on local
context (the current topic disposition of words in the same document) and on
global information (the current topic disposition of other words of the same
type). We find that contextualized representations offer similar local and
global information, but at a richer and more representationally powerful
level.

We argue that pretrained contextualized embeddings provide a simple, reliable
method for users to build fine-grained, semantically rich representations of
text collections, even with limited local training data. While for this study
we restrict our attention to English text, we see no reason contextualized
models trained on non-English data (e.g. Martin et al., 2020; Nguyen and
Nguyen, 2020) would not have the same properties. It is important to note,
however, that we make no claim that clustering contextualized word
representations is the optimal approach in all or even many situations.
Rather, our goal is to demonstrate the capabilities of contextualized
embeddings for token-level semantic clustering and to offer an additional
useful application in cases where models like BERT are already in use.

Term |  Model |  Top Words
---|---|---
land |  LDA |  sea coast Beach Point coastal land Long Bay m sand beach tide Norfolk shore Ocean Coast areas
land acres County ha facilities State location property acre cost lot site
parking settlers Department
BERT |  arrived arrival landing landed arriving arrive returning settled departed land leaving sailed arrives
land property rights estate acres lands territory estates properties farm
farmland Land fields acre
GPT-2 |  arrived landed arriving landfall arrive arrives arrival landing land departed ashore embarked Back
land sea ice forest rock mountain ground sand surface beach ocean soil hill
lake snow sediment
metal |  LDA |  metals metal potassium sodium + lithium compounds electron ions hydrogen chemical atomic – ion
metal folk bands music genre band debut Metal heavy musicians lyrics
instruments acts groups
BERT |  metals elements metal electron element atomic periodic electrons chemical atoms ions atom
rock dance pop metal Rock folk jazz punk comedy Dance heavy funk alternative
soul street club
GPT-2 |  rock pop hop dance metal folk hip punk jazz B soul funk alternative rap heavy disco electronic
plutonium hydrogen carbon sodium potassium metal lithium uranium oxygen
diamond radioactive
Table 1: Automatically selected examples of polysemy in contextualized
embedding clusters. Clusters containing “land” or “metal” as top words from
BERT $L[-1]$, GPT-2 $L[-2]$, and LDA with $K=500$. All models capture multiple
senses of the noun “metal”, but BERT and GPT-2 are better than LDA at
capturing the syntactic variation of “land” as a verb and noun.

##  2 Related Work

We selected three contextualized language models based on their general
performance and ease of accessibility to practitioners: BERT Devlin et al.
(2019), GPT-2 Radford et al. (2019), and RoBERTa Liu et al. (2019). All three
use similar Transformer Vaswani et al. (2017) based architectures, but their
objective functions vary in significant ways. These models are known to encode
substantial information about lexical semantics Petroni et al. (2019); Vulić
et al. (2020).

Clustering of vocabulary-level embeddings has been shown to produce
semantically related word clusters Sia et al. (2020). But such embeddings
cannot easily account for polysemy or take advantage of local context to
disambiguate word senses since each word type is modeled as a single vector.
Since these embeddings are not grounded in specific documents, we cannot
directly use them to track the presence of thematic clusters in a particular
collection. In addition, Sia et al. (2020) find that reweighting their type-
level clustering by corpus frequencies is helpful. In contrast, such
frequencies are “automatically” accounted for when we operate on the token
level. Similarly, clusterings of sentence-level embeddings have been shown to
produce semantically related document clusters Aharoni and Goldberg (2020).
But such models cannot represent topic mixtures or provide an interpretable
word-based representation without additional mapping from clusters to
documents to words. It is widely known that token-level representations of
single word types provide contextual disambiguation. For example, Coenen et
al. (2019) show an example distinguishing uses of die between the German
article, a verb for “perish” and a game piece. We explore this property on the
level of whole collections, looking at all word types simultaneously.

There are a number of models that solve the topic model objective directly
using contemporary neural network methods (e.g. Srivastava and Sutton, 2016;
Miao et al., 2017; Dieng et al., 2020). There are also a number of neural
models that incorporate topic models to improve performance on a variety of
tasks (e.g. Chen et al., 2016; Narayan et al., 2018; Wang et al., 2018;
Peinelt et al., 2020). Additionally, BERT has been used for word sense
disambiguation Wiedemann et al. (2019). In contrast, our goal is not to create
hybrid or special-purpose models but to show that simple contextualized
embedding clusters support token-level topic analysis in themselves with no
significant additional modeling. Since our goal is simply to demonstrate this
property and not to declare overall “winners”, we focus on LDA in empirical
comparisons because it is the most widely used and straightforward,
highlighting the similarities and differences between contextualized embedding
clusters and topics.

##  3 Data and Methods

We use three real-world corpora of varying size, content, and document length:
Wikipedia articles (Wikipedia), Supreme Court of the United States legal
opinions (Scotus), and Amazon product reviews (Reviews). We select Wikipedia
for its affinity with the training data of the pretrained models. Because its
texts are similar to ones the models have already seen, Wikipedia is a “best-
case” scenario for our clustering algorithms. If a clustering method performs
poorly on Wikipedia, we expect the method to perform poorly in general. In
contrast, we select Scotus and Reviews for their content variability. Legal
opinions tend to be long and contain many technical legal terms, while user-
generated product reviews tend to be short and highly variable in content and
vocabulary.

Corpus | Docs | Types | Tokens
---|---|---|---
Wikipedia | 1.0K | 22K | 1.2M
Scotus | 5.3K | 58K | 10.8M
Reviews | 100K | 52K | 9.4M
Table 2:  Corpus statistics for number of documents, types, and tokens.
Document and type counts are listed in thousands (K), token counts in millions
(M).

###  Wikipedia.

In this collection, documents are Wikipedia articles (excluding headings). We
randomly selected 1,000 articles extracted from the raw/character-level
training split of Wikitext-103 Merity et al. (2017). We largely use the
existing tokenization, but recombine internal splits on dot and comma
characters but not hyphens so that “Amazon @.@ com” becomes “Amazon.com”, “1
@,@ 000” becomes “1,000”, and “best @-@ selling” becomes “best - selling”.

###  Scotus.

In this collection, documents are legal opinions from the Supreme Court of the
United States filed from 1980 through 2019.111https://www.courtlistener.com/
These documents can be very long, but have a regular structure.

###  Reviews.

In this collection, documents are Amazon product reviews. For four product
categories (Books, Electronics, Movies and TV, CDs and Vinyl), we select
25,000 reviews from category-level dense subsets of Amazon product reviews He
and McAuley (2016); McAuley et al. (2015).

### Data Preparation.

For Scotus and Reviews, we tokenize documents using the spaCy NLP
toolkit.222https://spacy.io/ Tokens are case-sensitive non-whitespace
character sequences. For consistency across models, we also delete all
control, format, private-use, and surrogate Unicode codepoints since they are
internally removed by BERT’s tokenizer. We extract contextualized word
representations from BERT (cased version), GPT-2, and RoBERTa using pretrained
models available through the huggingface transformers library Wolf et al.
(2019). All methods break low-frequency words into multiple subword tokens:
BERT uses WordPiece Wu et al. (2016), while GPT-2 and RoBERTa use a byte-level
variant of byte pair encoding (BPE) Sennrich et al. (2016). For example, the
word disillusioned is represented by four subtokens “di -si -llus -ioned” in
BERT and by two subtokens “disillusion -ed” in GPT-2 and RoBERTa. One key
difference between these tokenizers is that byte-level BPE can encode all
inputs, while WordPiece replaces all Unicode codepoints it has not seen in
pretraining with the special token UNK. For simplicity, rather than using a
sentence splitter we divide documents into the maximum length subtoken blocks.
To make vocabularies comparable across models with different subword
tokenization schemes, we reconstitute the original word tokens by averaging
the vectors for subword units Bommasani et al. (2020).

### Clustering.

We cluster tokens using spherical $k$-means Dhillon and Modha (2001) with
spkm++ initialization Endo and Miyamoto (2015) because of its simplicity and
high-performance, and cosine similarities are commonly used in other embedding
contexts. Although we extract contextualized features for all tokens, prior to
clustering we remove frequent words occurring in more than 25% of documents
and rare words occurring in fewer than five documents. Each clustering is run
for 1000 iterations or until convergence. For LDA, we train models using
Mallet McCallum (2002) with hyperparameter optimization occurring every 20
intervals after the first 50. For each embedding model, we cluster the token
vectors extracted from the final layer $L[-1]$, the penultimate layer $L[-2]$,
and the antepenultimate layer $L[-3]$. Vulić et al. (2020) suggest combining
multiple layers, but no combination we tried provided additional benefit for
this specific task. We consider more than the final hidden layer of each model
because of the variability in anisotropy across layers Ethayarajh (2019). In a
space where any two words have near perfect cosine similarity, clustering will
only capture the general word distribution of the corpus. Since Ethayarajh
(2019) has shown GPT-2’s final layer to be extremely anisotropic, we do not
expect to produce viable topics in this case. For each test case, we build ten
models each of size $K\in\\{50,100,500\\}$.

##  4 Evaluation Metrics

We evaluate the quality of “topics” produced by clustering contextualized word
representations with several quantitative measures. For all models we use hard
topic assignments, so each word token has a word type $w_{i}$ and topic
assignment $z$. Note that we use “topic” and “cluster” interchangeably.

### Word Entropy.

As a proxy for topic specificity, we measure a topic’s word diversity using
the conditional entropy of word types given a topic:
$-\sum_{i}\Pr(w_{i}{\mid}z)\log\Pr(w_{i}{\mid}z)$. Topics composed of tokens
from a small set of types will have low entropy (minimum 0), while topics more
evenly spread out across the whole vocabulary will have high entropy (maximum
log of vocabulary size; approx. 10 for Wikipedia). There is no best fit
between quality and specificity, but extreme entropy scores indicate bad
topics. Topics with extremely low entropy are overly specialized, while those
with extremely high entropy are overly general.

### Coherence.

We measure the semantic quality of a topic using two word-cooccurrence-based
coherence metrics. These coherence metrics measure whether a topic’s words
actually occur together. Internal coherence uses word cooccurrences from the
working collection, while external coherence relies on word cooccurrences from
a held-out external collection. The former measures fit to a dataset, while
the latter measures generalization. For internal coherence we use Mimno et al.
(2011)’s topic coherence metric,
$\sum_{i}\sum_{j<1}\log\frac{D(w_{i},w_{j})+\epsilon}{D(w_{i})}$, where $D$
refers to the number of documents that contain a word or word-pair. For
external coherence we use Newman et al. (2010)’s topic coherence metric:
$\sum_{i}\sum_{j<1}\log\frac{\Pr(w_{i},w_{j})+\epsilon}{\Pr(w_{i})\Pr(w_{j})}$,
where probabilities are estimated from the number of 25-word sliding windows
that contain a word or word-pair in an external corpus. We use the New York
Times Annotated Corpus Sandhaus (2008) as our external collection with
documents corresponding to articles (headline, article text, and corrected
text) tokenized with spaCy. For both metrics, we use the top 20 words of each
topic and set the smoothing factor $\epsilon$ to $10^{-12}$ to reduce
penalties for non-cooccurring words Stevens et al. (2012). We ignore words
that do not appear in the external corpus and do not consider topics that have
fewer than 10 attested words. These “skipped” topics are often an indicator of
model failure. Higher scores are better.

### Exclusivity.

A topic model can attain high coherence by repeating a single high-quality
topic multiple times. To balance this effect, we measure topic diversity using
Bischof and Airoldi (2012)’s word-level exclusivity metric to quantify how
exclusive a word $w$ is to a specific topic $z$: $\frac{\Pr(w_{i}\mid
z)}{\sum_{z^{\prime}}\Pr(w_{i}\mid z^{\prime})}$. A word prevalent in many
topics will have a low exclusivity score near 0, while a word occurring in few
topics will have a score near 1. We lift this measure to topics by computing
the average exclusivity of each topic’s top 20 words. While higher scores are
not inherently better, low scores are indicative of topics with high levels of
overlap.

##  5 Results

We evaluate whether contextualized word representation clusters can group
together related words, distinguishing distinct uses of the same word based on
local context. Compared to bag-of-words LDA, we expect contextualized
embedding clusters to encode more syntactic information. As we are not doing
any kind of fine-tuning, we expect performance to be best on text similar to
the pretraining data. We also expect contextualized embedding clusters to be
useful in describing differences between partitions of a working collection.

Figure 1: Contextualized embedding clusters produce mean internal and external
coherence scores comparable to LDA (dashed line). BERT clusters (blue) have
high mean external coherence, better than LDA for large numbers of topics.
BERT clusters contain more unique words, while RoBERTa (red) and GPT-2 (green)
$L[-1]$ clusters tend to repeat similar clusters. BERT clusters have the
highest word concentrations.

### BERT produces meaningful topic models.

BERT cluster models consistently form semantically meaningful topics, with the
final layer performing marginally better for larger $K$. Figure 1 shows that
BERT clusterings have the highest external coherence, matching LDA for
$K\in\\{50,100\\}$ and beating LDA for $K=500$. For internal coherence, the
opposite is true, with BERT on par with LDA for smaller $K$, while LDA “fits”
better for $K=500$. This distinction suggests that at very fine-grained
topics, LDA may be overfitting to noise. BERT has relatively low word entropy,
indicating more focused topics on average. Figure 2 shows the number of word
types per cluster. BERT clusters are on average smaller than LDA topics
(counted from an unsmoothed sampling state), but very few BERT clusters fall
below our 10-valid-words threshold for coherence scoring. BERT clusters are
not only semantically meaningful, but also unique. Figure 1 shows that BERT
clusters have exclusivity scores as high if not higher than LDA topics on
average. Since there is little difference between layers, we will only
consider BERT L[-1] for the remainder of this work.

### GPT-2 can produce meaningful topic models.

As expected, the final layer clusterings of GPT-2 form bad topics. These
clusters tend to be homogeneous (low word entropy) and similar to each other
(low exclusivity). They also highlight the differences between our two
coherence scores. Since these clusters tend to repeatedly echo the background
distribution of Wikipedia, they perform relatively well for internal
coherence, but poorly for external coherence. Since the final layer of GPT-2
has such high anisotropy, we cannot expect vector directionalities to encode
semantically meaningful information. In contrast, the penultimate and
antepenultimate layer clusterings perform much better. We see a large
improvement in external coherence surpassing LDA for $K=500$. Topic word
entropy and exclusivity are also improved.

For $K=500$, GPT-2 L[-3] has surprisingly low mean internal coherence—the
worst scores in Figure 1 by a significant margin. The number of topics below
the 10-valid-words threshold is similar to BERT, so this result is comparable.
We posit that this layer is relying more on transferred knowledge from the
pretrained GPT-2 model than the working collection. Because of this less
explained behavior, we will only consider GPT-2 L[-2] going forward.

Figure 2: Distinct words per cluster for LDA, BERT $L[-1]$, GPT-2 $L[-2]$, and
RoBERTa $L[-1]$ for $K=500$. Although BERT clusters cover fewer word types on
average, RoBERTa produces more clusters with very few ($<20$) word types.

### RoBERTa clusters are noticeably worse.

Given BERT’s success and GPT-2’s partial success, we were surprised to find
that RoBERTa cluster models were consistently of poor quality, with very low
exclusivity scores and high word entropies. Although RoBERTa scores fairly
well in coherence, this is not indicative of collectively high quality topics
because of the correspondingly low exclusivity scores. As shown in Figure 1,
RoBERTa has the highest average number of distinct words per cluster, but also
large numbers of clusters that contain very few distinct words. For $K=500$,
25–50 clusters are skipped on average for different layer choices. For
example, one topic consists entirely of the words game, games, Game, another
just ago, and one simply the symbol. The remaining tokens are thus limited to
a smaller number of more general topics that are closer to the corpus
distribution.

While it is commonly accepted that RoBERTa outperforms BERT for a variety of
natural language understanding tasks Wang et al. (2019), we find the opposite
to be true for semantic clustering. There are a number of differences between
BERT and RoBERTa, but our experimental results do not mark a clear cause. The
tokenization method is a very unlikely source since GPT-2 uses the same
scheme.

Model | Perc. | Entr. |  Top Words (noun verb adj adv other)
---|---|---|---
LDA | 5% | 0.69 |  Valley Death valley Creek California mining ° Range Nevada Desert
25% | 0.97 |  army forces soldiers campaign troops captured defeated Battle victory commander
50% | 1.11 |  society News Week Good Spirit Fruit says Doug host free
75% | 1.28 |  Washington Delaware ceremony Grand Capitol building 156 Number laying Master
95% | 1.53 |  critics reviews review positive mixed list Entertainment Times style something
BERT | 5% | 0.00 |  1997 1996 1995 1937 1895 1935 96 1896 1795 97
25% | 0.61 |  Jewish Israel Jews Ottoman Arab Muslim Israeli Islamic Jerusalem Islam
50% | 0.86 |  captured defeated attacked capture attack siege destroyed surrender defeat occupied
75% | 1.09 |  hop dance hip B R Dance Hip Z Hop rapper
95% | 1.48 |  separate combined co joint shared divided common combination distinct respective
GPT-2 | 5% | 0.00 |  2004 2003 2015 2000 2014 1998 2001 2013 2002 1997
25% | 0.42 |  Atlantic Pacific Gulf Mediterranean Caribbean Columbia Indian Baltic Bay Florida
50% | 0.73 |  knew finds discovers learned reveals discovered know heard discover learns
75% | 1.02 |  Olympic League FA Summer Premier Division UEFA European Winter Tour
95% | 1.42 |  positive mixed critical negative garnered favorable mostly attracted commercial
Table 3: Contextualized embedding clusters are more syntactically aware than
LDA. Topics ranked by the entropy of POS distribution of the top 20 words (10
shown) with $K=500$.

### Contextualized embedding clusters capture polysemy.

A limitation of many methods that rely on vocabulary-level embeddings is that
they cannot explicitly account for polysemy and contextual differences in
meaning. In contrast, token-based topic models are able to distinguish
differences in meaning between contexts. There has already been evidence that
token-level contextualized embeddings are able to distinguish contextual
meanings for specific examples Wiedemann et al. (2019); Coenen et al. (2019),
but can they also do this for entire collections?

Instead of manually selecting terms we expect to be polysemous, we choose
terms that occur as top words for clusters with dissimilar word distributions
(high Jensen-Shannon divergence). While dissimilarity is not indicative of
polysemy—different topics can use a term in the same way—it narrows our focus
to words that are more likely to be polysemous. In Table 1, we see topics for
two such terms “land” and “metal”. All models are able to distinguish metal
the material from metal the genre, but BERT and GPT-2 are also able to
distinguish land the noun from land the verb.

### Contextualized embedding clusters are more syntactically consistent than
LDA topics.

Contextualized word representations are known to represent a large amount of
syntactic information that is not available to traditional bag-of-words topic
models Goldberg (2019). We therefore expect that token-level clusterings of
contextualized word representations will have more homogeneity of syntactic
form, and indeed we find that they do.

As a simple proxy for syntactic similarity, we find the most likely part of
speech (POS) for the top words in each cluster. We use this method because it
is easily implemented; inaccuracies should be consistent across models. To
quantitatively evaluate the homogeneity of POS within each topic, we count the
distribution of POS tags for the top 20 words of a cluster and calculate the
entropy of that distribution. If all 20 words are the same POS, this value
will be 0, while if POS tags are more diffuse it will be larger. We find that
BERT and GPT-2 clusters have consistently lower entropy. In Table 3, we see
that the 25th percentile for LDA topics has entropy 0.97, higher than the
median entropy for both BERT and GPT-2. We find that these results are
consistent across model sizes. Although contextualized embedding clusters are
more homogeneous in POS, LDA may appear more homogeneous because it is
dominated by nouns. For LDA, nouns and proper nouns account for 43.7% and
33.4% respectively of all the words in the top 20 for all topics, while verbs
make up 8.5% and adjectives 6.7%. These proportions are 39.0%, 25.3%, 14.9%,
and 8.1% for BERT, and 37.0%, 23.4%, 16.9%, and 9.5% for GPT-2.

### Compression improves efficiency while maintaining quality.

We have established that we can effectively learn topic models by clustering
token-level contextualized embeddings, and we have shown that there are
advantages to clustering at the token rather than vocabulary level. But for
token-level clustering to be more than a curiosity we need to address
computational complexity. Vocabularies are typically on the order of tens of
thousands of words, while collections often contain millions of words. Even
storing full 768-dimensional vectors for millions of tokens, much less
clustering them, can be beyond the capability of many potential users’
computing resources. Therefore, we investigate the effects of feature
dimensionality reduction to reduce the memory footprint of our method.

The hidden layers of deep learning models are known to learn representations
with dimensionalities much lower than the number of neurons Raghu et al.
(2017). We apply two methods for dimensionality reduction: principal component
analysis (PCA) and sparse random projection (SRP) Li et al. (2006); Achlioptas
(2001).333All implementations from Pedregosa et al. (2011).

We find that reducing our token vectors to as few as 100 dimensions can have
little negative effect. Figure 3 shows that reduced PCA features produce
improved internal coherence and little significant change in external
coherence, but reduced SRP features are worse in both metrics, especially for
BERT. We note that more clusters pass the 10-valid-words threshold for reduced
PCA and SRP features. Instead of skipping 10 BERT clusters and 7 GPT-2
clusters on average, no clusters are skipped for PCA reduced features and only
1 for 100-dimensional SRP features. For 300-dimensional SRP features there is
only a significant drop for GPT-2 with 3 skipped on average. This decrease in
skipped topics indicates that overly specific topics are being replaced with
less specific ones. We hypothesize that dimensionality reduction is smoothing
away “spikes” in the embeddings space that cause the algorithm to identify
small clusters. Finally, larger dimensionality reductions decrease
concentration and exclusivity, making clusters more general.

(a)

(b)

Figure 3: Mean internal and external coherence for reduced features of BERT
and GPT-2. Features reduced with PCA tend to have higher coherence than SRP.

PCA is significantly better than SRP, especially for more aggressive
dimensionality reductions. We find that mean-centering SRP features does not
significantly improve results. An advantage of SRP, however, is that the
projection matrix can be generated offline and immediately applied to
embedding vectors as soon as they are generated. To overcome the memory
limitations of PCA, we use a batch approximation, incremental PCA Ross et al.
(2008). Using 100 dimensions and scikit-learn’s default batch size of five
times the number of features (3840), we find no significant difference in
results between PCA and incremental PCA. For the remainder of experiments we
use 100-dimensional vectors which correspond to the top 100 components
produced by incremental PCA.

Figure 4:  BERT and GPT-2 produce coherent topics for less familiar (w.r.t. pretraining) collections. BERT consistently produces more unique clusters. LDA external coherence drops for $K=500$. books |  book books author novel novels work Book fiction by authors Kindle published volume works literature
---|---
read reading copy Read reads Reading readable reader reread across follow
opened researched study hand
problem children problems course power lives mystery questions issues words
death example reality battle
electronics |  use up than off used back over using there about work need down thing nice other full no easy small
screen quality sound device power battery unit system software remote video
player mode drive audio
setup remote battery mode card set range input signal support setting manual
stand menu GPS fan power
movies |  movie movies films flick theater Movie flicks game cinema film comedies Movies pictures westerns
into up over through between off down than about around during against under
found away along though
film movie picture screen documentary films Film cinema feature work
production filmmaker piece Picture
cds |  album albums record release Album LP releases records effort up label EP thing titled studio label titled
songs tracks hits tunes singles material stuff Songs ballads cuts ones numbers
sounds artists Hits versions
lyrics guitar vocals voice bass singing solo vocal sound work music piano
chorus style here live playing
Table 4: Unlike vocabulary-level clusters, token-level clusters are grounded
in specific documents and can be used to analyze collections. Here we show the
most prominent BERT topics ($K=500$) for the four product categories in
Reviews. This analysis is purely post hoc, neither BERT nor its clustering
have access to product labels. than beyond Than twice upon much except besides
half times less unlike nor yet per alone exceeded beside above within
---
day days Day morning today date daily Days month 19 basis night period 1979
shortly moment term rainy mood sunny
can Can able possible manage could knows lets capable allows can’t easily s
Cannot cant ability Can’t can‘t barely cannot
when When once time whenever Once soon everytime upon during Whenever moment
Everytime At near before anytime
too enough Too overly taste beyond tired sufficiently plenty somehow Enough
unnecessary sufficient sick half overkill
because since due Because Since considering cause meaning given thanks means
based result order therefore being
went took got happened came started did turned ended fell used kept left taken
gave stopped won ran made moved
would ’d Would might d normally I’d imagine otherwise happily woulda wouldn’t
Wouldn’t envision probably Iwould
up off Up ready upload ups along end forth uploading used down away unpacked
securely rope onto open unpacking
instead rather either matter Instead opposed other depending based choice
Rather than favor otherwise regardless no
Table 5: The ten BERT topics from Reviews with the most uniform distribution
over product categories.

### Pretrained embeddings are effective for different collections.

While BERT and GPT-2 cluster models produce useful topics for Wikipedia, will
this hold for collections less similar to the training data of these
pretrained models? Does it work well for collections of much shorter and much
longer texts than Wikipedia articles? We find that both BERT and GPT-2 produce
semantically meaningful topics for Scotus and Reviews, but BERT continues to
outperform GPT-2. As with Wikipedia, we find that contextualized embedding
clusters have the largest advantage over LDA for large $K$. Figure 4 shows
that for $K=500$ BERT and GPT-2 clusters have significantly higher external
coherence scores on average than LDA topics for Scotus and very similar scores
for Reviews. For smaller $K$, LDA has the highest external coherence scores
followed by BERT. Internal coherence is more difficult to interpret because of
the variability in exclusivity. With $K=500$, BERT clusters have substantially
worse internal coherence scores. In contrast, GPT-2 clusters tend to
experience a smaller drop in scores, but this can be partially explained by
their much lower average exclusivity. We find that BERT consistently produces
the most unique topics for Scotus and Reviews. BERT consistently has
significantly higher mean exclusivity scores for both Scotus and Reviews,
while GPT-2 tends to have scores as good as LDA for $K\in\\{50,100\\}$, but
significantly lower for $K=500$.

1980–2019 |  Top Words
---|---
10 0.000 0.495 0.026 1.000 0.051 0.962 0.077 0.937 0.103 0.700 0.128 0.637 0.154 0.643 0.179 0.734 0.205 0.575 0.231 0.807 0.256 0.484 0.282 0.557 0.308 0.147 0.333 0.170 0.359 0.282 0.385 0.204 0.410 0.245 0.436 0.203 0.462 0.321 0.487 0.240 0.513 0.080 0.538 0.153 0.564 0.212 0.590 0.204 0.615 0.117 0.641 0.107 0.667 0.119 0.692 0.142 0.718 0.185 0.744 0.222 0.769 0.147 0.795 0.240 0.821 0.234 0.846 0.160 0.872 0.264 0.897 0.159 0.923 0.095 0.949 0.043 0.974 0.260 1.000 0.152 |  union employment labor bargaining Labor workers job strike unions working
10 0.000 0.515 0.026 1.000 0.051 0.442 0.077 0.918 0.103 0.390 0.128 0.358 0.154 0.309 0.179 0.391 0.205 0.400 0.231 0.383 0.256 0.126 0.282 0.171 0.308 0.202 0.333 0.169 0.359 0.154 0.385 0.129 0.410 0.032 0.436 0.290 0.462 0.248 0.487 0.238 0.513 0.161 0.538 0.078 0.564 0.104 0.590 0.170 0.615 0.159 0.641 0.177 0.667 0.053 0.692 0.057 0.718 0.105 0.744 0.237 0.769 0.121 0.795 0.065 0.821 0.079 0.846 0.077 0.872 0.054 0.897 0.197 0.923 0.049 0.949 0.009 0.974 0.062 1.000 0.426 |  gas coal oil natural mining mineral fuel mine fishing hunting
10 0.000 0.739 0.026 1.000 0.051 0.550 0.077 0.749 0.103 0.536 0.128 0.492 0.154 0.461 0.179 0.435 0.205 0.505 0.231 0.441 0.256 0.466 0.282 0.146 0.308 0.408 0.333 0.261 0.359 0.123 0.385 0.312 0.410 0.229 0.436 0.592 0.462 0.141 0.487 0.224 0.513 0.119 0.538 0.376 0.564 0.245 0.590 0.231 0.615 0.120 0.641 0.156 0.667 0.211 0.692 0.177 0.718 0.174 0.744 0.148 0.769 0.146 0.795 0.164 0.821 0.381 0.846 0.158 0.872 0.259 0.897 0.305 0.923 0.177 0.949 0.085 0.974 0.176 1.000 0.657 |  compensation wages pension wage salary welfare compensate salaries retirement bonus
10 0.000 0.475 0.026 0.446 0.051 0.936 0.077 0.603 0.103 0.715 0.128 0.334 0.154 1.000 0.179 0.527 0.205 0.322 0.231 0.776 0.256 0.350 0.282 0.426 0.308 0.415 0.333 0.396 0.359 0.344 0.385 0.405 0.410 0.241 0.436 0.144 0.462 0.362 0.487 0.461 0.513 0.240 0.538 0.260 0.564 0.188 0.590 0.423 0.615 0.250 0.641 0.397 0.667 0.203 0.692 0.328 0.718 0.380 0.744 0.314 0.769 0.344 0.795 0.282 0.821 0.174 0.846 0.387 0.872 0.238 0.897 0.229 0.923 0.181 0.949 0.332 0.974 0.265 1.000 0.203 |  discrimination prejudice unfair bias harassment segregation retaliation boycott persecution
10 0.000 0.508 0.026 0.668 0.051 0.832 0.077 0.857 0.103 0.401 0.128 0.511 0.154 0.723 0.179 0.268 0.205 0.567 0.231 0.377 0.256 0.903 0.282 0.479 0.308 0.498 0.333 0.260 0.359 0.382 0.385 0.173 0.410 0.176 0.436 0.614 0.462 0.306 0.487 0.347 0.513 0.753 0.538 0.357 0.564 0.326 0.590 0.664 0.615 0.114 0.641 0.183 0.667 0.433 0.692 0.282 0.718 0.214 0.744 0.216 0.769 0.154 0.795 0.939 0.821 1.000 0.846 0.472 0.872 0.375 0.897 0.317 0.923 0.926 0.949 0.198 0.974 0.255 1.000 0.252 |  medical health care hospital physician patient Medical physicians clinic hospitals
10 0.000 0.437 0.026 0.360 0.051 0.532 0.077 0.606 0.103 1.000 0.128 0.434 0.154 0.625 0.179 0.317 0.205 0.508 0.231 0.260 0.256 0.598 0.282 0.244 0.308 0.522 0.333 0.538 0.359 0.334 0.385 0.153 0.410 0.186 0.436 0.544 0.462 0.158 0.487 0.219 0.513 0.063 0.538 0.114 0.564 0.356 0.590 0.122 0.615 0.218 0.641 0.223 0.667 0.255 0.692 0.494 0.718 0.371 0.744 0.094 0.769 0.345 0.795 0.318 0.821 0.307 0.846 0.587 0.872 0.416 0.897 0.304 0.923 0.718 0.949 0.079 0.974 0.877 1.000 0.168 |  market competition competitive markets compete demand marketplace trading competitor trade
10 0.000 0.635 0.026 0.367 0.051 0.493 0.077 0.471 0.103 0.267 0.128 0.156 0.154 1.000 0.179 0.141 0.205 0.090 0.231 0.182 0.256 0.121 0.282 0.386 0.308 0.544 0.333 0.192 0.359 0.555 0.385 0.346 0.410 0.641 0.436 0.475 0.462 0.138 0.487 0.266 0.513 0.743 0.538 0.152 0.564 0.106 0.590 0.594 0.615 0.184 0.641 0.154 0.667 0.361 0.692 0.187 0.718 0.643 0.744 0.296 0.769 0.430 0.795 0.135 0.821 0.071 0.846 0.337 0.872 0.364 0.897 0.411 0.923 0.216 0.949 0.130 0.974 0.531 1.000 0.215 |  election vote voting electoral ballot voter elected votes elect Election
10 0.000 0.312 0.026 0.138 0.051 0.231 0.077 0.403 0.103 0.234 0.128 0.345 0.154 0.310 0.179 0.206 0.205 0.138 0.231 0.130 0.256 0.251 0.282 0.190 0.308 0.316 0.333 0.400 0.359 0.301 0.385 0.288 0.410 0.074 0.436 0.197 0.462 0.385 0.487 0.176 0.513 0.345 0.538 0.138 0.564 0.171 0.590 0.203 0.615 0.258 0.641 0.291 0.667 0.352 0.692 0.275 0.718 0.923 0.744 0.415 0.769 1.000 0.795 0.542 0.821 0.325 0.846 0.224 0.872 0.941 0.897 0.408 0.923 0.346 0.949 0.229 0.974 0.364 1.000 0.654 |  violence firearm gun violent weapon firearms armed weapons arms lethal
10 0.000 0.442 0.026 0.168 0.051 0.029 0.077 0.050 0.103 0.280 0.128 0.323 0.154 0.020 0.179 0.046 0.205 0.193 0.231 0.156 0.256 0.288 0.282 0.085 0.308 0.030 0.333 0.082 0.359 0.079 0.385 0.041 0.410 0.085 0.436 0.079 0.462 0.159 0.487 0.184 0.513 0.019 0.538 0.210 0.564 0.107 0.590 0.417 0.615 0.022 0.641 0.101 0.667 0.153 0.692 0.197 0.718 0.131 0.744 0.012 0.769 0.568 0.795 0.239 0.821 1.000 0.846 0.905 0.872 0.445 0.897 0.318 0.923 0.277 0.949 0.473 0.974 0.278 1.000 0.381 |  patent copyright Copyright Patent patents trademark invention patented copyrighted patentee
Table 6: Grounded topics allow us to analyze trends in the organization of a
corpus using a BERT topic model with $K=500$. Here we measure the prevalence
of topics from 1980 to 2019 in US Supreme Court opinions by counting token
assignments. Topics related to unions and natural resources are more prevalent
earlier in the collection, while topics related to firearms and intellectual
property have become more prominent.

### Contextualized embedding clusters support collection analysis.

Token-level clusterings of contextualized word representations support
sophisticated corpus analysis with little additional complexity. In practice,
topic models are often used as a way to “map” a specific collection, for
example by identifying key documents or measuring the prevalence of topics
over time Boyd-Graber et al. (2017). A key disadvantage therefore of
vocabulary-level semantic clustering is that it is not grounded in specific
documents in a collection. Token-level clustering, in contrast, supports a
wide range of analysis techniques for researchers interested in using a topic
model as a means for studying a specific collection. We provide two case
studies that both use additional metadata to organize token-level clusterings
post hoc.

Given a partition of the working collection, we can count tokens assigned to
each topic within a given partition to estimate locally prominent topics.
Table 4 shows the three most prominent topics for the four product categories
in Reviews from a BERT cluster model with $K=500$. Many of these topics are
clearly interpretable as aspects of a particular product (e.g. full albums,
individual songs, descriptions of songs). Two topics contain mostly
prepositions. While we could have added these words to a stoplist before
clustering, these less obviously interpretable clusters can nevertheless
represent distinct discourses, such as descriptions of action in Movies (into,
up, over, through, …) or descriptions of physical objects in Electronics (use,
up, than, off, …). We can also find the topics that are least associated with
any one product category by calculating the entropy of their distribution of
tokens across categories. These are shown in Table 5, and appear to represent
subtle variations of subjective experiences: overkill, possibilities, reasons,
and time periods. We emphasize that this analysis requires no additional
modeling, simply counting.

For partitions that have a natural order, such as years, we can create time
series in the same post hoc manner. Thus, we can use a BERT clustering of
Scotus to examine the changes in subject of the cases brought before the US
Supreme Court. Table 6 shows time series for nine manually selected topics
from a BERT clustering of Scotus with $K=500$, ordered by the means of their
distributions over years. We find that topics related to labor and collective
bargaining, oil and gas exploration, and compensation have decreased in
intensity since the 1980s, while those related to medical care and elections
have remained relatively stable. It appears that competitive markets was a
less common subject in the middle years, but has returned to prominence.
Meanwhile, discrimination has remained a prominent topic throughout the
period, but with higher intensities in the 1980s. Additionally, topics related
to gun violence and patents and copyright appear to be increasing in
intensity.

##  6 Conclusion

We have presented a simple, reliable method for extracting mixed-membership
models from pretrained contextualized word representations. This result is of
interest in several ways. First, it provides insight into the affordances of
contextualized representations. For example, our result suggests a way to
rationalize seemingly ad hoc methods such as averaging token vectors to build
a representation of a sentence. Second, it suggests directions for further
analysis and development of contextualized representation models and
algorithms. The significant differences we observe in superficially similar
systems such as BERT and RoBERTa require explanations that could expand our
theoretical understanding of what these models are doing. Why, for example, is
RoBERTa more prone to very small, specific clusters, while BERT is not?
Furthermore, if models like BERT are producing output similar to topic model
algorithms, this connection may suggest new directions for simpler and more
efficient language model algorithms, as well as more representationally
powerful topic model algorithms. Third, there may be substantial practical
benefits for researchers analyzing collections. Although running BERT on a
large-scale corpus may for now be substantially more computationally
inefficient than running highly-tuned LDA algorithms, passing a collection
through such a system is likely to become an increasingly common analysis
step. If such practices could be combined with online clustering algorithms
that would not require storing large numbers of dense token-level vectors,
data analysts who are already using BERT-based workflows could easily extract
high-quality topic model output with essentially no additional work.

## Acknowledgments

This work was supported by NSF #1652536 and the Alfred P. Sloan foundation.

## References

  * Achlioptas (2001) Dimitris Achlioptas. 2001.  Database-friendly random projections.  In _PODS ’01_.
  * Aharoni and Goldberg (2020) Roee Aharoni and Yoav Goldberg. 2020.  Unsupervised domain clusters in pretrained language models.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7747–7763, Online. Association for Computational Linguistics.
  * Arora et al. (2013) Sanjeev Arora, Rong Ge, Yonatan Halpern, David Mimno, Ankur Moitra, David Sontag, Yichen Wu, and Michael Zhu. 2013.  A practical algorithm for topic modeling with provable guarantees.  In _International Conference on Machine Learning_ , pages 280–288.
  * Bischof and Airoldi (2012) Jonathan Bischof and Edoardo M Airoldi. 2012.  Summarizing topical content with word frequency and exclusivity.  In _Proceedings of the 29th International Conference on Machine Learning (ICML-12)_ , pages 201–208.
  * Blei et al. (2003) David M Blei, Andrew Y Ng, and Michael I Jordan. 2003.  Latent dirichlet allocation.  _Journal of machine Learning research_ , 3(Jan):993–1022.
  * Bommasani et al. (2020) Rishi Bommasani, Kelly Davis, and Claire Cardie. 2020.  Interpreting Pretrained Contextualized Representations via Reductions to Static Embeddings.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , Online. Association for Computational Linguistics.
  * Boyd-Graber et al. (2017) Jordan Boyd-Graber, Yuening Hu, David Mimno, et al. 2017.  Applications of topic models.  _Foundations and Trends in Information Retrieval_ , 11(2-3):143–296.
  * Brown et al. (1992) Peter F. Brown, Peter V. deSouza, Robert L. Mercer, Vincent J. Della Pietra, and Jenifer C. Lai. 1992.  Class-based n-gram models of natural language.  _Computational Linguistics_ , 18(4).
  * Chen et al. (2016) Wenhu Chen, Evgeny Matusov, Shahram Khadivi, and Jan-Thorsten Peter. 2016.  Guided alignment training for topic-aware neural machine translation.  _AMTA 2016, Vol._ , page 121.
  * Coenen et al. (2019) Andy Coenen, Emily Reif, Ann Yuan, Been Kim, Adam Pearce, Fernanda B. Viégas, and Martin Wattenberg. 2019.  Visualizing and measuring the geometry of BERT.  In _Advances in Neural Information Processing Systems 32_ , pages 8594–8603. Curran Associates, Inc.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Dhillon and Modha (2001) Inderjit S Dhillon and Dharmendra S Modha. 2001.  Concept decompositions for large sparse text data using clustering.  _Machine learning_ , 42(1-2):143–175.
  * Dieng et al. (2020) Adji B. Dieng, Francisco J. R. Ruiz, and David M. Blei. 2020.  Topic modeling in embedding spaces.  _Transactions of the Association for Computational Linguistics_ , 8:439–453.
  * Endo and Miyamoto (2015) Yasunori Endo and Sadaaki Miyamoto. 2015.  Spherical k-means++ clustering.  In _International Conference on Modeling Decisions for Artificial Intelligence_ , pages 103–114. Springer.
  * Ethayarajh (2019) Kawin Ethayarajh. 2019.  How contextual are contextualized word representations? comparing the geometry of BERT, ELMo, and GPT-2 embeddings.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 55–65, Hong Kong, China. Association for Computational Linguistics.
  * Goldberg (2019) Yoav Goldberg. 2019.  Assessing bert’s syntactic abilities.  _ArXiv_ , abs/1901.05287.
  * He and McAuley (2016) Ruining He and Julian McAuley. 2016.  Ups and downs: Modeling the visual evolution of fashion trends with one-class collaborative filtering.  In _proceedings of the 25th international conference on world wide web_ , pages 507–517.
  * Lancichinetti et al. (2015) Andrea Lancichinetti, M Irmak Sirer, Jane X Wang, Daniel Acuna, Konrad Körding, and Luís A Nunes Amaral. 2015.  High-reproducibility and high-accuracy method for automated topic classification.  _Physical Review X_ , 5(1):011007.
  * Li et al. (2006) Ping Li, Trevor J. Hastie, and Kenneth Ward Church. 2006.  Very sparse random projections.  In _KDD ’06_.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _ArXiv_ , abs/1907.11692.
  * Martin et al. (2020) Louis Martin, Benjamin Muller, Pedro Javier Ortiz Suárez, Yoann Dupont, Laurent Romary, Éric de la Clergerie, Djamé Seddah, and Benoît Sagot. 2020.  CamemBERT: a tasty French language model.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7203–7219, Online. Association for Computational Linguistics.
  * McAuley et al. (2015) Julian McAuley, Christopher Targett, Qinfeng Shi, and Anton Van Den Hengel. 2015\.  Image-based recommendations on styles and substitutes.  In _Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval_ , pages 43–52.
  * McCallum (2002) Andrew Kachites McCallum. 2002.  Mallet: A machine learning for language toolkit.
  * Merity et al. (2017) Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2017.  Pointer sentinel mixture models.  In _5th International Conference on Learning Representations, ICLR 2017_.
  * Miao et al. (2017) Yishu Miao, Edward Grefenstette, and Phil Blunsom. 2017.  Discovering discrete latent topics with neural variational inference.  In _PMLR_.
  * Mikolov et al. (2013a) Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013a.  Efficient estimation of word representations in vector space.  In _Proceedings of Workshop at ICLR_.
  * Mikolov et al. (2013b) Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013b.  Distributed representations of words and phrases and their compositionality.  In _Advances in Neural Information Processing Systems 26_ , pages 3111–3119. Curran Associates, Inc.
  * Mimno et al. (2011) David Mimno, Hanna Wallach, Edmund Talley, Miriam Leenders, and Andrew McCallum. 2011.  Optimizing semantic coherence in topic models.  In _Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing_ , pages 262–272, Edinburgh, Scotland, UK. Association for Computational Linguistics.
  * Narayan et al. (2018) Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018.  Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 1797–1807, Brussels, Belgium. Association for Computational Linguistics.
  * Newman et al. (2010) David Newman, Youn Noh, Edmund Talley, Sarvnaz Karimi, and Timothy Baldwin. 2010\.  Evaluating topic models for digital libraries.  In _Proceedings of the 10th annual joint conference on Digital libraries_ , pages 215–224.
  * Nguyen and Nguyen (2020) Dat Quoc Nguyen and Anh Tuan Nguyen. 2020.  PhoBERT: Pre-trained language models for Vietnamese.  _Findings of EMNLP_.
  * Pedregosa et al. (2011) F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011.  Scikit-learn: Machine learning in Python.  _Journal of Machine Learning Research_ , 12:2825–2830.
  * Peinelt et al. (2020) Nicole Peinelt, Dong Nguyen, and Maria Liakata. 2020.  tBERT: Topic models and BERT joining forces for semantic similarity detection.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7047–7055, Online. Association for Computational Linguistics.
  * Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014.  Glove: Global vectors for word representation.  In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 1532–1543, Doha, Qatar. Association for Computational Linguistics.
  * Petroni et al. (2019) Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019.  Language models as knowledge bases? In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 2463–2473, Hong Kong, China. Association for Computational Linguistics.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019.  Language models are unsupervised multitask learners.  _OpenAI Blog_ , 1(8):9.
  * Raghu et al. (2017) Maithra Raghu, Justin Gilmer, Jason Yosinski, and Jascha Sohl-Dickstein. 2017.  Svcca: Singular vector canonical correlation analysis for deep learning dynamics and interpretability.  In _Advances in Neural Information Processing Systems 30_ , pages 6076–6085. Curran Associates, Inc.
  * Ross et al. (2008) David A Ross, Jongwoo Lim, Ruei-Sung Lin, and Ming-Hsuan Yang. 2008.  Incremental learning for robust visual tracking.  _International journal of computer vision_ , 77(1-3):125–141.
  * Sandhaus (2008) Evan Sandhaus. 2008.  The new york times annotated corpus.  _Linguistic Data Consortium, Philadelphia_ , 6(12):e26752.
  * Sennrich et al. (2016) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016.  Neural machine translation of rare words with subword units.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1715–1725, Berlin, Germany. Association for Computational Linguistics.
  * Sia et al. (2020) Suzanna Sia, Ayush Dalmia, and Sabrina J. Mielke. 2020.  Tired of topic models? clusters of pretrained word embeddings make for fast and good topics too!  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing_.
  * Srivastava and Sutton (2016) Akash Srivastava and Charles Sutton. 2016.  Neural variational inference for topic models.  In _NeurIPS Bayesian deep learning workshop_.
  * Stevens et al. (2012) Keith Stevens, Philip Kegelmeyer, David Andrzejewski, and David Buttler. 2012.  Exploring topic coherence over many models and many topics.  In _Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning_ , pages 952–961, Jeju Island, Korea. Association for Computational Linguistics.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Advances in Neural Information Processing Systems 30_ , pages 5998–6008. Curran Associates, Inc.
  * Vulić et al. (2020) Ivan Vulić, Edoardo Maria Ponti, Robert Litschko, Goran Glavaš, and Anna Korhonen. 2020.  Probing pretrained language models for lexical semantics.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing_.
  * Wang et al. (2019) Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019.  Superglue: A stickier benchmark for general-purpose language understanding systems.  In _Advances in Neural Information Processing Systems 32_ , pages 3266–3280. Curran Associates, Inc.
  * Wang et al. (2018) Li Wang, Junlin Yao, Yunzhe Tao, Li Zhong, Wei Liu, and Qiang Du. 2018.  A reinforced topic-aware convolutional sequence-to-sequence model for abstractive text summarization.  In _Proceedings of the 27th International Joint Conference on Artificial Intelligence_ , pages 4453–4460.
  * Wiedemann et al. (2019) Gregor Wiedemann, Steffen Remus, Avi Chawla, and Chris Biemann. 2019.  Does bert make any sense? interpretable word sense disambiguation with contextualized embeddings.  In _KONVENS_.
  * Wolf et al. (2019) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R’emi Louf, Morgan Funtowicz, and Jamie Brew. 2019.  Huggingface’s transformers: State-of-the-art natural language processing.  _ArXiv_ , abs/1910.03771.
  * Wu et al. (2016) Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Gregory S. Corrado, Macduff Hughes, and Jeffrey Dean. 2016.  Google’s neural machine translation system: Bridging the gap between human and machine translation.  _ArXiv_ , abs/1609.08144.