# Dynamical Unification of Tidal Disruption Events

Lars Lund Thomsen Department of Physics, University of Hong Kong, Pokfulam
Road, Hong Kong, China  Tom Kwan Department of Physics, University of Hong
Kong, Pokfulam Road, Hong Kong, China  Lixin Dai Department of Physics,
University of Hong Kong, Pokfulam Road, Hong Kong, China  Samantha Wu
California Institute of Technology, Astronomy Department, Pasadena, CA 91125,
USA  Enrico Ramirez-Ruiz Department of Astronomy and Astrophysics, University
of California Santa Cruz, 1156 High Street, Santa Cruz, CA 95060, USA  DARK,
Niels Bohr Institute, University of Copenhagen, Denmark

###### Abstract

About a hundred tidal disruption events (TDEs) have been observed and they
exhibit a wide range of emission properties both at peak and over their
lifetimes. Some TDEs peak predominantly at X-ray energies while others radiate
chiefly at UV and optical wavelengths. While the peak luminosities across TDEs
show distinct properties, the evolutionary behavior can also vary between TDEs
with similar peak emission properties. At late time, some optical TDEs
rebrighten in X-rays, while others maintain strong UV/optical emission
components. In this Letter, we conduct three-dimensional general relativistic
radiation magnetohydrodynamics simulations of TDE accretion disks at varying
accretion rates ranging from a few to a few tens of the Eddington accretion
rate. We make use of Monte Carlo radiative transfer simulations to calculate
the reprocessed spectra at various inclinations and at different evolutionary
stages. We confirm the unified model proposed by Dai et al. (2018), which
predicts that the observed emission largely depends on the viewing angle of
the observer with respect to the disk orientation (X-ray strong when viewed
face-on and UV/optically strong when viewed disk-on). What is more, we find
that disks with higher accretion rates have elevated wind and disk densities,
which increases the reprocessing of the high-energy radiation and thus
generally augments the optical-to-X-ray flux ratio along a particular viewing
angle. This implies that at later times, as the accretion level declines, we
expect the funnel to effectively open up and allow more X-rays to leak out
along intermediate viewing angles. Such dynamical model for TDEs can provide a
natural explanation for the diversity in the emission properties observed in
TDEs at peak and along their temporal evolution.

††journal: ApJL

##  1 Introduction

The tidal disruption of stars by massive black holes (MBHs) offers a unique
probe of MBH demographics (Mockler et al., 2019; Gezari, 2021), host galaxy
properties (French et al., 2020), stellar dynamics (Stone et al., 2020;
Pfister et al., 2020), as well as black hole accretion and jet physics (Dai et
al., 2021). When a star with mass $m_{\star}$ and radius $r_{\star}$
approaches a MBH with mass $M_{\rm BH}$, the star is disrupted within the
tidal radius $R_{t}\approx(M_{\rm BH}/m_{\star})^{1/3}r_{\star}$, where the
MBH’s tidal force exceeds the stellar self-gravity. About half stellar debris
orbits back to the vicinity of the MBH following a characteristic pattern
which first quickly increases to a peak and then declines with time following
a mass fallback rate $\dot{M}_{\rm fb}\approx t^{-5/3}$ (Rees, 1988; Evans &
Kochanek, 1989; Guillochon & Ramirez-Ruiz, 2013). When $M_{\rm BH}\sim
10^{6}M_{\odot}$, $\dot{M}_{\rm fb}$ can exceed the Eddington accretion rate
by two orders of magnitude at peak and stays super-Eddington for years after
peak. Here we define the Eddington accretion rate of a black hole as
$\dot{M}_{\rm Edd}=L_{\rm Edd}/(\eta_{\rm NT}\ c^{2})$, where $L_{\rm
edd}=4\pi GM_{\rm BH}c/\kappa$ is the Eddington luminosity for an opacity
$\kappa$, and $\eta_{\rm NT}$ is the nominal accretion efficiency for the
Novikov-Thorne thin disk solution (Novikov & Thorne, 1973).

With the recent launches of all-sky transient surveys including ZTF and
eROSITA, the number of detected TDE candidates has reached around one hundred
(e.g., Gezari, 2021; Sazonov et al., 2021; Hammerstein et al., 2022). TDEs
have been categorized into two classes based on their main emission type at
peak: X-ray TDEs (Auchettl et al., 2017; Saxton et al., 2021) and optical TDEs
(van Velzen et al., 2020). Most X-ray-selected TDEs emit soft X-rays with
effective temperatures at $10^{5}-10^{6}$K, while only three of them emit
beamed, hard X-rays which are associated with relativistic jets (e.g., Bloom
et al., 2011; Burrows et al., 2011; Cenko et al., 2012; De Colle et al.,
2012). The optical TDEs have lower effective temperatures at few$\times
10^{4}$K, and are further characterized by their spectroscopic features
(producing strong and broad H, or He, or Bowen fluorescence emission lines)
(Leloudas et al., 2019; Charalampopoulos et al., 2022). Interestingly, as the
luminosity of a TDE typically declines by around one order of magnitude over
tens to hundreds of days after peak, its effective temperature usually
undergoes a peculiar non-evolution. Recently, a large number of TDEs have been
followed up for longer than a year and they show different late-time
behaviors. A few initially optically-strong TDEs rebrighten in X-rays (e.g.,
Gezari et al., 2017; Holoien et al., 2018; Wevers et al., 2019; Hinkle et al.,
2021; Liu et al., 2022), while many others maintain strong UV/optical
emissions over years (van Velzen et al., 2019).

Many theoretical models have been proposed to explain these TDE emission
properties (Roth et al., 2020). While the X-ray emissions have been predicted
by classical accretion disk models (Ulmer, 1999), UV/optical emissions are
argued to be produced from either the reprocessing of X-ray photons in an
extended envelope or outflows (Loeb & Ulmer, 1997; Strubbe & Quataert, 2009;
Lodato & Rossi, 2011; Coughlin & Begelman, 2014; Guillochon et al., 2014;
Metzger & Stone, 2016; Roth et al., 2016) or the shocks powered by debris
stream self-intersection (Piran et al., 2015; Bonnerot et al., 2021). The
late-time rebrightening of X-rays in TDEs can be accounted by either a change
in the disk morphology as the accretion rate declines from super-Eddington to
sub-Eddington, or the delayed onset of accretion (van Velzen et al., 2019;
Hayasaki & Jonker, 2021). The latter model, however, is disfavored by recent
simulations which show that a large fraction of the debris already settles
into a disk with moderate eccentricity within dynamical times for at least a
subset of TDE parameters (Bonnerot et al., 2021; Andalman et al., 2022).

In search for a unified model that can explain both the optical and X-ray
TDEs, Dai et al. (2018) have carried out the first three-dimensional (3D)
general relativistic radiation magnetohydrodynamics (GRRMHD) simulation
tailored for TDE super-Eddignton accretion flow. The simulated disk, around a
black hole with $M_{\rm BH}=5\times 10^{6}M_{\odot}$ and spin parameter
$a=0.8$, has an accretion rate of $\dot{M}_{\rm acc}\sim 10\ \dot{M}_{\rm
Edd}$, representing a typical accretion level in TDEs. The spectra of the disk
have been obtained by post-processing the simulated disk with a novel
radiative transfer code. It is found that the observed emission type largely
depends on the viewing angle of the observer with respect to the disk
orientation. When viewed face-on, X-ray emissions can escape from the
optically-thin funnel surrounded by winds. When viewed edge-on, X-ray
emissions are heavily reprocessed in the geometrically and optically thick
wind and disk, so only UV/optical emissions can be observed.

While this study gives a good first-order description of TDE disks and
spectra, recent simulations also show that the properties of a super-Eddington
disk depends on the accretion rate, the black hole mass and spin, as well as
the magnetic flux threading the disk (Jiang et al., 2014, 2019; McKinney et
al., 2015; Sa̧dowski & Narayan, 2016). Although the black hole spin and disk
magnetic flux might only mildly affect the main structure of the disk, they
determine whether a relativistic jet can be launched (Blandford & Znajek,
1977; Curd & Narayan, 2019). At first glance, the black hole mass is not
expected to affect the disk spectra significantly, since most TDE host MBHs
have masses in a narrow range $\approx 10^{5}-10^{7}\ M_{\odot}$. However, the
peak fallback rate of the TDE debris sensitively depends on the black hole
mass with the relation $\dot{M}_{\rm fb,peak}/\dot{M}_{\rm Edd}\propto
M^{-3/2}_{\rm BH}$. Therefore, TDEs from smaller MBHs should in general have
much higher accretion rates at peak than those from larger MBHs (Ramirez-Ruiz
& Rosswog, 2009). Also, further variance of the accretion rates in different
TDEs as $\dot{M}_{\rm fb,peak}$ can be introduced by the difference in the
masses and ages of the disrupted stars (Law-Smith et al., 2020).

In this Letter, we investigate how the TDE disk structure and the accompanied
emission are influenced by the accretion rate at super-Eddington rates. We
conduct three 3D GRRMHD simulations of super-Eddington disks with similar
structures but varying accretion rates, and post-process the simulated disks
to obtain their spectra at different inclination angles. These simulations are
useful for understanding the diversity of the emissions observed from
different TDEs, as well as the evolution of single TDEs as their fallback and
accretion rate decline after peak. In Section 2 we introduce the setup of the
GRRMHD and radiative transfer simulations. In Section 3 we give the main
results and compare with TDE key observables. In Section 4 we draw a summary
and discuss the implications and future work.

##  2 Methodology

###  2.1 Disk simulation setup

We carry out three dimensional simulations of super-Eddington disks using a
fully 3D GRRMHD code HARMRAD with M1 closure (McKinney et al., 2014). In all
simulations the MBH has $M_{\rm BH}=10^{6}M_{\odot}$ and $a=0.8$. The radial
grid has 128 cells spanning from $R_{\rm{}in}\approx 1.2R_{g}$ to $R_{\rm
out}=8500R_{g}$ ($R_{g}=GM_{\rm BH}/c^{2}$ is the gravitational radius of the
MBH), with cell size increasing exponentially until $R_{\rm break}\approx
500R_{g}$ and then increasing hyper-exponentially. The $\theta$-grid has 64
cells spanning from 0 to $\pi$ with finer resolution in the jet and disk
regions. The $\phi$-grid has 32 grids spanning uniformly from 0 to $2\pi$ with
periodic boundary conditions. The gas is assumed to have solar chemical
abundances (mass fractions of H, He, and “metals”, respectively,
$X=0.7,Y=0.28,Z=0.02$). Frequency-mean absorption and emission opacities are
used as in McKinney et al. (2015), except that the Chianti opacity is removed
as it is unimportant for the disk temperature of our TDE models. Thermal
Comptonization is also included.

We tailor the disk initial conditions to be consistent with realistic TDE
scenarios following the setup described in Dai et al. (2018). The accretion
disk is initialized with Keplerian velocity profile with a rest-mass density
which is Gaussian in angle with a height-to-radius ratio of $H/R\approx 0.3$,
so that the initial density profile is given by

$\rho(r,z)=\rho_{0}r^{-1.3}e^{-z^{2}/(2H^{2})},$ (1)

where $H$ is the scale height, $\rho_{0}$ is the initial reference density.
The disks are checked to have total masses and specific angular momentum
consistent to the first order with the conditions in TDEs. Adjusting
$\rho_{0}$ leads to different accretion rates after the disk reaches the
quasi-equilibrium state. As $\dot{M}_{\rm fb,peak}$ can reach $\approx
100\dot{M}_{\rm Edd}$ in a TDE around a $10^{6}M_{\odot}$ black hole, and a
large fraction of the debris mass is expected to escape in outflows, we set
the aimed accretion rates to be a few to a few tens of $\dot{M}_{\rm Edd}$
(Table 1).

A large-scale poloidal magnetic field is initially seeded. As adopted in
McKinney et al. (2015), for $r$ smaller than a breaking radius
$R_{b}=500R_{g}$, the $\phi$ component of the vector potential is given by
$A_{\phi}=MAX(r\ 10^{40}-0.02,0)(\sin\theta)^{5}$. For $r\geq R_{b}$, the
field transitions to a split monopolar field, which is given by
$A_{\phi}=MAX(R_{b}\ 10^{40}-0.02,0)(\sin\theta)^{1+4(R_{b}/r)}$. The field is
normalised so that the initial ratio of gas+radiation pressure to magnetic
pressure $\beta\equiv(p_{\rm{}gas}+p_{\rm{}rad})/p_{b}$ has a minimum value of
10.

###  2.2 Radiative transfer setup

We use the Monte Carlo radiative transfer code SEDONA (Kasen et al., 2006) to
post-process the simulated disks and calculate the escaping radiation. More
specifically, we calculate the bound electron level populations under non-
local thermal equilibrium (LTE) as in Roth et al. (2016) and the
Comptonization of electrons as in Roth & Kasen (2018). We track the free-free
interactions, as well as the bound-bound and bound-free interactions with
their opacities obtained from. the atomic database CMFGEN. The gas is assumed
to consist of only H, He, and O with solar abundances. We focus on calculating
the SED, and leave the line modeling to a future study which requires higher-
solution simulations.

For each accretion rate, we calculate the spectra for four inclination bins of
$\theta_{\rm bin}=$ 10, 30, 50 and $70^{\circ}$. In each bin, we take an
average over $\theta_{\rm bin}\pm 5^{\circ}$ for the already time-
and-$\phi-$averaged profile of the simulated disk. Since the simulated jet
density is likely numerically boosted, for the bins at $\theta_{\rm bin}=$ 10
and 30∘, we also reduce the jet density by 100 times before taking the
average. We note that the first-order behavior of the spectra does not depend
on this arbitrary choice of density re-scaling inside the jet. We set the
source to be a blackbody spectrum with a single temperature of $10^{6}$K,
which is consistent with the inner disk temperature. The source photons are
injected from an inner boundary set at the boundary between the inflow and
outflow, which is typically at a few $R_{g}$ except for the bin at
$i=70^{\circ}$ which is partly in the disk inflow region. For the latter, we
place the inner boundary at $R=20R_{g}$ instead, and set the gas velocity to
be always zero. The photons are then propagated in 3D, by assuming that the
gas density, temperature, and radial velocity profiles are spherically
symmetric. The photons propagate outwards until they leave the computational
domain set at $R=4000R_{g}$ or have become a part of the kinetic/thermal pool.

Based on the Monte Carlo calculations, we iterate the gas temperatures, gas
ionization state, bound electron states, and radiative transfer solution under
the assumption of radiative equilibrium until a steady solution has been
reached (after approximately 20 iterations). Since GRRMHD simulations show
that the luminosity of super-Eddington disks are not always capped by $L_{\rm
Edd}$, we also tweak the source photon luminosity and obtain the spectra under
two limits – the escaped bolometric luminosity is either $L_{\rm bol}=L_{\rm
Edd}\approx 10^{44}\rm\ ergs^{-1}$ or $L_{\rm bol}=0.1\ \dot{M}_{\rm
acc}c^{2}\approx(\dot{M}_{\rm acc}/\dot{M}_{\rm Edd})\ L_{\rm Edd}$.

##  3 Results

###  3.1 Properties of the accretion flow

In all three GRRMHD simulations, it takes an initial time period of $t={\rm
few}\times 1,000R_{g}/c$ for the accretion flow to get established. As the
disk evolves, magnetic fluxes accumulate near the MBH and their strength
further grow via the magnetorotational instability (MRI) (Balbus & Hawley,
1998). Here we focus on the final stage of the simulations when the accretion
flow has reached steady state and the wind has established equilibrium at most
inclination angles. The black hole parameters and some basic quantities of the
accretion disks, averaged over the period of $t=15,000-20,000R_{g}/c$, are
listed in Table 1. The disk profiles used for post-processing are also first
$\phi-$averaged and then time-averaged over the same period. More numerical
details of the simulated disks are given in Appendix A.

Table 1: Black hole and accretion disk parameters  Model | $M_{\rm{}BH}\ (M_{\odot})$ | $a$ | $\dot{M}_{\rm acc}\ (\dot{M}_{\rm{}Edd})$ | $\dot{M}_{w}\ [\dot{M}_{\rm{}Edd}]$ | $L_{\rm{}RAD}\ (L_{\rm{}Edd})$
---|---|---|---|---|---
M6a08-1 | $10^{6}$ | 0.8 | 7 | 1.4 | 5.4
M6a08-2 | $10^{6}$ | 0.8 | 12 | 4.5 | 3.3
M6a08-3 | $10^{6}$ | 0.8 | 24 | 14 | 8.1
Figure 1: The 2D vertical profiles of time-and-$\phi$-averaged gas rest-mass
density $\rho_{0}$ (upper panels, zoomed into the inner regions) and radiation
temperature $T_{\rm{}RAD}$ (lower panels, whole range of the simulation box),
for the three runs with different accretion rates (from left to right:
$\dot{M}_{\rm acc}=7\dot{M}_{\rm Edd}$, $12\dot{M}_{\rm Edd}$ and
$24\dot{M}_{\rm Edd}$) in the quasi-equilibrium state. In the upper panels, we
show the contours of constant lab-frame radial velocity ($v_{r}\equiv
u^{r}/u^{t}$) with white lines, and mark the jet regions where the
electromagnetic energy is larger than the rest-mass energy of the gas with
dark blue color. In the lower panels, the black lines indicate the electron-
scattering photosphere with $\tau_{\rm es}=1$, and the red lines indicate the
effective photosphere with $\tau_{\rm eff}=1$. Larger accretion rates induce
larger disk/wind density and higher gas/radiation temperature, while the gas
distribution and velocity structure remain rather robust against the variance
in accretion rates. The sizes of photospheres generally increase as accretion
rate increases.

The averaged accretion rates onto the event horizon of the MBH in these three
simulations are $\dot{M}_{\rm acc}\approx$ 7, 12 and 24 $\dot{M}_{\rm Edd}$.
To first order, these super-Eddington accretion flows have similar structures
as those described in Dai et al. (2018). As shown in the upper panels of Fig.
1, geometrically and optically thick disks are formed with half-angular
thickness $H/R\approx 0.20-0.25$, with, as expected, higher gas density
towards the equatorial plane. The rotation profile of the resultant thick
disks is sub-Keplerian. Wide-angle winds are launched by the large radiation
and magnetic pressure, which are also optically thick at most angles except
close to the polar axis. There are roughly two components of the wind: an
ultra-fast outflow (UFO) with speeds faster than $0.1c$ within $\lesssim
45^{\circ}$ from the pole, and a slower but denser wind outside $\gtrsim
45^{\circ}$ inclination. At larger accretion rates, the disk and wind
densities increase, and the winds become slower as a result of mass-loading.
However, the wind geometry stays fairly similar.

The emission properties of the accretion flow should be examined near the
photosphere. The optical depth for an opacity $\kappa$ is calculated using
$\tau(r)=\int\rho\kappa\text{d}l$ along the radial direction, $r$, from the
outer boundary $R_{\rm{}out}=8500R_{g}$ towards the event horizon. (Here
relativistic effects are included so that: $\text{d}l=-f_{\gamma}\text{d}r$,
with $f_{\gamma}\approx u^{t}(1-v/c)$ and $v/c\approx 1-1/(u^{t})^{2}$). The
electron scattering photosphere is defined by $\tau_{\rm es}(r)=1$ with
$\kappa=\kappa_{\rm{}es}\approx 0.2(1+X)~{}{\rm cm^{2}~{}g^{-1}}$. The
effective photosphere is then defined by $\tau_{\rm eff}(r)=1$ with
$\kappa=\kappa_{\rm{}eff}=\sqrt{3\kappa_{\rm{}ff}(\kappa_{\rm{}ff}+\kappa_{\rm{}es})}$,
where we have only considered free-free opacity $\kappa_{\rm{}ff}\approx
3.82\times 10^{22}Z(1+X)(1-Z)\rho T_{g}^{-3.5}$ in the scattering-dominated
gas, where $T_{g}$ is the gas temperature. Both photospheres are plotted over
the disk profiles in the lower panels of Fig. 1. As expected, we see the
presence of an optically thin “funnel” near the rotation axis where the wind
density is correspondingly lower. As the accretion rate increases, the wind
becomes more opaque, which reduces the angular size of the funnel.

The effective photospheres reside mostly within $r\sim 5000R_{g}$ except along
the equitorial direction for the disk with the largest $\dot{M}_{\rm acc}$.
Therefore, we select to evaluate various physical quantities at $r=5500R_{g}$.
This gives averaged wind mass rates of $\dot{M}_{w}=$ 1.4, 4.5 and 14
$\dot{M}_{\rm Edd}$, and averaged bolometric luminosities at $L_{\rm RAD}$=
5.4, 3.3 and 8.1 $L_{\rm Edd}$, respectively, for the three simulations. The
radiation temperature of the accretion flow is also plotted, which varies from
$\approx 10^{6}$ K near the black hole to $\approx 10^{4}$ K in the outer part
of the disk and wind. The radiation flux varies with the inclination as shown
in Dai et al. (2018) and leaks out through the region of least resistance,
which is the funnel.

Magnetic fluxes are dragged by the accretion flow and accumulate near the MBH.
This brings the inner regions to swiftly become a magnetically arrested
accretion disk (MAD) (Narayan et al., 2003). Relativistic jets are launched
magnetically through the Blandford-Znajek mechanism (Blandford & Znajek, 1977)
in all simulations. In this Letter we focus on calculating the emission
properties from the disk, and leave the analysis of the jets for future study.

###  3.2 Spectra from post-processing

In this Section we investigate how the emission reprocessing depends on a few
key parameters, namely, the viewing angle, the accretion rate, and the
luminosity. The dependence on viewing angle has been previously studied by Dai
et al. (2018). They show that for a fixed accretion rate, there is a clear
trend of the spectral energy distribution (SED) moving towards larger
wavelengths with increasing inclination angle. At low inclinations, the gas
density is lower, so X-rays produced from the inner disk easily escape. At
relatively large inclinations, on the other hand, the optically thick wind and
outer disk serve as an effective reprocessing envelope. More specifically, in
the fast wind region, the photons lose energy as they go through multiple
scatterings in the expanding outflow before escaping. In the disk and the slow
wind region, the reprocessing is mainly caused by the absorption of X-ray
photons and the thermal Comptonization of electrons.

Figure 2: The simulated escaping spectra of the accretion disk at different
accretion rates ($\dot{M}_{\rm acc}=$ 7, 12, and 24 $\dot{M}_{\rm Edd}$) and
inclinations ($i=10^{\circ},30^{\circ},50^{\circ},70^{\circ}$). The bolometric
luminosity of the spectra $L_{\rm bol}=L_{\rm Edd}$ for all spectra. The
purple shaded region indicates the X-ray band with an energy above 0.3 keV.
The orange shaded region corresponds to Swift UVOT band at 1700-6500 Å. Panel
(a)-(c) show the spectral evolution with inclination angles at fixed accretion
rate. All spectra change from X-ray strong to UV/optical strong as the
inclination goes from the polar direction to the disk direction. Panel (d) is
the same as panel (b), but only includes the spectrum at $i=10^{\circ}$ with a
blackbody spectrum fitting its X-ray continuum component, and the spectrum at
$70^{\circ}$ with another blackbody spectrum fitting its UV/optical continuum
component. Panel (e)-(h) show the spectral evolution with accretion rates at
fixed inclinations. Three types of evolution can happen as accretion rate
decreases: X-ray strong all the time (small inclination), optical/UV strong at
early time and X-ray rebrightening at late time (intermediate inclination),
optical/UV strong all the time (large inclination).

We re-examine the emission viewing-angle dependence in the three new simulated
disks. Fig. 2, 2 and 2 show how the escaped spectrum varies with inclination,
when $\dot{M}_{\rm acc}=7,12,{\rm and}\ 24\ \dot{M}_{\rm Edd}$ respectively.
At any of these accretion rates, the SED still evolves from X-ray to optically
dominated as the inclination increases. However, the exact angle at which this
transition occurs depends on the accretion rate. One can clearly observe that
at low accretion rates $\approx{\rm few}\times\dot{M}_{\rm Edd}$, the escaped
emission is dominated by X-ray emission at most inclinations unless the
inclination angle is substantial (e.g., $i\gtrsim 70^{\circ}$ for
$\dot{M}_{\rm acc}=7\ \dot{M}_{\rm Edd}$), while at very high accretion rates
$\approx{\rm few}\times 10\ \dot{M}_{\rm Edd}$, the emanating radiation is
mainly optically dominated except near the polar region (e.g., $i\lesssim
10^{\circ}$ for $\dot{M}_{\rm acc}=24\ \dot{M}_{\rm Edd}$).

We expect a TDE to be observed along a fix viewing angle during its entire
evolution, unless the disk experiences some axial precession. Therefore, we
show the spectral evolution with $\dot{M}_{\rm acc}$ at fixed inclination
angles in Fig. 2–2. As the accretion rate declines after peak, the amount of
reprocessing material is reduced, so the SED universally shifts towards lower
wavelengths. However, the exact behavior of the spectrum depends sensitively
on the inclination angle. At very small inclinations ($i=10^{\circ}$), the TDE
stays X-ray dominated throughout its evolution. At intermediate inclinations
($i=30^{\circ}$ and $50^{\circ}$), the TDE can be optically strong at early
times (if $\dot{M}_{\rm acc}$ can be sufficiently large), and then rebrightens
at X-ray energies at late time when the accretion rate diminishes. At very
large inclinations ($i=70^{\circ}$), the TDE stays UV/optically strong
throughout its entire evolution.

While we have assumed that the bolometric luminosities of the escaped
radiation always close to $L_{\rm Edd}$ in the analysis above, GRRMHD
simulations show that the true escaped luminosity from super-Eddington disks
can exceed the Eddington limit, with more flux leaking out through the funnel
(e.g., Sa̧dowski & Narayan, 2016). Therefore, we also calculated the spectra
when the escaped radiation has a bolometric luminosity $L_{\rm bol}=10\%\
\dot{M}_{\rm acc}c^{2}$, which are shown in Fig. B.1. When the reprocessing is
driven by adiabatic expansion, the SED shape stays unchanged while the
magnitude of the flux scales with the luminosity. When the reprocessing
mechanism is driven by absorption and Comptonization, increasing the
luminosity makes the gas more ionized and reduces bound-free and free-free
absorption, which shifts the spectral energy peak to slightly higher energies.
However, the spectral shape is rather insensitive to the luminosity. In
general, the setting of the luminosity within the explored range does not
alter how the escaped radiation depends on the viewing angle and accretion
rate.

###  3.3 Comparison with observations: blackbody luminosity, effective
temperature, and photosphere radius

In this section, we compare our model predictions to the observed properties
of TDEs. We start from the TDE catalog in Gezari (2021) which also lists the
observed blackbody luminosities and temperatures, and then only include TDEs
which have their masses estimated from the $M-\sigma$ relation as in Wong et
al. (2022). This gives us 7 X-ray-selected TDEs (Table C.1) and 16 optically-
selected TDEs (Table C.2). We plot their observed blackbody luminosities,
effective temperatures and photosphere radii as functions of $M_{\rm BH}$ in
Fig. 3 \- 3. The observed $L_{\rm BB}$ varies between $10^{-3}-10\ L_{\rm
Edd}$, and can exceed $L_{\rm Edd}$ when $M_{\rm BH}\lesssim 10^{6}M_{\odot}$.
Interestingly, $L_{\rm BB}$ commonly has a dependence on $M_{\rm BH}$
following the fallback rate trend: $\dot{M}_{\rm fb}/\dot{M}_{\rm Edd}\propto
M^{-3/2}_{\rm BH}$. The observed $T_{\rm BB}$ clearly depends on whether the
TDE is optically-selected ($\approx{\rm few}\times 10^{4}$ K) or X-ray-
selected ($\approx 10^{5}-10^{6}$ K). The observed $R_{\rm BB}$, calculated
from $L_{\rm BB}$ and $T_{\rm BB}$ using the Stefan-Boltzmann law, also has a
bimodal distribution. The optically-selected TDEs have $R_{\rm BB}$ reaching
$10^{2}-10^{4}R_{g}$ and always exceeding both the circularization radius
$R_{\rm circ}=2R_{t}$ and the stream self-intersection radius $R_{\rm int}$
(Dai et al., 2015). The X-ray-selected TDEs, however, sometimes have $R_{\rm
BB}$ even within the event horizon, which might be caused by absorption in
circumnuclear medium.

Figure 3: Comparison between the observed and modeled TDE blackbody
luminosity, temperature and radius. Upper row (a)-(c): The observed quantities
vs. $M_{\rm BH}$ for 16 optically-selected TDEs (orange dots) and 7 X-ray-
selected TDEs (purple triangles). (a) The observed $L_{\rm BB}/L_{\rm Edd}$
has a clear trend with $M_{\rm BH}$. A gray line showing $L=1\%\ \dot{M}_{\rm
fb}c^{2}$, with $\dot{M}_{\rm fb}$ being the fallback rate of a $0.1M_{\odot}$
star is overplotted to guide the eye. (b) X-ray TDEs have temperatures of
$10^{5}-10^{6}$K, and optical TDEs typically have lower temperature at
few$\times 10^{4}$K. (c) Optical TDEs have $R_{\rm BB}$ larger than the
circularization radius (red curve) or the stream self-intersection radius
(green curve) (both calculated using a $0.1M_{\odot}$ star). X-ray TDEs can
sometimes have $R_{\rm BB}$ smaller than the black hole Schwarzschild radius
(black line). Middle row (d)-(f) and lower row (g)-(i): The inferred
quantities based on the blackbody radiation spectrum fitting the simulated
spectra in the UV/Optical band or X-ray band respectively vs. inclination
angle $i$. Different symbols are used to mark different accretion rates:
$7\dot{M}_{\rm Edd}$ (blue circle), $12\dot{M}_{\rm Edd}$ (green triangle) and
$24\dot{M}_{\rm Edd}$ (red square). Vertical lines connect the values
calculated with an escaped luminosity of $L_{\rm bol}=L_{\rm Edd}$ (smaller
symbol size) and $L_{\rm bol}=10\%\dot{M}_{\rm acc}c^{2}$ (larger symbol size)
to indicate possible ranges. In (d)-(f), TDEs with $L_{\rm O,BB}<L_{\rm
X,0.3-10keV}$ are masked with low opacity to indicate they are less likely to
be selected optically. Similarly, in (g)-(i), TDEs with $L_{\rm O,BB}>L_{\rm
X,0.3-10keV}$ are masked with low opacity to indicate they are less likely to
be selected by X-ray instruments. The blackbody luminosity, temperature and
radius inferred from our modeling, to the first order, reproduce the observed
ones.

As the observed properties of TDEs are usually inferred from their spectra in
monochromatic bands, we also fit the simulated spectra in either the X-ray or
UV/optical band with blackbody spectra. For example, in Fig. 2, we show the
simulated escaped spectra for $\dot{M}_{\rm acc}=12\dot{M}_{\rm Edd}$ at
$i=10^{\circ}$ and $70^{\circ}$ and the blackbody radiation fits to the
spectra in the X-ray band (0.3-10 keV) or the part of the spectrum in the
UV/optical band (the 170-650 nm Swift UVOT band). It can be seen that the
X-ray and UV/optical emission can be individually well approximated by
blackbody radiation. Similarly, for each spectrum (at $\dot{M}_{\rm acc}=$ 7,
12, or 24 $\dot{M}_{\rm Edd}$ and inclination
$i=10^{\circ},30^{\circ},50^{\circ},\rm or\ 70^{\circ}$), we obtain two
blackbody radiation fits in the X-ray band and the UV/optical band. The
escaped bolometric luminosity is assumed to be varying between $L_{\rm Edd}$
and $0.1\%\ \dot{M}_{\rm acc}c^{2}$. The luminosities, temperatures and radii
of the best-fit blackbody radiation spectra are are listed in Table D.1 and
plotted in Fig. 3 \- 3 (UV/optical fit) and Fig. 3 \- 3 (X-ray fit). We
further categorize whether a modeled TDE spectrum is a X-ray strong or
optically strong by comparing the luminosity in the X-ray band (0.3-10 keV)
and the blackbody luminosity inferred from the UV/optical band. We compare the
model predictions to the observed properties as below:

1\. Luminosity: The modeled $L_{\rm BB}$ mostly lies between 0.01-few $L_{\rm
Edd}$. The simulated spectrum usually peaks in EUV and has a broader shape
than a single-temperature blackbody spectrum. Therefore, the inferred
blackbody luminosity ($L_{\rm O,BB}$ or $L_{\rm X,BB}$) is always smaller than
the bolometric luminosity $L_{\rm bol}$ of the escaped radiation. $L_{\rm
O,BB}/L_{\rm bol}$ for optically-strong TDEs and $L_{\rm X,BB}/L_{\rm bol}$
for X-ray-strong TDEs is typically few$\times(1-10)\%$. This naturally
explains the missing energy problem in TDE (Stone & Metzger, 2016; Lu & Kumar,
2018; Mockler & Ramirez-Ruiz, 2021) – the majority of the energy is emitted in
the EUV band which is difficult to be detected.

2\. Temperature: Our modeling reproduces the bimodal distribution of TDE
temperatures, i.e., optically-strong TDEs have temperature $T_{\rm
O,BB}\approx{\rm few}\times 10^{4}K$, and X-ray-strong TDEs have temperature
$T_{\rm X,BB}\lesssim 10^{6}K$. Furthermore, our modeling shows that the
inferred temperatures of optical TDEs are not highly sensitive to accretion
rates or observer inclination, which explains why TDEs have relatively
constant $T_{\rm BB}$ throughout the evolution (van Velzen et al., 2020).

3\. Photosphere radius: The optically-strong TDEs have far larger photosphere
radii ($R_{\rm O,BB}\approx 10^{2}-10^{4}R_{g}$) than X-ray-strong TDEs
($R_{\rm X,BB}\approx{\rm few}\times R_{g}$). A comparison between the
observed $R_{\rm O,BB}=10^{3}-10^{4}R_{g}$ and the modeled $R_{\rm O,BB}$
suggests that optically-selected TDEs are either commonly observed from large
inclinations, or have $L_{\rm bol}>L_{\rm Edd}$. The distinction between the
observed and modeled $R_{\rm X,BB}$ is possibly accounted by the absorption of
X-rays in the circumnuclear medium.

We note that our predictions for the X-ray quantities, in particularly in the
$i=10^{\circ}$ bin, are sensitive to the setting of the radiative transfer
calculations. Here we always inject a blackbody spectrum with a constant
$T=10^{6}K$, so our predicted $T_{\rm X,BB}$ at small inclinations also fall
into a very narrow range. However, the temperatures at the center of accretion
disks generally increase with increasing $\dot{M}_{\rm acc}$ and decreasing
$M_{\rm BH}$, which will induce more variance to the observed X-ray
temperatures. Also, for the setting with $L=L_{\rm Edd}$, the predicted
$L_{\rm X,BB}$ decreases as $\dot{M}_{\rm acc}$ increases, as a result of a
constant bolometric luminosity and more reprocessing from X-ray to UV/optical
emissions at higher densities. However, simulations show that the radiation
fluxes leaking out through the funnel are not Eddington-limited (McKinney et
al., 2015; Dai et al., 2018). Therefore, we expect that X-ray luminosities
should scale positively with accretion rates, as illustrated by the $L=0.1\
\dot{M}_{\rm acc}c^{2}$ case.

###  3.4 Temporal evolution of TDEs

We show in Fig. 4 the evolution of the modeled luminosity, temperature,
radius, as well as the ratio of optical to X-ray luminosity, as functions of
the accretion rate. In order to connect the snapshots at an specific accretion
rate to the temporal evolution of TDEs, we assume $\dot{M}_{\rm
fb}(t)=\dot{M}_{\rm acc}(t)+\dot{M}_{\rm wind}(t)$. This assumption is valid
only if the fallback timescale dominates over other timescales, such as the
disk viscous timescale and the photon diffusion/advection timescales. The
exact conversion from $\dot{M}_{\rm fb}$ to $t$ depends on the mass of the
MBH, the properties of the disrupted star, and the impact parameter (Law-Smith
et al., 2020). Focusing on the post-peak evolution, the three disk simulations
correspond to 45.9, 102.4 and 174 days after the peak of the flare, assuming a
solar-type star is fully disrupted by a $10^{6}M_{\odot}$ black hole.
Calculations of various timescales are given in Appendix E.

Figure 4: The post-peak temporal evolution of the modeled TDE UV/optical
luminosity (a), temperature (b), radius (c), and the ratio between the
UV/optical and X-ray luminosity (d). Different colors denote different
inclination angles. The escaped radiation has luminosity $L_{\rm bol}=L_{\rm
Edd}$ for all curves. The lower axis shows the accretion rate and the upper
axis shows the corresponding time elapsed since the peak, assuming a solar-
type star disrupted by a $10^{6}M_{\odot}$ black hole. In (a)-(c), we do not
include the evolution $i=10^{\circ}$, where the event is always X-ray strong.
In (a) the gray line shows the trend of $t^{-5/3}$ to guide the eye. In (d)
the X-ray luminosity includes only the flux in the 0.3-2 keV band for direct
comparison with observations. Also at $i=70^{\circ}$ the X-ray luminosities at
the two higher accretion rates are negligible.

As the accretion rate drops from $24\dot{M}_{\rm Edd}$ to $7\dot{M}_{\rm
Edd}$, the optical luminosity also decreases, and the UV/optical light curve
roughly follows the canonical $t^{-5/3}$ decay. As discussed in the previous
section, the fitted blackbody temperature stays rather constant.
Interestingly, at large inclinations $T_{\rm O,BB}$ slightly decreases with
declining $\dot{M}_{\rm acc}$, while at small to intermediate inclinations
$T_{\rm O,BB}$ shows the opposite trend. This can provide an explanation to
the different observed evolution of $T_{\rm O,BB}$ (van Velzen et al., 2020).
As a result, the photosphere radius shrinks as $\dot{M}_{\rm acc}$ decreases,
with a faster evolution at smaller inclinations.

The ratio between UV/optical luminosity ($L_{\rm O,BB}$) and X-ray luminosity
($L_{\rm X,0.3-2keV}$) also decreases as the accretion level drops and the
amount of obscuring material is reduced. The fastest $L_{\rm O,BB}/L_{\rm
X,0.3-2keV}$ evolution is observed at intermediate inclinations, and
especially if the TDE has a high accretion rate at peak. In such cases, we
expect to see strong X-ray re-brightening of initially optically-strong TDEs,
as reported in Gezari et al. (2017). The X-ray luminosity reaches the same
level as the UV/optical luminosity at $t\gtrsim 100$ days after peak, and
possibly even later if the disk formation or viscous timescale is long. At
very small inclinations, the event is always X-ray strong. At very large
inclinations, the event should stay optically strong for a long period,
although it is theoretically predicted that the disk should eventually become
geometrically thin and emit mostly in X-rays/UV when the accretion level drops
to around Eddington (Shakura & Sunyaev, 1973).

##  4 Summary and future work

Inspired by the unified model for TDEs proposed by in Dai et al. (2018), we
carry out three addtional 3D GRRMHD simulations of TDE super-Eddington
accretion flow at different Eddington ratios, and conduct radiative transfer
calculations to obtain the emanating spectra. Based on the results, we further
propose a dynamical unified model which can explain the diversity and
evolution of TDEs:

  * •

The viewing angle of the observer with respect to the orientation of the disk
is the most important parameter in determining whether we observe either an
X-ray or an optical bright TDE. At small inclinations, X-rays can escape from
the funnel of the super-Eddington disk. At large inclinations, X-rays are
mostly reprocessed into UV/optical radiation by the geometrically and
optically thick wind and disk.

  * •

The blackbody radiation fits of the TDE super-Eddington disk spectra produce
effective temperature, blackbody luminosity and photosphere radius
distributions consistent with the observed ones. Most radiative energy escapes
in the EUV range, and only a few to a few tens of percentage of radiative
energy can be detected, which provides a solution to the TDE missing energy
problem.

  * •

The observed diversity of the emission from different TDEs can be associated
with the different Eddington ratios of their accretion rates, $\dot{M}_{\rm
acc}={\rm few}\times(1-10)\ \dot{M}_{\rm Edd}$, at the flare peak conditions.
In general, higher accretion levels induce larger (fitted blackbody)
luminosities and larger photosphere radii, but do not significantly change the
fitted effective temperatures.

  * •

The early-time evolution ($t\approx 100$ days after peak) of optical TDEs can
be explained by this reprocessing model. As the luminosity drops by about
0.5-1 order of magnitude, the fitted temperature slightly increases at small
to intermediate inclinations or decreases at large inclinations.

  * •

The evolution of the optical-to-X-ray flux ratio also depends sensitively on
the viewing angle. At large inclinations, the TDE stays UV/optically strong
for a very long time. At intermediate inclinations, we expect to see the
fastest X-ray rebrightening, and $L_{O}/L_{X}$ reaches unity at few hundred
days. At small inclinations, the TDE is always X-ray strong. The exact
evolution timescale also depends on the accretion rate at peak, which further
depends on the black hole and stellar parameters.

In this study we focus on understanding the disk continuum emission. We will
investigate the disk spectroscopic features in a future work. Also, while our
GRRMHD simulations are conducted in 3D, we only do post-processing along
different inclination bins by assuming the profiles are spherically symmetric,
which means that photons have an overall radial motion. Implementing 2D or 3D
radiative transfer calculations can allow us to study the reprocessing of
photons along more realistic paths. Furthermore, the evolution of the disk and
jet emissions can be studied in more details by conducting simulations
covering a more complete parameter space for black hole accretion rate and
spin.

The unified model we have proposed for TDEs also sheds light on other types of
black hole accretion systems, especially other super-Eddington sources such as
ultra-luminous X-ray sources, narrow line Seyfert 1 galaxies, changing-look
AGN and quasars in the early universe. Works like this illustrate that novel
simulations are crucial for making direct comparisons between the observed
emissions and model predictions, which offers important insights to how black
holes grow through cosmic time, use accretion as a source of energy to power
outflows and radiation, and give feedback to their surrounding environment.

We thank K. Auchettl, M. Bulla, S. Gezari, D. Kasen, G. Leloudas, B. Mockler,
and N. Stone for useful discussions. LT, TK and LD acknowledge the support
from the Hong Kong Research Grants Council (HKU27305119, HKU17304821) and the
National Natural Science Foundation of China (HKU12122309). ER-R is grateful
for support from the Heising-Simons Foundation, NSF (AST-1615881, AST-1911206
and AST-1852393), Swift (80NSSC21K1409, 80NSSC19K1391) and Chandra
(GO9-20122X). The simulations carried out for this project were performed on
the HPC computing facilities offered by ITS at HKU and the Tianhe-2
supercluster.

## References

  * Andalman et al. (2022) Andalman, Z. L., Liska, M. T. P., Tchekhovskoy, A., Coughlin, E. R., & Stone, N. 2022, MNRAS, 510, 1627
  * Auchettl et al. (2017) Auchettl, K., Guillochon, J., & Ramirez-Ruiz, E. 2017, ApJ, 838, 149
  * Balbus & Hawley (1998) Balbus, S. A., & Hawley, J. F. 1998, Reviews of Modern Physics, 70, 1
  * Blandford & Znajek (1977) Blandford, R. D., & Znajek, R. L. 1977, MNRAS, 179, 433
  * Bloom et al. (2011) Bloom, J. S., Giannios, D., Metzger, B. D., et al. 2011, Science, 333, 203
  * Bonnerot et al. (2021) Bonnerot, C., Lu, W., & Hopkins, P. F. 2021, MNRAS, 504, 4885
  * Burrows et al. (2011) Burrows, D. N., Kennea, J. A., Ghisellini, G., et al. 2011, Nature, 476, 421
  * Cenko et al. (2012) Cenko, S. B., Krimm, H. A., Horesh, A., et al. 2012, ApJ, 753, 77
  * Charalampopoulos et al. (2022) Charalampopoulos, P., Leloudas, G., Malesani, D. B., et al. 2022, A&A, 659, A34
  * Coughlin & Begelman (2014) Coughlin, E. R., & Begelman, M. C. 2014, ApJ, 781, 82
  * Curd & Narayan (2019) Curd, B., & Narayan, R. 2019, MNRAS, 483, 565
  * Dai et al. (2021) Dai, J. L., Lodato, G., & Cheng, R. 2021, Space Sci. Rev., 217, 12
  * Dai et al. (2015) Dai, L., McKinney, J. C., & Miller, M. C. 2015, ApJ, 812, L39
  * Dai et al. (2018) Dai, L., McKinney, J. C., Roth, N., Ramirez-Ruiz, E., & Miller, M. C. 2018, ApJ, 859, L20
  * De Colle et al. (2012) De Colle, F., Guillochon, J., Naiman, J., & Ramirez-Ruiz, E. 2012, ApJ, 760, 103
  * Evans & Kochanek (1989) Evans, C. R., & Kochanek, C. S. 1989, ApJ, 346, L13
  * French et al. (2020) French, K. D., Wevers, T., Law-Smith, J., Graur, O., & Zabludoff, A. I. 2020, Space Sci. Rev., 216, 32
  * Gezari (2021) Gezari, S. 2021, ARA&A, 59, arXiv:2104.14580 [astro-ph.HE]
  * Gezari et al. (2017) Gezari, S., Cenko, S. B., & Arcavi, I. 2017, ApJ, 851, L47
  * Guillochon et al. (2014) Guillochon, J., Manukian, H., & Ramirez-Ruiz, E. 2014, ApJ, 783, 23
  * Guillochon & Ramirez-Ruiz (2013) Guillochon, J., & Ramirez-Ruiz, E. 2013, ApJ, 767, 25
  * Hammerstein et al. (2022) Hammerstein, E., van Velzen, S., Gezari, S., et al. 2022, arXiv e-prints, arXiv:2203.01461
  * Hayasaki & Jonker (2021) Hayasaki, K., & Jonker, P. G. 2021, ApJ, 921, 20
  * Hinkle et al. (2021) Hinkle, J. T., Holoien, T. W. S., Auchettl, K., et al. 2021, MNRAS, 500, 1673
  * Holoien et al. (2018) Holoien, T. W. S., Brown, J. S., Auchettl, K., et al. 2018, MNRAS, 480, 5689
  * Jiang et al. (2014) Jiang, Y.-F., Stone, J. M., & Davis, S. W. 2014, ApJ, 796, 106
  * Jiang et al. (2019) —. 2019, ApJ, 880, 67
  * Kasen et al. (2006) Kasen, D., Thomas, R. C., & Nugent, P. 2006, ApJ, 651, 366
  * Law-Smith et al. (2020) Law-Smith, J. A. P., Coulter, D. A., Guillochon, J., Mockler, B., & Ramirez-Ruiz, E. 2020, ApJ, 905, 141
  * Leloudas et al. (2019) Leloudas, G., Dai, L., Arcavi, I., et al. 2019, ApJ, 887, 218
  * Liu et al. (2022) Liu, X.-L., Dou, L.-M., Chen, J.-H., & Shen, R.-F. 2022, ApJ, 925, 67
  * Lodato & Rossi (2011) Lodato, G., & Rossi, E. M. 2011, MNRAS, 410, 359
  * Loeb & Ulmer (1997) Loeb, A., & Ulmer, A. 1997, ApJ, 489, 573
  * Lu & Kumar (2018) Lu, W., & Kumar, P. 2018, ApJ, 865, 128
  * McKinney et al. (2015) McKinney, J. C., Dai, L., & Avara, M. J. 2015, MNRAS, 454, L6
  * McKinney et al. (2012) McKinney, J. C., Tchekhovskoy, A., & Blandford, R. D. 2012, MNRAS, 423, 3083
  * McKinney et al. (2014) McKinney, J. C., Tchekhovskoy, A., Sadowski, A., & Narayan, R. 2014, MNRAS, 441, 3177
  * Metzger & Stone (2016) Metzger, B. D., & Stone, N. C. 2016, MNRAS, 461, 948
  * Mockler et al. (2019) Mockler, B., Guillochon, J., & Ramirez-Ruiz, E. 2019, ApJ, 872, 151
  * Mockler & Ramirez-Ruiz (2021) Mockler, B., & Ramirez-Ruiz, E. 2021, ApJ, 906, 101
  * Narayan et al. (2003) Narayan, R., Igumenshchev, I. V., & Abramowicz, M. A. 2003, PASJ, 55, L69
  * Novikov & Thorne (1973) Novikov, I. D., & Thorne, K. S. 1973, in Black Holes (Les Astres Occlus), ed. C. Dewitt & B. S. Dewitt, 343
  * Pfister et al. (2020) Pfister, H., Volonteri, M., Dai, J. L., & Colpi, M. 2020, MNRAS, 497, 2276
  * Piran et al. (2015) Piran, T., Svirski, G., Krolik, J., Cheng, R. M., & Shiokawa, H. 2015, ApJ, 806, 164
  * Ramirez-Ruiz & Rosswog (2009) Ramirez-Ruiz, E., & Rosswog, S. 2009, ApJ, 697, L77
  * Rees (1988) Rees, M. J. 1988, Nature, 333, 523
  * Rossi et al. (2021) Rossi, E. M., Stone, N. C., Law-Smith, J. A. P., et al. 2021, Space Sci. Rev., 217, 40
  * Roth & Kasen (2018) Roth, N., & Kasen, D. 2018, ApJ, 855, 54
  * Roth et al. (2016) Roth, N., Kasen, D., Guillochon, J., & Ramirez-Ruiz, E. 2016, ApJ, 827, 3
  * Roth et al. (2020) Roth, N., Rossi, E. M., Krolik, J., et al. 2020, Space Sci. Rev., 216, 114
  * Saxton et al. (2021) Saxton, R., Komossa, S., Auchettl, K., & Jonker, P. G. 2021, Space Sci. Rev., 217, 18
  * Sazonov et al. (2021) Sazonov, S., Gilfanov, M., Medvedev, P., et al. 2021, MNRAS, 508, 3820
  * Sa̧dowski & Narayan (2016) Sa̧dowski, A., & Narayan, R. 2016, MNRAS, 456, 3929
  * Shakura & Sunyaev (1973) Shakura, N. I., & Sunyaev, R. A. 1973, A&A, 24, 337
  * Stone & Metzger (2016) Stone, N. C., & Metzger, B. D. 2016, MNRAS, 455, 859
  * Stone et al. (2020) Stone, N. C., Vasiliev, E., Kesden, M., et al. 2020, Space Sci. Rev., 216, 35
  * Strubbe & Quataert (2009) Strubbe, L. E., & Quataert, E. 2009, MNRAS, 400, 2070
  * Tchekhovskoy et al. (2011) Tchekhovskoy, A., Narayan, R., & McKinney, J. C. 2011, MNRAS, 418, L79
  * Ulmer (1999) Ulmer, A. 1999, ApJ, 514, 180
  * van Velzen et al. (2020) van Velzen, S., Holoien, T. W. S., Onori, F., Hung, T., & Arcavi, I. 2020, Space Sci. Rev., 216, 124
  * van Velzen et al. (2019) van Velzen, S., Stone, N. C., Metzger, B. D., et al. 2019, ApJ, 878, 82
  * Wevers et al. (2019) Wevers, T., Pasham, D. R., van Velzen, S., et al. 2019, MNRAS, 488, 4816
  * Wong et al. (2022) Wong, T. H. T., Pfister, H., & Dai, L. 2022, ApJ, 927, L19

##  Appendix A Properties of simulated disks

A few more time-averaged quantities of the three simulated disks are listed in
Table A.1. $\Phi_{H}$ is the normalised magnetic flux at horizon $\Phi_{H}$
(Tchekhovskoy et al., 2011), with $\Phi_{H}\gtrsim 30-40$ being the condition
for MAD disks. The net accretion efficiency $\eta_{H}$ evaluates how much
rest-mass energy going into the black hole is converted to out-going energy
near the event horizon. $L_{\rm{}jet}$ is the total power of the relativistic
jet at $r=5500r_{g}$, most of which is in the form of electromagnetic energy.
$\Gamma_{\rm{}jet,max}$ is the maximum Lorentz factor of the jet. $L_{K}$ is
the thermal+kinetic luminosity of the wind calculated at $r\sim 5500r_{g}$.
$\alpha_{\rm{}eff}$ is the effective $\alpha$-parameter of the disk as defined
in McKinney et al. (2012). $R_{\rm eq}$ indicates the radius within which the
disk inflow equilibrium has been established. In our simulations, the winds
are launched mostly from the inner disk regions and have traveled beyond the
photospheres at most inclination angles.

Table A.1: More disk quantities  Model | $\dot{M}_{\rm acc}\ (\dot{M}_{\rm{}Edd})$ | $\Phi_{H}$ | $\eta_{H}\ (\%)$ | $L_{\rm{}jet}\ (L_{\rm{}Edd})$ | $\Gamma_{\rm jet,max}$ | $L_{K}\ (L_{\rm Edd})$ | $\alpha_{\rm eff}$ | $R_{\rm eq}\ (R_{g})$
---|---|---|---|---|---|---|---|---
M6a08-1 | 7 | 34.5 | 35.5 | 1.1 | 2.27 | 0.5 | 2.7 | 340
M6a08-2 | 12 | 39 | 40.1 | 2.3 | 2.39 | 1.0 | 3.3 | 320
M6a08-3 | 24 | 55 | 69.3 | 9.7 | 2.66 | 4.8 | 2.3 | 230

##  Appendix B Simulated spectra at higher luminosity

Figure B.1: The simulated escaping spectra of the accretion disk at different
accretion rates and inclinations, similar to Fig. 2-2, except that the
bolometric luminosity of the spectra $L_{\rm esc}=10\%\times\dot{M}_{\rm
acc}c^{2}$.

##  Appendix C List of observed TDEs reported in literature

For plotting Fig. 3-3 we use 16 optically-selected TDEs and 7 X-ray-selected
TDEs. For completeness we list their names and relevant parameters as reported
from pervious literature in Table C.1 and C.2.

Table C.1: Names and parameters of optical TDEs Name | $\log\ M_{\rm BH}\ (M_{\odot})$ | $\log\ L_{\rm BB}\ (\rm erg\ s^{-1})$ | $\log\ T_{\rm BB}\ (\rm K)$ | log $R_{\rm BB}$ ($R_{g}$)
---|---|---|---|---
ASASSN-14ae | 5.42 | 43.87 | 4.29 | 4.34
ASASSN-14li | 6.31 | 43.66 | 4.52 | 2.88
ASASSN-15lh | 8.47 | 45.34 | 4.30 | 2.00
ASASSN-15oi | 5.71 | 44.45 | 4.60 | 3.72
AT2018dyb/ASASSN-18pg | 6.67 | 44.08 | 4.40 | 2.97
AT2018hyz/ASASSN-18zj | 5.68 | 44.10 | 4.25 | 4.27
GALEXD1-9 | 6.51 | 43.48 | 4.59 | 2.45
GALEXD23H-1 | 6.39 | 43.95 | 4.70 | 2.59
GALEXD3-13 | 7.36 | 43.98 | 4.66 | 1.71
iPTF-15af | 6.88 | 44.10 | 4.85 | 1.87
iPTF-16axa | 6.34 | 43.82 | 4.46 | 3.05
PS1-10jh | 5.85 | 44.47 | 4.59 | 3.61
PTF-09axc | 5.68 | 43.46 | 4.08 | 4.29
PTF-09djl | 5.82 | 44.42 | 4.41 | 3.97
PTF-09ge | 6.34 | 44.04 | 4.08 | 3.92
SDSS-TDE1 | 7.24 | 43.64 | 4.42 | 2.14
Table C.2: Names and parameters of X-ray TDEs Name | $\log\ M_{\rm BH}\ (M_{\odot})$ | $\log\ L_{\rm BB}\ (\rm erg\ s^{-1})$ | $\log\ T_{\rm BB}\ (\rm K)$ | log $R_{\rm BB}$ ($R_{g}$)
---|---|---|---|---
SDSSJ1323+48 | 6.15 | 44.30 | 5.91 | 0.80
SDSSJ1201+30 | 7.18 | 45.00 | 6.06 | -0.29
RXJ1624+75 | 7.68 | 43.38 | 6.05 | -1.57
RXJ1420+53 | 7.33 | 43.38 | 5.67 | 0.03
RBS1032 | 5.25 | 41.70 | 6.11 | -0.13
3XMMJ1521+07 | 5.61 | 43.51 | 6.30 | -0.02
3XMMJ1500+01 | 5.64 | 43.08 | 6.06 | 0.29

##  Appendix D Modeled TDE Parameters

The values of the modeled observable as in Fig. 3-3 are listed in Table D.1.

Table D.1: Modeled TDE observables $\dot{M}_{\rm acc}$ |  $L_{\rm bol}$ | inc | log $L_{\rm O,BB}$ | $L_{\rm O,BB}/L_{\rm bol}$ | log $T_{\rm O,BB}$ | log $R_{\rm O,BB}$ | log $L_{\rm X,BB}$ | $L_{\rm X,BB}/L_{\rm bol}$ | log $T_{\rm X,BB}$ | log $R_{\rm X,BB}$
---|---|---|---|---|---|---|---|---|---|---
$(\dot{M}_{\rm Edd})$ |  ($L_{\rm Edd}$) | (∘) | (erg $\rm s^{-1}$) |  | (K) | (cm) | (erg $\rm s^{-1}$) |  | (K) | (cm)
7 |  1 | 10 | 40.650 | 0.000 | 4.47 | 12.71 | 43.952 | 0.710 | 5.99 | 11.38
30 | 42.212 | 0.013 | 4.89 | 12.64 | 43.742 | 0.438 | 5.91 | 11.45
50 | 42.480 | 0.024 | 4.67 | 13.48 | 42.475 | 0.024 | 5.72 | 11.85
70 | 43.179 | 0.120 | 4.30 | 14.55 | 38.879 | 0.000 | 5.25 | 13.11
7 | 10 | 42.861 | 0.008 | 5.13 | 12.50 | 44.799 | 0.713 | 5.99 | 11.80
30 | 43.434 | 0.031 | 5.04 | 12.95 | 44.587 | 0.438 | 5.91 | 11.87
50 | 43.691 | 0.056 | 4.85 | 13.72 | 43.407 | 0.029 | 5.74 | 12.29
70 | 43.175 | 0.017 | 4.30 | 14.55 | 41.422 | 0.000 | 5.35 | 13.55
12 |  1 | 10 | 40.989 | 0.001 | 4.55 | 12.73 | 43.914 | 0.651 | 5.98 | 11.38
30 | 42.503 | 0.025 | 4.74 | 13.09 | 43.334 | 0.171 | 5.78 | 11.61
50 | 42.684 | 0.038 | 4.54 | 13.84 | 41.451 | 0.002 | 5.57 | 12.05
70 | 43.508 | 0.255 | 4.38 | 14.57 | - | - | - | -
12 | 10 | 43.364 | 0.015 | 5.13 | 12.75 | 44.994 | 0.652 | 5.99 | 11.91
30 | 43.921 | 0.055 | 4.89 | 13.50 | 44.413 | 0.171 | 5.78 | 12.15
50 | 44.130 | 0.089 | 4.78 | 14.08 | 42.973 | 0.006 | 5.65 | 12.40
70 | 43.558 | 0.024 | 4.34 | 14.66 | - | - | - | -
24 |  1 | 10 | 41.407 | 0.002 | 4.44 | 13.15 | 43.786 | 0.483 | 5.98 | 11.30
30 | 42.994 | 0.078 | 4.53 | 13.75 | 41.452 | 0.002 | 5.40 | 12.17
50 | 43.193 | 0.123 | 4.48 | 14.20 | - | - | - | -
70 | 43.784 | 0.482 | 4.41 | 14.64 | - | - | - | -
24 | 10 | 43.427 | 0.009 | 4.83 | 13.37 | 45.167 | 0.485 | 5.99 | 11.97
30 | 44.790 | 0.204 | 4.77 | 14.18 | 42.977 | 0.003 | 5.43 | 12.80
50 | 44.401 | 0.083 | 4.76 | 14.26 | 43.026 | 0.004 | 5.50 | 13.22
70 | 44.444 | 0.092 | 4.67 | 14.46 | - | - | - | -

##  Appendix E Characteristic Timescales

There are various physical timescales relevant fo TDEs: debris mass fallback
timescale, disk formation/circularization timescale, disk viscous timescale,
and photon transport timescale. The longest timescale of all governs the
temporal evolution of TDE emissions.

The photon transport time through optically thick medium is the shorter one
between the diffusion timescale and advection timescale. The photon diffusion
timescale is calculated as $t_{\rm diff}=\tau_{\rm es}\ R_{\rm es}/c$, where
$R_{\rm es}$ is the size of the electron-scattering photosphere along a
particular inclination, and $\tau_{\rm es}$ is the electron-scattering optical
depth integrated radially from $r=0$ to $r=R_{\rm es}$. The advection
timescales is calculated as $t_{\rm adv}\approx R_{\rm es}/v_{r}$, since the
photons trapped by the optically-thick gas has a similar speed as the gas.
Here the gas radial velocity $v_{r}$ is averaged over the radial path within
$R_{\rm es}$ and weighted by gas density. The values for the $\dot{M}_{\rm
acc}=12\dot{M}_{\rm Edd}$ disk are given in Table E.1. One can see that for
the inclinations considered in this work, photons are preferably advected out
by the optically-thick wind. The photon transport time varies from $\sim$0.1
day to a few days depending on the inclination.

The disk viscous timescale can be analytically calculated by $t_{\rm
visc}=t_{\rm dyn}\ \alpha^{-1}(H/R)^{-2}$, where $t_{\rm dyn}$ is the orbital
timescale of the disk, $\alpha$ is a free parameter between 0 and
approximately 1, and $H/R$ is the disk thickness (Shakura & Sunyaev, 1973).
For our disk parameters, we have $t_{\rm visc}\approx 5.44{\rm\ days}\ (R_{\rm
disk}/8500r_{g})\ (M_{\rm
BH}/10^{6}M_{\odot})(\alpha/1)^{-1}((H/R)/0.3)^{-2}$. The viscous time is
therefore only a few days. We caution the readers that our simulated disks are
MAD which typically have effective $\alpha\gtrsim 1$. The viscous time is
potentially longer if the disks do not have such large magnetic fluxes.

The disk formation/circularization timescale induces uncertainty into the
evolution of TDEs. Recent simulations show that a large fraction of the debris
materials can form a disk within dynamical timescale but the disk still
pocesses some moderate eccentricity (Bonnerot et al., 2021). As the topic is
out of the scope of this paper, we assume here that the disk forms quickly for
the calculation of the emission evolution.

Stellar debris typically falls back on timescales of $t_{\rm mb}$, which is
the orbital time of the most bound debris (Evans & Kochanek, 1989; Guillochon
& Ramirez-Ruiz, 2013; Rossi et al., 2021):

$t_{\rm mb}\approx 41\ {\rm days}\ \Big{(}\frac{M_{\rm
BH}}{10^{6}M_{\odot}}\Big{)}^{1/2}\Big{(}\frac{m_{\star}}{M_{\odot}}\Big{)}^{-1}\Big{(}\frac{r_{\star}}{R_{\odot}}\Big{)}^{3/2}.$
(E1)

For our case with $M_{\rm BH}=10^{6}M_{\odot}$, $t_{\rm mb}$ varies between 25
days and 41 days for $m_{\star}=(0.1-10)\ M_{\odot}$.

Lastly, assuming that the fallback timescale governs the evolution of TDEs, we
calculate the time corresponding to each simulated disk with a further
assumption that the fallback rate $\dot{M}_{\rm fb}$ equals the instataneous
accretion rate $\dot{M}_{\rm acc}$ plus the wind mass rate $\dot{M}_{w}$. Here
we focus on the post-peak evolution. By setting that the fallback rate peaks
at $t_{\rm mb}$, the post-peak time $t_{\rm pp}$ associated with a particular
$\dot{M}_{\rm fb}$ can be calculated with:

$\dot{M}_{\rm fb}\approx
133\Big{(}\frac{M_{BH}}{10^{6}M_{\odot}}\Big{)}^{-3/2}\Big{(}\frac{m_{\star}}{M_{\odot}}\Big{)}^{2}\Big{(}\frac{r_{\star}}{R_{\odot}}\Big{)}^{-3/2}\Big{(}\frac{t}{t_{\rm
mb}}\Big{)}^{-5/3}\dot{M}_{\rm Edd}.$ (E2)

With the wind mass rate given in Table 1, the three simulated disks correspond
to $\dot{M}_{\rm fb}=\dot{M}_{\rm acc}+\dot{M}_{w}=(7+1.4,12+4.5,24+14)\
\dot{M}_{\rm Edd}$. The post-peak time obtained with $M_{\rm
BH}=10^{6}M_{\odot}$ and a few different $m_{\star}$ are shown in Table E.2.

Table E.1: Photon diffusion and advection timescales for the simulated disk at $\dot{M}_{\rm acc}=12\dot{M}_{\rm Edd}$ inclination (∘) | 10 | 30 | 50 | 70
---|---|---|---|---
$R_{\rm es}\ (R_{g})$ | 896 | 1278 | 5425 | 8377
$\tau_{\rm es}$ | 7.0 | 74.0 | 137.2 | 710.2
$v_{r}\ (c)$ | 0.50 | 0.35 | 0.13 | 0.013
$t_{\rm adv}$ (day) | 0.07 | 0.13 | 1.06 | 13.6
$t_{\rm diff}$ (day) | 0.35 | 5.4 | 42.2 | 337.4
Table E.2: Time elapsed since peak corresponding to the three simulated TDE disks, assuming $M_{\rm BH}=10^{6}M_{\odot}$ and various different $m_{\star}$ $m_{\star}$ ($M_{\odot}$) | 0.2 | 0.5 | 1 | 1.5
---|---|---|---|---
$\dot{M}_{\rm fb,peak}\ (\dot{M}_{\rm Edd})$ | 36.7 | 76.4 | 133.0 | 211.6
$t_{\rm pp}(\dot{M}_{\rm fb}=(24+14)\ \dot{M}_{\rm Edd})$ (day) | NA | 18.6 | 45.9 | 69.7
$t_{\rm pp}(\dot{M}_{\rm fb}=(12+4.5)\ \dot{M}_{\rm Edd})$ (day) | 18.3 | 53.8 | 102.4 | 140.0
$t_{\rm pp}(\dot{M}_{\rm fb}=(7+1.4)\ \dot{M}_{\rm Edd})$ (day) | 42.3 | 98.5 | 174.0 | 229.2


# Will we run out of data? An analysis of the limits of scaling datasets in
Machine Learning

Pablo Villalobos1, Jaime Sevilla12, Lennart Heim14, Tamay Besiroglu13, Marius
Hobbhahn 15, Anson Ho1

###### Abstract

We analyze the growth of dataset sizes used in machine learning for natural
language processing and computer vision, and extrapolate these using two
methods; using the historical growth rate and estimating the compute-optimal
dataset size for future predicted compute budgets. We investigate the growth
in data usage by estimating the total stock of unlabeled data available on the
internet over the coming decades. Our analysis indicates that the stock of
high-quality language data will be exhausted soon; likely before 2026. By
contrast, the stock of low-quality language data and image data will be
exhausted only much later; between 2030 and 2050 (for low-quality language)
and between 2030 and 2060 (for images). Our work suggests that the current
trend of ever-growing ML models that rely on enormous datasets might slow down
if data efficiency is not drastically improved or new sources of data become
available. ††1Epoch, 2University of Aberdeen, 3MIT Computer Science &
Artificial Intelligence Laboratory, 4Centre for the Governance of AI,
5University of Tübingen

## Key Takeaways

  * •

We project the growth of training datasets for vision and language models
using both the historical growth rate and the compute-optimal dataset size
given current scaling laws and existing compute availability estimates
(Section III-A).

  * •

We also project the growth in the total stock of unlabeled data, including
high-quality language data (Section III-B).

  * •

Language datasets have grown exponentially by more than 50% per year, and
contain up to 2e12 words as of October 2022. (section IV-A)

  * •

The stock of language data currently grows by 7% yearly, but our model
predicts a slowdown to 1% by 2100. This stock is currently between 7e13 and
7e16 words, which is 1.5 to 4.5 orders of magnitude larger than the largest
datasets used today (Section IV-B1).

  * •

Based on these trends, we will likely run out of language data between 2030
and 2050 (Section IV-D).

  * •

However, language models are usually trained on high-quality data. The stock
of high-quality language data is between 4.6e12 and 1.7e13 words, which is
less than one order of magnitude larger than the largest datasets (Section
IV-B2).

  * •

We are within one order of magnitude of exhausting high-quality data, and this
will likely happen between 2023 and 2027 (Section IV-D).

  * •

Projecting the future growth of image datasets is less obvious than for
language, because the historical trend stopped in the past four years111New
models appeared which use much more data than what was the case in the
previous years, see [1].. However, the growth rate seems likely to be around
18% to 31% per year. The current largest dataset is 3e9 images (Section IV-A).

  * •

The stock of vision data currently grows by 8% yearly, but will eventually
slow down to 1% by 2100. It is currently between 8.11e12 and 2.3e13 images –
three to four orders of magnitude larger than the largest datasets used today
(Section IV-C).

  * •

Projecting these trends highlights that we will likely run out of vision data
between 2030 to 2070 (Section IV-D).

(a) Projections for low-quality
language data

(b) Projections for high-quality
language data

(c) Projections for vision data

Figure 1: Projections of data usage. Each graph shows two extrapolations of
data usage, one from past trends and one from compute availability estimations
plus scaling laws. Both projections are constrained to be lower than the
estimated data stock. In all three cases, this constraint causes a slowdown in
data usage growth.

##  I Introduction

Training data is one of the three main factors that determine the performance
of Machine Learning (ML) models, together with algorithms and compute. Current
understanding of scaling laws suggests that future ML capabilities will
strongly depend on the availability of large amounts of data to train large
models [2, 3].

Previous work compiled a database of more than 200 training datasets used in
ML models [1] and estimated historical rates of growth in dataset size for
vision and language models.

To learn about the limits of this trend, we developed probabilistic models to
estimate the total amount of image and language data that will be available
between 2022 and 2100. Based on our dataset size trend projections, we then
estimated the limit of these trends due to the exhaustion of available data.

##  II Previous work

Stock of data: There have been several estimates of the size of the internet
and the total amount of information available [4, 5, 6]. However, in recent
years, these types of reports have not provided breakdowns of different data
modalities (for example into the number of images, videos, or blog posts), and
instead aggregated all data modalities into a single number in bytes [7].

Data bottleneck in ML: In [8], the author estimated the stock of high-quality
data and used the scaling laws [3] to predict that the stock of data is not
enough to scale language models more than 1.6x the size of DeepMind’s
Chinchilla language model [3] using compute-optimal scaling. We improve this
analysis by creating explicit models of dataset size growth and more detailed
estimations of the stock of data over time, which allows us to predict the
date that datasets will become as large as the total stock of data.

##  III Methods

###  III-A Projecting growth in training dataset sizes

Previous work compiled historical trends of dataset size work for different
application domains222The domains that were included were vision, language,
recommendation, speech, drawing, and games. However, there is only significant
data for vision and language. [1].

Our definition of _dataset size_ is the number of unique datapoints on which
the model is trained. The definition of "datapoint" is different for each
domain. In particular, for language data we define a datapoint as a word, and
for image data we define a datapoint as an image. Additional details on this
choice of dataset size metric can be found in [1].

Using the historical trend, together with the size of the largest datasets
used to date, we can estimate the future evolution of dataset sizes. However,
this projection naively assumes that the past trend will be sustained
indefinitely. In reality, there are constraints on the amount of data that a
model can be trained on. One of the most important constraints is compute
availability. This is because increasing the amount of training data for a
given model requires additional compute, and the amount of compute that can be
used is limited by the supply of hardware and the cost of buying or renting
that hardware.

To account for this constraint, we make another projection, based on compute
availability and the compute-optimal dataset size. Scaling laws can be used to
predict the optimal balance of model size and dataset size for a given compute
budget (measured in FLOP) [2, 3]. Concretely, the optimal dataset size is
proportional to the square root of the compute budget ($D~{}\sqrt{C}$).

Previous work [9], projected the available compute to the largest training
into the future333Note that this projection has a wide range of uncertainty
and includes scenarios in which spending on compute grows orders of magnitude
over current levels, up to 1% of GDP.. We use those projections to estimate
the optimal training dataset size that will be achievable in each future year.

###  III-B Estimating data accumulation rates

In recent years, unsupervised learning has successfully created foundation
models that can be fine-tuned for several tasks using small amounts of labeled
data and large amounts of unlabeled data. In addition, unsupervised models
have also proved able to generate valuable pseudo-labels for unlabeled data
[10]. For these reasons, we will focus on the stock and accumulation rates of
unlabeled data, even if the amount of labeled data is much lower444Note that
while transfer learning vastly reduces the need for labeled data, it does not
eliminate it. In addition, labeled data is usually much harder to acquire than
unlabeled data. Therefore, labeled data might turn out to be a bottleneck even
though it is required in smaller quantities..

Before delving into the details, let us consider a theoretical framework of
what we expect the data accumulation rate to look like. The vast majority of
data is user-generated and is stored in social media platforms, blogs, forums,
etc. There are three factors that determine how much content is produced in a
given period: human population, internet penetration rate, and the average
amount of data produced by each internet user.

Human population has been extensively studied so we use the standard United
Nations projections [11]. Internet penetration (the percentage of the
population who uses the Internet) grows as an S-curve from 0% in 1990 to 50%
in 2018 to over 60% today [12]. We model this as a sigmoid function of time
and fit it to the data in [12].

The average amount of data produced by users changes over geography and time
according to internet usage trends, and is not easy to analyze555Doing so
would require taking into account the effects of culture, demography and
socioeconomic development in different countries and times, which is out of
the scope of this paper.. For simplicity, let us assume the average amount of
data produced by users is constant over time.

Figure 2: Real and projected evolution of internet users.

This model of Internet population (the number of Internet users) closely
matches the historical number of Internet users (Figure 2). To test its
ability to predict Internet data generation, we conducted an empirical test by
fitting this model to Reddit submission data. This model fit the data better
than both exponential and sigmoid models (see Appendix C).

###  III-C High-quality data

We have developed a model for the accumulation rate of user-generated content.
However, for language data, this kind of content tends to be of lower quality
than more specialized language data like books or scientific papers. Models
trained on the latter kind of data perform better [13], so it is common
practice to use it for training language models [14, 15, 3]. We have little
insight on data quality for image models nor how to identify high-quality
image data666Other than very crude metrics like image resolution. For example,
comparing robustness across distributional shifts of image-text models trained
on different commonly-used datasets shows that there is no single dataset that
induces better robustness across all shifts [16], so we will focus on language
in this section.

Because of our limited insights and unawareness of research into the tradeoffs
involved in using high- versus low-quality data, we provide estimates and
growth projections for both high- and low-quality data separately. To identify
high-quality date we defer to the expertise of practitioners and look at the
composition of the datasets used to train large language models. The most
common sources in these datasets are books, news articles, scientific papers,
Wikipedia, and filtered web content777Filtered web content is regular web
content selected using a proxy measure of quality, like the number of upvotes
of a link shared in Reddit. The MassiveWeb and WebText datasets were built in
this way.888Other common sources are GitHub (for code), subtitles and
transcriptions of educational videos, podcasts or parliamentary sessions, and
emails..

A common property of these sources is that they contain data that has passed
usefulness or quality filters. For example, in the case of news, scientific
articles, or open-source code projects, the usefulness filter is imposed by
professional standards (like peer review). In the case of Wikipedia, the
filter is standing the test of time in a community of dedicated editors. In
the case of filtered web content, the filter is receiving positive engagement
from many users. While imperfect, this property can help us identify
additional sources of high-quality data, so we will use it as our working
definition of high-quality data.

Some high-quality data, such as filtered web content and Wikipedia, is
generated by dedicated internet contributors. This means we can use the same
model developed for general user-generated content.

However, other sources of high-quality data are generated by subject matter
experts (such as scientists, authors, and open-source developers). In this
case, the generation rate is not determined by human population or internet
penetration but by the size of the economy and the share of the economy
devoted to creative sectors (like science and art).

OECD countries have spent roughly 2% of their GDP on R&D over the past 20
years [17]. This number is increasing slowly, but we will assume it is mostly
constant. So the data accumulation rate should be roughly proportional to the
size of the world economy, which grows around 4% each year. This prediction is
consistent with the observed growth in scientific publications [18].

We estimate the proportion of these two classes of data (dedicated
contributors and professionals) in high-quality datasets by looking at
existing datasets and classifying each of their subcomponents into a class.

###  III-D Limitations

There are a number of reasons why our estimates of the growth rate of dataset
sizes might be incorrect:

  * •

There might be less need for data in the future to achieve equivalent levels
of performance. This is particularly likely since there have previously been
large increases in data efficiency in other domains[19, 8].

  * •

Compute availability might grow slower than expected for a number of reasons,
including technical obstacles to efficiency increases, supply chain
disruptions, or reduced willingness to spend.

  * •

Current scaling laws could be wrong, as has happened in the past999In [2] the
authors recommended increasing the training dataset size fivefold for each
tenfold increase in compute. In the more recent [3], they revisit the problem
and recommend instead increasing the training dataset size threefold for each
tenfold increase in compute.. Even if there is no additional increase in data
efficiency, perhaps there are better ways of scaling that use less data.

  * •

Multimodal models might prove to perform better than models with a single
modality via transfer learning. This would effectively increase the data stock
to encompass a combination of the stocks of all data modalities.

(a) Comparison of the different data stock models.

(b) Aggregated data stock model.

(c) Projection of historical trend of training dataset sizes and available
data stock.

Figure 3: Models of low-quality language data.

In addition, there are some limitations in our estimates of the stock of data:

  * •

The use of synthetic data could make the stock of data virtually infinite. We
are uncertain about the usefulness and cost of synthetic data for training.

  * •

Big economic shifts might significantly impact the production of data. For
example, large-scale adoption of self-driving cars would result in an
unprecedented amount of road video recordings.

  * •

Similarly, actors with big budgets (such as governments or large corporations)
might be able to increase the production of data with enough spending,
especially in the case of high-quality data for niche domains. Some
possibilities are widespread screen recording or mass surveillance.

  * •

We might find better ways to extract high-quality data from low-quality
sources, for example, by coming up with robust automatic quality metrics.

##  IV Analysis

###  IV-A Trends in dataset size

Previous work [1] identified the historical rate of growth for training
datasets in different domains. Since there is only significant data for the
language and vision domains, we will limit our analysis to those two domains.
The trends are summarized in Table I.

(a) Composition of high-quality datasets: The Pile (left), PaLM (top-right),
MassiveText (bottom-right)

(b) Data stock model.

(c) Projection of historical trend of training dataset sizes and available
data stock.

Figure 4: Models of high-quality language data. Domain |  Doubling time median and CI (months) |  Largest training dataset (datapoints)
---|---|---
Language |  15.8 |  [11.2; 20.9] |  2e12
Vision |  41.5 |  [30.4; 48.3] |  3e9
TABLE I: Trends in training dataset size for vision and language models.

###  IV-B Language

####  IV-B1 Low-quality data

We have used five different models to estimate the amount of data and the
accumulation rate. Table II summarizes these different models, which are
further illustrated in Figure 3(a) and explained in more detail in Appendix A.
The aggregated model finds an estimated current total stock between 6.9e13 and
7.1e16 words, and current growth between 6.41% and 17.49% per year.

Note that the high end of this estimate comes from two highly theoretical
models that we trust the least. The way we interpret this range is: 1e14 words
is what is very likely available to single, well-funded actors such as Google;
1e15 words is what is available to the combined group of all major actors (all
tech companies); 1e16 words is what humanity might be able to collectively
produce with a worldwide, multiyear effort, employing practices such as
recording all text messages, phone calls and video meetings, practices which
are currently very far outside the Overton window.

Using the aggregated data stock model as an upper bound for scaling datasets,
we project the size of training datasets and find that it grows rapidly until
it exhausts the stock of data. After this point, growth slows down
substantially (Figure 3(c)).

Model | Stock of data (#words) | Growth rate
---|---|---
Recorded speech | 1.46e17 | 5.2%
[3.41e16; 4.28e17] | [4.95%; 5.2%]
Internet users | 2.01e15 | 8.14%
[6.47e14; 6.28e15] | [7.89%; 8.14%]
Popular platforms | 4.41e14 | 8.14%
[1.21e14; 1.46e15] | [7.89%; 8.14%]
CommonCrawl | 9.62e13 | 16.68%
[4.45e13; 2.84e14] | [16.41%; 16.68%]
Indexed websites | 2.21e14 | NA
[5.16e13; 6.53e15]
Aggregated model | 7.41e14 | 7.15%
[6.85e13; 7.13e16] | [6.41%; 17.49%]
TABLE II: Median and 90% CI of estimates of high-quality language data
accumulation.

####  IV-B2 High-quality data

We studied high-quality data by looking at the composition of several high-
quality datasets and determining how much each component can be scaled. We
considered three datasets: The Pile [13], MassiveText [3], and the PaLM
pretraining dataset [15].

From these, we can see that high-quality datasets are usually composed of 50%
scraped user-generated content (Pile-CC, OpenWebText2, social media
conversations, filtered webpages, MassiveWeb, C4), 15-20% books, 10-20%
scientific papers, <10% code and <10% news. In addition, they all incorporate
known small very-high-quality datasets like Wikipedia (Figure 4(a)).

We estimated the amount of available text in digitized books, public GitHub
repositories, and scientific papers. Assuming all of these form between 30% to
50% of a hypothetical high-quality dataset, we could reach 9e12 [4.6e12;
1.7e13] words. We assume the amount of high-quality data grows at 4-5% per
year in line with the world economy, as explained in the Introduction (see
Figure 4(b)). Details of the model can be found in Appendix A.

Projecting the growth of language datasets using the high-quality stock
instead of the low-quality stock as an upper bound, we find the same pattern
of slowdown, with the distinction that the slowdown happens much earlier,
before 2026 (Figure 4(c)).

###  IV-C Vision

We used two different estimates for vision: one produced by Rise Above
Research [20], and one using the combined images and videos posted to the most
popular social media platforms. The aggregated model shows there are between
8.11e12 and 2.3e13 images on the internet today, with the current yearly
growth rate around 8%. The models are summarized in Table III and Figure 5(a).

Using the aggregated data stock model as an upper bound for scaling datasets,
we project the size of training datasets from both the historical trend and
the compute-optimal extrapolation. The historical projection is very uncertain
since we do not yet know if the recent high outliers indicate a new higher-
growth trend. The compute projection is also more uncertain than the
corresponding projection for language because we do not have a great
understanding of scaling laws for vision101010This is because images can have
different resolutions, so image tokenization is more variable than text
tokenization..

Similarly to the case of language, dataset sizes grow exponentially until
reaching the size of the data stock, at which point they revert to a much
slower rate of growth (Figure 5(c)).

We do not understand the impact of data quality for unlabeled vision data and
how to distinguish high-quality data, so we did not attempt to estimate it.

(a) Comparison of data stock models.

(b) Aggregated data stock model.

(c) Projection of historical trend of training dataset sizes and available
data stock.

Figure 5: Models of vision data. Model | Stock of data (#images) | Growth rate
---|---|---
Popular platforms | 9.48e12 | 8.14%
[5.02e12 ; 1.79e13] | [7.89% ; 8.14%]
External estimate | 1.28e12 | 8.14%
[6.5e11 ; 2.58e12] | [7.89% ; 8.14%]
Aggregated model | 4.36e12 | 8.14%
[8.11e12 ; 2.3e13] | [7.89% ; 8.14%]
TABLE III: Summary of estimates of image data accumulation. The bottom row
contains an aggregate of all the models.

###  IV-D Will data become a bottleneck?

So far we have found that data stocks grow much slower than training dataset
sizes (see Figures 3(c), 4(c), and 5(c)). This means that exhausting our data
stocks is inevitable if current trends continue. In addition, the high-quality
data stock is much smaller than the low-quality stock. The two dataset size
projections, based on historical trends and compute availability
extrapolations, are very similar in the first years, but later diverge.

We computed the probability that exhaustion will happen each year for each of
our projections of data stock and dataset size (Figure 6). While there is
significant uncertainty in the exhaustion dates for low-quality language and
vision stocks, it seems unlikely that it will happen before 2030 or after
2060. However, the high-quality language stock will almost surely be exhausted
before 2027 if current trends continue. The quantiles for these distributions
are shown in Table IV.

Figure 6: Distribution of exhaustion dates for each intersection of the data
availability trend and data consumptiin trend. Note that the time scale is
different for each kind of data.

[2023.55, 2024.5, 2025.75]

|  Historical projection |  Compute projection
---|---|---
Low-quality language stock |  2032.4 [2028.4 ; 2039.2] |  2040.5 [2034.6 ; 2048.9]
High-quality language stock |  2024.5 [2023.55 ; 2025.75] |  2024.1 [2023.2 ; 2025.3]
Vision stock |  2046 [2037 ; 2062.8] |  2038.8 [2032.0 ; 2049.8]
TABLE IV: Median and 90% CI of exhaustion year for each of the intersections.

##  V Discussion

Scaling laws for language models indicate that scaling is dependent on the
amount of available data [3, 8]. Under this view, around half of the
improvement in language models over the past four years comes from training
them on more data. Without further room to scale datasets, this would lead to
a slowdown in AI progress.

The accumulation rate of data for both language and vision models is much
slower than the growth in dataset size that we have observed so far, both
historically and taking compute constraints into account. As a consequence, we
might be headed for a bottleneck in training data. This would happen between
2030 and 2040 for language models and between 2030 and 2060 for image models
(Figure 6).

This is particularly true for high-quality language data, which seems likely
to be exhausted by 2027. It is unclear whether large enough datasets can
substitute for poor data quality, but even if this is the case, it would not
be enough to completely avoid the slowdown, since our ability to scale
training datasets is also limited by compute availability.

Given these projections, it might be tempting to conclude that a slowdown is
inevitable. However, we have significant reasons to believe that our models
are not adequately capturing the evolution of ML progress (see Limitations).

In particular, the future evolution of data efficiency and the impact of data
quality on performance are crucial to predict future data requirements.
Unfortunately, our understanding of these variables is insufficient to provide
detailed forecasts. Future work could try to incorporate these considerations
into the analysis.

##  VI Conclusion

We have projected the growth of training dataset sizes and data stocks. Data
stocks grow at a much slower pace than dataset sizes, so if current trends
continue, datasets will eventually stop growing due to data exhaustion. Our
models show this is likely to happen between 2030 and 2040 for language data,
and between 2030 and 2060 for vision data. In addition, high-quality language
data will be exhausted by 2026.

If our assumptions are correct, data will become the main bottleneck for
scaling ML models, and we might see a slowdown in AI progress as a result.
However, as outlined, there are multiple reasons to doubt that these trends
will continue as projected, such as the possibility of algorithmic innovations
in data efficiency.

## Acknowledgments

Thanks to Ben Cottier, Michael Aird, David Atkinson, Matthew Barnett and Keith
Wynroe for their comments on earlier drafts of this article. Also thanks to
Adam Papineau for reviewing and editing.

## References

  * Villalobos and Ho [2022] P. Villalobos and A. Ho, “Trends in training dataset sizes,” https://epochai.org/blog/trends-in-training-dataset-sizes, 2022, accessed: 2022-09-27.
  * Kaplan et al. [2020] J. Kaplan _et al._ , “Scaling laws for neural language models,” 2020.
  * Hoffmann et al. [2022] J. Hoffmann _et al._ , “Training compute-optimal large language models,” 2022\.
  * Coffman and Odlyzko [1998] K. Coffman and A. Odlyzko, “The size and growth rate of the internet,” 1998.
  * Murray H. and Moore [2000] B. Murray H. and A. Moore, “Sizing the internet,” Cyveillance, Tech. Rep., 7 2000\.
  * Lyman and Varian [2003] P. Lyman and H.R. Varian, “How much information,” 2003.
  * Reinsel et al. [2018] D. Reinsel, J. Gantz, and J. Rydning, “The digitization of the world from edge to core,” International Data Corporation, Tech. Rep., 11 2018.
  * nostalgebraist [2022] nostalgebraist, “chinchilla’s wild implications,” 2022.
  * Tamay Besiroglu and Sevilla [2022] L.H. Tamay Besiroglu and J. Sevilla, “Projecting compute trends in machine learning,” https://epochai.org/blog/projecting-compute-trends, 2022, accessed: 2022-09-27.
  * Pham et al. [2020] H. Pham _et al._ , “Meta pseudo labels,” 2020.
  * UNp [2022] “World population prospects 2022, online edition,” 2022.
  * Ritchie and Roser [2017] H. Ritchie and M. Roser, “Technology adoption,” _Our World in Data_ , 2017, https://ourworldindata.org/technology-adoption.
  * Gao et al. [2021] L. Gao _et al._ , “The pile: An 800gb dataset of diverse text for language modeling,” 2021.
  * Du et al. [2021] N. Du _et al._ , “Glam: Efficient scaling of language models with mixture-of-experts,” 2021.
  * Chowdhery et al. [2022] A. Chowdhery _et al._ , “Palm: Scaling language modeling with pathways,” 2022\.
  * Nguyen et al. [2022] T. Nguyen _et al._ , “Quality not quantity: On the interaction between dataset design and robustness of clip,” 2022.
  * OECD [2022] OECD, “Gross domestic spending on R&D (indicator),” 2022.
  * nsc [2019] “Publications output: U.s. trends and international comparisons,” 2019.
  * Ye et al. [2021] W. Ye _et al._ , “Mastering atari games with limited data,” 2021.
  * Lee [2021] E. Lee, “2021 worldwide image capture forecast: 2020 – 2025,” 2021.

###  -A Appendix A: Models of language data accumulation

In this Appendix we provide a brief overview of the models and the sources for
their parameters. The full code is available in https://github.com/epoch-
research/data-stock.

####  -A1 Total recorded human speech

Consider the amount of raw speech generated by humans. Assume each person
pronounces or writes between 5k and 20k words per day.111111This corresponds
to talking for 30 minutes and 2 hours at 150wpm, respectively. This study
(https://doi.org/10.1126/science.1139940) found an average of 16000 words per
day, with a standard deviation of 7300.  Supoosing that between 0.5% and 50%
of those words are digitally recorded121212That is, they are processed by a
digital device, be it in a phone conversation, text messages or video
meetings., then the production of words per person-year would be between 160k
and 2.6M. Multiplying this by the number of person-years (the integral of the
population size) over a given period we can get the total production of words.

####  -A2 Total text produced by internet users

As described in the introduction, the amount of text uploaded to the internet
can be modeled as a product of three factors:

  * •

Human population

  * •

Internet penetration

  * •

Amount of text generated in a year by the average internet user

Assume the third factor is constant over time and its value is between 10k and
100k words per year.131313Professional writers can write a book of 100k words
in half a year, and sending 5 text messages per day with ten words each
already produces almost 20k words. Then we can fit a sigmoid to internet
penetration data, and use existing human population projections as
approximations for the first two factors. The product of those three is the
total stock.

####  -A3 Popular platforms

In an unspecified recent year, around 500 hours of video were uploaded to
YouTube each minute.141414This is an old figure from a few years ago that
YouTube published in their blog. Can also be found on Statista. Plausibly
higher nowadays, maybe up to 800h. Assuming between 5% and 50% of those
contain speech, and 9k words per hour,151515150 words per minute for 60
minutes. we get that between 130B and 1.3T words are uploaded to YouTube per
year.

Since 2012 there have been around 500M tweets per day.161616Source:
https://www.internetlivestats.com/twitter-statistics/. Outdated, could be a
bit higher today Assuming between 10 and 50 words per tweet, Twitter produces
2-20T words each year.

Several sources estimate the number of daily published blog posts to be around
7.5M.171717This number is repeated over the internet, and it’s consistent with
a single-digit percentage of the world population posting one blog post a
week. An example source is https://www.websiterating.com/research/internet-
statistics-facts/#chapter-3. Assuming an average blog post length of 100-1000
words,181818CommonCrawl documents have around this length, this can be seen in
the ’Mean Document Size’ column on Table 1 of the The Pile paper. It is also
easy to find claims on the internet that the optimal blog post size to get
more engagement is 1000 words. in a single year between 0.2T and 2T words are
produced.

Adding all of those, and assuming they account for between 5% and 40% of the
whole internet, between 8.9T and 110T words are produced per year. Multiplying
this by the number of internet users normalized to 2022 and integrating, we
get the stock at any given year.

####  -A4 Indexed websites

The indexed web size has been roughly constant in size at tens of billions of
websites191919According to estimates from https://www.worldwidewebsize.com/.
Assuming between 500 and 50k words per website, this means the indexed web
contains between 1e13 and 1e15 words.

####  -A5 CommonCrawl

CommonCrawl releases monthly crawls which for the past four years have
contained 5-10TB of compressed plaintext.202020Example:
https://commoncrawl.org/2022/07/june-july-2022-crawl-archive-now-available/
Half of the URLs in each monthly crawl are visited for the first time, so
let’s assume around half of the plaintext is new data. Assuming a compression
rate for English between 30% and 85%, the new total yearly amount of
uncompressed text is 160TB. At 200M words per GB of text, CommonCrawl produces
between 8.3e12 and 4.4e13 words per year.

####  -A6 High-quality data

##### Code

In 2020, GitHub archived 21TB of public repositories in the Arctic Vault.
Software Heritage has archived 13B source code files. Assuming each of them is
5KB,2121215KB is the average for GitHub, this can be seen from The Pile and
MassiveText. this gives 65TB of code. So the size of GitHub in 2022 is
reasonably between 21TB and 65TB.

The size of a git repository includes all the blobs used for storing the
history of the repository. This storage overhead ranges from 100% for a one-
year-old repo with <100 commits up to 600% for a very old repository like the
Linux kernel. In the MassiveText dataset, the overhead is 3.1TB / 844GB - 1 =
267%. This means that approximately 1/4 of the size is actual source code.

It’s not clear how much code constitutes a “word,” but to be consistent with
natural language we’ll take a TB of code to be 200B “words.” Note that
tokenizers usually have worse compression rates for code than natural
text,22222250% vs 80% in the case of the Gopher tokenizer. so the size in
tokens will be larger than in “words.” In total this is between 320B and 4.2T
words.

##### Papers

In 2014, there were an estimated 114M English scientific articles on the
web.232323Source: https://journals.plos.org/plosone/article?id=10.1371/
journal.pone.0093949 The Web of Science lists 82M papers in total, with 55M
published before 2014. Assuming the coverage of Web of Science is constant
over time, the total number of papers has increased by 50% since.242424This
translates to a yearly growth of 4.5%, which matches our expectations. This
means today there are up to 114M*1.5 = 170M papers.

Average paper length is 6k words,252525From the Pile paper. Mean document
length is 30KB, so 30*200 words. so we get between 600B and 1T words.

##### Books

1M books were published yearly as of 1996 (How much information, 1996). At 4%
yearly growth, for 25 years, this is 2.6M published yearly today. But I don’t
trust this estimate. Other unreliable Internet sources claim between 500k and
4M books published per year.

It’s not clear which fraction of published books is digitized. E-book sales
are around 10% those of print books,262626Source:
https://www.cnbc.com/2019/09/19/physical-books-still-outsell-e-books-and-
heres-why.html so this seems like a reasonable guess. Another source estimated
that there were 3.4M ebooks and 48.5M print books on Amazon a few years
ago.272727Source: https://justpublishingadvice.com/how-many-kindle-ebooks-are-
there So the fraction of digitized books is probably between 2% and 20%.

Using the estimate of yearly publishing and assuming exponential economic
growth, the stock of books is the yearly published books divided by the
logarithm of the growth rate.

We also have a direct estimate of the stock in 2022: there are 12M ebooks in
Amazon kindle,282828Source: https://justpublishingadvice.com/how-many-kindle-
ebooks-are-there/#2022_Update_and_new_methodology and the Internet Archive has
20M books in its digital library.292929Source:
https://archive.org/details/texts So the number of ebooks is probably between
10M and 30M.

Taking the average of these two estimates and a length of 100k words per book,
we get between 620B and 1.8T words.

##### Total

Assuming books, papers and code represent between 30% and 50% of high-quality
data (as found in current datasets) and exponential growth of 4%, we get the
estimate of the stock of high-quality data.

Figure 7: Monthly user submissions to Reddit, in linear scale(up) and log
scale(down). While the three functions appear to fit the data well in the log
scale, the linear plot shows that the sigm*exp function predicts much better
the recent years.

### Appendix B: Models of vision data accumulation

In this Appendix we provide a brief overview of the models and the sources for
their parameters. The full code is available in https://github.com/epoch-
research/data-stock.

####  -A7 Popular platforms

If we count one second of YouTube video as a single image and assume 10% to
50% of hours contain usable images (not a still background, etc), then the
number of images uploaded to YouTube in 2022 is between 120B and
620B.303030Using the figure of 500h of video uploaded per minute found in the
popular platforms model for language data.

In 2015, the number of pictures shared daily on Instagram + Snapchat +
WhatsApp + Facebook was around 3.2B.313131Page 90 of this report:
https://www.kleinerperkins.com/perspectives/2016-internet-trends-report/.
Extrapolating that to the current year, the number is probably between 5B and
20B. A single image might be shared on average between 5 and 15 times, so
dividing by that we get between 170B and 1T images per year published on
social media.

Taking the sum of those two estimates and assuming the yearly production grows
in proportion to the internet population, we get the estimate of the total
stock.

####  -A8 External estimate

This estimate was produced by Rise Apart Research. The number of images in
2022 is between 5e10 and 2e11.323232Numbers found here:
https://photutorial.com/photos-statistics/, primary source here:
https://riseaboveresearch.com/rar-reports/2021-worldwide-image-capture-
forecast-2020-2025/ (paywalled commercial research) We extrapolate this stock
by assuming it is proportional to the number of person-years on the internet.

###  -B Appendix C: Test of the theoretical growth model

We check our theoretical model of data accumulation rates developed in the
Methods section on Reddit submission data. The combined sigmoid times
exponential function fits the monthly submission history better than either
sigmoid or exponential functions alone (Figure 7).