∎

11institutetext: M.S. Turner 22institutetext: Kavli Institute for Cosmological
Physics
The University of Chicago
5640 S. Ellis Avenue
Chicago, IL 60637-1433
USA 22email<EMAIL_ADDRESS>
#  $\rm\Lambda$CDM

Much more than we expected, but now less than what we want

Michael S. TURNER

(Proceedings of the Lemaitre Workshop on Black Holes, Gravitational Waves and
Space Time Singularities, published in Foundations of Physics (2018))

###### Abstract

The $\rm\Lambda$CDM cosmological model is remarkable: with just 6 parameters
it describes the evolution of the Universe from a very early time when all
structures were quantum fluctuations on subatomic scales to the present, and
it is consistent with a wealth of high-precision data, both laboratory
measurements and astronomical observations. However, the foundation of
$\rm\Lambda$CDM involves physics beyond the standard model of particle
physics: particle dark matter, dark energy and cosmic inflation. Until this
‘new physics’ is clarified, $\rm\Lambda$CDM is at best incomplete and at worst
a phenomenological construct that accommodates the data. I discuss the path
forward, which involves both discovery and disruption, some grand challenges
and finally the limits of scientific cosmology.

##  1 The ${\rm\Lambda}$CDM paradigm

###  1.1 Some history

For me, cosmology began in the late 1970s. Then, the Cosmic Microwave
Background (CMB) was well established but only the dipole anisotropy had been
measured. The redshifts of a few thousand galaxies had been determined, and a
high redshift galaxy was $z\sim 0.3$ and the QSO record holder was $z\sim
3.7$. CCD cameras were just entering the astronomical scene, $H_{0}$ was
either 50 or 100 km/s/Mpc, each with tiny claimed error bar, and the best bet
value for $\Omega_{0}$ was $\sim 0.1$. On the theoretical side, big-bang
nucleosynthesis (BBN) was well established and stood as the earliest
cosmological outpost at about $10^{-2}\,$sec. Gravitational instability was
the default model for structure formation, but there was no evidence for the
seed density perturbations or any compelling ideas about their origin. There
were two schools of thought on structure formation: Zel’dovich’s adiabatic
(now called curvature) school that favored the scale-invariant Harrison
spectrum of primeval fluctuations with structure forming ‘top-down’ and
Peebles’ isothermal (now called isocurvature) school with structure forming
‘bottom-up.’

While cosmology is a young science, it dates back further than the 1970s! On
the theoretical side, the starting point is Einstein’s General Relativity
(1915): without it, it is not possible to sensibly discuss the Universe and
interpret the observations. Hubble’s two big contributions followed:
establishing the existence of other galaxies in 1925 and establishing the
relationship between redshifts and distances in 1929. While the current
interpretation of the Hubble diagram as the expansion of space and redshifting
of light seems so straightforward, it took a good ten years to arrive at such
a consensus and involved much confusion.111I think it is fair to say that it
took almost 100 years to fully understand General Relativity. Years ago,
Chandrasekhar captured the enormity of the task in a quip to an experimental
colleague, “you see Bruce, Einstein’s understanding of General Relativity was
very limited,” which evoked a loud laugh. The work of Lemaitre, deSitter and
others was crucial to establishing our current understanding of the expansion,
and Kragh has discussed Lemaitre’s seminal contributions at this meeting
Kraghe.

Shortly before his death in 1953, Hubble summarized the state of observational
cosmology Hubble1953 . Between the Lick program led by Mayall and the Mt.
Wilson/Palomar program led by Humason, just over 800 galaxy redshifts had been
measured Humason1956 . For field galaxies, redshifts were $z\leq 0.04$, with a
few cluster galaxies at $z\sim 0.2$. The best estimate for the Hubble constant
was 180 km/s/Mpc. Shortly thereafter, in two influential papers
SandagePhysToday ; Sandage200inch , Allan Sandage characterized cosmology as
the quest for two numbers222Most younger cosmologists today would be hard
pressed to define $q_{0}$, and in fact, it is not measurable Neben ! – $H_{0}$
and $q_{0}$ – and argued that they would be determined with the 200-inch Hale
telescope at Mt. Palomar. The discovery of the CMB in 1965 and its
interpretation as evidence for a hot beginning changed everything: Before
galaxies, there was a radiation-dominated phase where the light elements were
synthesized (from $10^{-2}\,$sec to $200\,$sec) and the oldest light
originated (last scattering of the CMB at $380,000\,$yrs and $z\simeq 1100$).

This brings us to the late 1970s, but not to $\rm\Lambda$CDM. The big change
in the 1980s was the realization of the profound connections between quarks
and the cosmos. The discovery that the fundamental constituents are point-like
quarks and leptons with perturbatively-weak interactions opened up the very
early Universe, times earlier than $10^{-6}\,$sec, for exploration
TheEarlyUniverse . Speculations about the earliest moments, motivated by ideas
about unification of the particles and forces, changed cosmology forever and
led to the ideas underpinning $\rm\Lambda$CDM and modern cosmology.

The first shift was to view the large photon (or entropy) to baryon ratio
(around 2 billion) as a small net baryon number to photon ratio that arose
dynamically due to $B$, $C$ and $CP$ violating processes that occurred out of
thermal equilibrium (baryogenesis). Next, the dark matter problem of cosmology
became an opportunity for particle physics: stable particles (beyond baryons)
left over from the quark soup beginning account for the bulk of the mass
density of the Universe.

Inflation, a period of accelerated expansion caused by a slowly-evolving
scalar field initially displaced from the minimal of its scalar potential, was
proposed in 1981 to explain the smoothness of the Universe, the absence of
magnetic monopoles and other unwanted relics as well as explaining the small
density perturbations needed to seed the formation of structure in the
Universe Guth .

The final piece of current paradigm, dark energy, was added most recently. It
was proposed in 1984, as a smooth, highly-elastic form of energy that would
not interfere with the formation of structure and would account for 70% of the
critical density. It was needed because because dark matter was coming up
short of the critical density necessary for the flat Universe predicted by
inflation TSK . By virtue of its highly elastic nature – equation-of-state
$w\simeq-1$ – dark energy has repulsive gravity and leads to accelerated
expansion. That accelerated expansion was discovered in 1998, as discussed by
Filippenko Filippenko  and Suntzeff Suntzeff .

The deep connections between the quarks and the cosmos go far beyond the fact
that the early Universe was comprised of quark soup, and they have led to a
paradigm shift in cosmology and the coming together of the study of the very
big (cosmology) and the very small (particle physics). That profound change of
view is embodied both in $\rm\Lambda$CDM and the big questions we are asking
about the Universe today that go far beyond the simple kinematics of $H_{0}$
and $q_{0}$ (see §4).

The three major features of $\rm\Lambda$CDM have testable consequences that
have driven observational cosmology since the 1980s: 1. Inflation predicts a
flat (critical density) Universe, with almost scale-invariant, Gaussian
density perturbations and gravitational waves; 2. Cold dark matter (slowly-
moving, dark-matter particles), together with the inflationary density
perturbations, leads to a very detailed picture of structure formation – the
cold dark matter paradigm (CDM); and 3. $\rm\Lambda$ as the dark energy has a
host of consequences and can be readily falsified.

Figure 1: CMB anisotropy as measured by Planck (upper) and polarization as
measured by Planck, SPT and ACTPol (lower); in both panels, the curve is the
best-fit $\rm\Lambda$CDM model.

After three decades of intense testing $\rm\Lambda$CDM is consistent with an
enormous body of diverse, high-quality observational data, from measurements
of large-scale structure to the anisotropy of the CMB, from light-element
abundances to supernovae determinations of the expansion rate, as well as a
host of laboratory measurements. $\rm\Lambda$CDM is characterized by just six
numbers Planck2013 :

  1. 1.

Baryon density: $\omega_{B}\equiv\Omega_{B}h^{2}$

  2. 2.

Matter density: $\omega_{M}\equiv\Omega_{M}h^{2}$

  3. 3.

Spectral index density perturbations: $n_{S}$

  4. 4.

Amplitude of density perturbations: $A_{S}$

  5. 5.

Optical depth to the CMB due to re-ionization: $\tau$

  6. 6.

Sound horizon at recombination: $\theta_{MC}$

The excellent fit to the CMB anisotropy data is shown in Fig. 1. Fitted values
for these parameters, which include $\omega_{B}=0.0222\pm 0.00023$,
$\omega_{M}=0.142\pm 0.0022$, $\tau=0.078\pm 0.019$ and $n_{S}=0.965\pm
0.006$, indicate the percent-level precision that is now being achieved. From
these (and other cosmological data) one can derive the following set of
cosmological parameters that describe our Universe:

$\displaystyle T_{0}$ $\displaystyle=$ $\displaystyle 2.7255\pm 0.0006\,{\rm
K}$ (1) $\displaystyle t_{0}$ $\displaystyle=$ $\displaystyle 13.80\pm
0.02\,{\rm Gyr}$ (2) $\displaystyle\Omega_{0}$ $\displaystyle=$ $\displaystyle
1.00\pm 0.005$ (3) $\displaystyle\Omega_{M}$ $\displaystyle=$ $\displaystyle
0.316\pm 0.01$ (4) $\displaystyle\Omega_{B}$ $\displaystyle=$ $\displaystyle
0.048\pm 0.001$ (5) $\displaystyle\Omega_{\Lambda}$ $\displaystyle=$
$\displaystyle 0.685\pm 0.01$ (6) $\displaystyle H_{0}$ $\displaystyle=$
$\displaystyle 67.3\pm 0.7\,{\rm km/s/Mpc}$ (7)

Beyond its impressive precision $\rm\Lambda$CDM describes in detail the
Universe from quantum fluctuations to quark soup to neutrons and protons and
then nuclei, from the recombination of ions and electrons and photon last
scattering to the growth of structure and the formation of astrophysical
objects that can be studied today with telescopes – stars, galaxies, clusters
of galaxies, voids, walls and superclusters.

In particular, $\rm\Lambda$CDM provides the background space-time and initial
conditions for what I call ‘astrophysical cosmology,’ namely, the story of the
Universe from lumpy, ionized gas to the formation of the stars, galaxies and
large-scale structure. With the remarkable computing power that is being
brought to bear and the powerful array of instrumentation and telescopes that
allow us to probe the Universe back to redshift $z\sim 10$ and higher,
progress in reconstructing the history of the Universe from the first stars
until today is equally impressive. For me, this success is best summarized by
the Madau plot of the star formation history of the Universe; see Fig. 2.

In 1978 I would never have imagined that we would find ourselves having
progressed so far in our knowledge about and understanding of the Universe.
While I cannot speak for Lemaitre, Hubble, Einstein and Sandage, I suspect
that they would be similarly surprised and awed. $\rm\Lambda$CDM is a
remarkable achievement, one that has raised our expectations even higher.

Figure 2: Madau plot of the star formation rate MadauDickinson .

##  2 The missing physics: discovery opportunities or surprises ahead?

So why isn’t a model that agrees with a wealth of observational data, takes us
back to within a tiny, tiny fraction of a second of the beginning, and
explains the Universe that we see today not enough? Just three words: dark-
matter, dark-energy and inflation. They are the three pillars of
$\rm\Lambda$CDM, and each involves physics beyond what we know from laboratory
experiments – ‘physics beyond the standard model of particle physics.’ Science
is much more than a list of facts or curve-fitting formulae; it is at its best
a deeper understanding of the physical world. The success of $\rm\Lambda$CDM
has left us wanting more, asking deeper questions, and expecting answers! In
science no good deed goes unpunished: answering questions usually leads to a
new set of more profound questions.

Of course, there is another way to look at $\rm\Lambda$CDM’s success: dark
matter, dark energy and inflation are simply proxies for the real, even more
revolutionary physics needed to describe our Universe; some have even called
them epicycles. For example, dark energy and dark matter could use be patches
for the shortcomings of General Relativity. If this is correct, the path ahead
in cosmology will be even more exciting.

The conservative view today is that dark matter, dark energy, and inflation
are simply physics beyond the standard model waiting to be clarified and
understood: a new particle, a new form of elastic energy and a new scalar
field. Other than neutrino mass, they are the only phenomena that cannot be
explained by the standard model.333Baryogenesis is also likely in this
category; I have not included it though, because there is $B$ and $CP$
violation within the standard model and some still hold out the hope that the
baryon asymmetry of the Universe can be explained by the standard model or a
minimal extension thereof.

###  2.1 Dark Matter

The evidence for non-baryonic dark matter began to develop in the 1980s with
the emerging gap between the baryon density determined by BBN ($\Omega_{B}\leq
0.1$) and the total matter density determined by a variety of astrophysical
methods ($\Omega_{M}\geq 0.2$). Today, there is an airtight case based upon
the 50$\sigma$ discrepancy between the BBN/CMB determination of the baryon
density, $\omega_{B}=0.0222\pm 0.00023$, and the CMB/LSS determination of the
matter density, $\omega_{M}=0.142\pm 0.0022$. We have additional astrophysical
information about the dark matter as well: it interacts weakly with ordinary
matter and is slowly moving (‘cold’). And we know that a small amount of the
dark matter – somewhere between 0.1% and 1% of critical density – exists in
the form of fast-moving (‘hot’) relic neutrinos.

On the theoretical side, attempts to go beyond the standard model of particle
physics to unify the forces and particles of Nature have led to two compelling
ideas for the dark matter particle: the neutralino, the lightest superpartner
in supersymmetric extensions of the standard model (mass 100 GeV to several
TeV), and the axion, a very light particle (mass of order $10^{-6}\,$eV),
which is a consequence of the most attractive idea about solving the strong
$CP$ problem of QCD. There is a more generic version of the neutralino – the
WIMP for Weakly-Interacting Massive Particle – whose motivations trace to the
fact that a stable particle whose annihilation cross section is ‘weak,’ more
specifically $<\sigma v>_{\rm ann}\sim 3\times 10^{-26}\,{\rm
cm^{3}sec^{-1}}$, will have the relic abundance needed to be the dark matter
(sometimes called the WIMP miracle).

On the experimental side, there is a vigorous, three-pronged approach to
identifying the dark matter particle: direct detection of dark matter
particles in our halo, indirect detection (detection of annihilation products
from halo dark-matter annihilations) and accelerator production. Direct-
detection experiments are now probing the theoretically favored parts of the
parameter space for neutralinos and axions; for indirect detection and
accelerator production the same is true for WIMPs and neutralinos.

This is moment of truth for the particle dark matter hypothesis: we have an
airtight case for this bold hypothesis, a broad and sensitive-enough
experimental effort to test the hypothesis, and no compelling evidence for
particle dark matter – yet.

There are several possible futures, all interesting, though some less
satisfying. First, a detection could be right around the corner! Second, our
compelling ideas could be just that – compelling, but not correct! Third,
there are new ideas, including asymmetric dark matter, low-mass dark matter (1
GeV and down to tiny fractions of an eV), and the most expansive idea yet, a
whole dark sector, one with its own particles and interactions, to be
discovered. It could be that rather than being close to a solution, we have
just seen the tip of the iceberg of a whole new sector that interacts only
very weakly with the standard model particles that we are familiar with.

It should also be said that none of the astrophysical and cosmological
evidence for dark matter requires it to have other than gravitational
interactions with standard model particles. Thus, it could be that detection
of the dark matter particles is far beyond our current capabilities and for
the foreseeable future particle dark matter will just be an economical
description of all the dark-matter phenomena.

Finally, it is possible that the dark matter problem is actually telling us
that Newton’s theory – recall, most of the astrophysical evidence for dark
matter only involves Newtonian physics – is not correct on scales much larger
than the solar system, where it is well tested. (Of course, General Relativity
would have to be modified as well.) There have been and are a host of ideas,
starting with Milgrom’s MOND in 1983 Milgrom  and extending to string-inspired
ideas (e.g., those of Verlinde Verlinde ) about how this might be. I must
confess, I find none of these ideas as compelling as the simple hypothesis of
particle dark matter. Of course, Nature may not have chosen what I find what I
find compelling!

###  2.2 Dark Energy

The history of dark energy and cosmic acceleration is more recent, but its
roots in the form of $\rm\Lambda$ go back to deSitter, Einstein, Lemaitre and
Eddington and the beginning of cosmology. $\rm\Lambda$ was invoked again by
Bondi, Hoyle and Gold in their steady state cosmology in 1948 and in the 1960s
to explain the great abundance of QSOs at redshift $z\sim 2$. In the 1980s
cosmic acceleration returned in the form of inflation, and beginning in 1984,
$\rm\Lambda$ returned as a means to achieving a flat Universe with a matter
density of order 30% of the critical density TSK .

One can’t discuss $\rm\Lambda$ without mentioning the greatest embarrassment
of theoretical physics: the mis-prediction of quantum vacuum energy, which is
mathematically equivalent to $\rm\Lambda$. The quantum zero-point energies,
when summed over all modes, diverges; this is a fact that Pauli first realized
in the 1930s. Truncating the sum at 1 TeV, leads to a value that is 55 orders-
of-magnitude larger than what is required to explain cosmic acceleration. For
many years this big problem – known as the cosmological constant problem – was
ignored, with the tacit assumption that the correct answer would eventually
turn out to be zero and theorists just needed to figure out how. The discovery
of accelerated expansion in 1998 brought the cosmological constant problem
front and center and made it a much richer and more urgent question: a small
vacuum energy could be the explanation; or something else. One problem or two
(or more)!

The term dark energy refers to a smooth component of energy density with an
equation-of-state $w$ sufficiently negative to cause the expansion to
accelerate ($w<-1/3$); it was introduced to remind astronomers (and others)
that cosmic acceleration might well involve more than just $\rm\Lambda$
Stromlo . Because of its familiarity and its mathematical equivalence to
vacuum energy, $\rm\Lambda$ is the ‘simplest’ example of dark energy, with
$w\equiv-1$. Moreover, the data are consistent with this:
$w=-1.00^{+0.04}_{-0.05}$ (sys+stat) DES1 . There is even theoretical
motivation: some argue that the landscape of string theory, with its more than
$10^{500}$ vacua, all with different values of the vacuum energy, can
accommodate the seemingly small value we find Linde . The next level of
complexity, involves the idea of a time-varying vacuum energy, implemented by
a slowly-rolling, ultra-light scalar field. While General Relativity + dark
energy can accommodate the observed accelerated expansion, it could be that
there is no dark energy and that the correct gravity theory will explain
cosmic acceleration.

Unlike dark matter, where there are compelling theoretical ideas and a robust
experimental program to test them, the situation here is different: No
compelling theoretical idea, and $\rm\Lambda$, with and all its baggage, fits
the data well (better than 5%). For the moment, $\rm\Lambda$ serves as a good
proxy for the real explanation. Moreover, with the observational program in
place, there are opportunities for surprises and new physics ahead.

###  2.3 Inflation

Inflation is the most important idea in cosmology since the big bang itself.
It offers a compelling explanation for the origin of the smoothness of the
Universe as well as the small inhomogeneities that seed cosmic structure, and
with particle dark matter, it has given us the highly successful CDM paradigm
for structure formation. There is evidence for two of its three basic
predictions: flat Universe and almost scale-invariant Gaussian density
perturbations. Further, heroic efforts are being mounted to test its third big
prediction, an almost scale-invariant spectrum of gravitational waves Kallosh
. Lastly, the discovery of the Higgs has provided some motivation for the
scalar field(s) that are used in most models of inflation.

All that being said, inflation is not yet a well-formulated, complete theory,
and there is no standard model of inflation. At best, we have a rudimentary
description of inflation. An analogy I like, is the one to superconductivity:
Ginzburg-Landau theory vs BCS theory. At this stage we have a Ginzburg-Landau
description, one with three robust predictions.

There are also important unresolved issues: the initial conditions for
inflation; if it predicts the multiverse, how is one to interpret the
multiverse and the testing of inflation?; and with regard to the
smoothness/horizon problem, it is at best a postponement since inhomogeneity
is merely pushed outside the horizon, eventually coming back inside the
horizon (unless the expansion of the Universe continues to accelerate). Some
believe that these issues are telling us that inflation is not bold enough
Penrose .

While the simplest models of inflation (single-field) do an excellent job of
fitting the extant data – flat Universe with Gaussian, almost scale-invariant
density perturbations – at this point, the best we can say is that inflation
is a compact description of the earliest moments that dynamically sets some of
the most fundamental features of the Universe. The opportunities for a deeper
understanding of the beginning of the Universe as well as big surprises are
great.

##  3 Moving forward: discovery and disruption

I see cosmology moving forward in two distinct ways: discovery and disruption.
Both are made possible by the growing number of precision measurements and
correspondingly precise theoretical predictions. Some of the datasets being
assembled are large and complex, and often the cross correlations between
large datasets will be crucial in moving forward.

Disruption and discovery are not mutually exclusive. While disruption refers
to upsetting the $\rm\Lambda$CDM paradigm, a disruptive measurement could also
lead to a discovery. A discovery could lead to clarifying and establishing the
current paradigm, or disrupting it. I will illustrate with a few examples of
discovery and disruption that I see on the horizon.

###  3.1 Inflationary B-modes: discovery

The third key prediction of inflation is a spectrum of almost scale-invariant
gravitational waves. The amplitude of the spectrum depends upon the energy
scale of inflation, and thus the detection of these gravitational waves would
reveal the epoch of inflation. The most promising means of detection is their
imprint on the CMB, in the form of the odd-parity mode (B-mode) of CMB
polarization, which is not excited by scalar (density) perturbations.

But it won’t be easy; current limits to the B-mode polarization from
inflation, tensor-to-scalar ratio $r<0.1$, correspond to a B-mode signal that
is less than $100\,$nK in amplitude. B-mode polarization can be produced from
E-mode (even parity) polarization arising from density perturbations by the
gravitational lensing effects of large-scale structure along the line of
sight. Further, foreground dust emission is polarized with roughly equal
E-mode and B-mode components. Finally, there is no compelling theoretical
prediction for the inflationary B-mode amplitude – worst yet, some theorists
say it will be undetectably small.

In spite of all this, there is an impressive and growing effort to reach a
sensitivity of $r\leq 10^{-3}$ Kallosh . That effort involves multi-wavelength
measurements to separate foreground emission from CMB contributions;
measurements on angular scales from more than one degree to a tenth of a
degree (multipole ${\ell}$ from 10’s to a few 1000); and the use of powerful
analysis techniques to remove B-mode polarization arising from lensed E-modes.

Here is where we stand: B-mode polarization from both dust and lensing has
been detected and mapped out from $\ell\sim 40-2000$, with a current upper
limit, $r\leq 0.1$; see Fig. 3. The importance of the measurement is reflected
in the very simple relationship between $r$ and the value of the inflationary
potential and Hubble constant during inflation:

$\displaystyle V^{1/4}$ $\displaystyle=$ $\displaystyle 10^{16}\,{\rm
GeV}\,\left(r\over 10^{-2}\right)^{1/4}$ (8) $\displaystyle H_{\rm
inflation}^{-1}$ $\displaystyle=$ $\displaystyle 2\times 10^{-38}\,{\rm
sec}\,\left(r\over 10^{-2}\right)^{-1/2}$ (9)

Figure 3: Top: B-mode power spectrum including theoretical expectations for
$\rm\Lambda$CDM with $r=0$: lensing (solid line), dust (dotted) and total
(broken) SPTBBmode . Bottom: E-mode spectrum, B mode measurements and
predictions for $r=0.2$ and $=0.01$ Araujo

###  3.2 $H_{0}$: disruption (and discovery?)

There is currently a more than 3$\sigma$ discrepancy between direct
measurements of the Hubble constant, using type Ia SNe calibrated by the
astronomical distance scale, and measurements of $H_{0}$ inferred from CMB
measurements, assuming $\rm\Lambda$CDM: $73\pm 1.7$ km/s/Mpc (direct) vs
$67\pm 0.5$ km/s/Mpc (CMB); see Fig. 4 Freedman . There could be an error in
one or both measurements; or, they could both be correct and the problem is
with the assumption of $\rm\Lambda$CDM. If it is the latter, $\rm\Lambda$CDM
is disrupted and possibly something new is discovered (e.g., neutrino mass or
dark energy beyond $\rm\Lambda$). At the moment, there is no compelling
theoretical explanation of the two different values waiting to be verified, so
a discovery would be an even bigger surprise.

Figure 4: $H_{0}$ tension: Direct determinations in blue and CMB
determinations in red differ by more than $3\sigma$ Freedman .

Allan Sandage must be happy: Just when we thought the Hubble constant had
nothing more interesting to tell us, this discrepancy reminds us that it is
the most important number in cosmology! One promising new approach to
determining $H_{0}$ is gravitational-wave standard sirens: the gravitational
waveform of coalescing black holes (recently discovered!) or neutron stars
determine the distance to the coalescing binary. If an electromagnetic
counterpart to the event is detected and the redshift can be measured, a
single event gives a direct determination of $H_{0}$ that is independent of
the astronomical distance scale.444A few months after this meeting, the first
detection of, and determination of the Hubble constant by, a standard siren
was made by the LIGO/VIRGO Collaboration abbott .

###  3.3 $\Lambda$ or GR: disruption

$\rm\Lambda$CDM assumes the correctness of General Relativity with a
cosmological constant as the dark energy. A measurement of $w\not=-1$ or a
change in time ($w_{a}\not=0$) would falsify $\rm\Lambda$ and disrupt the
paradigm; measurements from the Dark Energy Survey, or the upcoming Euclid or
WFIRST missions have the potential to do this. A new technique, the growth of
cosmic structure, has the additional benefit of being able to test the
validity of the GR as the underlying theory of gravity. In both cases, the
absence of an compelling theoretical alternative makes it unlikely the
disruption would also lead to a clear-cut discovery.

###  3.4 Dark matter: discovery

As mentioned in §2, there are ambitious experimental programs searching for
evidence of axions, neutralinos and WIMPs. They are improving in sensitivity
and already have sufficient sensitivity to cover interesting parts of the
theoretical parameter space. My favorite scenario is a triple detection of a
WIMP or neutralino – direct detection, indirect detection through annihilation
products and accelerator production. Not only would that confirm the particle
dark matter hypothesis, but it would also allow for redundancy checks (e.g.,
consistency of the relic abundance with its measured annihilation cross
section).

###  3.5 CDM: disruption

CDM is the theory that has at least nine lives! Since its introduction in
1983, there have been many attempts to kill it, all unsuccessful. (The only
significant change in the paradigm came with the addition of $\rm\Lambda$,
lowering the matter content to about 30% of the critical density.) CDM is a
strong theory in the sense of Karl Popper: it makes many predictions that can
be used to falsify it. Two current problems that are frequently mentioned are
that the galactic rotation curves it predicts are too cuspy and that it
predicts too many small satellites. There is much controversy about how
serious these problems are Silk , and because baryonic ‘gastrophysics’ can be
important on these small scales it will be hard to rule out CDM this way.
Further, none of the alternatives, e.g., keV sterile neutrino dark matter or
self-interacting dark matter, are compelling.

Finally, because the successes of CDM are so numerous, especially on large
scales where gastrophysics cannot confuse the situation, at the very least,
CDM must have much of the truth, and the correct theory must be very similar
to it. Nonetheless, because CDM is a strong theory, we should keep an open
mind to the possibly of its disruption.

###  3.6 CMB anomalies: disruption

Precision measurements of the anisotropy of the CMB have dramatically advanced
cosmology and played a major role in establishing $\rm\Lambda$CDM. However,
they have also revealed some anomalies: small quadrupole; correlations between
the first few multipoles; and a striking ‘dent’ around $\ell\sim 20-40$; see
Fig. 1. We have but one CMB sky to infer the underlying distribution and so
sample variance – here known as cosmic variance – introduces an irreducible
uncertainty. Because of the absence of an a priori prediction, the first two
anomalies reveal little because of significant cosmic variance. The dent could
be different and was discussed at this meeting by Bouchet Bouchet .

##  4 Some grand challenges beyond $\rm\Lambda$CDM

In the previous sections I have focused on the three pillars of
$\rm\Lambda$CDM because of their central importance to the paradigm itself and
their ripeness for testing. Here, I mention some grander aspirations for
cosmology. Big questions, whose answers may not yet be within our reach. And
needless to say, my list is idiosyncratic and incomplete.

###  4.1 Baryogenesis

The revolution in cosmology initiated by bringing in ideas from particle
physics began with baryogenesis in 1978. At the time, baryogenesis, while
attractive, was optional: one could still imagine a net baryon number as an
initial condition. Once inflation is in the mix, the tremendous entropy
produced by the re-heating process makes baryogenesis required because any
initial baryon asymmetry would be reduced exponentially. While we have
compelling – and testable – models for the origin of dark matter, we can’t say
the same for the baryon asymmetry needed to ensure the existence of ordinary
matter.

The standard model has two of the necessary ingredients: $B$ violation; $C$
and $CP$ violation, and the expansion of the Universe can provide the needed
departure from thermal equilibrium. Electroweak baryogenesis once looked
promising, but appears to fail on two accounts: not enough $CP$ violation in
the standard model and the dynamics of the phase transition do not provide the
requisite departure from equilibrium. Currently, much attention is focused on
lepto/baryogenesis, where a lepton asymmetry is produced early on and then
transformed into a baryon asymmetry by $B+L$ violating electroweak processes.
In either case, it appears that baryogenesis involves physics at scales that
are inaccessible in the laboratory, making progress and testing difficult. The
grand challenge is a detailed model of baryogenesis that is testable.

###  4.2 Who ordered that?!#

The history of cosmology is growing complexity in the composition of the
Universe: baryons (circa 2500BCE), photons (1965), neutrinos (1967), exotic
dark matter (1981), cold dark matter (1983), massive neutrinos (1998), and
dark energy (1998).

In I.I. Rabi’s words, who ordered all that? Are there more relics? What’s
next? The cosmic mix raises interesting questions: why is the ratio of CDM to
baryons about 5? and the ratio of baryons to neutrinos about 10? (both ratios
are of course constant in time) carrturner . Why did dark energy come to
dominate the energy density so recently? The grand challenge is explaining the
mix and/or finding more relics.

###  4.3 Inflation: up or out!

Inflation has moved cosmology forward for more than three decades, but it is
an incomplete theory and has teased us with some very big ideas, e.g., the
multiverse. The grand challenge is simple: a fundamental theory that makes
sharp predictions that can be falsified, or an even better theory to replace
it.

###  4.4 Making sense of the multiverse

The multiverse presents us with a big dilemma: it may be the most important
idea in science since Copernicus removed Earth from the center of the
Universe, but it is inherently untestable, and so isn’t science. I say that it
is inherently untestable because the fundamental idea behind the multiverse is
that the different pieces of the multiverse are causally distinct, which makes
direct testing not just hard, but impossible in principle. There are other
important issues to be addressed to make sense of the multiverse, including
that of defining a measure, but for me, the issue of whether or not it is
within the realm of science is the go/no-go question. The grand challenge is
to make the multiverse science or to stop talking about it!

##  5 Limits of scientific cosmology

Cosmology is an enterprise that requires hutzpah: the Universe is enormous and
often beyond the reach of our instruments and ideas. Armed with no small
amount of arrogance, bold ideas and powerful instruments, we have been
remarkably successful in determining the basic features of the Universe and
reconstructing much of its history. We hope to do even more. However, there
are limits to what we can learn.

###  5.1 Destiny

For years, we were confident that geometry and destiny were linked and that we
could determine the fate of our Universe, cf. Sandage200inch . Cosmic
acceleration and dark energy have humbled us. We know the geometry – flat to
within 0.5% – but because of dark energy the question of destiny is even more
unsettled. First, there is a new possibility: the big rip. If $w<-1$, dark
energy will lead to a singular ending in a finite amount of time. Even if dark
energy is explained by a scalar field that eventually relaxes to the minimum
of its potential, returning the Universe to a new, dark matter-dominated era,
we still cannot predict the destiny unless we solve the cosmological constant
problem. There could be a small – really tiny – cosmological constant of
either sign that would change the outcome. The destiny question is likely to
remain open for a long time.

###  5.2 Our past light cone

G.F.R. Ellis has emphasized how the smallness of our past light cone severely
limits our knowledge of the entire space-time Ellis . While invoking the
Cosmological Principle may sound good, it is not a substitute for
observational facts. Moreover, if inflation is correct, the large-scale
structure of the Universe could be very, very different from what we see in
our observable Universe. Because the Universe has undergone at least two
periods of accelerated expansion, even the question of our past light cone is
not a simple one. The small fraction of space-time occupied by our past light
cone fundamentally limits what we can learn about the Universe.

###  5.3 The opaque screen of inflation

A – perhaps the – fundamental theoretical motivation for something like
inflation was to lessen the dependence of the present state of the Universe
upon initial data. Inflation has done its job well, maybe too well: The
entropy produced during reheating exponentially dilutes the abundance of any
potential relic from the pre-inflation era; the accelerated expansion
guarantees that the Universe will be appear to be flat, pushes any initial
large-scale inhomogeneity outside our current horizon and impresses new
density perturbations on small scales (i.e., those accessible today). In
short, if the Universe inflated, we have lost access to its initial state, and
inflation has put up a very opaque screen that prevents us from seeing further
back than the epoch of inflation.

###  5.4 Multiverse

Enough said earlier.

###  5.5 Closing the circle

Cosmology faces the challenge of being a ‘historical science,’ concerned with
reconstructing the timeline of the Universe (both backward and forward), and
thus we cannot do experiments to directly test the timeline. Of course,
cosmology also allows us to use the ‘Heavenly Laboratory’ to make discoveries
beyond the reach of terrestrial laboratories – e.g., the discoveries of 4He,
dark matter and dark energy – that can be later followed up on Earth.

Establishing the evolution of the Universe is not impossible, but can be very
challenging. The process typically involves a theoretical framework with
testable predictions and observational evidence for those predictions, with
the level of confidence depending upon both. The theoretical framework may
involve theories that range from well-tested to highly speculative. Few would
argue that the expansion of the Universe is not well-established and at the
other end of the spectrum we have inflation, which is supported by some
evidence, but far from being well-established.

I consider big-bang nucleosynthesis to be ‘well-established’ and it
illustrates my ‘closing the circle’ criterion. The theoretical framework
includes General Relativity (and its FLRW model for which there is much
evidence), nuclear physics, and statistical mechanics. All three elements of
the theoretical framework are well-tested in the laboratory; e.g., the crucial
cross sections for BBN are all measured at the energies of relevance. Further,
BBN makes predictions that can be tested: the primordial abundances of D, 3He,
4He and 7Li. Within the standard model of particle physics, the predictions
depend upon one cosmic parameter: the baryon-to-photon ratio $\eta$, which is
directly related to $\Omega_{B}$.

With the exception of 7Li,5557Li is a complicated and continuing story. The
theoretical predictions have the largest error bars and it is not clear that
the primordial abundance has been determined. the predicted BBN light-element
abundances agree with the measured primordial abundances for a single value of
$\eta$, one which is consistent with the other determinations of the baryon
density, most notably that from the CMB. In spite of the fact that BBN
involves events that occurred when the Universe was more than a billion times
smaller and a billion times hotter, seconds after the beginning of the
Universe, because the underlying physics is well established in the laboratory
and its predictions have been verified, most cosmologists consider BBN a well-
established part of the timeline of the Universe. It is the connection to
physics well-established in the laboratory and verified predictions that
closes the circle.

Inflation presents a bigger challenge. On the one hand, two of the key
predictions of the inflationary paradigm – flat Universe with almost scale-
invariant, Gaussian density perturbations – have been confirmed, and great
effort is being put forward to testing the third prediction, gravitational
waves. On the other hand, there is no standard model of inflation, some models
have their roots in very speculative physics (e.g., superstring theory), and
others are simply ad hoc.

To illustrate the challenge inflation faces, let’s suppose that the B-mode
polarization signature of inflation-produced gravitational waves is detected.
This determines the Hubble constant during inflation, and allows the
reconstruction of the inflationary potential. So, would I include inflation in
the well-established history of the Universe? No: Without a fundamental model
of inflation rooted in laboratory experiments (e.g., the production of the
inflaton particle at an accelerator), I believe the strongest statement I
could make is that inflation holds much of the truth about the earliest
moments of the Universe.

Finally, a few quick words about dark matter. Until we discover the dark
matter particle, dark matter and CDM occupy a similar position: a very
attractive description of the formation of structure in the Universe that has
much of the truth. A triple detection of the dark matter particle would change
that and close the circle, bringing CDM and particle dark matter into the well
established history of the Universe.

###  5.6 Limits of science

As I like to remind my students (and the general public), science is the most
powerful means we have found to explore the physical world. Science has not
only helped us understand the physical world and the laws that govern it, but
it has also allowed us to use the physical laws for the betterment of
humanity.

However, there are limits to science. It is all about how and not about why.
While we can aspire to discover the laws that govern the Universe and
understand how they have shaped its evolution, science cannot explain why the
laws are what they are, where they came from, or why we have something rather
than nothing. Some questions are simply beyond the reach of science.

###  5.7 Final words

No need to worry, there is plenty more that science can do in cosmology;
$\rm\Lambda$CDM has only whet our appetite for more by allowing us to frame
even more profound questions about the Universe.

###### Acknowledgements.

This work was supported in part by the Kavli Institute for Cosmological
Physics at the University of Chicago through grant NSF PHY-1125897 and an
endowment from the Kavli Foundation and its founder Fred Kavli.

## References

  * (1) H. Kragh, in this volume (2017).
  * (2) E. Hubble, MNRAS, 113, 658 (1953).
  * (3) M.L. Humason, N.U. Mayall, and A.R. Sandage, AJ, 61, 97 (1956).
  * (4) A.R. Sandage, Physics Today, Feb 1970, 34 (1970).
  * (5) A.R. Sandage, ApJ, 133, 355 (1961).
  * (6) A.R. Neben and M.S. Turner, ApJ 769, 133 (2013).
  * (7) P. Madau and M. Dickinson, ARAA, 52, 415 (2014).
  * (8) E.W. Kolb and M.S. Turner, The Early Universe (Addison-Wesley, 1990).
  * (9) M.S. Turner, G. Steigman and L.M. Krauss, PRL, 52, 2090 (1984).
  * (10) A. Guth, PRD, 23, 346 (1981).
  * (11) A. Filippenko, in this volume (2017).
  * (12) N. Suntzeff, in this volume (2017).
  * (13) Planck Collaboration, A&A, 594, 13 (2016)
  * (14) M. Milgrom, ApJ, 270, 365 (1983).
  * (15) E. Verlinde, SciPostPhys, 2, 016 (2017).
  * (16) M.S. Turner, AstonSocPacConf, 165, 431 (1999) (astro-ph/9811454).
  * (17) DES Collaboration, axXiv:1708.01530 (2017).
  * (18) A. Linde, in this volume (2017).
  * (19) R. Kallosh, in this volume (2017).
  * (20) R. Penrose, in this volume (2017).
  * (21) R. Keisler, et al., ApJ, 807, 151 (2015), arXiv:1503.02315.
  * (22) D.C. Araujo et al, Proc SPIE Int Soc Opt Eng, 9153, 91530W (2014), arXiv:1407.6249.
  * (23) W.L. Freedman, Nature Astronomy, 1, 0121 (2017).
  * (24) B.P. Abbott, Nature 551, 85 (2017).
  * (25) J. Silk, in this volume (2017).
  * (26) F. Bouchet, in this volume (2017).
  * (27) M.S. Turner and B.J. Carr, ModPhysLett A, 2, 1 (1987)
  * (28) G.F.R. Ellis, in this volume (2017).


# The Wiki Music dataset:
A tool for computational analysis of popular music

Fabio Celli  Profilio Company s.r.l.
via sommarive 18,
38123 Trento, Italy
Email<EMAIL_ADDRESS>
###### Abstract

Is it possible use algorithms to find trends in the history of popular music?
And is it possible to predict the characteristics of future music genres? In
order to answer these questions, we produced a hand-crafted dataset with the
intent to put together features about style, psychology, sociology and
typology, annotated by music genre and indexed by time and decade. We
collected a list of popular genres by decade from Wikipedia and scored music
genres based on Wikipedia descriptions. Using statistical and machine learning
techniques, we find trends in the musical preferences and use time series
forecasting to evaluate the prediction of future music genres.

###### Index Terms:

Popular Music, Computational Music analysis, Wikipedia, Natural Language
Processing, dataset

##  I Motivation, Background and Related Work

Until recent times, the research in popular music was mostly bound to a non-
computational approach [21] but the availability of new data, models and
algorithms helped the rise of new research trends. Computational analysis of
music structure [18] is focused on parsing and annotate patters in music
files; computational music generation [4] trains systems able to generate
songs with specific music styles; computational sociology of music analyzes
databases annotated with metadata such as tempo, key, BPMs and similar
(generally referred to as sonic features); even psychology of music use data
to find new models.
Recent papers in computational sociology investigated novelty in popular
music, finding that artists who are highly culturally and geographically
connected are more likely to create novel songs, especially when they span
multiple genres, are women, or are in the early stages of their careers [15].
Using the position in Billboard charts and the sonic features of more than 20K
songs, it has been demonstrated that the songs exhibiting some degree of
optimal differentiation in novelty are more likely to rise to the top of the
charts [3]. These findings offer very interesting perspectives on how popular
culture impacts the competition of novel genres in cultural markets. Another
problem addressed in this research field is the distinction between what is
popular and what is significative to a musical context [16]. Using a user-
generated set of tags collected through an online music platform, it has been
possible to compute a set of metrics, such as novelty, burst or duration, from
a co-occurrence tag network relative to music albums, in order to find the
tags that propagate more and the albums having a significative impact.
Combining sonic features and topic extraction techniques from approximately
17K tracks, scholars demonstrate quantitative trends in harmonic and timbral
properties that brought changes in music sound around 1964, 1983 and 1991
[14]. Beside these research fields, there is a trend in the psychology of
music that studies how the musical preferences are reflected in the dimensions
of personality [11]. From this kind of research emerged the MUSIC model [20],
which found that genre preferences can be decomposed into five factors: Mellow
(relaxed, slow, and romantic), Unpretentious, (easy, soft, well-known),
Sophisticated (complex, intelligent or avant-garde), Intense (loud,
aggressive, and tense) and Contemporary (catchy, rhythmic or danceable).
Is it possible to find trends in the characteristics of the genres? And is it
possible to predict the characteristics of future genres? To answer these
questions, we produced a hand-crafted dataset with the intent to put together
MUSIC, style and sonic features, annotated by music genre and indexed by time
and decade. To do so, we collected a list of popular music genres by decade
from Wikipedia and instructed annotators to score them. The paper is
structured as follows: In section II we provide a brief history of popular
music, in section III we describe the dataset and in section IV we provide the
results of the experiments. In the end we draw some conclusions.

##  II Brief introduction to popular music

We define ”popular music” as the music which finds appeal out of culturally
closed music groups, also thanks to its commercial nature. Non-popular music
can be divided into three broad groups: classical music (produced and
performed by experts with a specific education), folk/world music (produced
and performed by traditional cultures), and utility music (such as hymns and
military marches, not primarily intended for commercial purposes). Popular
music is a great mean for spreading culture, and a perfect ground where
cultural practices and industry processes combine. In particular the cultural
processes select novelties, broadly represented by means of underground music
genres, and the industry tries to monetize, making them commercially
successful. In the following description we include almost all the genres that
reach commercial success and few of the underground genres that are related to
them.
Arguably the beginning of popular music is in the USA between 1880s and 1890s
with spirituals, work and shout chants [7], that we classify half-way between
world music and popular music. The first real popular music genres in the
1900s were ragtime, pioneer of piano blues and jazz, and gospel, derived from
religious chants of afro-american communities and pioneer of soul and RnB. The
1910s saw the birth of tin pan alley (simple pop songs for piano composed by
professionals) and dixieland jazz, a spontaneous melting pot of ragtime,
classical, afroamerican and haitian music [19]. In the 1920s, blues and
hillbilly country became popular. The former was born as a form of expression
of black communities and outcasts, while the latter was a form of
entertainment of the white rural communities. Tin pan alley piano composers
soon commercialized tracks in the style of blues, generating boogie-woogie as
a reaction, an underground and very aggressive piano blues played by black
musicians. In Chicago and New York jazz became more sophisticated and spread
to Europe, where gipsy jazz became popular soon after. Both in US and Europe,
the 1930s were dominated by swing, the most popular form of jazz, which was at
the same time danceable, melanchonic, catchy and intelligent. In the US the
west swing, a mellow and easy type of country music, became popular thanks to
western movies. The 1940s in the US saw a revival of dixieland jazz, the rise
of be-bop (one of the most mellow and intelligent forms of jazz), the advent
of crooners (male pop singers) and the establishment of back-to-the-roots
types of country music such as bluegrass, a reaction against west swing,
modernity and electric guitars. In the underground there was honky-tonk, a sad
kind of country music that will influence folk rock. In the 1950s rock and
roll was created by black communities with the electric fusion of blues,
boogie-woogie and hillbilly and soon commercialized for large white audiences.
Beside this, many things happened: urban blues forged its modern sound using
electric guitars and harmonicas; cool jazz, played also by white people,
launched a more commercial and clean style; gospel influenced both doo-wop,
(a-cappella music performed by groups of black singers imitating crooners) and
RnB, where black female singers played with a jazz or blues band. The 1960s
saw an explosion of genres: countrypolitan, an electric and easy form of
country music, became the most commercialized genre in the US; the first
independent labels (in particular the Motown) turned doo-wop into well-
arranged and hyper-produced soul music with a good commercial success [10];
ska, a form of dance music with a very typical offbeat, became popular outside
of Jamaica; garage (and also surf) rock arose as the first forms of
independent commercial rock music, sometimes aggressive and sometimes easy; in
the UK, beat popularized a new style of hyper-produced rock music that had a
very big commercial success; blues rock emerged as the mix of the two genres;
teenypop was created in order to sell records to younger audiences;
independent movements like beat generation and hippies helped the rise of folk
rock and psychedelic rock respectively [6]; funk emerged from soul and jazz
(while jazz turned into the extremely complex free jazz as a reaction against
the commercial cool jazz, but remained underground). In the 1970s progressive
rock turned psychedelia into a more complex form, independent radios
contribute to its diffusion as well as the popularity of songwriters, an
evolution of folk singers that proliferated from latin america (nueva canción)
to western Europe. In the meanwhile, TV became a new channel for music
marketing , exploited by glam rock, that emerged as a form of pop rock music
with a fake trasgressive image and eclectic arrangements; fusion jazz begun to
include funk and psychedelic elements; the disillusion due to the end of
hippie movement left angry and frustrated masses listening to hard rock and
blues rock, that included anti-religious symbols and merged into heavy metal.
Then garage and independent rock, fueled by anger and frustration, was
commercialized as punk rock at the end of the decade, while disco music (a
catchy and hyper-danceable version of soul and RnB) was played in famous clubs
and linked to sex and fun, gathering the LGBT communities. The poorest black
communities, kept out from the disco clubs, begun to perform in house-parties,
giving rise to old skool rap, whose sampled sounds and rhythmic vocals were a
great novelty but remained underground. The real novelties popularized in this
decade were ambient (a very intelligent commercial downtempo music derived
from classical music), reggae (which mixed ska, rock and folk and from Jamaica
conquered the UK) and above all synth electronica, a type of industrial
experimental music that became popular for its new sound and style, bridging
the gap between rock and electronic music. This will deeply change the sound
of the following decades [2]. The 1980s begun with the rise of synth pop and
new wave. The former, also referred to as ”new romantics”, was a popular music
that mixed catchy rhythms with simple melodies and synthetic sounds while the
latter was an hipster mix of glam rock and post-punk with a positive view (as
opposed to the depressive mood of the real post-punk), with minor influences
from synth electronica and reggae. The music industry created also glam metal
for the heavy metal audiences, that reacted with extreme forms like thrash
metal; a similar story happened with punk audiences, that soon moved to
extreme forms like hardcore, which remained underground but highlighted a
serious tensions between industry and the audiences that wanted spontaneous
genres [17]. In the meanwhile discopop produced a very catchy, easy and
danceable music mix of disco, funk and synthetic sounds, that greatly improved
the quality of records, yielding to one of the best selling genres in the
whole popular music history. In a similar way smooth jazz (a mix of mellow and
easy melodies with synthetic rhythmical bases) and soft adult (a mellow and
easy form of pop) obtained a good commercial success. Techno music emerged as
a new form of danceable synthetic and funky genre and hard rap became popular
both in black and white audiences, while electro (break dance at the time) and
(pioneering) house music remained underground for their too much innovative
sampled sounds. In the 1990s alternative/grunge rock solved the tension
between commercial and spontaneous genres with a style of rock that was at the
same time aggressive, intelligent and easy to listen to. The same happened
with skatepunk (a fast, happy and commercial form of rock) and rap metal (a
mix of the two genres) while britpop continued the tradition of pop rock
initiated with beat. RnB evolved into new jack swing (a form of softer,
rhythmical and easy funk) and techno split into the commercial eurodance (a
mix of techno and disco music with synthetic sounds, manipulated RnB vocals
and strong beats) and the subculture of rave (an extremely aggressive form of
techno played in secret parties and later in clubs), which helped the creation
of goa trance, that new hippie communities used for accompany drug trips [5].
An intelligent and slow mix of electro and RnB became popular as trip hop
while an aggressive and extremely fast form of electro with reggae influences
became popular as jungle/DnB. By the end of the decade the most commercially
successful genres were dancepop (a form of pop that included elements of funk,
disco and eurodance in a sexy image) and gangsta rap/hip hop that reached its
stereotypical form and became mainstream, while independent labels (that
produced many subgenres from shoegaze/indie rock to electro and house)
remained in the underground. In the underground -but in latin america- there
was also reggaetón, a latin form of rap. The rise of free download and later
social networks websites in 2000s opened new channels for independent genres,
that allowed the rise of grime (a type of electro mixing DnB and rap), dubstep
(a very intelligent and slow mix of techno, DnB and electro low-fi samples),
indietronica (a broad genre mixing intelligent indie rock, electro and a lot
of minor influences) and later nu disco (a revival of stylish funk and disco
updated with electro and house sounds) [8]. In the meanwhile there were
popular commercial genres like garage rock revival (that updated rock and punk
with danceable beats), emo rock/post grunge (aggressive, easy and even more
catchy), urban breaks (a form of RnB with heavy electro and rap influences)
and above all electropop (the evolution of dancepop, that included elements of
electro/house and consolidated the image of seductive female singers, also
aimed at the youngest audiences of teens). Among those genres epic trance (an
euphoric, aggressive and easy form of melodic techno) emerged from the biggest
dedicated festivals and became mainstream with over-payed DJ-superstars [22].
In the underground remained various forms of nu jazz, hardcore techno, metal
and house music. Then in 2010s finally euro EDM house music (a form of sample-
based and heavily danceable mix of house and electro) came out of underground
communities and, borrowing the figure of DJ-superstar from trance, reached
commercial success, but left underground communities unsatisfied (they were
mostly producing complex electro, a mix of dubstep and avant-garde house).
Also drumstep (a faster and aggressive version of dubstep, influenced by EDM
and techno) and trap music (a form of dark and heavy techno rap) emerged from
underground and had good commercial success. Genres like indiefolk (a modern
and eclectic folk rock with country influences) and nu prog rock (another
eclectic, experimental and aggressive form of rock with many influences from
electro, metal and rap) had moderate success. The availability of websites for
user-generated contents such as Youtube helped to popularize genres like
electro reggaetón (latin rap with new influences from reggae and electro),
cloud rap (an eclectic and intelligent form of rap with electro influences)
and JK-pop (a broad label that stands for Japanese and Korean pop, but emerged
from all over the world with common features: Youtubers that produce easy and
catchy pop music with heavy influences from electropop, discopop and
eurodance) [12]. Moreover, technologies helped the creation of mainstream
genres such as tropical house (a very melodic, soft and easy form of house
music singed in an modern RnB style). In the underground there are yet many
minor genres, such as bro country (an easy form of country played by young and
attractive guys and influenced by electro and rap), future hardstyle (a form
of aggressive trance with easy vocals similar to tropical house) and afrobeat
(a form of rap that is popular in western africa with influences from
reggaetón and traditional african music).
From this description we can highlight some general and recurrent tendencies,
for example the fact that music industry converts spontaneous novelties into
commercial success, but when its products leave audiences frustrated (it
happened with west swing, glam metal, cool jazz, punk and many others), they
generate reactions in underground cultures, that trigger a change into more
aggressive versions of the genre. In general, underground and spontaneous
genres are more complex and avant-garde. Another pattern is that media allowed
more and more local underground genres to influence the mainstream ones,
ending in a combinatorial explosion of possible new genres, most of which
remain underground. We suggest that we need to quantify a set of cross-genre
characteristics in order to compute with data science techniques some weaker
but possibly significative patterns that cannot be observed with qualitative
methods. In the next section we define a quantitative methodology and we
annotate a dataset to perform experiments.

##  III Data Description

From the description of music genres provided above emerges that there is a
limited number of super-genres and derivation lines [9, 13], as shown in
figure 1.

Figure 1: Distribution of genre derivation by super-genres and decade.

From a computational perspective, genres are classes and, although can be
treated by machine learning algorithms, they do not include information about
the relations between them. In order to formalize the relations between genres
for computing purposes, we define a continuous genre scale from the most
experimental and introverted super-genre to the most euphoric and inclusive
one. We selected from Wikipedia the 77 genres that we mentioned in bold in the
previous paragraph and asked to two independent raters to read the Wikipedia
pages of the genres, listen to samples or artists of the genres (if they did
not know already) and then annotate the following dimensions:

  * •

genre features: genre scale (a score between 0 and 1 where
0=downtempo/industrial, 0.1=metal, 0.15=garage/punk/hardcore, 0.2=rock,
0.25=pop rock, 0.3=blues, 0.4=country, 0.5=pop/traditional, 0.55=gospel,
0.6=jazz, 0.65=latin, 0.7=RnB/soul/funk, 0.75=reggae/jamaican, 0.8=rap,
0.85=DnB, 0.9=electro/house, 0.95=EDM, 1=techno/trance) and category of the
super-genre (as defined in figure 1) and influence variety 0.1=influence only
from the same super-genre, 1=influences from all the supergenres

  * •

perceived acoustic features: sound (0=acoustic, 0.35=amplified,
0.65=sampled/manipulated, 1=synthetic), vocal melody (1=melodic vocals,
0=rhythmical vocals/spoken words), vocal scream (1=screaming, 0=soft singing),
vocal emotional (1=emotional vocals, 0=monotone vocals), virtuous (0.5=normal,
0=not technical at all, 1=very technical); richbass 1=the bass is loud and
clear, 0=there is no bass sound; offbeat 1=the genre has a strong offbeat,
0=the genre has not offbeat

  * •

time: decade (classes between 1900s and 2010s) and year representative of the
time when the genre became meainstream

  * •

place features: origin place 0=Australia, 0.025=west USA, 0.05=south USA,
0.075=north/east USA, 0.1=UK, 0.2=jamaica, 0.3=carribean, 0.4=latin america,
0.5=africa, 0.6=south EU, 0.65=north/east EU, 0.7=middle east, 0.8=India,
0.9=China/south asia, 1=Korea/north asia; place urban (0=the origin place is
rural, 1=the origin place is urban), place poor (0=the origin place is poor,
1=the origin place is rich)

  * •

media features: media mainstream (0=independent media, 1=mainstream media,
0.5=both), media live 0=sell recorded music, 1=sell live performance)

  * •

emotion features: joy/sad (1=joy, 0=sad), anticipation/surprise
(1=anticipation or already known, 0=surprise), anger/calm (1=anger, 0=calm).

  * •

style features: novelty 0=derivative, 0.5=normal, 1=totally new
characteristics and type retro 1=the genre is a revival, 0.5=normal, 0=the
genre is not a revival, lyrics love/explicit 0.5=normal, 1=love lyrics,
0=explicit lyrics, style upbeat 1=extroverted and danceable, 0=introverted and
depressive, style instrumental 1=totally instrumental, 0=totally singed, style
eclecticism 1=includes many styles, 0=has a stereotypical style, style
longsongs 0.5=radio format (3.30 minutes), 1=more than 6 minutes by average,
0=less than 1 minute by average; largebands 1=bands of 10 or more people,
0.1=just one musician; subculture 1=the audience one subculture or more, 0=the
audience is the main culture; hedonism 1=the genre promotes hedonism, 0=the
genre does not promote hedonism; protest 1=the genre promotes protest, 0=the
genere does not promote protest; onlyblack 1=genere produced only by black
communities, 0=genre produced only by white communities; ; 44beat 1=the genre
has 4/4 beat, 0=the genre has other types of measures; outcasts 1=the audience
is poor people, 0=the audience is rich people; dancing 1=the genre is for
dancing, 0=the genre is for home listening; drugs 1=the audience use drugs,
0=the audience do not use drugs

  * •

MUSIC features: mellow (1=slow and romantic, 0=fast and furious),
sophisticated (1=culturally complex, 0=easy to understand), intense
(1=aggressive and loud, 0=soft and relaxing), contemporary (1=rhythmical and
catchy, 0=not rhythmical and old-fashioned), uncomplicated (1=simple and well-
known, 0=strange and disgustive)

We computed the agreement between the two annotators using Cronbach’s alpha
statistics [23]. The average between all features is $\alpha=0.793$, which is
good. Among the most agreed features there are genre, place, sound and MUSIC
features. In particular, the genre scale got an excellent $\alpha=0.957$,
meaning that the genre scale is a reliable measure. In the final annotation
all the divergences between the two annotators were agreed upon and the scores
were averaged or corrected accordingly. The final dataset is available to the
scientific community111The dataset can be downloaded from
http://personality.altervista.org/fabio.htm.

##  IV Experiments

What are the tendencies that confirm or disconfirm previous findings? We
noticed very interesting remarks just from the distributions of the features,
reported in figure 2.

Figure 2: Distributions of some of the features annotated in the dataset.

We can see that most of the popular music genres have a novelty score between
0.5 and 0.65, which is medium-high. This confirms the findings of previous
work about the optimal level of innovation and acceptance. It is interesting
to note that almost all the popular genres come from an urban context, where
the connections between communities are more likely to create innovations.
Moreover, we can see that the distribution of mainstream media is bi-modal:
this means that an important percentage of genres are popularized by means of
underground or new media. This happened many times in music history, from the
the free radios to the web of the user-generated content. Crucially, popular
music genres strongly tend to be perceived as technically virtuous.
Why the sound changed from acoustic to synthetic during the last century? To
answer this question we used a correlation analysis with the sound feature as
target. It emerged that the change towards sampled and synthetic sound is
correlated to dancing, to intensity/aggressiveness, to a larger drug usage and
to a large variety of infleunces, while it is negatively correlated to large
bands and mellow tones. In summary a more synthetic sound allowed a more
intense and danceable music, reducing the number of musicians (in other words
reducing costs for the industry).

Figure 3: Trend lines (dashed) of the MUSIC features from 1900.

How the music taste of the audience of popular music changed in the last
century? The trend lines of the MUSIC model features, reported in figure 3,
reveal that audiences wanted products more and more contemporary, intense and
a little bit novel or sophisticated, but less and less mellow and
(surprisingly) unpretentious. In other words, the audiences of popular music
are getting more demanding as the quality and variety of the music products
increases.
Is it possible to predict future genres by means of the genre scale? To answer
this question we used time series forecasting. In particular, we exploited all
the features in the years from 1900 to 2010 to train a predictive model of the
scores from 2011 to 2018. As the year of the genre label is arbitrary,
predicted scores and labels can be not aligned, thus MAE or RSME are not
suitable evaluation metrics. As evaluation metric we defined average accuracy
as $a=\frac{\sum count(|l-h|<0.1)}{count(t)}$, where the label (_l_) and the
prediction (_h_) can be anywhere within the year serie (_t_). Table I, shows
the results of the prediction of genre scale for the years 2011 to 2018 with
different algorithms: linear regression (LR), Support Vector Machine (SVM),
multi layer perceptron (MPL), nearest neighbors (IBk), and a meta classifier
(stacking) with SVM+MLP+IBk.

year | genre | LR | SVM | MLP | IBk | meta
---|---|---|---|---|---|---
2011 | 0.95 (edm) | 0.612 | 0.570* | 0.255* | 0.81 | 0.8*
2012 | 0.5 (jk-pop) | 0.732 | 0.728 | 1* | 0.51* | 0.24*
2013 | 0.25 (indiefolk) | 0.738 | 0.690 | 1.407 | 0.70 | 0.89*
2014 | 0.9 (drumstep) | 0.686 | 0.601 | 0.591 | 0.81* | 0.49*
2015 | 0.91 (tropical house) | 0.747* | 0.747* | 0.512 | 0.81 | 0.6
2016 | 0.18 (nu prog) | 0.739* | 0.666 | 0.862* | 0.51 | 0.7*
2017 | 0.76 (reggaeton) | 0.735 | 0.670 | 0.354 | 0.70* | 0.23*
2018 | 0.82 (cloudrap) | 0.765 | 0.748* | 0.279* | 0.81* | 0.7
| avg accuracy | 0.25 | 0.375 | 0.5 | 0.5 | 0.75
Table I: Results. *=scores considered for computing avg accuracy

The results reveal that the forecasting of music genres is a non-linear
problem, that IBk predicts the closest sequence to the annotated one and that
a meta classifier with nearest neighbors[1] is the most accurate in the
prediction. Deep Learning algorithms does not perform well in this case
because the dataset is not large enough. Last remark: feature reduction (from
41 to 14) does not affect the results obtained with IBk and meta classifiers,
indicating that there is no curse of dimensionality.

##  V Conclusion Acknowledgments and Future

We annotated and presented a new dataset for the computational analysis of
popular music. Our preliminary studies confirm previous findings (there is an
optimal level of novelty to become popular and this is more likely to happen
in urban contexts) and reveal that audiences tend to like contemporary and
intense music experiences. We also performed a back test for the prediction of
future music genres in a time series, that turned out to be a non-linear
problem. For the future we would like to update the corpus with more features
about audience types and commercial success. This work has also been inspired
by Music Map222https://musicmap.info/.

## References

  * [1] David W Aha, Dennis Kibler, and Marc K Albert.  Instance-based learning algorithms.  Machine learning, 6(1):37–66, 1991.
  * [2] Sean Albiez and David Pattie.  Kraftwerk: Music non-stop.  Bloomsbury Publishing USA, 2010.
  * [3] Noah Askin and Michael Mauskapf.  What makes popular culture popular? product features and optimal differentiation in music.  American Sociological Review, 82(5):910–944, 2017.
  * [4] Jean-Pierre Briot, Gaëtan Hadjeres, and François Pachet.  Deep learning techniques for music generation-a survey.  arXiv preprint arXiv:1709.01620, 2017.
  * [5] Sebastian Chan.  Music (ology) needs a context-re-interpreting goa trance.  Perfect Beat, 3(4):93–98, 2015.
  * [6] John Rudolph Covach and Andrew Flory.  What’s that sound?: an introduction to rock and its history.  WW Norton & Company New York., 2006.
  * [7] Robert Darden and Bob Darden.  People Get Ready!: A New History of Black Gospel Music.  Bloomsbury Publishing, 2005.
  * [8] Mike D’Errico.  Electronic dance music in the dubstep era.  Oxford Handbooks, 2015.
  * [9] Michael Fell and Caroline Sporleder.  Lyrics-based analysis and classification of music.  In Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, pages 620–631, 2014.
  * [10] Andrew Flory.  I Hear a Symphony: Motown and Crossover R&B.  University of Michigan Press, 2017.
  * [11] David M Greenberg, Michal Kosinski, David J Stillwell, Brian L Monteiro, Daniel J Levitin, and Peter J Rentfrow.  The song is you: Preferences for musical attribute dimensions reflect personality.  Social Psychological and Personality Science, 7(6):597–605, 2016\.
  * [12] Jin Ha Lee, Kahyun Choi, Xiao Hu, and JH Downie.  K-pop genres: A cross-cultural exploration.  In Proceedings of the 14th Conference of the International Society for Music Information Retrieval (ISMIR). The International Society for Music Information Retrieval (ISMIR)., 2013.
  * [13] Jennifer C Lena and Richard A Peterson.  Classification as culture: Types and trajectories of music genres.  American sociological review, 73(5):697–718, 2008.
  * [14] Matthias Mauch, Robert M MacCallum, Mark Levy, and Armand M Leroi.  The evolution of popular music: Usa 1960–2010.  Royal Society open science, 2(5):150081, 2015.
  * [15] Michael Mauskapf, Eric Quintane, Noah Askin, and Joeri Mol.  Embeddedness and the production of novelty in music: A multi-dimensional perspective.  In Academy of Management Proceedings. Academy of Management Briarcliff Manor, NY 10510, 2017.
  * [16] Bernardo Monechi, Pietro Gravino, Vito DP Servedio, Francesca Tria, and Vittorio Loreto.  Significance and popularity in music production.  Royal Society open science, 4(7):170433, 2017.
  * [17] Keith Negus.  Music genres and corporate cultures.  Routledge, 2013.
  * [18] Oriol Nieto and Juan Pablo Bello.  Systematic exploration of computational music structure research.  In ISMIR, pages 547–553, 2016.
  * [19] Bruce Boyd Raeburn.  New Orleans style and the writing of American jazz history.  University of Michigan Press Ann Arbor, 2009.
  * [20] Peter J Rentfrow, Lewis R Goldberg, David J Stillwell, Michal Kosinski, Samuel D Gosling, and Daniel J Levitin.  The song remains the same: A replication and extension of the music model.  Music Perception: An Interdisciplinary Journal, 30(2):161–185, 2012\.
  * [21] Roy Shuker.  Popular music: The key concepts.  Routledge, 2017.
  * [22] Timothy D Taylor.  Strange sounds: Music, technology and culture.  Routledge, 2014.
  * [23] P Wessa.  Cronbach alpha (v1. 0.5) in free statistics software (v1. 2.1).  Office for Research Development and Education, 2017.