# A Network Based Approach to Characterize Twenty-First-Century Populism in
Colombia

Juan D. Garcia-Arteaga<EMAIL_ADDRESS>Valentina Pellegrino  School
of Medicine,
Universidad Nacional de Colombia,
Bogotá, Colombia  LASC: Laboratory for the Anthroplogy of the State in
Colombia,
Bogotá, Colombia

###### Abstract

Populism is a political phenomenon of democratic illiberalism centered on the
figure of a strong leader. By modeling person/node connections of prominent
figures of the recent Colombian political landscape we map, quantify, and
analyze the position and influence of Alvaro Uribe as a populist leader. We
found that Uribe is a central hub in the political alliances networks, cutting
through traditional party alliances, but is not the most central figure in the
state machinery. The article first presents the framing of the problem,
followed by the historical context of the case in study, the methodology
employed and data collection, analysis, conclusions and further research
paths. This study has implications for offering a new way of applying
quantitative methods to the studies of populist regimes.

###### keywords:

Populism , Political , Networks , Colombia

##  1 Introduction

Populism is a phenomenon broadly defined as the pursuit of politicial power by
use of a confontational narrative founded on the concept of “us, the people”
against “them, the elite”. Once considered a result of politcally immature
systems, the rise of populist figures in well established liberal democracies
in Europe and North America[39] has renewed interest in the subject and in
understanding how and why these regimes come to power [15][32].

Although populist focused studies date back to the 1960’s, there is still not
a consensus on whether populism should be defined as a communication style, an
ideology or a political strategy. The research question in contemporary
studies has then shifted from what populism is to how may populism be
quantified. Recently the field has added studies about the degree of populism
in specific contexts, recurring to textual and content analysis of discourses,
speeches or manifestos [26][31][41][42]. Additionally, there has been an
increase in the studies of the attitudes in citizens that can be supportive of
populist regimes [10] [1] [44] [28].

These studies are valuable as there is much gain in understanding the
discourse patterns of populist leaders and how masses are incorporated into
them. However, these approaches tend to ignore both the impact of populism in
the party system [27] and, more generally, how populists leaders fit in in the
landscape of political power relationships. It is possible that, by centering
in the figure of the leader and his or her discourse, we are falling into the
trap of methodological individualism denounced by [49]: a tendency to focus on
individuals instead of relationships, which entails unrealistic assumptions of
the independence of these individuals.

In this article we present an alternative to a widespread tendency to study
populism and the networks that support it separately [43]. Specifically, we
will analyze the network of political alliances and work relationships
surrounding a populist leader: Colombia’s former president Alvaro Uribe whose
ongoing influence and politics have been used as an example of neo-populism by
various authors [37, 20].

We offer a novel analysis based on the Colombian political networks to show
how Uribe has a central role in them, contesting the commonly held belief of
the source of populist power being the direct contact of the leader with the
masses without the mediation of traditional political machinery. Our research
is based on publicly available profiles documenting different relationships
types such as political alliances, work relationships, rivalries and family
ties of the main actors of Colombian politics.

The article is organized as follows. In the rest of the Introduction we
provide a general context of Colombian politics in the last two decades and
briefly review the use of networks in the analysis of political problems. We
then describe the dataset used to capture the relationships between different
actor of the Colombian political scene. The data collected is then analyzed
using well known concepts of network analysis such as centrality and community
detection. We show how these concepts relate to political strategies put in
place to consolidate power.

###  1.1 Alvaro Uribe and the Colombian Political Landscape

A former congressman, Governor and Mayor of Medellín, Alvaro Uribe distanced
himself from the Liberal Party because of his hard-line security politics.
Running as an independent in the 2002 presidential elections and winning by a
landslide, Uribe became the first president in Colombian history outside of
the traditional two-party system. Once in office, he used his immense
popularity and political influence to change the constitution and remove the
ban on presidential reelection, thus allowing himself to be reelected for a
second term (2006-2010).

Unable to run for a third term, Uribe supported the candidacy of Juan Manuel
Santos, his former minister of Defense and assumed ally. Santos was elected
for the 2010-2014 term by cashing in Uribe’s electoral force and was re-
elected for the 2014-2018 term on a completely different platform based on
signing a peace treaty with FARC guerrilas. Uribe then became the most vocal
and visible opponent to the peace project and to the Santos administration,
creating a right wing opposition party, Centro Democrático (Democratic Center,
hereafter CD), and being elected as a Senator in the Colombian Congress twice
(2014-2018 and 2018-2022).

The CD now constitutes the majority of the Congress. Colombia’s current
president (Iván Duque, 2018-2022) belongs to the CD and was elected under the
strong support and campaigning of Uribe.

Uribe is one of the most interesting examples of neo-populism as, on the one
hand, he and his communcation style fulfill the commonly accepted
characteristics of a populist leader: charismatic, mobilizes masses appealing
to a direct communication style, presents himself as an outsider of the
traditional politics, promotes an “us against them” adversarial politics that
minimizes the space for democratic deliberation, and promotes the idea of his
manifest destiny as the only possible “saviour” of the country from its
perceived enemies. On the other hand, his status as populist has been
challenged mainly because his origins are strongly rooted in the traditional
Colombian economic and political elite, undermining the “people vs the elites”
opposition that characterizes many populist regimes.

###  1.2 Individual Power Networks

There is a widespread structuralist approach in the social sciences which
defines power as relational rather than inherent to the agent [33], i.e. actor
$A$’s power is defined in relation to his influence over actor $B$. Political
science may then be seen as the study of the relationships that create,
access, use, accumulate and/or preserve power within a society and politics
themselves may be understood fundamentally as a network phenomenon[35]. This
relational approach is slowly gaining foothold in a discipline mostly grounded
on theories that assume independence of actors.

The use of networks as an adaptable tool to describe various relationships
between individuals, as opposed to other collective actors such as states or
political parties, has been adopted by various researchers to describe
political data[50][33]. An important part of this research is based on the
analysis of digital data from social networks and focuses on individuals and
potential voters. The relatively immediate availability of this type of data
has promoted a large corpus of research describing the formation of
communities with similar political points of view, the relationship between
political leader and their followers and predicting the political alignment of
potential voters [13, 12].

The research of networks the inner structures of power wielding political
instituions are rarer, possibly due to the difficulty of accessing descriptive
data [45]. In some cases, relational data may be manually collected from
public information [36, 23] or from direct surveys [4, 29], but the process is
work intensive, slow and difficult to keep up to date [47].

Affiliation networks [7, 34], bipartite graphs in which one set corresponds to
individuals and the other to groups to which the individuals belong, have been
used as an alternative to direct relationships. When available, it is possible
to use groups corresponding to formal and well defined affiliations, e.g.
families [14] or political parties [18]. In other cases, especially in
political systems without a strong culture of information openness, authors
have to use more flexible relation definitions to build the networks, e.g.
alumni-connections of US congressmen [5] or intelligence reports of the
simultaneous presence in public events of Soviet Politicians during the
Breshnev era [17].

Relationships between members of deliberative assemblies, such as parliaments
and legislative bodies, may be uncovered by analyzing the coincidences of
their voting pattern in support of bills [46, 40, 21].

In the following Section we will describe a semi-automatic method to collect
direct relational data (“works with”, “is allied with”) of prominent figures
of recent Colombian history.

##  2 Data

Data was collected from the “Quien es quien” (“Who is Who”) section of “La
Silla Vacia” (“The Empty Chair”), an independent Colombian news and political
journalism website 111http://lasillavacia.com/quienesquien using web scrapping
tools implemented in the Python 3 programming language [48]. The raw data
consisted on 344 individual profiles saved as plain html text webpages.

The webpage of each person in the network includes a brief profile and
hyperlinks to related profiles. There are five types of relationships between
profiles: Work, alliance, friendship, family and rivalry. These relationships
are, in general, non-exclusive, meaning two nodes may be related by more than
one type of edge. The total number of edges by relationship type is presented
in Figure 1.

Figure 1:  Number of edges per relationship type.

It is clear from the figure that the most common type of relationship is
“Work”, with more edges (783) than all the other edge types combined (613),
followed by “Alliance” (381). Although all edge information could be
represented either by a multiplex network, i.e. one in which each
individual/node is connected by different edge/relationships [8], the
difference in size would bias results towards “Work”, the most populated edge
type.

We have opted instead to analyze separately only the “Work” network and
“Alliance” networks (hereafter WN and AN, respectively) as they are the only
edge types with a significant number of edges with respect to the number of
nodes. Additionally, in both of the chosen networks there is a giant connected
component (one in which a path exists between any pair of nodes) connecting
more than half of the networks elements (Figure 2).

Figure 2:  Number of nodes in the largest connected component per relationship
type graph.

##  3 Analysis

###  3.1 Degree

Figure 3 shows the normalized frequencies of node degree (number of edges
connected to each vertex) of WN and AN. Both networks show a very long-tail
and nodes with degrees an order of magnitude above the average,
characteristics of power law distributions [11, 25].

Formally, a distribution is said to follow a power law if the fraction of
nodes of degree $k$ is proportional to an exponential function:

$P(k)\sim k^{-\gamma}.$ (1)

Instances of natural, social and man-made networks spontaneously showing this
behaviour have been extensively reported in the literature[11].

Although there is still no consensus over the exact mechanisms which result in
this type of distribution, preferential attachment, also referred to as a
“Yule Process”[52] or “the rich get richer effect”, has been proposed as a
possible explanation. In this type of processes, new edges added to the
network by a random or partly random process will attach with a higher
probability to nodes with higher degrees in a self reinforcing loop[2].

|
---|---
|
Figure 3:  Frequecies of node degrees for AN (left) and WN (right). The fitted
power law curves using $\gamma$ values of $0.95$ for AN and $0.85$ for WN are
overlain. Bottom row show the graphics in a logarithmic scale.

###  3.2 Centrality Measures

A widely accepted, although loose, definition of a node’s centrality coming
from the study of social networks and organizations states that centrality is
related to the “importance” of the node within the graph [22]. The most
central nodes of AN and WN networks were calculated using four common
measures: Degree, Betweenness, Closeness and Eigenvector Centrality. By all
four metrics Alvaro Uribe is the most central node of the AN and former
President Juan Manuel Santos the most central node of the WN network. Results
are shown in Table 1.

Alliance Network
---
Degree | Betweenness | Closeness | Eigencentrality
AU | 31 | AU | 0.387 | AU | 0.378 | AU | 0.288
GV | 19 | EP | 0.187 | EP | 0.376 | EP | 0.276
Work Network
Degree | Betweenness | Closeness | Eigencentrality
JMS | 113 | JMS | 0.530 | JMS | 0.571 | JMS | 0.489
AU | 57 | AU | 0.203 | AU | 0.466 | AU | 0.196
Table 1: Top centrality nodes of AN and WN according to four metrics. In all
measures Alvaro Uribe (AU) is the most central node of AN and the second most
important of the WN. Former President Juan Manuel Santos (JMS) is the most
important node by all measures of the WN. Other nodes are Germán Vargas Lleras
(GV, Vice President under Juan Manuel Santos)) and Enrique Peñalosa (EP,
former Mayor of Bogota).

####  3.2.1 AN Centrality

It is interesting to note that, on the one hand, there is a large proportional
gap between Alvaro Uribe’s score in Degree and Betweenness measurements and
the second most central nodes.

Degree centrality (DC) is a simple yet effective way of measuring the
importance of a node and corresponds to the number of edges connected to it.

Betweeness centrality (BC) assumes that information travels from node to node
following the shortest path[16]. The BC of a node is defined as:

$BC(i)=\sum_{i}\sum_{j}\frac{\rho(i,j,k)}{\rho(j,k)},i\neq j\neq k$ (2)

where $\rho(j,k)$ is the number of shortest paths connecting nodes $i$ and $k$
and $\rho(j,i,k)$ is the number of shortest paths connecting $j$ and $k$
passing through $i$.

A rapid inspection of the network shows that Alvaro Uribe has many alliances
with low-degree nodes, as showin in Figure 4. Since DC measures the number of
edges connected to a node (independently of the importance of the nodes it
attaches with) and BC measures the number of shortest paths passing through a
given node (again, independently of the importance of the paths), it is
logical that having many connections, even to unimportant nodes, will boost
these scores.

---
Figure 4:  DC histogram for the immediate neighbors of the Alvaro Uribe node
in the AN. Most of the nodes connected to Alvaro Uribe have a very low degree.

On the other hand, Alvaro Uribe’s Eigencentrality score (EC) is only
marginally larger than other nodes. This measure is based on the concept that
being connected to high-scoring nodes will contribute more to the centrality
than being connected to low-scoring nodes, resulting in a more distributed
centrality. The EC of a vertex $v_{i}$ in a graph $G$ is defined as:

$EC(v_{i})={\frac{1}{\lambda}}\sum_{j\in G}a_{i,j}EC(v_{j})$ (3)

where $a_{i,j}$ is the value in row $i$ and column $j$ of $A$, the adjacency
matrix of $G$. One may rewrite Equation 3 as the eigenvector definition:

$A\mathbf{x}=\lambda\mathbf{x}.$ (4)

Variations of EC in which a node’s centrality is determined by the centrality
of its neighbors and being connected to high-ranking nodes increases the
influence in the network, are used to determine the ranking of webpages, e.g.
Google’s Pagerank algorithm [38] or the citation impact of academic articles
[51, 6]. In the political context, it provides a strong incentive for actors
with a limited political power to establish alliances with central nodes such
as Uribe, thus increasing the latter’s influence in a self perpetuating
process consistent with the preferential attachment mechanism.

####  3.2.2 WN Centrality

The WN is dominated by the figure of Juan Manuel Santos, who, as we previously
explained, rose to the Presidency with the support of his former ally Uribe.
Alvaro Uribe criticized Juan Manuel Santos’s peace talk efforts, announced
shortly after being sworn in, and has become one of his fiercest critics.

Besides Juan Manuel Santos and Alvaro Uribe, the top positions of the Work
network are dominated either by former Presidents or by high-ranking ministers
who have served under different Presidents. This, together with the difference
in size of the AN and WN network (Figure 2), tends to confirm the diagnostics
of how, despite the changes stated in the 1991 constitution, Colombia remains
a highly centralized nation with a large state-bureaucracy orbiting, mainly,
around the executive branch of the government.

---
Figure 5:  DC histogram for the immediate neighbors of the Juan Manuel Santos
node in the WN. The highest connected node corresponds to Alvaro Uribe under
whom Juan Manuel Santos served as a Minister of Defense.

The higher centrality of Juan Manuel Santos over Alvaro Uribe in the WN is
also consistent with their public discourses. Whereas Juan Manuel Santos
insisted on the need to work towards peace construction and, consequently,
created bureaucratic positions to negotiate and implement peace accords,
Alvaro Uribe’s discourse and public figure are centered on efficiency, action
over reflection and the reduction of the size of the government, all points
which align with the description of neo-populism given by Fierro [20].

It is interesting to note that one of the main criticisms of Alvaro Uribe and
his party allies against Juan Manuel Santos is the alleged wide-spread use of
pork barrel politics, i.e. the appropriation of government spending for
political gain. The use of key word repetition tactics by the CD on this point
has made the Spanish word for “sweet jam” (“mermelada”) a functional synonym
of pork barrel in Colombia and popularized the use of the expression “spread
the sweet jam” (“repartir mermelada”) to describe this practice [3].

###  3.3 Communities and Network Topology

The complete AN and WN are displayed in Figures 6 and 8, respectively. The
position of the nodes is calculated using a “force based” layout algorithm
[30] which iteratively attracts linked nodes and repels non-attached nodes.

Figure 6: AN graphical representation. Size is proportional to the node degree
and colors correspond to communities detected by the first cut of a Girvan-
Newman algorithm. A clear fracture between the close supporters of Alvaro
Uribe and the rest of the political scene may seen.

####  3.3.1 AN Community Analysis

The most notable characteristic of the AN is how the node corresponding to
Alvaro Uribe, despite having the high centrality values shown in Table 1,
distances itself from most other nodes with which it does not share edges.

The polarizing tendency is confirmed via community analysis using the Girvan-
Newman (GN) algorithm [24]. GN calculates the number of shortest paths between
all nodes passing through each edge and removes the edge with the highest
count. When an edge removal makes the graph disconnected the resulting
connected components are considered communities and the process repeats for
each component. In AN we consider only the graph’s first partition.

The first cut of AN results in two communities containing $69.27\%$ and
$30.73\%$ of the nodes. The smaller community (shown in Figure 6) centers
around Alvaro Uribe and contains many members of his party’s inneokr circle
including Colombia’s current president Ivan Duque.

The algebraic connectivity, also known as the the Fiedler eigenvalue [19, 9],
of a graph $G$ may be used to partition the graph partition. The Fiedler
vector is the eigenvector corresponding to the second smallest eigenvalue of
the Laplacian matrix $L$ of $G$ where

$L=D-A,$ (5)

$A$ corresponds to the adjacency matrix of $G$ and $D$ to the degree matrix of
$A$:

$D_{i,j}:=\left\\{\begin{matrix}deg(v_{i})&\mbox{if}&i=j\\\
0&\mbox{otherwise.}&\end{matrix}\right.$ (6)

The sign (positive or negative) of each value of the Fiedler vector can be
used to partition graph $G$. For the AN this results in a similar separation
with only 5 nodes migrating from the largest to the smallest community. The
proportional number of nodes remain similar resulting in $64.58\%$ and
$35.42\%$ of nodes in each community.

In both the GN and the Fiedler vector partitions, the community containing
Alvaro Uribe also contains the three maximum cliques (subgraphs in which every
node is connected to every other node). The cliques are formed by 5 nodes each
corresponding to current or former members of congress of the CD (including
Alvaro Uribe and current President Ivan Duque). As seen in Figure 7, these
cliques have an overlap of 4 nodes (equivalent to $80\%$ of the clique’s
nodes) between any pair of cliques indicating a strong and well connected
inner network community within the AN.

Figure 7:  There are three maximum cliques consisting of 5 nodes in the AN.
All nodes correspond to members of the CD who are or have been members of the
congress including former president Alvaro Uribe (AU) and current president
Ivan Duque (ID).

####  3.3.2 WN Community Analysis

Unlike the AN, the WN does not present any easily detectable communities. The
network revolves around Juan Manuel Santos and Alvaro Uribe, as predicted by
the centrality values, but there is a dense network of shared vertices (people
who have had work relationships with both) which bind them into one large
community.

The first three cuts of GN correspond to very small marginal communities
($1.51\%$,$1.51\%$ and $1.13\%$ of all the nodes) connected by single edges to
the main component. It then separates a sizeable community ( $8.68\%$ of the
nodes) seen in the right side of Figure 8. This community is formed mostly by
politicians and technocrats who worked with Gustavo Petro, a prominent left-
wing opposition leader, during his tenure as Bogota Mayor (2012-2015).

Figure 8: WN graphical representation. Node colors correspond to the first
five communities detected by the Girvan-Newman algorithm.

The WN has 5 maximum cliques containing 6 nodes each. The cliques form two
distinct communities bridged by Juan Manuel Santos who is a member of 4 of
them (see Figure 9). The communities correspond to two distinct time periods:
the negotiating team for the Havana Peace Accords (2012-2016) and members of
former President Cesar Gaviria’s cabinet (1990-1994). Unlike the maximal
cliques of the AN, the WN is formed by representatives of various parties and
sectors.

Figure 9:  There are 5 maximum cliques consisting of 6 nodes in the WN. Two
communities bridged by former President Juan Manuel Santos (JMS) may be seen:
A community formed by members of the Havana Peace Accords negotiating teams
(left) and a clique formed by members of the cabinet of former President Cesar
Gaviria (CG, right).

Whereas AN is formed by two easily separable communities, WN forms a tightly
knit network without any strong separations. This would lead to think that,
unlike the political arena, in a a state as large and centralized as Colombia
it is necessary through time to draw people from all political tendencies to
keep the machinery moving.

##  4 Conclusions

A network-based approach to populism is necessary because it brings nuance to
both the overly structural analysis and to the pairing of influential
leader/faithful followers that so often is the backbone of the studies in
populism. The massive popular support of a populist leader becomes the social
force that allows eroding democratic institutions and mechanisms under his
will, which is the most notorious danger of populism. That explains the
interest in understanding both the leader’s followers, and the leader’s means
to obtain support, by analyzing voting dispositions, discourse, ideology, and
even personality traits in both voters and the leader. Additionally, studies
of populist regimes also seek for explanations in structural terms: what are
the social and economic conditions that allow a populist movement to thrive.
These are all fundamental questions. However, they oscillate between a micro
and a macro level. The network approach allows us to address the meso level by
rendering visible the paths that connect the leader and the voters. These
paths are made of politicians, advisors, and high-rank State officials that
cement and increase or erode and diminish a populist leader’s power.

A characteristic of populist leaders is their presentation as outsiders.
Uribe, launching his presidential campaign as an independent candidate, was
not an exception. But just because someone presents him or herself as a
political outsider does not mean that it is not playing the political game of
forging alliances. Mapping the political landscape of Colombia allowed us to
see the extent of Uribe’s partnerships and the uniqueness of his connections
in comparison to other former Presidents. They are not nearly as interlinked
as him and correspondingly their political presence is more marginal. Populist
leaders need not only charisma, discourse, or ideology to get and maintain
political power but some good old networking.

##  5 Future research

The data provided by the news website takes into account the most notable
figures in Colombian politics based on their significance in a given news
cycle, which makes it necessary to explain to the readers who the person is.
Therefore, it is essential to build a database that includes not only the
notable figures but also the ones that are less visible and equally powerful.
This database would require collecting information about all the members both
from the Congress and from the top Government positions. Additionally, the
network approach to populism would benefit from tracing the changes in the
alliances throughout the emergence and decline of a populist regime. Our long-
term goal is to use the network morphology to create a populism indicator by
strengthening the database and incorporating a historical perspective in the
analysis.

## References

  * [1] Agnes Akkerman, Cas Mudde, and Andrej Zaslove.  How populist are the people? measuring populist attitudes in voters.  Comparative political studies, 47(9):1324–1353, 2014.
  * [2] Réka Albert and Albert-László Barabási.  Statistical mechanics of complex networks.  Reviews of modern physics, 74(1):47, 2002.
  * [3] Camila Angulo Amaya, Anthony M Bertelli, and Eleanor Woodhouse.  The political cost of public-private partnerships: Theory and evidence from colombian infrastructure development.  Forthcoming, Governance, 2019.
  * [4] Mahnoor Asif and Azam Chaudhry.  Connections and elections in lahore: How network centrality affects electoral politics in pakistan.  2016\.
  * [5] Marco Battaglini, Valerio Leone Sciabolazza, and Eleonora Patacchini.  Effectiveness of connected legislators.  American Journal of Political Science, 2019.
  * [6] Johan Bollen, Marko A Rodriquez, and Herbert Van de Sompel.  Journal status.  Scientometrics, 69(3):669–687, 2006.
  * [7] Ronald L Breiger.  The duality of persons and groups.  Social forces, 53(2):181–190, 1974.
  * [8] Ronald L Breiger and Philippa E Pattison.  Cumulated social roles: The duality of persons and their algebras.  Social networks, 8(3):215–256, 1986.
  * [9] Fan RK Chung and Fan Chung Graham.  Spectral graph theory.  Number 92. American Mathematical Soc., 1997.
  * [10] Angélica Abad Cisneros.  El voto populista en la región andina: los casos de colombia, ecuador y perú/the populist vote in the andean region: the cases of colombia, ecuador and peru.  Revista CIDOB d’Afers Internacionals, (119):209–244, 2018.
  * [11] Aaron Clauset, Cosma Rohilla Shalizi, and Mark EJ Newman.  Power-law distributions in empirical data.  SIAM review, 51(4):661–703, 2009.
  * [12] Elanor Colleoni, Alessandro Rozza, and Adam Arvidsson.  Echo chamber or public sphere? predicting political orientation and measuring political homophily in twitter using big data.  Journal of communication, 64(2):317–332, 2014.
  * [13] Michael D Conover, Bruno Gonçalves, Jacob Ratkiewicz, Alessandro Flammini, and Filippo Menczer.  Predicting the political alignment of twitter users.  In 2011 IEEE third international conference on privacy, security, risk and trust and 2011 IEEE third international conference on social computing, pages 192–199. IEEE, 2011.
  * [14] Cesi Cruz, Julien Labonne, and Pablo Querubin.  Politician family networks and electoral outcomes: Evidence from the philippines.  American Economic Review, 107(10):3006–37, 2017.
  * [15] Carlos De la Torre and Federico Finchelstein.  Democraduras?: Venezuela and national-populism in latin america.  Democraduras?: Venezuela and National-Populism in Latin America, pages 147–160, 2018.
  * [16] Ernesto Estrada and Philip A Knight.  A first course in network theory.  Oxford University Press, USA, 2015.
  * [17] Katherine Faust, Karin E Willert, David D Rowlee, and John Skvoretz.  Scaling and statistical models for affiliation networks: patterns of participation among soviet politicians during the brezhnev era.  Social networks, 24(3):231–259, 2002.
  * [18] Josemar Faustino, Hugo Barbosa, Eraldo Ribeiro, and Ronaldo Menezes.  A data-driven network approach for characterization of political parties’ ideology dynamics.  Applied Network Science, 4(1):48, 2019.
  * [19] Miroslav Fiedler.  Laplacian of graphs and algebraic connectivity.  Banach Center Publications, 25(1):57–70, 1989.
  * [20] Marta I Fierro.  Alvaro uribe populism and neo-populism.  Análisis político, 27(81):127–147, 2014.
  * [21] Manuel Fischer, Frédéric Varone, Roy Gava, and Pascal Sciarini.  How mps ties to interest groups matter for legislative co-sponsorship.  Social Networks, 57:34–42, 2019.
  * [22] Linton C Freeman.  Centrality in social networks conceptual clarification.  Social networks, 1(3):215–239, 1978.
  * [23] Jorge Gil-Mendieta and Samuel Schmidt.  The political network in mexico.  Social Networks, 18(4):355–381, 1996.
  * [24] Michelle Girvan and Mark EJ Newman.  Community structure in social and biological networks.  Proceedings of the national academy of sciences, 99(12):7821–7826, 2002.
  * [25] Taichi Haruna and Yukio-Pegio Gunji.  ordinal preferential attachment: A self-organizing principle generating dense scale-free networks.  Scientific Reports, 9(1):1–8, 2019.
  * [26] Kirk A Hawkins, Rosario Aguilar, Bruno Castanho Silva, Erin K Jenne, Bojana Kocijan, and Cristóbal Rovira Kaltwasser.  Measuring populist discourse: The global populism database.  Presentado en el 2019 en EPSA Annual Conference en Belfast, Reino Unido, June, 2019.
  * [27] Kirk A Hawkins and Cristobal Rovira Kaltwasser.  What the (ideational) study of populism can teach us, and what it can’t.  Swiss Political Science Review, 23(4):526–542, 2017.
  * [28] Kirk Andrew Hawkins, Scott Riding, and Cas Mudde.  Measuring populist attitudes.  2012\.
  * [29] Narisong Huhe, Daniel Naurin, and Robert Thomson.  The evolution of political networks: Evidence from the council of the european union.  European Union Politics, 19(1):25–51, 2018.
  * [30] Mathieu Jacomy, Sebastien Heymann, Tommaso Venturini, and Mathieu Bastian.  Forceatlas2, a continuous graph layout algorithm for handy network visualization.  Medialab center of research, 560:4, 2011.
  * [31] Jan Jagers and Stefaan Walgrave.  Populism as political communication style: An empirical study of political parties’ discourse in belgium.  European journal of political research, 46(3):319–345, 2007.
  * [32] Cristóbal Rovira Kaltwasser, Paul A Taggart, Paulina Ochoa Espejo, and Pierre Ostiguy.  The Oxford handbook of populism.  Oxford University Press, 2017.
  * [33] David Knoke.  Political networks: the structural perspective, volume 4.  Cambridge University Press, 1994.
  * [34] Silvio Lattanzi and D Sivakumar.  Affiliation networks.  In Proceedings of the forty-first annual ACM symposium on Theory of computing, pages 427–434, 2009.
  * [35] Scott D McClurg and David Lazer.  Political networks.  Social Networks, (36):1–4, 2014.
  * [36] Nuno Moniz, Francisco Louçã, Márcia Oliveira, and Renato Soeiro.  Empirical analysis of the portuguese governments social network.  Social Network Analysis and Mining, 6(1):43, 2016.
  * [37] Carlo Nasi.  Derechización ‘a la colombiana’en tiempos confusos: un ensayo especulativo.  Colombia internacional, (66):162–183, 2007.
  * [38] Lawrence Page, Sergey Brin, Rajeev Motwani, and Terry Winograd.  The pagerank citation ranking: Bringing order to the web.  Technical report, Stanford InfoLab, 1999.
  * [39] Takis S Pappas.  Modern populism: Research advances, conceptual and methodological pitfalls, and the minimal definition.  In Oxford research encyclopedia of politics. 2016.
  * [40] Paolo Parigi and Laura Sartori.  The political party as a network of cleavages: Disclosing the inner structure of italian political parties in the seventies.  Social Networks, 36:54–65, 2014.
  * [41] Teun Pauwels.  Measuring populism: A quantitative text analysis of party literature in belgium.  Journal of Elections, Public Opinion and Parties, 21(1):97–119, 2011.
  * [42] Roel Popping.  Measuring populist discourse using semantic text analysis: a comment.  Quality & quantity, 52(5):2163–2172, 2018.
  * [43] Kenneth M Roberts.  Populism and political parties.  In The Oxford handbook of populism. 2017.
  * [44] Matthijs Rooduijn.  What unites the voter bases of populist parties? comparing the electorates of 15 populist parties.  European Political Science Review, 10(3):351–368, 2018.
  * [45] David A Siegel.  Social networks in comparative perspective.  PS: Political Science and Politics, 44(1):51–54, 2011.
  * [46] Mirko Signorelli and Ernst C. Wit.  A penalized inference approach to stochastic block modelling of community structure in the italian parliament.  Journal of the Royal Statistical Society: Series C (Applied Statistics), 67(2):355–369, Jul 2017.
  * [47] Mariela Szwarcberg.  Revisiting clientelism: A network analysis of problem-solving networks in argentina.  Social Networks, 34(2):230–240, 2012.
  * [48] Guido Van Rossum and Fred L. Drake.  Python 3 Reference Manual.  CreateSpace, Scotts Valley, CA, 2009.
  * [49] Jennifer Nicoll Victor, Alexander H Montgomery, and Mark Lubell.  Introduction: The emergence of the study of networks in politics.  The oxford handbook of political networks, pages 3–58, 2017.
  * [50] J.N. Victor, A.H. Montgomery, and M. Lubell.  The Oxford Handbook of Political Networks.  Oxford handbooks online : Political science. Oxford University Press, 2016\.
  * [51] Erjia Yan and Ying Ding.  Applying centrality measures to impact analysis: A coauthorship network analysis.  Journal of the American Society for Information Science and Technology, 60(10):2107–2118, 2009.
  * [52] George Udny Yule.  Ii.—a mathematical theory of evolution, based on the conclusions of dr. jc willis, fr s.  Philosophical transactions of the Royal Society of London. Series B, containing papers of a biological character, 213(402-410):21–87, 1925\.


# SD-HuBERT: Self-Distillation Induces Syllabic Organization in HuBERT

###### Abstract

Data-driven unit discovery in self-supervised learning (SSL) of speech has
embarked on a new era of spoken language processing. Yet, the discovered units
often remain in phonetic space, limiting the utility of SSL representations.
Here, we demonstrate that a syllabic organization emerges in learning
sentence-level representation of speech. In particular, we adopt “self-
distillation” objective to fine-tune the pretrained HuBERT with an aggregator
token that summarizes the entire sentence. Without any supervision, the
resulting model draws definite boundaries in speech, and the representations
across frames show salient syllabic structures. We demonstrate that this
emergent structure largely corresponds to the ground truth syllables.
Furthermore, we propose a new benchmark task, Spoken Speech ABX, for
evaluating sentence-level representation of speech. When compared to previous
models, our model outperforms in both unsupervised syllable discovery and
learning sentence-level representation. Together, we demonstrate that the
self-distillation of HuBERT gives rise to syllabic organization without
relying on external labels or modalities, and potentially provides novel data-
driven units for spoken language modeling.

Index Terms—  Self-Supervised Learning; Unsupervised Unit Discovery; Spoken
Language Understanding;

##  1 Introduction

Self-supervised learning (SSL) of speech has been extremely successful in
learning rich representations of speech which are transferable to many
downstream tasks [1, 2]. In particular, discrete units discovered by internal
clustering of SSL models have been actively utilized for various domains,
including spoken language modeling (“text-less NLP”) [3, 4, 5] and speech
synthesis [6, 7]. Though such units are often referred to as “semantic units”,
they are highly correlated with phonetic information [8, 9], being fine-
grained subphonemic units. Therefore, linguistic components beyond phonemes
would not be well-captured by the previously discovered units, though such
higher-order components are implicitly inferred by speech SSL models [10].

Discovering true semantic units is challenging since the relationships between
phonetics and semantics are highly non-linear. For example, the word “orange”
is closer to “apple” than “strange” in semantic space, but the relationship is
opposite in phonetic space. Due to this non-linearity, phonetics and semantics
can not be effectively disentangled by a naive clustering method. Moreover,
the transition from phonetics to semantics and beyond is language-specific,
while semantics are universal foundations of meaning. Therefore, disentangling
higher-order linguistic units from speech models is critical to building a
more generalizable spoken language system.

Here, we propose to induce higher-order structure by fine-tuning sentence-
level representation of HuBERT. The self-distillation approach [11] is adopted
to fine-tune HuBERT without relying on any label or external modality,
resulting Self-Distilled HuBERT (SD-HuBERT). We demonstrate that a syllabic
organization emerges in SD-HuBERT representation with salient boundaries,
which allows the deployment of an efficient syllable discovery algorithm.

Fig. 1: Diagram of the model architecture and self-distillation framework. An
aggregator token (Agg) is inserted to summarize the entire input speech.

We hypothesize that the emergent properties are driven by the enhanced
semantic representation of the model promoted by sentence-level learning. To
verify this hypothesis, we propose a new benchmark, Spoken Sentence ABX
(SSABX), for evaluating the semantic discriminability of the models on spoken
sentences. The SSABX task is performed by comparing similarities between
sentence-level embeddings, which does not require any fine-tuning. To the best
of our knowledge, this is the first benchmark task that can easily evaluate
the sentence-level representation of speech models. The proposed model shows a
higher SSABX accuracy than the baseline models including some representative
speech models.

Fig. 2: Frame similarity matrices for ”intelligence in others” from the 9th
layer of HuBERT before fine-tuning, and the 9th and 11th layers after fine-
tuning. The similarity is measured by dot product. The white dotted lines
denote the ground truth syllable boundaries and the red dotted lines are the
predicted boundaries. The syllables become clearly visible in SD-HuBERT after
self-distillation. The frames are knocked out in the 11th layer of SD-HuBERT,
drawing definite boundaries.

Our major contributions are:

  * •

We propose a sentence-level speech representational model, SD-HuBERT, by fine-
tuning pretrained HuBERT with the self-distillation objective.

  * •

We demonstrate that syllabic organization emerges in SD-HuBERT, and the model
outperforms the baseline models in both syllable boundary detection and
syllabic unit discovery.

  * •

SD-HuBERT infers definite sub-word boundaries by knocking out the boundary
frames, which can be utilized to speed up the previous segmentation algorithm.

  * •

We propose a new benchmark task, Spoken Sentence ABX (SSABX), for simple,
tuning-free evaluation of the sentence-level discriminability of speech
models.

  * •

When evaluated on the SSABX task, SD-HuBERT outperforms previous speech SSL
models by a large margin.

##  2 Related Work

The speech processing community has long sought to discover linguistic units
from speech audio. Diverse unsupervised methods have been proposed to find
word boundaries and lexical semantic embeddings in speech [12, 13, 14, 15],
and recent speech SSL models have been actively leveraged to discover phonetic
units from speech [16, 17]. However, most of the previous works have been
focused on lexical or phonetic units, and the unsupervised discovery of
syllabic units remains underexplored. This leaves a gap in the transition from
phonetics to lexical semantics in the speech hierarchy.

Some studies have suggested that visually grounding speech representation can
reveal lexical structure in speech. They have demonstrated that words can be
discovered from raw speech by learning the linkage between spoken words and
their visual entities [18, 19, 20]. Furthermore, Peng et al. [21] suggest that
syllabic structure emerges from visually grounded HuBERT (VG-HuBERT) [20].
However, we claim that cross-modal grounding is not necessary for such
emergent property, and rather, the key is the emphasis on semantic information
in the model over phonetic information, which can be achieved by learning
sentence-level representation. Indeed, a similar emergent behavior is reported
in a single-modality model in the image domain [11]. We empirically
demonstrate this claim by fine-tuning HuBERT on sentence-level embedding,
using only speech data.

Several studies utilize an aggregator token that can condense information of
the entire input sequence into a single embedding [22, 23, 24, 20]. Still,
most of the previous studies for speech rely heavily on cross-modal grounding,
and such approaches in a single modality still need to be explored in the
speech domain. Given that text or image labels of speech are expensive to
collect, the text-less, speech-only model should be developed.

##  3 Methods

###  3.1 Sentence-Level Fine-Tuning of HuBERT

Our approach is to fine-tune a pre-trained speech SSL model, HuBERT [16], with
a sentence-level self-supervised learning objective.111We use the base model
with 90M parameters. The model is composed of a CNN feature extractor followed
by a Transformer encoder. To attain a sentence-level representation, an
aggregator token with learnable embedding is concatenated to the inputs to the
Transformer encoder [20, 23, 24].222This token is often named [CLS] token in
computer vision and NLP. The aggregator token aggregates information across
the frames into a single, representative embedding of the entire audio input.
The final output of the aggregator token is passed to non-linear mapping and
softmax function to parametrize the probability of the given spoken sentence.
This probability is denoted as $p_{\theta}(\cdot)$ where $\theta$ is the model
weights.

Self-distillation We follow the self-distillation framework suggested by Caron
et al. [11]. This framework distills the student model,
$p_{\theta_{S}}(\cdot)$, to the teacher model, $p_{\theta_{T}}(\cdot)$, in
which $\theta_{T}$ is an exponential moving average (EMA) of $\theta_{S}$. For
the student model, random data augmentation, $\tau(\cdot)$, is applied to the
output frames of the feature extractor, which is either random masking or
random time warping of the frames [25]. The masked frames are replaced with a
learnable mask token. The model minimizes the cross-entropy of the
probabilities inferred from the aggregator token, using the teacher inference
as a target reference: $\sum_{x\in
X}-p_{\theta_{T}}(x)\>\text{log}(p_{\theta_{S}}(\tau(x)))$. As suggested in
[11], recentering teacher output and stop gradient are applied to prevent
degeneration. We name the resulting model SD-HuBERT (Self-distilled HuBERT).

The model weights are initialized with the weights from the official
checkpoint of HuBERT, which is trained on 960 hours of English speech from
LibriSpeech data [26]. We reinitialize the last three layers of the
Transformer encoder with random weights. As the data augmentation is applied
after the feature extractor, we freeze the feature extractor and positional
encoding layer.333Otherwise, the model adapts to the data augmentation and
degenerates.

LibriSpeech [26] is used for training the model and five-second windows are
sampled as training inputs. We set the decay rate of EMA as 0.999. AdamW [27]
is utilized for the optimizer with the batch size of 100 for 200K iterations.
The learning rate starts with 1e-4 and decays to 1e-5 by the Cosine learning
rate schedule.

###  3.2 Unsupervised Syllable Discovery

As shown in the frame similarity matrices in Fig. 2, the proposed self-
distillation shapes the embedding space with interesting topology, which
roughly corresponds to the syllabic structure in the input speech. To test
this hypothesis, we apply an unsupervised syllable discovery framework
suggested by Peng et al. [21].

In addition to the emergent syllabic organization, the later layers of SD-
HuBERT infer definite boundaries within speech (Fig. 2, SD-HuBERT-L11). The
frames near the boundaries are knocked out in the last randomly initialized
layers, which is most salient in the 11th layer.444This is not observable when
we remove the random reinitialization. Leveraging these signals, speech can be
easily segmented by thresholding the frame norm with a constant value (“Norm
thresholding” in Fig. 2). However, the resulting segments are not yet
syllables. In the example of speaking “intelligence in others” (Fig. 2), the
norm thresholding assigns a single segment for “T-EH” and “L-AH”. The same
issue happens with “AH” and “DH-ER-Z” in the later frames. Therefore, these
segments may span more than one syllable, thus we applied the minimum cut
algorithm [21] to refine each segment.555In general, the number of syllables
found in one segment is at most three.

This novel emergent behavior of SD-HuBERT provides a first cut of segmentation
for free, reducing search space by a large margin. The original method has
$O(kN^{2})$ time complexity where $k$, $N$ is the number of syllables and
frames, respectively. As the norm thresholding divides the frames by the
number of syllables asymptotically, our model can reduce time complexity down
to $O(N^{2}/k)$.

Other than the segmentation algorithm, the rest of the procedure largely
follows Peng et al. [21]. To evaluate the detected syllable boundaries, we
measured precision (Pr), recall (Re), F1, and R scores with the 50 ms
tolerance window. Although the ground truth syllable boundaries are seamlessly
annotated, the predicted boundaries are not due to the knocked-out frames
(Fig. 2). Therefore, we use the front onsets of the segments as the detected
boundaries.

The segments are clustered into a set of data-driven syllabic units, to
evaluate correspondence to the ground truth syllables. First of all, the frame
embeddings within the segment are averaged to get segment-level embedding.
Then, we apply the two steps of clustering by initially assigning a large
number of clusters (# = 16384) and then merging clusters by agglomerative
clustering on the cluster centers (# = 16384 $\rightarrow$ 4096). The quality
of clustering is evaluated by syllable purity (SP) and cluster purity (CP).
The same Hungarian matching algorithm is leveraged to find matching pairs with
the ground truth syllables, maximizing the temporal intersection-of-union.

We use LibriSpeech for the evaluation where the ground truth labels are
obtained by Montreal Forced Alignment [28] and the syllabification of the
transcribed texts.

###  3.3 Spoken Sentence ABX (SSABX)

Inspired by Semantic Textual Similarity (STS) tasks in NLP [29], we propose a
new benchmark task by carefully mining triplets from the LibriSpeech test set.
Unlike the multi-categorical rating in STS, we design a semantic ABX task to
focus on semantic discriminability of the model. First of all, each audio in
the LibriSpeech test set is segmented into smaller pieces of sentences by
cutting silence. Then, we leverage a pretrained sentence-level text embedding
model, SimCSE [22], to extract the ground truth semantic embedding of the
transcribed texts. The similarity between two sentences is measured by cosine
similarity of their sentence embeddings, and a pair with higher similarity is
regarded as the positive pair in an ABX triplet. We carefully designed the
following criteria for curating the benchmark test set of $(X,Pos,Neg)$
triplets.

  * •

The matching condition of the positive pair has cosine similarity higher than
or equal to 0.8 in text sentence embedding, i.e.,
$\text{cosine\\_sim}(X,Pos)\geq 0.8$.

  * •

To balance the difficulty of the ABX task, the range of similarity of negative
samples is divided into three groups, [-1, 0.2], [0.2, 0.4], [0.4, 0.6], and
1K samples are sampled for each group.

  * •

The difference in the number of words between $X$ and $Pos$, and $X$ and $Neg$
is less than four words.

  * •

Every speech in the triplet is from different speakers.

  * •

To prevent making decisions based on acoustic or phonetic similarity, we
rejected samples with a high Levenshtein similarity ratio ($>$ 0.7) on the
text between $X$ and $Pos$.

  * •

Each sentence has at least five words and the speech does not exceed five
seconds.

The final test set includes 3K triplets of spoken sentences.666We will open
source this task and codes for generating the samples. An example triplet is:
– $X$: “and must have locked the door when you went out”
– $Pos$: “She found the door but it was locked outside”
– $Neg$: “and the horse a going like a house afire too”

##  4 Results

###  4.1 Evaluation on Syllable Boundaries and Clustering

For syllable discovery, we compare the proposed SD-HuBERT with HuBERT and VG-
HuBERT. The 11-th layer of SD-HuBERT is used for norm thresholding and the
9-th layer is used for the minimum cut algorithm. For HuBERT, we use the 9-th
layer since that layer encodes semantic information the most (Fig. 3 bottom),
and for VG-HuBERT, we followed the exact configuration using the checkpoint
released by the authors.777https://github.com/jasonppy/syllable-discovery.git
Table 1 compares the syllable boundaries and clustering scores by SD-HuBERT
and the baseline models. As shown in the table, SD-HuBERT outperforms the
baselines in all evaluation metrics. Moreover, the time complexity of the
segmentation of the proposed method is much faster than the baselines. For a
typical sentence with 25-30 syllables, our method can boost up to several
hundred times compared to the previous method. This is even faster without the
minimum cut algorithm with the time complexity of $O(k)$, where the precision
is slightly better. This indicates that the sentence-level fine-tuning of
HuBERT can induce syllabic organization in speech, without any supervision or
cross-modal grounding.

Table 1: Performance of syllable boundary detection and clustering by different models [%]. TC is time complexity. Model | Pr | Re | F1 | R | SP | CP | TC
---|---|---|---|---|---|---|---
HuBERT | 47 | 27 | 35 | 47 | 28 | 30 | $O(kN^{2})$
VG-HuBERT | 61 | 62 | 61 | 67 | 53 | 43 | $O(kN^{2})$
SD-HuBERT | 64 | 70 | 67 | 70 | 54 | 46 | $O(N^{2}/k)$
– mincut | 66 | 57 | 60 | 66 | 38 | 43 | $O(k)$

###  4.2 Evaluation on Sentence-level Speech Embedding

To evaluate the proposed SSABX task, the sentence-level embeddings are
extracted from the models in comparison, in which some representative speech
SSL models are included [30, 31, 32]. For VG-HuBERT and SD-HuBERT, the
aggregator token embeddings are utilized to extract sentence-level embedding,
and for the models without such tokens, embeddings are averaged across frames
(“average pooling”). For Wav2Vec2 and HuBERT, the base models are tested. We
test every layer in the models and the scores from the best layers are
reported.

Table 2 shows the performance of the different models. When compared to other
speech models, SD-HuBERT outperforms by a large margin, achieving 90%
accuracy. The accuracy significantly drops in VG-HuBERT, indicating that
visual grounding would harm the semantic representation of speech. This is
because the visual grounding actually limits the semantic space because not
all semantic concepts have visual entities (e.g., abstract concepts like
“love”). Among the baselines, HuBERT and its extension, WavLM, show higher
scores than others. Since Wav2Vec2 and XLS-R share the same training
objective, this failure would be due to Wav2Vec2 style training objective. The
major difference from HuBERT is extracting target units from the feature
extractor outputs, while HuBERT uses units from the Transformer encoder. This
discrepancy of targets in the information hierarchy may affect the SSABX
performance.

Without reinitializing the last three layers, the score gets slightly better
(90 $\rightarrow$ 91, “– re-init” in Table 2). However, the model fails
severely with all Transformer layers reinitialized randomly, showing a score
even below the chance level (“+ all-re-init” in Table 2). This suggests that
the initial starting point as pretrained HuBERT is critical to train the model
properly. Furthermore, the significant gap from the text-based models suggests
potential room for improvement.

Table 2: Performance of the SSABX task by different models [%]. Modality | Model | Acc
---|---|---
Text | SimCSE | 100
GloVe | 97
Speech |  $\text{Wav2Vec2}_{\text{}}$ [30] | 74
$\text{HuBERT}_{\text{}}$ | 84
WavLM [32] | 87
XLS-R [31] | 63
VG-HuBERT | 77
SD-HuBERT | 90
– re-init | 91
\+ all-re-init | 46

###  4.3 Why does syllabic structure emerge in SD-HuBERT?

Syllable is the closest element of speech to morpheme which is by definition
the minimal orthographical unit of meaning, though it is originally defined
from a phonological perspective. Therefore, learning the semantic
representation of speech naturally encourages the model to be aware of the
syllabic structure. From the SSABX evaluation, we demonstrate that learning
sentence-level representation effectively captures the semantic information of
speech, discriminating semantically different sentences. Moreover, such
semantic information is readily present in the original HuBERT, but entangled
with phonetic, or articulatory, information [33, 34]. Here, the sentence-level
self-distillation disentangles out the semantics by suppressing phonetic
information and enhancing semantic information. Indeed, the layer-wise
analysis reveals that the articulatory information diminishes in the later
layer after the fine-tuning, while the semantic ABX score increases (Fig. 3).
This mechanism makes the syllabic structure more emphasized than articulatory
information, which is well-visualized in the frame similarity matrix (Fig. 2).

Fig. 3: Layer-wise analysis of articulatory correlation [34] (top) and SSABX
performance (bottom) of HuBERT (orange) and SD-HuBERT (blue).

##  5 Discussion

By fine-tuning HuBERT with self-distillation, a syllabic organization emerges
without any supervision or relying on cross-modal reference. Furthermore, the
new benchmark task validates the superior semantic discriminability of the
proposed model.

Our novel, data-driven syllables can potentially serve as new input tokens to
the text-less NLP models. By allowing the generative models to operate on
phonetics and semantics separately, we may achieve better controllability,
interpretability, and cross-lingual generalizability for spoken language
models.

In this paper, we only analyzed syllable structures, but other linguistic
components in speech hierarchy, such as lexical semantics or syntactic
structures, are likely to be encoded in the model, as demonstrated in text-
based language models [35]. Our work made an important step toward revealing
how information hierarchy can be naturally built upon spoken language.

##  6 Acknowledgements

This research is supported by the following grants to PI Anumanchipalli — NSF
award 2106928, BAIR Commons-Meta AI Research, the Rose Hills Innovator
Program, and UC Noyce Initiative, at UC Berkeley.

## References

  * [1] Shu-wen Yang, Po-Han Chi, Chuang, et al.,  “Superb: Speech processing universal performance benchmark,”  arXiv preprint arXiv:2105.01051, 2021.
  * [2] Abdelrahman Mohamed, Hung-yi Lee, Lasse Borgholt, et al.,  “Self-supervised speech representation learning: A review,”  arXiv preprint arXiv:2205.10643, 2022.
  * [3] Kushal Lakhotia, Eugene Kharitonov, Wei-Ning Hsu, Yossi Adi, Adam Polyak, Benjamin Bolte, Tu-Anh Nguyen, Jade Copet, Alexei Baevski, Abdelrahman Mohamed, et al.,  “On generative spoken language modeling from raw audio,”  Transactions of the Association for Computational Linguistics, vol. 9, pp. 1336–1354, 2021.
  * [4] Eugene Kharitonov, Ann Lee, Polyak, et al.,  “Text-free prosody-aware generative spoken language modeling,”  arXiv preprint arXiv:2109.03264, 2021.
  * [5] Zalán Borsos, Raphaël Marinier, Damien Vincent, Eugene Kharitonov, et al.,  “Audiolm: a language modeling approach to audio generation,”  IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2023.
  * [6] Chengyi Wang, Sanyuan Chen, Yu Wu, et al.,  “Neural codec language models are zero-shot text to speech synthesizers,”  arXiv preprint arXiv:2301.02111, 2023.
  * [7] Adam Polyak, Yossi Adi, Jade Copet, Eugene Kharitonov, Kushal Lakhotia, Wei-Ning Hsu, Abdelrahman Mohamed, and Emmanuel Dupoux,  “Speech resynthesis from discrete disentangled self-supervised representations,”  arXiv preprint arXiv:2104.00355, 2021.
  * [8] Amitay Sicherman and Yossi Adi,  “Analysing discrete self supervised speech representation for spoken language modeling,”  in ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023, pp. 1–5.
  * [9] Badr M Abdullah, Mohammed Maqsood Shaik, Bernd Möbius, and Dietrich Klakow,  “An information-theoretic analysis of self-supervised discrete representations of speech,”  arXiv preprint arXiv:2306.02405, 2023.
  * [10] Ankita Pasad, Chung-Ming Chien, Shane Settle, and Karen Livescu,  “What do self-supervised speech models know about words?,”  arXiv preprint arXiv:2307.00162, 2023.
  * [11] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin,  “Emerging properties in self-supervised vision transformers,”  in Proceedings of the IEEE/CVF international conference on computer vision, 2021, pp. 9650–9660.
  * [12] Alex S Park and James R Glass,  “Unsupervised pattern discovery in speech,”  IEEE Transactions on Audio, Speech, and Language Processing, vol. 16, no. 1, pp. 186–197, 2007.
  * [13] Chia-ying Lee, Timothy J O’donnell, and James Glass,  “Unsupervised lexicon discovery from acoustic input,”  Transactions of the Association for Computational Linguistics, vol. 3, pp. 389–403, 2015.
  * [14] Herman Kamper, Karen Livescu, and Sharon Goldwater,  “An embedded segmental k-means model for unsupervised segmentation and clustering of speech,”  in 2017 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU). IEEE, 2017, pp. 719–726.
  * [15] Herman Kamper,  “Truly unsupervised acoustic word embeddings using weak top-down constraints in encoder-decoder models,”  in ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2019, pp. 6535–3539.
  * [16] Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed,  “Hubert: Self-supervised speech representation learning by masked prediction of hidden units,”  IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 29, pp. 3451–3460, 2021.
  * [17] Benjamin Van Niekerk, Leanne Nortje, and Herman Kamper,  “Vector-quantized neural networks for acoustic unit discovery in the zerospeech 2020 challenge,”  arXiv preprint arXiv:2005.09409, 2020.
  * [18] David Harwath, Adria Recasens, Dídac Surís, Galen Chuang, Antonio Torralba, and James Glass,  “Jointly discovering visual objects and spoken words from raw sensory input,”  in Proceedings of the European conference on computer vision (ECCV), 2018, pp. 649–665.
  * [19] David Harwath, Wei-Ning Hsu, and James Glass,  “Learning hierarchical discrete linguistic units from visually-grounded speech,”  arXiv preprint arXiv:1911.09602, 2019.
  * [20] Puyuan Peng and David Harwath,  “Word discovery in visually grounded, self-supervised speech models,”  in Interspeech, 2022.
  * [21] Puyuan Peng, Shang-Wen Li, Okko Räsänen, Abdelrahman Mohamed, and David Harwath,  “Syllable segmentation and cross-lingual generalization in a visually grounded, self-supervised speech model,”  in Interspeech, 2023.
  * [22] Tianyu Gao, Xingcheng Yao, and Danqi Chen,  “Simcse: Simple contrastive learning of sentence embeddings,”  arXiv preprint arXiv:2104.08821, 2021.
  * [23] Paul-Ambroise Duquenne, Hongyu Gong, and Holger Schwenk,  “Multimodal and multilingual embeddings for large-scale speech mining,”  Advances in Neural Information Processing Systems, vol. 34, pp. 15748–15761, 2021.
  * [24] Paul-Ambroise Duquenne, Holger Schwenk, and Benoît Sagot,  “Sentence-level multimodal and language-agnostic representations,”  arXiv preprint arXiv:2308.11466, 2023.
  * [25] Daniel S Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin D Cubuk, and Quoc V Le,  “Specaugment: A simple data augmentation method for automatic speech recognition,”  arXiv preprint arXiv:1904.08779, 2019.
  * [26] Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur,  “Librispeech: an asr corpus based on public domain audio books,”  in 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE, 2015, pp. 5206–5210.
  * [27] Ilya Loshchilov and Frank Hutter,  “Decoupled weight decay regularization,”  arXiv preprint arXiv:1711.05101, 2017.
  * [28] Michael McAuliffe, Michaela Socolof, Sarah Mihuc, Michael Wagner, and Morgan Sonderegger,  “Montreal forced aligner: Trainable text-speech alignment using kaldi.,”  in Interspeech, 2017, vol. 2017, pp. 498–502.
  * [29] Eneko Agirre, Carmen Banea, Daniel Cer, et al.,  “Semeval-2016 task 1: Semantic textual similarity, monolingual and cross-lingual evaluation,”  in SemEval-2016. 10th International Workshop on Semantic Evaluation; ACL; 2016. p. 497-511., 2016.
  * [30] Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli,  “wav2vec 2.0: A framework for self-supervised learning of speech representations,”  Advances in neural information processing systems, vol. 33, pp. 12449–12460, 2020.
  * [31] Arun Babu, Changhan Wang, Andros Tjandra, Kushal Lakhotia, Qiantong Xu, Naman Goyal, Kritika Singh, Patrick von Platen, Yatharth Saraf, Juan Pino, et al.,  “Xls-r: Self-supervised cross-lingual speech representation learning at scale,”  arXiv preprint arXiv:2111.09296, 2021.
  * [32] Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, et al.,  “Wavlm: Large-scale self-supervised pre-training for full stack speech processing,”  IEEE Journal of Selected Topics in Signal Processing, vol. 16, no. 6, pp. 1505–1518, 2022.
  * [33] Ankita Pasad, Bowen Shi, and Karen Livescu,  “Comparative layer-wise analysis of self-supervised speech models,”  in ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023, pp. 1–5.
  * [34] Cheol Jun Cho, Peter Wu, Abdelrahman Mohamed, and Gopala K Anumanchipalli,  “Evidence of vocal tract articulation in self-supervised learning of speech,”  in ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023, pp. 1–5.
  * [35] Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D Manning,  “What does bert look at? an analysis of bert’s attention,”  in Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, 2019, pp. 276–286.