∎

11institutetext:  Sudhir Malik
<EMAIL_ADDRESS>

1University of Puerto Rico Mayaguez, USA
2CERN, Geneva, Switzerland
3Ludwig Maximilian University of Munich, Germany
4University of Sussex, Brighton, UK
5University of Mississippi, Oxford, MS, USA
6University of Illinois at Urbana-Champaign, IL, USA
7Princeton University, Princeton, NJ, USA
8Université Paris-Saclay, CNRS/IN2P3, IJCLab, Orsay, France
9Santa Cruz Institute for Particle Physics, UC Santa Cruz, CA, USA
10Instituto de Fśsica Corpuscular / Consejo Superior de Investigaciones
Científicas - University of Valencia (IFIC / CSIC - UV), Spain
11LAPP, Université Savoie Mont Blanc, CNRS/IN2P3, Annecy, France
12Deutsches Elektronen-Synchrotron DESY, Hamburg, Germany
13University of British Columbia, Vancouver, Canada
14Université de Strasbourg, CNRS, IPHC UMR 7178, Strasbourg, France
15Humboldt University of Berlin, Berlin, Germany
16Karlsruhe Institute of Technology (KIT), Karlsruhe, Germany
17Simon Fraser University, Burnaby BC, Canada
18Siena College, Loudonville, NY, USA
19University of Washington, Seattle, WA, USA
20University of Amsterdam, Amsterdam, The Netherlands
21University of Nebraska, Lincoln, NE, USA
22University of Michigan, Ann Arbor, MI, USA
23University of California Davis, Davis, CA, USA
24Oklahoma State University, Stillwater, OK 74078, USA
25University of Pittsburgh, Pittsburgh, PA, USA
26Universität Zürich, Zürich, Switzerland
27University of Massachusetts, Amherst MA, USA
28Institut Ruđer Bošković, Zagreb, Croatia
29Boston University, Boston, USA
30University of Bologna, Bologna, Italy
31INFN (Istituto Nazionale di Fisica Nucleare), Italy
32SingleStore, New York
34University of Oklahoma, Norman, OK, USA
35Center for Advanced Systems Understanding, Saxony, Germany
36Technische Universität Dresden, Dresden, Germany

# Software Training in HEP

Sudhir Malik (editor) 1 Samuel Meehan 2 Kilian Lieret3 Meirin Oan Evans4
Michel H. Villanueva5 Daniel S. Katz6 Graeme A. Stewart2 Peter Elmer7 Sizar
Aziz8 Matthew Bellis18 Riccardo Maria Bianchi25 Gianluca Bianco*30  31]  Johan
Sebastian Bonilla23 Angela Burger24 Jackson Burzynski27 David Chamont8 Matthew
Feickert6 Philipp Gadow12 Bernhard Manfred Gruber*2  35  36]  Daniel Guest15
Stephan Hageboeck2 Lukas Heinrich2 Maximilian M. Horzela16 Marc Huwiler26
Clemens Lange2 Konstantin Lehmann17 Ke Li19 Devdatta Majumder28 Judita
Mamužić10 Kevin Nelson22 Robin Newhouse13 Emery Nibigira14 Scarlet Norberg1
Arturo Sánchez Pineda11 Mason Proffitt19 Brendan Regnery23 Amber Roepe34
Stefan Roiser2 Henry Schreiner7 Oksana Shadura21 Giordon Stark9 Stephen
Nicholas Swatman*2  20]  Savannah Thais7 Andrea Valassi2 Stefan Wunsch*2  16]
David Yakobovitch32 Siqi Yuan29

(Received: 1 August 2021 / Accepted: date)

###### Abstract

The long term sustainability of the high energy physics (HEP) research
software ecosystem is essential to the field. With new facilities and upgrades
coming online throughout the 2020s, this will only become increasingly
important. Meeting the sustainability challenge requires a workforce with a
combination of HEP domain knowledge and advanced software skills. The required
software skills fall into three broad groups. The first is fundamental and
generic software engineering (e.g., Unix, version control, C++, continuous
integration). The second is knowledge of domain-specific HEP packages and
practices (e.g., the ROOT data format and analysis framework). The third is
more advanced knowledge involving specialized techniques, including parallel
programming, machine learning and data science tools, and techniques to
maintain software projects at all scales. This paper discusses the collective
software training program in HEP led by the HEP Software Foundation (HSF) and
the Institute for Research and Innovation in Software in HEP (IRIS-HEP). The
program equips participants with an array of software skills that serve as
ingredients for the solution of HEP computing challenges. Beyond serving the
community by ensuring that members are able to pursue research goals, the
program serves individuals by providing intellectual capital and transferable
skills important to careers in the realm of software and computing, inside or
outside HEP.

###### Keywords:

SoftwareTraining HEP

##  1 Introduction

Particle physics in the coming decades will continue to explore the
fundamental workings of the universe. This requires upgrading existing major
facilities like the Large Hadron Collider (LHC) to the High Luminosity LHC hl-
lhc  and building new facilities like the Long-Baseline Neutrino Facility
(LBNF) papadimitriou2017design  and Deep Underground Neutrino Experiment
(DUNE) dune , among many others. To realise the full physics potential of this
work, an equivalent investment must be made into the software required to
collect, process, and analyse the deluge of the data recorded. Recent efforts
like the HSF HSF-homepage  and IRIS-HEP iris-hep  are facilitating cooperation
and common efforts in HEP software and computing worldwide to develop state-
of-the-art software cyberinfrastructure required to meet the challenges of the
upcoming HEP experiments’ data-intensive scientific research. The rapid
evolution of computing technology with a concomitant increase in the
complexity of software algorithms for analysis requires developers to acquire
a broad portfolio of programming skills in order to enable future discoveries.

It is critical that all stakeholders across HEP make a major effort to provide
a strong foundation for new researchers entering the field. The researchers
must be brought up to date with new software technologies, concurrent
programming, and artificial intelligence, and must maintain, improve, and
sustain the existing HEP software. However, young researchers graduating from
universities worldwide currently do not receive adequate preparation in modern
computing practices to respond to the growing needs related to the above
experimental challenges. A community white paper hsfwptraining  outlined the
initiatives to address training needs and issues that need to be taken into
account for these to be successful. In the last two years, the HSF Training
working group, together with IRIS-HEP and FIRST-HEP first-hep  and partnering
with The Carpentries carpentries , has begun development of a software
training program. The efforts of this group have been focused on two specific
goals: (1) developing material for an introductory HEP software curriculum,
and (2) teaching this curriculum to HEP scientists. Thus far, over 1000 people
in HEP and related computing areas have been trained. This paper describes the
activities, the curriculum, and future directions of HEP software training.

##  2 Organization

The HSF Training working group, which is led by three co-conveners, engages
with different experimental collaborations and initiatives such as IRIS-HEP,
FIRST-HEP, and The Carpentries. The training group has weekly public meetings
HSF-training-meetings  to plan and assess progress. This is where ideas and
proposals are discussed and events are planned. These meetings are held
remotely using Zoom and live notes are maintained for anyone unable to join.
Training events are announced via several email lists, with registration and
timetables organized using Indico HSF-training-events .

The style and pedagogy of the training is heavily inspired by The Carpentries.
The training is student-centric, suitable for self-study, and experiment
agnostic, with reusable study material that is open source and open access,
and hosted in the HSF’s Training repositories on GitHub HSF-training-materials
. We encourage participants to provide feedback and suggestions for
improvement by opening issues in these repositories or to directly help with
the development by opening pull requests. In most cases the training material
is in the form of a website that is built from files written in the easy-to-
learn Markdown language. The website is automatically built using the static
site generator Jekyll jekyll  via GitHub Pages github-pages  and an adapted
and extended template from The Carpentries hsf-styles ; carpentry-styles .
Thus the entry barrier to contribute to the material is fairly low, as only
basic knowledge of git is required (and in most cases, all necessary steps can
be performed via the GitHub web interface). All lessons are listed in the HSF
Training Center HSF-curriculum , which provides an overview of the available
training modules and serves as an entry point for anyone wishing to learn.

Based on our experiences, we have also formalized the procedure used to
organize a training event and have compiled our knowledge in a compact guide
HSF-training-how-to . As organization is all about dividing work, we
distinguish between three relevant roles at our events:

  * •

_Instructors_ are subject-matter experts who develop training material and
then teach it, either in person, in recorded live sessions, or by recording
videos before the event. Instructors are the primary academic drivers of the
program at large and provide guidance to mentors and students alike. They gain
experience in curriculum design with a focus on optimizing pedagogy for all
learning styles.

  * •

_Mentors_ work closely with participants, for example, by conducting small
group mentoring sessions with ideally only five students per mentor. They
optimize the learning environment for individual participants and help them
persevere. They are critical to the success of any event and through
participation as a mentor not only serve the community, but develop
pedagogical communication skills that are transferable to other aspects of
their research/teaching portfolio.

  * •

_Facilitators_ take care of organizational aspects. They are responsible for
putting together all of the pieces of the puzzle to successfully execute the
full event while serving as the primary point of reference for participants to
communicate. They take on a dynamic responsibility beyond the “core content”
of the training event itself, and they also learn the essential “soft skills”
necessary to be a leader in the academic community and beyond.

All three groups are collectively referred to as _educators_. As creating
training material and teaching requires a lot of commitment and time, it is of
great importance to acknowledge the efforts of everyone involved. Currently
this is is mostly achieved by listing helping community members on the pages
of the relevant training and on a central community page HSF-training-
community .

Finally, Blueprint workshops HSF-training-blueprint  and hackathons HSF-
training-hackathon  are organised to brainstorm new training events, develop
content, and discuss improvements. The travel cost for educators and video
captioning of training material have been supported by IRIS-HEP and FIRST-HEP.

##  3 Curriculum

An initial survey of the software and training needs of the HEP community was
conducted in February of 2019 david_lange . This was followed by the
development of “prototype” course modules and pilot training events from which
feedback from participants was solicited.

Based on the surveys and the experiences gathered at the events, the course
structure was extended into a full curriculum consisting of a variety of
training modules. Each training module is independent from the others (except
for some clearly marked requirements), so that students can prioritize certain
skills before others. This is especially important in academia because
students are often expected to work directly towards scientific results with
minimal time given for acquiring software knowledge or best practices.

The most basic skill set (Unix shell, Python, git) is covered by modules
directly developed by the Software Carpentry softcarp . A large module that
covers the basics of modern C++ is currently in development and other modules
focusing on development in C++ such as CMake have already been taught with
great success.

This is complemented by a series of broader software engineering topics, such
as continuous integration and deployment using both GitHub Actions and GitLab
CI as examples. These modules are also particularly relevant for analysis
preservation, for which modules covering domain specific software such as
REANA reana  are in development.

A lesson on machine learning and a lesson specifically targeting machine
learning with GPUs started a data analysis techniques curriculum section.
Similarly important are HEP specific tools, especially ROOT root_cern  and
integration such as uproot uproot .

Finally, development is ongoing of modules that cover advanced topics that are
important for students striving to become core developers, such as code
documentation, performance optimization and parallel programming.

The module list HSF-curriculum  and the material evolves continuously
depending on input from participants and person-power available; as it is open
source, any interested stakeholder can contribute.

##  4 Training Events

During the initial period of training, 150 people received “introductory”
software skills training at Fermilab (FNAL), Argonne National Lab (ANL),
Lawrence Berkeley Lab (LBNL), and CERN HSF-training-SC-FNAL-2019 ; HSF-
training-ATLAS-ANL-2019 ; HSF-training-ATLAS-LBNL-2019 ; HSF-training-SC-
CERN-2019 . National labs are the hub of the HEP community and provide an
environment where it is easier to reach a diverse population of participants
with good infrastructure for in-person training. At the CoDaS-HEP school
codas-hep , over 50 people participated in the advanced “computing bootcamp”
software training. These training events were in-person.

However, the COVID-19 pandemic necessitated a rapid adjustment to virtual
platforms, which evolved throughout the course of 2020 as we gained
experience. The events that we had to pivot to use a virtual environment
include training on continuous integration and deployment cicd_gitlab ;
cicd_github , Docker docker , machine learning on GPUs mlgpu  and C++
cplusplus1  (organized together with SIDIS sidis ).

To date, nearly 100 educators have taught over 1000 participants in about a
dozen training events. Valuable lessons have been learned regarding in-person
and virtual training. There is very clear and detailed guidance for anyone
willing to host, request or organize a training while staying aligned with the
approach, philosophy, and code of conduct of the HSF Training group so as to
make the tools and techniques that are developed persistent, reuseable, and
broadly accessible HSF-training-how-to .

While in-person events offer more opportunities for active and efficient
engagement of participants and community building, they are generally more
exclusive: Participants need sufficient funding and extra preparation time to
arrange travel to the venue. Hosts have to book specially arranged/equipped
rooms with multiple projectors and screens to simultaneously show teaching
materials and slides. The space constraints typically limit the number of
participants to a few dozen and a long lead time is required for the
logistics. Our in-person events have been managed by about five educators,
which is necessary for the “hands-on” aspect to be successful. These educators
also need to make a large time commitment; they cannot just present their
material and leave. Virtual events have a broader reach of participant
attendance that is much higher compared to in-person events and enable a
considerably more equitable service to the community. Because the teaching
materials are fully preserved via lesson creation and YouTube videos
beforehand, an inability to attend during the scheduled time does not
considerably degrade learning. Finally, these video materials are captioned to
be inclusive of those with hearing impairments. Captioning videos for a week-
long event ($\sim$$50/day) is considerably more economical than the cost of a
hired sign language interpreter ($\sim$$1000/day).

The disadvantage of virtual events, however, is that it is difficult for
educators and participants to interact closely – you just can’t recreate the
in-person environment on Zoom. Educators and participants have to plan and act
upon their spread across time zones in the best possible way. It is also
challenging to keep everyone engaged and on the same page due to the pervasive
culture of “multi-tasking” within HEP. Due to this issue, although initial
registrations for these events are very high, the actual attendance is
typically only 50% of those who have registered. The online experience is more
prone to be distracted by other professional duties. However, it should be
noted that this does not mean that there is a lesser degree of learning
occurring at the training event. Tools like Mattermost, discord, and Slack
have been effectively deployed for asynchronous communication, both during and
after the event.

In general, devoting full time to training is always challenging because of a
combination of two factors. Though there is widespread desire to engage in
training, there is an institutional culture that prioritizes immediate
research activity over dedicated professional development, even though the
latter will lead to higher productivity in the long term.

##  5 Feedback

Feedback is required for us to evaluate if we are effectively facilitating
learning and to ensure the success of future training. Every training has a
pre- and post- survey to collect feedback from the participants. This includes
a set of baseline questions pertaining to demographics and questions to assess
the quality and method of training. These questions can be adapted to the
nature and topic of each training event. Additionally we organize a “post-
mortem” session among the educators to internally discuss the successes and
failures. This typically occurs after completion of the results of the post-
(and pre-) workshop surveys, which guide the discussion. Finally, a short
presentation about the training experience is presented at the HSF Training
weekly meeting and/or at the HSF all-working-groups planning meeting.

Figure 1 shows feedback on a training event involving containerization with
Docker HSF-training-Docker-2020 : clearly the training made a difference.
However, we are aware that this type of “learning evaluation” does not fully
encompass the impact of our training. It only probes the perceived and self-
reported learning of a skill. Instead, what is needed is a survey that is
conducted sufficiently later to understand how well the learned skill is being
applied in the context of research.

Figure 1: The self-reported pre- and post- training level of knowledge on the
topic of Docker (a software container technology)

##  6 Community

The solutions to future computing challenges require a large workforce trained
in a wide range of software skills. In order to train this workforce, we rely
on an active community whose members are enthusiastic and motivated to teach.
Our members include people with various roles and backgrounds in HEP, such as
experimental physicists from different collaborations, as well as software
engineers from different institutes. As we scale our training activities, we
also have members from nuclear physics and computer science as well. Members
of The Carpentries teach part of our very basic curriculum by an agreement via
membership subscription through IRIS-HEP. The overall diversity of the
background of the instructors and mentors adds great value to the training.
Each educator brings their own flavor of experience from a different computing
environment with a common goal of creating, teaching, and sustaining a common
set of software skills.

As the success of our mission depends crucially on the motivation and
participation of the community, we cultivate a strong sense of community
ownership and pay special attention to acknowledge contributions of all kinds.
We also encourage the participants in our training events to remain active or
become more active, share feedback, and in particular, to sign up to be a
mentor in one of the next iterations of the same training module. If former
participants do not yet feel confident about their mentoring skills, we offer
to match them with a more senior mentor. In the same way, we encourage mentors
to become instructors or facilitators and to become more and more active in
our organization. By actively engaging participants and educators throughout
the training community, we can sustain and nurture a culture of intentional
learning and grow our community in an organic fashion hsf-community-building .

Educators not only provide an invaluable service to the HEP community, but
they also get the opportunity to develop and sharpen their pedagogical skills
and enhance their professional portfolio. About two-thirds of the HEP
workforce eventually works outside of HEP, such as in the software industry
and in data science. The training makes a meaningful difference in the
preparation for such careers in terms of software knowledge and experience,
and enhances the employability for both the educators and participants. The
skills taught and learned, like Python, machine learning, and data analysis,
align with the needs of the software industry and strengthen the job profile
of a physicist to work in industry. At the same time, recognizing the
importance of software skills within HEP may hopefully help to provide more
incentives and clearer career paths in academia to those who want to pursue
their career within HEP, or in other scientific research fields. In
particular, strengthening the research software engineer career path rse_dan ;
rse_irishep  could significantly help retain the expertise within the HEP
community.

##  7 Sustainability

Sustainable software dan_katz  is essential for HEP. A sustainable training
program sustainability_flashtalk  is key to pursuing this goal. While
continuing the existing work, it will be essential to spread the training
events and training expertise geographically to keep the costs low and move to
an online training model to reduce financial burdens that accompany in-person
training. In parallel, it is important that as the curriculum grows, it begins
to include material specifically aimed at making software sustainable.

Training should be structured so that a minimal set of people are needed for
maintenance and costs per event are minimized. Growing the community is an
important aspect of sustaining the workforce. Providing recognition and
possible financial incentives can keep the community vibrant and motivated.
The community should recognize and appreciate the broader value of our
software training, which prepares a workforce to solve computing challenges
that are essential to advance our field and society at large.

To lead software training across HEP and related communities over the long
run, we need a core team whose main focus is to support the overall mission of
HEP software training. To scale up training efforts, we need to build
mentorship and leadership at the local and regional level supported by the
core team. Specifically, while we have started the following set of
activities, we need to scale up by:

  * •

Engaging more HEP labs, institutes, and universities in this endeavor.

  * •

Promoting equity, diversity, inclusion, and accessibility in participation
across HEP communities and being mindful of under-resourced institutions in
different geographical regions.

  * •

Establishing a mechanism to get feedback from our communities and improve the
training.

  * •

Ensuring that our core team and volunteers are afforded opportunities to grow
professionally and have career paths.

  * •

Exploring ways to manage a financial support model to share costs in the long
term.

##  8 Broader Impacts

HSF-led training is multilayered, with a basic HEP software curriculum
progressing to HEP-specific physics tools. Integrated with this is a growing
outreach program that is essential to building an influx of software workforce
and training young minds, catching them early in their educational
development. For example, several outreach events are organised on introducing
Python programming to K-12 teachers HSF-training-DA-STEM-PR-2020  under IRIS-
HEP and FIRST-HEP. The teachers can turn this into a classroom experience for
their students where physics, astronomy, and math courses can have problem
solving components that integrate programming with Python. In outreach events,
the teachers analyze and interpret physics data with Python using Google Colab
google-colab , which allows them to work directly in the web browser without
requiring any additional setup. Workshops teaching the basics of machine
learning to school teachers are also organized HSF-training-ML-STEM-PR-2021 .
We plan to scale this experience by partnering with other stakeholders in HEP
outreach, for example, Quarknet quarknet , which already has a well developed
network of teachers and schools taking part in HEP outreach programs.

##  9 Summary

HSF and IRIS-HEP are creating software training and ensuring sustainability of
software in HEP for years to come. The training material is open source and
open access, shared publicly via GitHub. This allows anyone to join the
discussion and make contributions by proposing changes, thereby continuously
improving the available material. This process is guided by continual feedback
solicited from the participants of the training events. Finally, we have
established a growing community of educators to broadly promote a culture
within HEP that goes beyond valuing software skills, but also values the
teaching of those skills to others. In doing so, we aim to foster a more
active, inclusive, and diverse scientific community. By leading software
training across HEP and related communities, we will be able to meet the
challenges in the field and beyond.

## Declarations

Funding:

This work is supported in part by National Science Foundation Cooperative
Agreement OAC-1836650 and grants OAC-1829707 and OAC-1829729.

Conflicts of interest:

The authors have no conflicts of interest to declare that are relevant to the
content of this article.

Availability of data and material:

Data sharing not applicable to this article as no datasets were generated or
analysed during the current study.

Code availabiliy:

Not applicable.

## References

  * (1) High-luminosity LHC.  https://home.cern/science/accelerators/high-luminosity-lhc
  * (2) V. Papadimitriou, K. Ammigan, J.A.J. au2, K.E. Anderson, R. Andrews, V. Bocean, C.F. Crowley, N. Eddy, B.D. Hartsell, S. Hays, P. Hurh, J. Hylen, J.A. Johnstone, P. Kasper, T. Kobilarcik, G.E. Krafczyk, B. Lundberg, A. Marchionni, N.V. Mokhov, C.D. Moore, D. Pushka, I. Rakhno, S.D. Reitzner, P. Schlabach, V. Sidorov, A.M. Stefanik, S. Tariq, L. Valerio, K. Vaziri, G. Velev, G. Vogel, K. Williams, R.M. Zwaska, C. Densham.  Design of the LBNF beamline.  https://arxiv.org/abs/1704.04471 (2017)
  * (3) Deep Underground Neutrino Experiment.  https://lbnf-dune.fnal.gov/
  * (4) High Energy Physics Software Foundation.  https://hepsoftwarefoundation.org/
  * (5) Institute for Research and Innovation in Software for High Energy Physics (IRIS-HEP).  https://iris-hep.org
  * (6) HEP Software Foundation, D. Berzano, R.M. Bianchi, P. Elmer, S.V. Gleyzer, J. Harvey, R. Jones, M. Jouvin, D.S. Katz, S. Malik, D. Menasce, M. Neubauer, F. Psihas, A.P. Navarro, G.A. Stewart, C. Tunnell, J.A. Vasel, S.J. Wang.  HEP Software Foundation community white paper working group - training, staffing and careers.  https://arxiv.org/abs/1807.02875 (2019)
  * (7) Framework for Integrated Research Software Training in High Energy Physics.  https://first-hep.org
  * (8) The Carpentries.  https://carpentries.org/
  * (9) HSF Training and Careers Working Group meetings.  https://indico.cern.ch/category/10294/
  * (10) HSF training events.  https://indico.cern.ch/category/11386/
  * (11) HSF training and educational material GitHub organization.  https://github.com/hsf-training
  * (12) Jekyll: Transform your plain text into static websites and blogs.  https://jekyllrb.com/
  * (13) GitHub Pages: Websites for you and your projects.  https://pages.github.com/
  * (14) HSF Training module template repository.  https://github.com/hsf-training/hsf-styles
  * (15) The Carpentries training module template repository.  https://github.com/carpentries/styles
  * (16) Hsf training center.  https://hepsoftwarefoundation.org/training/curriculum.html
  * (17) How to host an HSF training event.  https://hepsoftwarefoundation.org/training/howto-event.html
  * (18) The HSF training community.  https://hepsoftwarefoundation.org/training/community.html
  * (19) IRIS-HEP and HSF training blueprint meeting.  https://indico.cern.ch/event/889665/ (2020)
  * (20) The HSF training hackathon.  https://indico.cern.ch/event/997485/ (2021)
  * (21) D. Lange.  Selected results from HSF training survey.  https://indico.cern.ch/event/759388/contributions/3315848/attachments/1816082/2968198/training_how2019.pdf (2019).  2019 Joint HSF/OSG/WLCG Workshop HOW2019
  * (22) Software Carpentry – Teaching Basic Lab Skills for research computing.  https://software-carpentry.org/lessons/
  * (23) Reproducible Research Data Analysis Platform (REANA).  https://reanahub.io/
  * (24) ROOT: analyzing petabytes of data, scientifically.  https://root.cern/ (2021)
  * (25) Getting started with uproot.  https://uproot.readthedocs.io/ (2021)
  * (26) Software carpentry workshop (Fermilab).  https://indico.fnal.gov/event/20233/ (2019)
  * (27) FIRST-HEP/ATLAS training (Argonne).  https://indico.cern.ch/event/827231/ (2019)
  * (28) FIRST-HEP/ATLAS training (LBNL).  https://indico.cern.ch/event/827232/ (2019)
  * (29) Software carpentry workshop (CERN).  https://indico.cern.ch/event/834411/ (2019)
  * (30) Computational and Data Science Training for High Energy Physics.  http://codas-hep.org/
  * (31) Virtual pipelines training with GitLab.  https://indico.cern.ch/event/904759/ (2020)
  * (32) Virtual pipelines training with GitHub.  https://indico.cern.ch/event/1001128/ (2020)
  * (33) Virtual Docker training.  https://indico.cern.ch/event/934651/ (2020)
  * (34) Machine learning on GPUs training.  https://indico.cern.ch/event/958112/ (2020)
  * (35) 1st HEP C++ course and hands-on training.  https://indico.cern.ch/event/946584/ (2020)
  * (36) Software Institute for Data-Intensive Sciences.  https://sidis.web.cern.ch/
  * (37) HSF virtual Docker training.  https://indico.cern.ch/event/934651/ (2020)
  * (38) K. Lieret.  Community building.  https://indico.cern.ch/event/941278/contributions/4084356/ (2021).  HSF WLCG Virtual Workshop
  * (39) D.S. Katz, K. McHenry, C. Reinking, R. Haines.  Research Software Development & Management in Universities: Case Studies from Manchester’s RSDS Group, Illinois’ NCSA, and Notre Dame’s CRC.  https://doi.org/10.1109/SE4Science.2019.00009 (2019)
  * (40) Building a career path for research software engineers.  https://iris-hep.org/2021/05/12/career-path-rse.html (2021)
  * (41) D.S. Katz, S. Malik, M.S. Neubauer, G.A. Stewart, K.A. Assamagan, E.A. Becker, N.P. Chue Hong, I.A. Cosden, S. Meehan, E.J.W. Moyse, A.M. Price-Whelan, E. Sexton-Kennedy, M.O. Evans, M. Feickert, C. Lange, K. Lieret, R. Quick, A. Sánchez Pineda, C. Tunnell.  Software sustainability & high energy physics.  https://doi.org/10.5281/zenodo.4095837 (2020).  DOI 10.5281/zenodo.4095837
  * (42) S. Malik, S. Thais, M. Villanueva, K. Lieret, G. Stark, E.N. Nibigira, M.O. Evans, C. David.  Software training and Sustainable HEP.  https://indico.cern.ch/event/1004432/contributions/4377762/ (2021).  Sustainable HEP Workshop
  * (43) Data analysis for STEM teachers.  https://indico.cern.ch/event/927162/ (2020)
  * (44) Google Colaboratory.  https://colab.research.google.com/
  * (45) Machine learning basics for STEM teachers.  https://indico.cern.ch/event/998732/ (2021)
  * (46) Quarknet.  https://quarknet.org/


# TreeSeg: Hierarchical Topic Segmentation of Large Transcripts

Dimitrios C. Gklezakos Timothy Misiak Diamond Bishop
Augmend
{dimi, tim<EMAIL_ADDRESS>
###### Abstract

From organizing recorded videos and meetings into chapters, to breaking down
large inputs in order to fit them into the context window of commoditized
Large Language Models (LLMs), topic segmentation of large transcripts emerges
as a task of increasing significance. Still, accurate segmentation presents
many challenges, including (a) the noisy nature of the Automatic Speech
Recognition (ASR) software typically used to obtain the transcripts, (b) the
lack of diverse labeled data and (c) the difficulty in pin-pointing the
ground-truth number of segments. In this work we present TreeSeg, an approach
that combines off-the-shelf embedding models with divisive clustering, to
generate hierarchical, structured segmentations of transcripts in the form of
binary trees. Our approach is robust to noise and can handle large transcripts
efficiently. We evaluate TreeSeg on the ICSI and AMI corpora, demonstrating
that it outperforms all baselines. Finally, we introduce TinyRec, a small-
scale corpus of manually annotated transcripts, obtained from self-recorded
video sessions.

##  1 Introduction

The wide availability of video conferencing platforms, together with the rapid
surge in the volume of hosted videos McGrady et al. (2023), have resulted in
the proliferation of self-recorded content in the form of meetings and videos.
Often transcribed into text via Automatic Speech Recognition (ASR), this
content offers a wealth of information waiting to be extracted.

In this work we focus on segmenting large transcripts originating from
automatically transcribed, self-recorded content, into temporally contiguous
but semantically distinct segments. The goal of segmentation in this context
is two-fold: (a) To display content in an organized manner (i.e. automatic
chapter generation) and (b) to break down large transcripts in order to
satisfy the size constraints of downstream models, such as the context window
limitations of commoditized Large Language Models (LLMs).

In this context, topic segmentation poses many challenges due to: (a) The
noisy nature of ASR software, resulting in errors due to poor transcription or
out-of-dictionary technical terms, (b) the scarcity of labeled data of a
diverse distribution and the difficulty in obtaining it Gruenstein et al.
(2008) and (c) the difficulty in pin-pointing the ground-truth number of
segments, which can vary between human annotators even for the same
transcript.

In this paper we propose TreeSeg, a novel hierarchical topic segmentation
approach. TreeSeg combines utterance embeddings with divisive clustering to
filter the input and identify segment transition points. Our approach is
completely unsupervised, has no learnable parts and utilizes readily available
off-the-shelf embedding models. TreeSeg partitions the input in a hierarchical
manner and is accurate at multiple levels of segmentation resolution. In the
context of automatically generating and displaying video or meeting
segmentations, this hierarchical aspect of TreeSeg provides the user with the
affordance to dynamically choose the desired number and resolution of the
generated chapters/segments.

We evaluate our approach on two standard large meeting corpora: ICSI (Janin et
al., 2003) and AMI (Mccowan et al., 2005). We demonstrate that TreeSeg
outperforms its competition across the board. We also contribute a small-scale
corpus of our own, TinyRec, consisting of $21$ self-recorded sessions with
technical content, transcribed via ASR and manually annotated. We plan to
gradually extend TinyRec over time with more annotated sessions.

###  1.1 Related Work

Koshorek et al. (2018) adopt a supervised learning approach to topic
segmentation of written text by applying an LSTM-based (Hochreiter and
Schmidhuber, 1997) model to WIKI-727K, a dataset extracted from Wikipedia.
Models based on transformers (Vaswani et al., 2017) such as BERT (Devlin et
al., 2019) and its variants (Liu et al., 2019) are considered by Lukasik et
al. (2020) and Ghosh et al. (2024). Retkowski and Waibel (2024) use
annotations obtained from Youtube videos to train a transformer-based
segmentation model in a supervised manner.

Bayomi and Lawless (2018) apply agglomerative clustering to extract a
hierarchy of segments from text. Hazem et al. (2020) use a bottom-up approach
to segment text from medieval manuscripts. Grootendorst (2022) uses Sentence-
BERT (Reimers and Gurevych, 2019) embeddings to cluster documents and extract
latent topics.

Unsupervised topic segmentation of meetings has generated a lot of interest in
the recent years. Most recent approaches are essentially modern variants of
TextTiling (Hearst, 1997), a technique that relies on similarities between
adjacent utterances. TextTiling identifies topic changes by finding local
similarity minima. Perhaps the closest work to our own is BertSeg, introduced
in Solbiati et al. (2021). BertSeg is a modern version of TextTiling that
embeds utterances using a pre-trained model, extracts overlapping blocks of
embeddings and aggregates them to compute utterance similarities. HyperSeg,
introduced in Park et al. (2023), also follows the TextTiling paradigm, while
replacing learned embeddings with hyper-dimensional vectors derived from
random word embeddings. CohereSeg (Xing and Carenini, 2021) and
$\text{M}^{3}$Seg (Wang et al., 2023) are unsupervised segmentation approaches
that fine-tune embeddings using what is, in essence, a contrastive learning
technique. CohereSeg focuses on dialogue topic segmentation and is shown in
Park et al. (2023) to perform worse than or on par with HyperSeg. At the time
of writing this manuscript there is no publicly available code base for
$\text{M}^{3}$Seg. Finally, Ghosh et al. (2022) compare various topic
segmentation approaches on semi-structured and unstructured conversations and
show that pre-training on structured data does not transfer well to
unstructured data.

##  2 Method

Consider the linear topic segmentation setting, where the input is a temporal
sequence of transcript entries/utterances $U=[U_{1},...,U_{T}]$. We will
henceforth refer to $U$ as the ‘timeline’. The underlying organization of the
transcript into topics is modeled as a partition $P=\\{P_{k}\\}_{k=1}^{K}$ of
$U$ into segments. Each such segment covers a temporally contiguous set of
utterances:

$P_{k}=\\{U_{t}:t_{s}(k)\leq t\leq t_{e}(k)\\}$

starting with $U_{t_{s}(k)}$ and ending with $U_{t_{e}(k)}$ (endpoints
included). Each utterance belongs to exactly one segment. The goal of linear
topic segmentation is to approximate this ground-truth partition.

###  2.1 Hierarchical Topic Segmentation

We extend the linear setting to a hierarchical version by considering nested
partitions of the timeline, of increasing resolution, represented by trees.

(a) Flat partition tree: Linear topic segmentation.

(b) Deep partition tree: Hierarchical topic segmentation.

Figure 1: From linear to hierarchical topic segmentation: (a) A partition tree
of depth equal to $1$ corresponding to linear topic segmentation and (b) a
deeper partition tree corresponding to hierarchical topic segmentation. The
root node always covers the full timeline. Note that in both cases, the
children of a node form a partition of the node’s segment.

A flat partition $P$ of the timeline can be viewed as a tree of depth equal to
$1$, where the root is the timeline itself and each child is an utterance set
$P_{k}$ in the partition (see Figure 1(a)). A nested partition is represented
by a deeper tree where each node $v$ corresponds to a temporally contiguous
set of utterances $P_{v}$, while its children, denoted by $N(v)$ form a
partition of $P_{v}$, such that: $P_{v}=\bigcup_{u\in N(v)}P_{u}$ and $\forall
i,j\in N(v)\;:\;i\neq j$, $P_{i}\cap P_{j}=\varnothing$.

Figure 1(b) shows an example of such a nested partition and its corresponding
tree. A partition tree has the following properties:

  1. 1.

Every sub-tree containing the root is a valid nested partition of the
timeline.

  2. 2.

The leaves of every sub-tree containing the root form a valid flat partition
of the timeline.

The partition tree also induces a natural order in which segments are divided
into sub-segments. Consider the sub-tree that contains all nodes of depth
smaller than $\tau\leq D(P)$, where $D(P)$ is the maximum depth of the
partition tree and let $P_{\leq\tau}$ denote its corresponding valid, nested
partition. The leaves of this sub-tree form a flat partition of the timeline.
As $\tau$ increases, the resolution $|P_{\leq\tau}|$ of this flat partition
increases as well.

###  2.2 From Linear to Hierarchical Partitions

Suppose that we have access to a linear topic segmentation model that takes in
the desired partition length $K$ and identifies $K$ segments. Suppose also
that as $K$ increases, additional segment boundaries are added but not
deleted111Note that techniques based on TextTiling (Hearst, 1997) such as
BertSeg (Solbiati et al., 2021) and HyperSeg (Park et al., 2023) naturally
exhibit this property.. Then we can evaluate this model on the hierarchical
segmentation task, as follows:

  * •

Choose a depth threshold $\tau$ and construct the corresponding partition
$P_{\leq\tau}$ of maximum depth $\tau$.

  * •

Query the model with $K=|P_{\leq\tau}|$.

  * •

Evaluate the result by comparing with the flat partition induced by the leaves
of $P_{\leq\tau}$.

  * •

Repeat for all $1\leq\tau\leq D(P)$.

Figure 2: Inaccurate hierarchical segmentation: An example of an accurate
linear, but inaccurate hierarchical approximation of the tree in Figure 1(b).
Note that the leaves of the output partition match those of the ground-truth
partition, however the order in which the nodes are partitioned is not
respected and the hierarchical structure of the segments is not properly
identified.

Intuitively, a model that accurately captures the hierarchical relations
between segments, will output intermediate partitions that match those induced
by the limited-depth sub-trees of the ground truth partition. Figure 2 shows
an example where an output partition perfectly matches the ground-truth
bottom-level partition in the linear topic segmentation setting, but fails to
accurately capture its hierarchical structure.

###  2.3 TreeSeg

TreeSeg first embeds the transcript timeline entries $U$. Then these
embeddings are combined with a divisive clustering approach, to identify
appropriate splitting points and construct a deep partition tree.

####  2.3.1 Embedding the Transcript

TreeSeg uses an embedding model to convert transcript entries to embeddings.
For the results in this paper we used OpenAI’s text-embedding-ada-002 (ADA)
(Neelakantan et al., 2022), an embedding model trained in an unsupervised
manner, utilizing contrastive learning techniques. The choice of an off-the-
shelf, commoditized embedding model, results in a pipeline with no trainable
parts. Note that TreeSeg does not depend on this particular choice. Any
suitable embedding model such as Liu et al. (2019) can be directly plugged
into our approach in the same manner.

A single utterance might not contain enough context to be embedded on its own
in a meaningful way, especially in the presence of automatic transcription
errors. To address this issue we extract overlapping blocks of utterances. For
each utterance $u_{t}$ at position $t$ we extract a block that consists of
$u_{t}$ itself and up to $W$ utterances in the immediate past. This block
$[u_{t-W},...,u_{t}]$ is passed through the embedding model $f$ to obtain the
block embedding $e_{t}=f([u_{t-W},...,u_{t}])$. Note that this is a point of
deviation from BertSeg (Solbiati et al., 2021). While BertSeg embeds each
utterance separately and aggregates them using max-pooling, TreeSeg embeds the
whole block of utterances together, resulting in additional context passed to
the embedding model. Repeating for every utterance in the transcript results
in a temporal sequence of utterance embeddings that maintain local context.
The utterance block width $W$ is the sole hyperparameter of TreeSeg.

####  2.3.2 Divisive Clustering

(a) Candidate splitting points: An example utterance embedding timeline with
valid candidate splitting points shown in green. Candidates in red are
excluded due to minimum size constraints.

(b) First split: The original timeline is divided into two sub-segments.

(c) Second split: A sub-segment is sub-divided again and the partition tree
deepens.

Figure 3: Dividing the timeline: (a) At each step, valid candidate splitting
points are identified for all leaves. (b) & (c) The optimal splitting point
across all leaves is used to divide the corresponding segment into two sub-
segments. The process continues until a termination criterion is met.

TreeSeg utilizes a divisive clustering approach to recursively split segments
into two sub-segments, constructing a binary partition tree in the process.
Below, we provide a high-level outline of this segment division process:

  * •

For each leaf in the current partition tree, identify the optimal splitting
point according to the loss function, while respecting any constraints.

  * •

Pick the leaf with the best scoring candidate splitting point and split it
into two sub-segments.

  * •

Repeat until the termination condition is met.

We use a one-dimensional clustering objective as our loss function. Consider a
timeline of embeddings $E=[e_{1},...,e_{T}]$. A candidate splitting point $i$
partitions this timeline into two segments $[e_{1},...,e_{i-1}]$ and
$[e_{i},...,e_{T}]$. In practice we want to avoid ending up with trivial
segments, therefore we enforce a minimum viable segment size denoted by $M$
and only consider candidate points in the range $M<i\leq T-M$. The loss
function for candidate point $i$ is given by:

$\mathcal{L}(i)=\sum_{t=1}^{i-1}\|e_{t}-\mu_{\text{L}}\|_{2}^{2}+\sum_{t=i}^{T}\|e_{t}-\mu_{\text{R}}\|_{2}^{2}$

with $\mu_{\text{L}}=\frac{1}{i-1}\sum_{t=1}^{i-1}e_{t}$ and
$\mu_{\text{R}}=\frac{1}{T-i+1}\sum_{t=i}^{T}e_{t}$. The process stops when we
reach the desired number of segments $K$ or when all leaf segments are of size
$<2M$ 222Further splitting such a segment will result in at least one sub-
segment below the size threshold $M$.. The division process of TreeSeg is
outlined in Figure 3. Figure 3(a) shows an example timeline together with the
corresponding set of candidate splitting points, while Figures 3(b) and 3(c)
demonstrate two successive division steps.

Note that agglomerative hierarchical clustering approaches are typically
preferred to divisive ones for computational efficiency reasons. We opt for
the divisive approach for the following reasons:

  * •

Divisive clustering naturally matches the hierarchical segmentation task for
transcripts. The termination condition is typically met long before the
utterance level is reached.

  * •

Since timelines are one-dimensional, the optimal splitting point can be
identified with a single linear pass. Efficient implementations using
cumulative sums of embedding vectors allow for fast loss function computation.
Optimal splits are computed only once for every node and are maintained in a
min-heap data structure.

Our hypothesis is that the divisive approach utilizes global information to
identify strong candidates for topic shifts, with averaging over multiple
embedding vectors functioning as a candidate splitting point filter.

##  3 Evaluation

###  3.1 Datasets

We evaluate TreeSeg on three datasets:

  * •

ICSI (Janin et al., 2003) A corpus of $75$ transcribed meetings, containing
manual hierarchical topic annotations up to $4$ levels deep.

  * •

AMI (Mccowan et al., 2005) Over $100$ hours of transcribed meetings,
containing manual hierarchical topic annotations up to $3$ levels deep.

  * •

TinyRec We introduce TinyRec, a dataset consisting of transcripts obtained
from $21$ self-recorded sessions. Each transcript contains spoken utterances
as transcribed via ASR and was manually annotated with two-level topic
annotations. For more details on the content and annotation guidelines for
TinyRec, refer to Appendix A

We enforce a minimum size of five utterances per segment. Segments with sizes
below this threshold are automatically merged to the segment that comes
immediately after them. Table 1 shows the topic annotation statistics for each
dataset. Almost all transcripts in the ICSI and AMI corpora contain second-
level annotations, while several of them contain third- or even fourth-level
annotations. All TinyRec transcripts are annotated at two different
resolutions: ‘coarse’ and ‘fine’. Table 2 shows the average number of segments
at each partition level, after pruning segments below the size threshold.

Dataset | Avg. $|U|$ | L1 | L2 | L3 | L4
---|---|---|---|---|---
ICSI | 1453.7 | 75 | 75 | 52 | 3
AMI | 636.4 | 139 | 125 | 21 | 0
TinyRec | 267.4 | 21 | 21 | 0 | 0
Table 1: Annotation statistics: Number of transcripts with available topic annotations per partition level (L1 through L4), together with the average utterance timeline length. Dataset | L1 | L2 | L3 | L4
---|---|---|---|---
ICSI | 5.8 | 19.05 | 28.08 | 34.33
AMI | 6.81 | 14.44 | 26.61 | -
TinyRec | 4.18 | 14.12 | - | -
Table 2: Segment statistics: Average number of segments per partition level
(L1 through L4), after pruning.

###  3.2 Methodology

We compare TreeSeg against four baselines on the hierarchical topic
segmentation task. We adapt BertSeg and HyperSeg to output the top $K$
splitting points on the timeline, as described in Section 2.2. We also compare
with two naive baselines: RandomSeg and EquiSeg. RandomSeg generates $K$
random segments by picking $K-1$ segment transition points at random. EquiSeg
splits the timeline into equidistant segments.

For evaluation we use the standard $P_{k}$ (Beeferman et al., 1999) and
WinDiff (Pevzner and Hearst, 2002) metrics. For each level of resolution we
query each model with the ground-truth number of segments $K$ and compare the
obtained partition with the ground-truth one. We average metrics across all
possible partitions, as well as on a per-level basis. We run RandomSeg $100$
times and average the results.

The adapted version of HyperSeg has no hyperparameters. TreeSeg and BertSeg
both use a hyperparameter that regulates the utterance embedding block width.
BertSeg uses two additional hyperparameters related to utterance similarity
score smoothing. The first five transcripts from each dataset were denoted as
the ‘development’ set and were used to determine reasonable values for these
hyperparameters.

###  3.3 Results

| ICSI | AMI | TinyRec
---|---|---|---
Method | Pk | Wd | Pk | Wd | Pk | Wd
RandomSeg | 0.464 | 0.503 | 0.464 | 0.503 | 0.465 | 0.492
EquiSeg | 0.482 | 0.508 | 0.478 | 0.506 | 0.505 | 0.513
HyperSeg | 0.453 | 0.499 | 0.48 | 0.519 | 0.485 | 0.515
BertSeg | 0.388 | 0.432 | 0.443 | 0.48 | 0.473 | 0.486
TreeSeg (ours) | 0.31 | 0.353 | 0.355 | 0.396 | 0.367 | 0.382
Table 3: Hierarchical topic segmentation results: The performance of all
techniques is evaluated on the ICSI, AMI and TinyRec datasets. $P_{k}$ (Pk)
and WinDiff (Wd) metrics (lower is better) are aggregated over all
segmentation resolution levels. Our approach (TreeSeg) clearly outperforms all
baselines.

| ICSI (Pk) | AMI (Pk) | TinyRec (Pk)
---|---|---|---
Method | L1 | L2 | L3 | L4 | L1 | L2 | L3 | L1 | L2
RandomSeg | 0.445 | 0.472 | 0.48 | 0.487 | 0.455 | 0.471 | 0.486 | 0.44 | 0.49
EquiSeg | 0.446 | 0.5 | 0.505 | 0.492 | 0.464 | 0.492 | 0.492 | 0.492 | 0.518
HyperSeg | 0.442 | 0.456 | 0.465 | 0.417 | 0.47 | 0.491 | 0.481 | 0.48 | 0.49
BertSeg | 0.343 | 0.416 | 0.41 | 0.422 | 0.441 | 0.442 | 0.461 | 0.462 | 0.484
TreeSeg (ours) | 0.28 | 0.325 | 0.326 | 0.392 | 0.35 | 0.356 | 0.38 | 0.336 | 0.399
(a) $P_{k}$

| ICSI (Wd) | AMI (Wd) | TinyRec (Wd)
---|---|---|---
Method | L1 | L2 | L3 | L4 | L1 | L2 | L3 | L1 | L2
RandomSeg | 0.474 | 0.516 | 0.527 | 0.534 | 0.489 | 0.515 | 0.532 | 0.461 | 0.523
EquiSeg | 0.47 | 0.531 | 0.53 | 0.519 | 0.493 | 0.52 | 0.515 | 0.502 | 0.524
HyperSeg | 0.473 | 0.508 | 0.523 | 0.481 | 0.504 | 0.534 | 0.533 | 0.506 | 0.524
BertSeg | 0.386 | 0.462 | 0.453 | 0.466 | 0.477 | 0.481 | 0.498 | 0.479 | 0.493
TreeSeg (ours) | 0.314 | 0.375 | 0.372 | 0.441 | 0.387 | 0.403 | 0.421 | 0.352 | 0.413
(b) WinDiff

Table 4:  Hierarchical topic segmentation results aggregated per level: The
performance of all techniques is evaluated on the ICSI, AMI and TinyRec
datasets. (a) WinDiff (Wd) and (b) $P_{k}$ (Pk) metrics are aggregated per
segmentation resolution level. Our approach (TreeSeg) maintains strong
performance across all segmentation resolutions.

Table 3 shows the $P_{k}$ and WinDiff scores for all approaches, averaged over
all topic annotation resolutions as described in Section 2.2. TreeSeg clearly
outperforms all baselines on all three datasets. Table 4 shows the $P_{k}$
(Table 4(a)) and WinDiff (Table 4(b)) scores of all approaches aggregated per
segmentation resolution level. Note that TreeSeg maintains strong performance
across all segmentation resolutions.

Our results demonstrate that TreeSeg adequately captures the hierarchical
relations between segments at all levels of the hierarchy. Note also that the
performance of BertSeg and HyperSeg degrades on TinyRec, a dataset that is
less structured in nature than ICSI or AMI. While small in scale, TinyRec
might be more representative of self-recorded content in the wild. TreeSeg
maintains strong performance across all three datasets.

##  4 Conclusion

We introduced TreeSeg, a hierarchical segmentation approach suitable for
segmenting large meeting and video transcripts. TreeSeg generates structured
segmentations in the form of binary trees, capturing the hierarchical
relations between segments. Our approach utilizes off-the-shelf components,
contains no learnable parts and only a single hyperparameter. We provided a
rigorous definition and evaluation methodology for the hierarchical topic
segmentation task. We compared TreeSeg with two related embeddings-based
approaches ; BertSeg (Solbiati et al., 2021) and HyperSeg (Park et al., 2023),
as well as two naive baselines. We introduced TinyRec, a small-scale
collection of transcripts obtained from self-recorded sessions via ASR.
Evaluating on ICSI, AMI and TinyRec, we demonstrated the superior performance
of TreeSeg. Our work constitutes, to our knowledge, the first divisive
clustering variant of TextTiling (Hearst, 1997). A promising future direction
for research is that of utilizing the structure of the generated partition in
downstream tasks such as summarization (Park et al., 2024) or knowledge
extraction.

##  5 Limitations

One limitation of this work is related to the diversity of the datasets in our
evaluation. ICSI and AMI are large corpora but are unlikely to capture the
full diversity of self-recorded content. Contributing TinyRec is one attempt
at mitigating this limitation. Another limitation is the lack of comparison
with $\text{M}^{3}$Seg (Wang et al., 2023). At the time of writing this
manuscript, there is no publicly available code base for $\text{M}^{3}$Seg.
Evaluation metrics on ICSI and AMI vary with the resolution of the
segmentation and the data extraction process, making evaluating models on the
same exact data a necessity for a fair comparison. Finally another limitation
of this work is our restricted focus on ADA embeddings (Neelakantan et al.,
2022). A more thorough comparison of various embedding models might yield
interesting insights into the function of divisive clustering as a filter for
strong topic shift candidate points.

## References

  * Bayomi and Lawless (2018) Mostafa Bayomi and Séamus Lawless. 2018.  C-HTS: A concept-based hierarchical text segmentation approach.  In _Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)_ , Miyazaki, Japan. European Language Resources Association (ELRA).
  * Beeferman et al. (1999) Doug Beeferman, Adam L. Berger, and John D. Lafferty. 1999.  Statistical models for text segmentation.  _Machine Learning_ , 34:177–210.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  Bert: Pre-training of deep bidirectional transformers for language understanding.  In _North American Chapter of the Association for Computational Linguistics_.
  * Ghosh et al. (2022) Reshmi Ghosh, Harjeet Kajal, Sharanya Kamath, Dhuri Shrivastava, Samyadeep Basu, and Soundararajan Srinivasan. 2022.  Topic segmentation in the wild: Towards segmentation of semi-structured & unstructured chats.
  * Ghosh et al. (2024) Reshmi Ghosh, Harjeet Singh Kajal, Sharanya Kamath, Dhuri Shrivastava, Samyadeep Basu, Hansi Zeng, and Soundararajan Srinivasan. 2024.  Topic segmentation of semi-structured and unstructured conversational datasets using language models.  In _Intelligent Systems and Applications_ , pages 91–104, Cham. Springer Nature Switzerland.
  * Grootendorst (2022) Maarten Grootendorst. 2022.  Bertopic: Neural topic modeling with a class-based tf-idf procedure.  _arXiv preprint arXiv:2203.05794_.
  * Gruenstein et al. (2008) Alexander Gruenstein, John Niekrasz, and Matthew Purver. 2008.  _Meeting Structure Annotation_, pages 247–274.
  * Hazem et al. (2020) Amir Hazem, Beatrice Daille, Dominique Stutzmann, Christopher Kermorvant, and Louis Chevalier. 2020.  Hierarchical text segmentation for medieval manuscripts.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pages 6240–6251, Barcelona, Spain (Online). International Committee on Computational Linguistics.
  * Hearst (1997) Marti A. Hearst. 1997.  Texttiling: segmenting text into multi-paragraph subtopic passages.  _Comput. Linguist._ , 23(1):33–64.
  * Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jürgen Schmidhuber. 1997.  Long short-term memory.  _Neural computation_ , 9(8):1735–1780.
  * Janin et al. (2003) A. Janin, D. Baron, J. Edwards, D. Ellis, D. Gelbart, N. Morgan, B. Peskin, T. Pfau, E. Shriberg, A. Stolcke, and C. Wooters. 2003.  The icsi meeting corpus.  In _2003 IEEE International Conference on Acoustics, Speech, and Signal Processing, 2003. Proceedings. (ICASSP ’03)._ , volume 1, pages I–I.
  * Koshorek et al. (2018) Omri Koshorek, Adir Cohen, Noam Mor, Michael Rotman, and Jonathan Berant. 2018.  Text segmentation as a supervised learning task.  _ArXiv_ , abs/1803.09337.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized BERT pretraining approach.  _CoRR_ , abs/1907.11692.
  * Lukasik et al. (2020) Michal Lukasik, Boris Dadachev, Kishore Papineni, and Gonçalo Simões. 2020.  Text segmentation by cross segment attention.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 4707–4716, Online. Association for Computational Linguistics.
  * Mccowan et al. (2005) Iain Mccowan, J Carletta, Wessel Kraaij, Simone Ashby, S Bourban, M Flynn, M Guillemot, Thomas Hain, J Kadlec, V Karaiskos, M Kronenthal, Guillaume Lathoud, Mike Lincoln, Agnes Lisowska Masson, Wilfried Post, Dennis Reidsma, and P Wellner. 2005.  The ami meeting corpus.  _Int’l. Conf. on Methods and Techniques in Behavioral Research_.
  * McGrady et al. (2023) Ryan McGrady, Kevin Zheng, Rebecca Curran, Jason Baumgartner, and Ethan Zuckerman. 2023.  Dialing for videos: A random sample of youtube.  _Journal of Quantitative Description: Digital Media_ , 3.
  * Neelakantan et al. (2022) Arvind Neelakantan, Tao Xu, Raul Puri, Alec Radford, Jesse Michael Han, Jerry Tworek, Qiming Yuan, Nikolas Tezak, Jong Wook Kim, Chris Hallacy, Johannes Heidecke, Pranav Shyam, Boris Power, Tyna Eloundou Nekoul, Girish Sastry, Gretchen Krueger, David Schnurr, Felipe Petroski Such, Kenny Hsu, Madeleine Thompson, Tabarak Khan, Toki Sherbakov, Joanne Jang, Peter Welinder, and Lilian Weng. 2022.  Text and code embeddings by contrastive pre-training.  _CoRR_ , abs/2201.10005.
  * Park et al. (2024) Seongmin Park, Kyungho Kim, Jaejin Seo, and Jihwa Lee. 2024.  Unsupervised extractive dialogue summarization in hyperdimensional space.  In _ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE.
  * Park et al. (2023) Seongmin Park, Jinkyu Seo, and Jihwa Lee. 2023.  Unsupervised dialogue topic segmentation in hyperdimensional space.  In _INTERSPEECH 2023_. ISCA.
  * Pevzner and Hearst (2002) Lev Pevzner and Marti A. Hearst. 2002.  A critique and improvement of an evaluation metric for text segmentation.  _Computational Linguistics_ , 28(1):19–36.
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019.  Sentence-BERT: Sentence embeddings using Siamese BERT-networks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3982–3992, Hong Kong, China. Association for Computational Linguistics.
  * Retkowski and Waibel (2024) Fabian Retkowski and Alexander Waibel. 2024.  From text segmentation to smart chaptering: A novel benchmark for structuring video transcriptions.  _ArXiv_ , abs/2402.17633.
  * Solbiati et al. (2021) Alessandro Solbiati, Kevin Heffernan, Georgios Damaskinos, Shivani Poddar, Shubham Modi, and Jacques Calì. 2021.  Unsupervised topic segmentation of meetings with bert embeddings.  _ArXiv_ , abs/2106.12978.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Proceedings of the 31st International Conference on Neural Information Processing Systems_ , NIPS’17, page 6000–6010, Red Hook, NY, USA. Curran Associates Inc.
  * Wang et al. (2023) Ke Wang, Xiutian Zhao, Yanghui Li, and Wei Peng. 2023.  M3Seg: A maximum-minimum mutual information paradigm for unsupervised topic segmentation in ASR transcripts.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 7928–7934, Singapore. Association for Computational Linguistics.
  * Xing and Carenini (2021) Linzi Xing and Giuseppe Carenini. 2021.  Improving unsupervised dialogue topic segmentation with utterance-pair coherence scoring.  In _Proceedings of the 22nd Annual Meeting of the Special Interest Group on Discourse and Dialogue_ , pages 167–177, Singapore and Online. Association for Computational Linguistics.

##  Appendix A The TinyRec Dataset

TinyRec consists of $21$ self-recorded video sessions with screen-sharing that
were transcribed using Automatic Speech Recognition (ASR) and manually
annotated with topic annotations at two levels of resolution. The selection
criteria were the following:

  * •

A session needs to be at least $8$ minutes long or contain at least $80$
transcript entries.

  * •

The content of the session must be technical in nature and non-trivial.

The dataset was annotated by four different annotators. The concept of a
partition was explained to each annotator. The annotators were asked to first
identify a ‘coarse’ partition typically consisting of $2$-$5$ segments for a
ten-minute session. Then they were asked to further partition each ‘coarse’
segment, wherever that made sense according to their judgement, to obtain the
‘fine’, two-level partition.

The transition from ‘coarse’ to ‘fine’ segments was explained with the
following example scenario:

_You are recording your work update for a week during which you worked on
three features. Suppose that you identified a coarse segment covering the
discussion on one of these features. The fine segmentation would segment this
coarse segment again into sub-segments, each discussing parts of the feature
implementation or nuances in its design._

as well as an example of a segmented transcript.