# Project ThaiPASS: International Outreach Blending Astronomy and Python

James D. Keegans, Richard J. Stancliffe, Lawrence E. Bilton, Claire R.
Cashmore, Brad K. Gibson, Mikkel T. Kristensen, Thomas V. Lawson, Marco
Pignatari, Iraj Vaezzadeh  E.A. Milne Centre for Astrophysics, The University
of Hull, Cottingham Road, Kingston-Upon-Hull, HU6 7RX, United Kingdom
<EMAIL_ADDRESS>Benoit Côté  Konkoly Observatory, Research Centre
for Astronomy and Earth Sciences, MTA Centre for Excellence, Budapest H-1121,
Hungary  ELTE Eötvös Loránd University, Institute of Physics, Budapest 1117,
Hungary  and Siri Chongchitnan  Warwick Mathematics Institute, University of
Warwick, Coventry, CV4 7AL, United Kingdom.

###### Abstract

We present our outreach program, the Thailand-UK Python+Astronomy Summer
School (ThaiPASS), a collaborative project comprising UK and Thai institutions
and assess its impact and possible application to schools in the United
Kingdom. Since its inception in 2018, the annual ThaiPASS have trained around
60 Thai high-school students in basic data handling skills using Python in the
context of various astronomy topics, using current research from the teaching
team. Our impact assessment of the 5 day summer schools show an overwhelmingly
positive response from students in both years, with over 80% of students
scoring the activities above average in all activities but one. We use this
data to suggest possible future improvements. We also discuss how ThaiPASS may
inspire further outreach and engagement activities within the UK and beyond.

###### pacs:

00.00, 20.00, 42.10

Keywords: Advanced, 16-19, A-level, Atomic/nuclear, Astronomy/cosmology

##  1 Introduction

ThaiPASS (Thailand-UK Python+Astronomy Summer School) is an overseas
development assistance (ODA) outreach project focusing on building students’
programming skills for the future. The project was spearheaded by the
University of Hull in collaboration with the National Astronomy Research
Institute of Thailand, and has since garnered additional participating
institutions. ThaiPASS brings UK professional astronomers to Thailand, to
train 16-18 year-old Thai students in using Python for data handling, analysis
and mining. Student engagement was fostered through the use of astronomy and
astrophysics as the bulk of the teaching matter. Astronomy and astrophysics
have been shown, through initiatives such as Galaxy Zoo [1], to capture the
interest of a broad range of individuals. Using astronomy, we also highlight
the importance of data science for the modern economy and STEM subjects. The
first ThaiPASS was launched in 2018 under the auspices of the STFC (Science
and Technology Facilities Council) Newton Fund scheme, running annually, with
funding now secured through 2022. ThaiPASS activities have now been extended
for teacher Career Professional Development (CPD) training events in the UK.

In this report, we discuss findings from the first two events, held in 2018
and 2019, in Chiang Mai, Thailand 111ThaiPASS 2020 has been postponed to 2021,
due to the Covid19 pandemic.. In total, 60 students and 19 teachers from 30
Thai schools participated in the first two ThaiPASS events. Teachers
participated in the activities with the goal of applying this experience in
the classroom, increasing the effective impact of the summer school to include
students at each of these 30 schools. Each summer school comprised a series of
lectures and hands-on sessions over the course of a week, focusing on various
topics in astronomy. Students brought their own laptops, and additional online
computing facilities were provided in collaboration with the University of
Victoria, Canada, as part of their Cyberhubs initiative [2]. All data and data
mining tools used in the summer school were produced by our lecturing team and
their collaborators, much of which is currently published [3, 4, 5].

This paper is structured as follows. We first discuss the motivation for
ThaiPASS in Section 2. We then present the astronomy and computing material
covered at the summer schools in Section 3. In Section 4, we discuss the
feedback from participants, and the challenges faced. Finally, we outline our
plans for future events in Section 5. A full overview of ThaiPASS can also be
found at the respective 2018 and 2019 summer school
websites222https://indico.narit.or.th/event/81 and
https://indico.narit.or.th/event/120, respectively..

##  2 Motivation

###  2.1 Why ThaiPASS?

Project ThaiPASS is a human capacity building project with a strong ODA
resonance: ICT and data handling capabilities are two of the key technical
skills young people need to further their professional development and obtain
the highly skilled jobs which drive the modern economy.

One of the most effective ways to achieve the goals of ODA is to strengthen
Thailand’s knowledge-based economy. ThaiPASS addresses this by directly
reaching out to young people in their final years of secondary education.
These students may still be undecided about university and career choices, and
they may not have had any experience with programming and data-handling. By
attending ThaiPASS, they will be equipped with these skills and STEM knowledge
that may not be readily available to them otherwise. We accomplish all this by
using STEM frontier research disciplines like astronomy and computational
astrophysics, which many school students find to be an extremely exciting and
inspirational subject.

From an ODA perspective, the combination of ICT education and astronomy
presents a highly cost-effective and high-impact way to further economic
development, since STEM knowledge and skills are transferable to many
different sectors and underpins economic advances in health systems, education
and infrastructure. The kind of training offered at ThaiPASS aligns with and
supports existing initiatives concerned with the development of a knowledge-
based high-tech economy which Thailand envisages itself to be.

The motivations for ThaiPASS also resonate with the United Nation’s
Sustainable Development Goals (SDG) to end poverty, protect the planet, and
ensure prosperity for all. Project ThaiPASS addresses the following SDGs.

  * •

SDG4: Ensuring inclusive and quality education for all and promoting lifelong
learning. ThaiPASS brings Thai school students and teachers in contact with
international researchers, who will train them in scientific skills that they
may not have access to (or afford) otherwise. The training received at
ThaiPASS will be current and relevant, and will be a seed for lifelong
learning of STEM subjects. Our target students are those who have a passion
and aptitude in science, maths and English from Thai schools across the
country (those in highly exclusive, international schools are not targeted).
We made concerted effort in targeting schools that are in underprivileged
areas, and strongly urged schools to nominate both male and female
participants. We note that the gender ratio was 50:50 for ThaiPASS’18.

  * •

SDG8: Promoting inclusive and sustainable economic growth, employment and
decent work for all. A young person with ICT, data-handling and visualisation
skills is a highly employable person within the STEM sector and beyond. A
STEM-enabled generation of young people is essential for the future of
Thailand’s information-based economy.

  * •

SDG9: Build resilient infrastructure, promote sustainable industrialization
and foster innovation. ThaiPASS represents an investment in human capacity
building, which is crucial to achieving sustainable development, empowering
communities, and generating a high sustainable income with a low environmental
impact.

###  2.2 Why Python?

The Python programming language is one of the most popular programming
languages in astronomy and the wider sciences. Its position as an open source,
high level, simple-to-use programming language has made it a popular choice in
many areas of research. Although other programming languages (such as C,
FORTRAN) are still used for intensive computational tasks, data science needs
are widely carried out using Python. Its popularity and widespread use in
science research make it invaluable as a tool for any student who wishes to
pursue a career in astronomy. Python is also highly desirable in industry, and
as such also provides a useful learning opportunity for students wishing to
pursue careers outside of academia.

Python, being a high-level programming language, often uses natural language
for built-in functions, making it more approachable that some other languages.
The use of Jupyter notebooks also improves accessibility, as the interactive
environment allows the students to easily test and edit code with a simple
GUI. Python also has a huge array of modules available which are invaluable in
scientific computing. These include modules such as:

  1. 1.

Astropy - packages containing astronomy related functions

  2. 2.

Matplotlib - plotting functionality

  3. 3.

NumPy - mathematical operations not native to Python

  4. 4.

SciPy - contains functions, statistical tools, numerical solvers for
differential equations and many other features

All of the above factors make Python programming a highly desirable and useful
skill for students, particularly those planning to go into a career in STEM.

##  3 Content of ThaiPASS

All of the materials covered at ThaiPASS’18 and 19 are available at the E. A.
Milne Centre’s GitHub webpages333 https://github.com/Milne-Centre/ThaiPASS2018
and https://github.com/Milne-Centre/ThaiPASS2019 respectively.. These
materials comprise Jupyter notebooks and data files, which formed the basis of
activities at ThaiPASS, as well as basic introductory tutorials consisting of
scaffolded notebooks with a variety of simple tasks. As the most current
version of much of the material, the authors suggest using the 2019 repository
where possible (a description of the contents of each repository is provided
in their respective read-me). We present a summary of some of the materials
here.

###  3.1 Astronomy content

Astronomy has been a compulsory part of secondary education in Thailand since
2001, and a surprisingly wide range of topics are covered in Thai
schools444Detail of astronomy education in Thailand (and many other countries)
can be found at the IAU Office of Astronomy for Education https://www.haus-
der-astronomie.de/oae/worldwide. Nevertheless, there are disparities in the
depth and quality of astronomy education, notably amongst non-private schools,
mainly due to the lack of teachers with specialist training in physics and
astronomy, and inadequate provision for after-school astronomy activities. At
ThaiPASS, we made a decision to ensure that as much of the astronomy content
is as self-contained and accessible as possible to 16-18 year olds. We also
included introductory discussions of modern research topics outside the
standard syllabus. To date, the topics covered include:

  * •

nucleosynthesis, the production of elements and isotopes in stars;

  * •

galactic chemical evolution (GCE), the process by which material synthesised
by stars is mixed into the wider galaxy;

  * •

galaxies and dark matter, which discusses how the rotation curves of galaxies
lead to the idea that there is unseen matter in the Universe;

  * •

supernova, describing how stars explode and how ejecta material spreads out
into space.

During morning lectures, students were given an introduction to the main ideas
and concepts of each topic, and in the afternoon they were given tasks to
complete using data from recent publications (See for example [3] and [6]).

We note that the lectures on stellar evolution build on and extend topics
found in the Scottish Advanced Higher Specifications, the AQA A-level syllabus
and other exam boards in the UK (with some intersection with the Thai
syllabus). Common elements in these specifications concern the competition
between two opposing forces: the burning of nuclear fuel in a star, and its
collapse under gravity. The material covered here allows for a deeper
understanding of the processes and pathways involved in the production of new
nuclei in stars. This topic is expanded further in the second set of lectures
on nucleosynthesis described below.

####  3.1.1 Nucleosynthesis

This portion of the summer school discussed the production of elements in
stars - from low-mass stars like the Sun to massive core-collapse supernova
explosions (CCSNe). Lectures covered all of the major burning regimes found in
main sequence stars, and beyond to the products of stellar collapse and
explosion. This material complements and extends the material covered in some
A-level courses - for example the optional AQA astrophysics module which
covers the Hertzsprung-Russell (HR) diagram (which also featured at ThaiPASS-
see section 3.2) and supernova. Figure 1 shows HR diagram tracks for various
masses of stars, an example of some of the material that was covered during
the two ThaiPASS summer schools.

The students were then presented with a data-mining task, in which they were
asked to investigate stellar models computed by the NuGrid collaboration [3].
They were asked to find various points in the evolution of a star, and show
how the abundances of various isotopes changed as the star evolved using
published tools [5]. The nugridpy python package was used by the students, in
the same way that it is used to interface with real data-mining research. This
parallels the work of IRIS (Institute for Research in Schools) which promotes
cutting edge research in UK schools in collaboration with UK universities [7,
8].

Figure 1: Example of the content covered in ThaiPASS 2019 - an HR diagram of
the evolution of various masses of stars plotted with Python.

####  3.1.2 Galactic Chemical Evolution and Galactic Dynamics

Whilst not directly addressed in A-level courses, the natural progression
after discussing how isotopes and elements are produced in stars is to ask how
those products are distributed around the galaxy, and indeed how a galaxy
forms. A series of lectures was delivered in the 2018 ThaiPASS school
addressing this, and the effects of supernova explosions on the formations of
galaxies were discussed. Students were then trained in using OMEGA [9] a one
dimensional galactic chemical evolution code, stellar data library and data
handling package which is used in current GCE research. Students were tasked
with fitting the evolution of different elements in the Milky Way through
time, by adjusting the ratio of different stars and the products of burning
from these. Students enjoyed this task and showed good progress in using the
code to achieve the expected evolution of a number of key elements.

####  3.1.3 Galaxies and dark matter

Two lectures were given regarding galaxy rotation curves (orbital velocity as
a function of galactic radius) and how the discrepancy between theoretical
predictions and observed rotation curves led to the theory of dark matter.
Though these topics are not covered in A-level syllabuses, the level of the
lectures was not beyond that of an A-level student.

The task consisted of three parts, which drew on python skills learned in
previous workshops during the week. Specifically, the tasks required data
handling, graph plotting and defining functions. Students were provided with
rotation curve data for the galaxy M31 (Andromeda) obtained from [10] and
density profiles calculated from [11]. Task one required students to plot the
rotation curve using the data from [10]; task two was to plot the
theoretically predicted rotation curve from only baryonic matter
contributions. The two graphs from tasks one and two would not match, which
indicates baryonic matter is not sufficient to explain observations. Task
three was then intended for students to fix the theoretical rotation curve by
adding a dark matter component.

The purpose of the task was to consolidate the programming skills learned over
the week by presenting a guided enquiry-based task. The physics of the task
was signposted (and the data curated to avoid complexity that would distract
from the overall aims) whereas the method of programming was left to the
student.

Tasks one and two, which involved loading and plotting data along with a
simple calculation, were completed by all students with little to no help
required. Task three proved more difficult. Most students approached the task
correctly from a programming perspective; however, the increased conceptual
and mathematical complexity gave rise to erroneous results.

####  3.1.4 Supernova

This topic covered the ends of the lives of massive stars, their explosions as
supernova and the evolution of supernova remnants. During the lecture,
supernova were discussed in general including an explanation of the different
types of supernova, the physics of the supernova event and examples of
observations. A large part of the lecture focused on the details of supernova
remnants (SNRs), discussing the three phases of the expansion of the shell of
ejected material and showing examples of observations of SNRs at each phase.
The analytic equations that describe the evolution of the first two phases
(the free expansion and Sedov-Taylor phases) were introduced as a way to
estimate the radius and velocity of the SNR as a function of time. Modeling
supernova and their remnants was discussed, both using the analytical
equations and hydrodynamical simulations. The end of the lecture included a
brief explanation of how the evolution of galaxies is effected by supernova
feedback.

The first task involved modeling the dynamical evolution of a SNR starting
from the Sedov-Taylor phase. Using python, the students were required to use
the given equations to calculate the initial kinetic energy and radius
resulting from the end of the free expansion phase. Then using this as a
starting point for the Sedov-Taylor phase, use the given equations to
calculate the radius and velocity of the expanding shell of gas as a function
of time and plot their results. The second task used a module called yt
(https://yt-project.org) to analyse data which had been produced by a
hydrodynamical simulation of a supernova blast wave. Using this they could
visualise different properties (e.g. density, temperature) in “slices” through
the simulation box and compare the expansion over time to that estimated from
the analytical equations. As an extra task for those who had completed the
first two, the students could download the plots produced from analysing the
simulation data and combine them into a movie.

###  3.2 Programming content

One of the goals of ThaiPASS is to train students in data handling. Ideally,
students should have a basic knowledge of Python before they arrive, so that
the week can be devoted to more complex tasks. Therefore, we developed a
number of introductory ‘Starter Pack’ materials which were provided to
students before the school. These included a guide on installation of Python
on their own machines, a Jupyter notebook and exercises covering basic Python
commands, and an advanced notebook covering more difficult topics in Python
(e.g. reading and writing a file). These materials are available online on the
Milne Centre GitHub page, and together constitute a first introduction to
Python programming. For ThaiPASS’19, video tutorials555The videos were
produced by Dr Teeraparb Chantavat (one of the Thai PIs for ThaiPASS), and
made available as a MOOC through https://thaimooc.org. were also provided as
part of the Starter Pack.

The Starter Pack was found to have been a valuable tool for those students
which engaged with them beforehand. These notebooks would be appropriate for
any pupils of ages 11-18, and could be supplied to schools easily as a first
introduction for students with no prior knowledge of Python.

The main goal of the programming section of ThaiPASS was to introduce the
fundamental tools of Python, and some of the ways in which those tools could
be applied in the context of data-intensive astronomy. Although it was not our
primary intention to give a comprehensive course on computer programming, the
skills required for the set tasks led to students becoming much more familiar
and confident in their application of programming skills.

The Python tutorials and sessions during the summer schools covered a variety
of tasks - producing HR diagrams of sample spectroscopic data, calculating the
mechanics of orbital motion, handling large data sets during the GCE,
nucleosynthesis tasks and galactic dynamics sessions, and further skills such
as producing function, using loops to complete tasks and more. These tasks are
largely modular, and lend themselves to being changed between sessions to
suite the interests and capabilities of the students.

####  3.2.1 Data handling and plotting with Python

One of the more common introductory topics for astrophysics is the
Hertzsprung-Russell (HR) diagram. Colour-magnitude diagrams are directly
related to these HR diagrams, which show the luminosity (or brightness) of an
object as a function of its colour (a proxy for temperature). These are
fundamental tools in observational astronomy and allow classification of stars
by their evolutionary stages. Therefore, we provided the students with a
sample of real observational data obtained from the Sloan Digital Sky Survey
(SDSS)[12] SEGUE value added catalogue, which contains a mixture of well-
studied open and globular stellar clusters[13]. Students were tasked with
replicating the colour-magnitude diagrams for each stellar cluster from this
sample in order to illustrate the observed differences between open and
globular cluster systems. Figure 2 shows an example Python plot of the M13
cluster using the SDSS data supplied to the students.

Figure 2: Sample python plot of the HR diagram of the M13 cluster. All
students were able to produce a similar plot with varying degrees of
assistance.

This task calls on a wide range of data-handling skills such as accessing data
in a file, manipulating that data, and finally presenting it in a legible way,
all of which are key skills for any fields involving data. Students took to
this task well, with all participants being able to produce their own colour
magnitude plot from the data provided, though varying degrees of help were
required. By comparison with any of the widely available images of M13 and its
accompanying HR diagram, Python plotting tools produce publication-quality
figures. Students were able to produce plots similar to that shwon in Figure 2
after only a few hours tuition on the first day, and were instructed to verify
their work by comparing with any published HR diagrams of M13 that they could
find.

####  3.2.2 Forward Euler Method

Another computational topic covered at ThaiPASS was producing a Forward-Euler
solver - a numerical method for solving ordinary differential equations (for a
full discussion see, for example, [14, 15]). Because of its complexity, this
task lends itself well to discussing the idea of building an algorithm, and
the idea of planning out what needs to be coded up. Future ThaiPASS events may
extend this task further to allow students the opportunity to experience
algorithm development and gain proficiency with planning code methodology as
it is well suited to this application.

Numerical modelling of ordinary differential equations are the backbone of
many codes used in computational astrophysics and a key tool in all of the
computational work undertaken in STEM subjects. In both ThaiPASS’18 and 19,
students were tasked with producing their own numerical solver for integrating
ordinary differential equations.

This task is particularly difficult, and was set with the expectation that
students would work collaboratively. Most students succeeded, with assistance
either from peers or the teaching staff, in the programming of the forward
Euler solver, a particularly impressive feat considering the difficulty. The
full task of modelling the orbits of an example planet was completed only by a
few students, however the difficulty of this task is more appropriate for a
university level course with undergraduates at the University of Hull
performing similar computational tasks in years 2 and 3. This task was largely
included as a stretch and challenge goal for the most capable. Based on the
experience built in ThaiPASS 2018, the 2019 ThaiPASS summer school, included a
brief introduction to basic calculus (as finding the area under a curve) which
was not known by all students at the start of the summer school. We noted
significant increase in attainment and engagement from students with the
addition of this material. Future introductory material should include either
a basic description of the principals of calculus or a thorough description of
the Euler method if this is to be retained. Another option to increase
confidence in participating students would be to make available a model answer
with solutions., which they may then compare to in order to verify their
results.

###  3.3 STEM-career content

Each ThaiPASS also featured a ‘University and Career Day’, with the aim of
inspiring students to take up STEM education at university level and perhaps
as a career.

During the event, Thai professionals (STEM graduates) gave inspirational and
interactive talks on ways in which STEM subjects can be taken further at
university level and as a career, and the career paths available to graduates
with STEM skills. Previous lectures include: ‘How to become an astronomer?’
and ‘What do astronomers do?’ (both by Thai astronomy researchers) as well as
‘Why STEM? Career paths of STEM graduates’ by a speaker from the British
Council.

##  4 Lessons learned from ThaiPASS

We now discuss the impact of the program, as based on surveys of the
participants and discuss some of the challenges faced in presenting the
material.

###  4.1 Impact of ThaiPASS

At the end of each five-day summer school, participants were asked to complete
an anonymous questionnaire in order to assess the impact of the summer school
and to improve the next iteration of ThaiPASS. We asked the students to rate
their experiences of the various activities, and the usefulness of those
activities.

####  4.1.1 Results from 2018

Results for each of the questions are presented below:

(a) Did you enjoy the lecture and activities related to Kepler’s laws of
motion and the Hertzsprung-Russel Diagram?

(b) Did you enjoy the session focusing on nuclear astrophysics and the
production of the elements?

(c) Did you enjoy the session focusing on the Euler method and solving ODE?

(d) Did you enjoy the galactic chemical evolution lectures and activities?

Figure 3: Student responses of how much they enjoyed each day’s subject
material, with 1 being ‘not at all’ and 5 being ‘very much’.

(a) Has the summer school increased your knowledge of Python syntax and
application?

(b) Has the summer school increased your understanding of the application of
computer science?

(c) Enjoyment of lectures

(d) Did you enjoy the coding sessions?

(e) How confident are you plotting with python after the summer school?

(f) Would you now study a STEM subject at university?

Figure 4: Student response to questions, on a scale of 1 (worst) to 5 (best).

As can be seen from Figures 3 and 4, the feedback from the week long school is
largely positive. Figure 3 shows student enjoyment of the various activities
presented during the 2018 summer school on a scale of one to 5, 5 being the
most positive response. Overwhelmingly students enjoyed the activities. The
Euler method received the worst response however the majority of students
still rated this activity from average to good. The galactic chemical
evolution lectures are clearly the most well received. The material for this
session was accessible, students were engaged, and progress was identifiable
in the tasks that they were set. Future ThaiPASS summer school lecture
material should also be accessible, engaging and relevant, with measurable
outcomes for student progression which they can appreciate at the end of the
course. Further development of teaching materials is underway to facilitate
this.

Figure 4 asks students to rate their experience according to the various
questions outlined. As is shown in (a) and (b), almost all students report a
large increase in their technical skills in Python and the applications of
computer science more generally. Almost all students also rate the enjoyment
of lectures highly too, showing that material is engaging to the target
audience. The response to the increase in programming and plotting skills are
more varied, however this is likely due to the difficult nature of the
material, and the response still shows that students believe their skills have
improved significantly. The likelihood of students studying a STEM subject
further is also encouragingly high.

####  4.1.2 Results from 2019

For ThaiPASS’19, some changes to the questionnaire were made, and a pre- and
post-school survey were undertaken with the intention of trying to better
measure the impact of the school. The full questionnaire is provided in the
Appendix.

We see from Figure 5 that students have engaged with the lecture material and
found it useful and informative, feedback from NARIT (National Astronomical
Research Institute of Thailand) and teaching staff is also very positive.
Students were generally well disposed to the subject material before the
school, with most giving a score of 4 or 5. The post-school surveys showed a
higher proportion of 5s, suggesting we have helped to enhance students’
perceptions. Particularly gratifying is the response to the question ‘Are you
interested in a STEM career?’: after the school, the student response was
overwhelmingly of the highest response. While students said they enjoyed both
the astronomy lectures and the coding sessions, they expressed a clear
preference for the hands-on activities. This suggests it may be beneficial to
either change the balance of lectures to activities, or to intersperse the two
so as not to lose students’ interest.

Figure 5: Some of the questions and responses from the pre- and post-school
surveys conducted during ThaiPASS 2019. All responses includes questionnaires
from teachers and NARIT participants.

###  4.2 Challenges

Although ThaiPASS’18 and 19 were delivered without major difficulties, it is
worth discussing two aspects which are important to consider for those
organising outreach events, especially international ones.

####  4.2.1 Language barrier

Although English is a compulsory subject in all Thai schools, participants
came to us with various levels of skill in English. Most students had little
trouble comprehending written instructions in English, although oral
communication was occasionally challenging. It was helpful to have Thai
instructors on hand in the practical sessions to help lessen the language
barrier. Perhaps the most important mitigating measure was The Starter Pack
which ensured that participants were familiar with key Python and astronomical
terms in English before arrival. A good number of students were able to
enthusiastically communicate with the UK team and asked interesting questions
(both orally and through Mentimeter an anonymous online forum).

####  4.2.2 Gender balance

ThaiPASS participation is based on nominations from schools by teachers. From
the very beginning, we have always strongly encouraged each school to nominate
at least one female student to attend ThaiPASS (unless the school is single
gender). Tables 1-3 break down the participating students, teachers and
instructors respectively by gender, for the 2018 and 2019 workshops.

| Male | Female | M:F ratio
---|---|---|---
ThaiPASS’18 | 15 | 15 | 50:50
ThaiPASS’19 | 20 | 10 | 67:33
Table 1: Student participants by gender. | Male | Female | M:F ratio
---|---|---|---
ThaiPASS’18 | 6 | 4 | 60:40
ThaiPASS’19 | 7 | 2 | 78:22
Table 2: Teacher participants by gender. | Male | Female | M:F ratio
---|---|---|---
ThaiPASS’18 | 6 | 2 | 75:25
ThaiPASS’19 | 6 | 4 | 60:40
Table 3: Instructors by gender.

Although ThaiPASS’19 performed worse in terms of overall F:M ratio, we were
able to award ‘outstanding participant’ prizes to 3 male and 3 female
students. In terms of speakers, we have had a slight increase over the past
two years, though more work needs to be done to achieve an even balance.
Feedback from female participants clearly praised the positive impact of
having female instructors, who were regarded as positive role models for women
in STEM. A female participant at ThaiPASS’19 expressed this as follows:

> “I got a chance to see my dream self. I have a dream to be a woman who
> graduated with a doctoral degree. After seeing Dr. Claire’s lectures, I have
> a feeling that this is possible. She looks special and even more
> extraordinary when she is talking about something educational; which
> inspired me to try harder to become like that.”

Funding received for ThaiPASS’20 and ’21 allows us to plan for a far bigger
event. We therefore hope for a much larger pool of applicants from which we
hope to be able to draw a better gender balance. We will explicitly ask
schools to nominate female students, as we have done with the past schools. We
will also try to recruit more female instructors, particularly when recruiting
Ph.D. assistants, with the aim of achieving a 50:50 balance.

Finally, because teachers were not the primary target audience for the 2018-19
workshops, we did not monitor the gender balance nor actively encouraged
participation from female teachers. However, the next ThaiPASS will feature a
separate 5-day Python training for teachers (around 20 expected), and we will
work to improve the gender balance within this group.

##  5 Plans for impact in the UK

A high-impact legacy of the ThaiPASS project is for a similar summer school to
be held in the UK, particularly in Hull. Hull is an area with 47 Lower layer
Super Output Areas (LSOA) in the bottom 20% nationally [16], the highest
national proportion of deprived areas [17], and remains one of the cities in
the UK with a highest number of under-performing schools. The teaching
material developed for ThaiPASS can be adapted to organise UK-based events
targeting under-performing schools in Yorkshire and hosted by the University
of Hull. The University of Hull has a very engaged outreach team, with
participation from undergraduates, postgraduates, teaching and research staff.
We anticipate that the university could easily provide one summer school
initiative based in Hull each year, dependent on funding. With a similar model
to the ThaiPASS initiative, this could reach students in up to 30 local
authority schools each year, with a cohort of 40-60 students.

ThaiPASS also offers an opportunity for continuous professional development
for teaching professionals, who may wish to incorporate some of the material
in their classrooms, if similar interest is shown in these projects then a
further 20 teachers may attend each summer school. Further programs using
these materials may be rolled out across the country, with one such event
being held in Glasgow as part of a teacher career professional development
(CPD) event. Further summer schools, teacher training events and outreach
activities in targeted schools are possible with the continued engagement of
academics and research students at participating universities.

Due to the flexible nature of the materials and activities, this model may be
adapted by any research team who wishes to present their data and tools to
interested pupils and teachers. Our hope is that we have provided a viable
framework that allows students to engage with cutting-edge research from many
different universities, whilst building on topics in the GCSE and A-level
syllabi. Specifically in the area of astronomy and astrophysics, one could
develop projects or workshops based on these topics, amongst others:

  * •

Telescope optics and design

  * •

Spectroscopy and stellar classification

  * •

Compact-object physics

  * •

Cosmology

  * •

Exoplanet detection.

The Physics group at Hull (particularly the Milne Centre) has a strong track-
record of commitment to outreach within the Yorkshire region and beyond, and
new participants from other institutes offers the possibility of producing a
comprehensive suite of high-quality astronomy education materials, suitable
for a broad range of students at all levels of education, and accessible to
all schools no matter their performance or available funding.

Assessing the impact of these summer schools will be essential to the
improvement and development of the project. Future events hosted at
universities would be assessed in a similar way as the current ThaiPASS summer
schools - a mixture of continuous assessment during the course of the summer
school, questionnaires and written statements from participating students and
teachers. If the project is developed further to be used more independently in
schools and over a longer timescale, a number of other options for assessment
are possible. One such method could be the instigation of a competition where
students complete a project of 4-6 weeks using current research data,
producing a finished suite of code used to analyse data or to model simple
astrophysical systems. A similar initiative could include a symposium or
poster presentation, which introduces students to another key aspect of
scientific research - the presentation of data and results. Significant
progress on projects may also be evidenced through publication, similar to
some of the projects run by IRIS.

Given the challenging nature of the material, currently the resources are best
suited to students in KS4 and 5. However, it is our intention to produce
material suitable for both younger and more mature students, with the
flexibility of the material lending itself to even post A-level study.

##  6 Conclusions

We have presented here an assessment of the first two years (2018,2019) of
operation of Project ThaiPASS, a Thai-UK collaborative science outreach
project. From both written and oral feedback (from students and teachers),
ThaiPASS was an extremely positive and enriching experience for the
participants, in addition to being a unique learning opportunity for us as
education providers. The teaching resources developed for ThaiPASS are now
publicly available and can be used as an introduction to one of the fastest
growing programming languages in the world. These materials are also an
excellent primer to a number of inter-disciplinary topics, including nuclear
astrophysics and the chemical evolution of galaxies. Astronomy topics covered
are also relevant to and expand upon the syllabi of various UK exam boards.
This suite of resources provides both an introduction to new techniques and
skills in programming, as well as extension material which can stretch and
challenge even the most gifted students, whilst remaining relevant to their
studies.

## Acknowledgments

The authors gratefully acknowledge the financial support of the Science &
Technology Facilities Council (STFC), through the awards of the Project
ThaiPASS (ST/R006547/1) and of the University of Hull Consolidated Grant
(ST/R000840/1). We thank the Thai PIs (Dr. Teeraparb Chantavat and Dr. Apimook
Watcharangkool) for their collaboration on ThaiPASS. We also thank Dr. Kevin
Pimbblet for helpful advice on the gathering of appropriate and relevant
feedback. For the scientific data and access provided we acknowledge the
support of the NuGrid collaboration, the National Science Foundation (NSF,
USA) under grant No. PHY-1430152 (JINA Center for the Evolution of the
Elements), from the ”Lendulet-2014” Program of the Hungarian Academy of
Sciences (Hungary), from the ERC Consolidator Grant (Hungary) funding scheme
(Project RADIOSTAR, G.A. n. 724560) and the University of Hull High
Performance Computing Facility viper. We thank Prof. Falk Herwig and the
University of Victoria for their support and the access provided to their
outreach platforms within the Cyberhubs initiative.

## Ethical Statement

This investigation carried out in accordance with the principles outlined in
the IoP ethical policy. Students’ responses were collected anonymously, and
consent to participate in the summer school was required from parents or legal
guardians for all students.

## References

## References

  * [1] M Jordan Raddick, Georgia Bracey, Pamela L Gay, Chris J Lintott, Phil Murray, Kevin Schawinski, Alexander S Szalay, and Jan Vandenberg.  Galaxy zoo: Exploring the motivations of citizen science volunteers.  arXiv preprint arXiv:0909.2925, 2009.
  * [2] Falk Herwig, Robert Andrassy, Nic Annau, Ondrea Clarkson, Benoit Côté, Aaron D’Sa, Sam Jones, Belaid Moa, Jericho O’Connell, David Porter, et al.  Cyberhubs: Virtual research environments for astronomy.  The Astrophysical Journal Supplement Series, 236(1):2, 2018.
  * [3] M Pignatari, F Herwig, R Hirschi, M Bennett, G Rockefeller, C Fryer, FX Timmes, C Ritter, A Heger, S Jones, et al.  Nugrid stellar data set. i. stellar yields from h to bi for stars with metallicities z= 0.02 and z= 0.01.  The Astrophysical Journal Supplement Series, 225(2):24, 2016.
  * [4] C Ritter, F Herwig, S Jones, M Pignatari, C Fryer, and R Hirschi.  Nugrid stellar data set–ii. stellar yields from h to bi for stellar models with m zams= 1–25 m⊙ and z= 0.0001–0.02.  Monthly Notices of the Royal Astronomical Society, 480(1):538–571, 2018.
  * [5] Christian Ritter and Benoit Côté.  Nupycee: Nugrid python chemical evolution environment.  ascl, pages ascl–1610, 2016.
  * [6] Benoit Côté, Chris L Fryer, Krzysztof Belczynski, Oleg Korobkin, Martyna Chruślińska, Nicole Vassh, Matthew R Mumpower, Jonas Lippuner, Trevor M Sprouse, Rebecca Surman, et al.  The origin of r-process elements in the milky way.  The Astrophysical Journal, 855(2):99, 2018.
  * [7] Peter Hatfield, W Furnell, A Shenoy, E Fox, B Parker, L Thomas, and EAC Rushton.  Iris opens pupils’ eyes to real space research.  Astronomy & Geophysics, 60(1):1–22, 2019.
  * [8] Becky Parker.  Real science, real classrooms.  School Science Review, 98(365):116–117, 2017.
  * [9] Benoit Côté, Brian W. O’Shea, Christian Ritter, Falk Herwig, and Kim A. Venn.  The impact of modeling assumptions in galactic chemical evolution models.  The Astrophysical Journal, 835(2):128, 2017.
  * [10] Yoshiaki Sofue.  Radial distributions of surface mass density and mass-to-luminosity ratio in spiral galaxies.  Publications of the Astronomical Society of Japan, 70(2):31, March 2018.
  * [11] A. Tamm, E. Tempel, P. Tenjes, O. Tihhonova, and T. Tuvikene.  Stellar mass map and dark matter distribution in M 31.  Astronomy & Astrophysics, 546:A4, October 2012.
  * [12] Donald G. York, J. Adelman, Jr. Anderson, John E., Scott F. Anderson, James Annis, Neta A. Bahcall, J. A. Bakken, Robert Barkhouser, and et al. Bastian, Steven.  The sloan digital sky survey: Technical summary.  The Astronomical Journal, 120(3):1579–1587, September 2000.
  * [13] Deokkeun An, Jennifer A. Johnson, James L. Clem, Brian Yanny, Constance M. Rockosi, Heather L. Morrison, Paul Harding, James E. Gunn, and et al. Allende Prieto.  Galactic globular and open clusters in the sloan digital sky survey. i. crowded-field photometry and cluster fiducial sequences in ugriz.  The Astrophysical Journal Supplement Series, 179(2):326–354, December 2008.
  * [14] Kendall E Atkinson.  An introduction to numerical analysis.  John wiley & sons, 2008.
  * [15] Uri M Ascher and Linda R Petzold.  Computer methods for ordinary differential equations and differential-algebraic equations, volume 61.  Siam, 1998.
  * [16] Richard Prothero.  Towns and cities analysis, england and wales, march 2016, Mar 2016.
  * [17] Communities and; Local Government Ministry of Housing.  English indices of deprivation 2019, Sep 2019.

##  Appendix A Questionnaire

Below are the questions used for the ThaiPASS’19 pre- and post-school surveys.

1\. Who are you? Please tick.
A student
A teacher
NARIT staff
Others (please specify)

2\. Please rate the following activities on a scale of 1 to 5, where:

5 = very interesting
4 = interesting
3 = ok
2 = uninteresting
1 = very uninteresting

Please give comments if possible. e.g. what did you find interesting or not so
interesting? Leave blank if you are unsure.

a) Day 1: Why is Python useful + revision
Rating: 1 2 3 4 5 (please circle)
Comments:

b) Day 2: Nuclear astrophysics
Rating: 1 2 3 4 5 (please circle)
Comments:

c) Day 3: Half-day Python activities
Rating: 1 2 3 4 5 (please circle)
Comments:

d) Day 4: Supernovae/Galaxies/Dark matter
Rating: 1 2 3 4 5 (please circle)
Comments:

e) Day 5: Half-day “University and career” talks
Rating: 1 2 3 4 5 (please circle)
Comments:

The following questions were only used in the post-school survey.

3\. Did you learn any new skills / knowledge while at ThaiPASS? If YES, choose
which skills. You can choose more than one answer.
New programming skills
New knowledge about astronomy
New knowledge about data science
Improved English communication skills
Improved teamwork and collaboration skills
I did not gain any new skills/knowledge
Other (please describe)

If NO, please explain why.

4\. Please choose one option per question. (1 = Strongly disagree, 3 = neither
agree nor disagree, 5 = strongly agree). Leave blank if you are unsure.

a) ThaiPASS has improved my knowledge of Python.
Rating: 1 2 3 4 5

b) ThaiPASS has improved my knowledge on astronomy.
Rating: 1 2 3 4 5

c) ThaiPASS has increased my understanding of why data science is important
for astronomy and other STEM subjects.
Rating: 1 2 3 4 5

c) I enjoyed the lectures at ThaiPASS.
Rating: 1 2 3 4 5

d) I enjoyed the Python coding sessions.
Rating: 1 2 3 4 5

e) I have a better understanding of how to use Python to plot graphs and
present scientific data.
Rating: 1 2 3 4 5

f) ThaiPASS has encouraged me to study a STEM subject at university.
Rating: 1 2 3 4 5

g) ThaiPASS has made me interested in studying in the UK.
Rating: 1 2 3 4 5

5\. What was the most enjoyable thing about ThaiPASS?

6\. What could be improved for the next ThaiPASS?


# Developing a Production System for Purpose of Call Detection in Business
Phone Conversations

Elena Khasanova, Pooja Hiranandani, Shayna Gardiner,
Cheng Chen, Xue-Yong Fu, Simon Corston-Oliver
Dialpad Canada Inc
1100 Melville St #400
Vancouver, BC, Canada, V6E 4A6
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>

###### Abstract

For agents at a contact centre receiving calls, the most important piece of
information is the reason for a given call. An agent cannot provide support on
a call if they do not know why a customer is calling. In this paper we
describe our implementation of a commercial system to detect Purpose of Call
statements in English business call transcripts in real time. We present a
detailed analysis of types of Purpose of Call statements and language patterns
related to them, discuss an approach to collect rich training data by
bootstrapping from a set of rules to a neural model, and describe a hybrid
model which consists of a transformer-based classifier and a set of rules by
leveraging insights from the analysis of call transcripts. The model achieved
88.6 F1 on average in various types of business calls when tested on real life
data and has low inference time. We reflect on the challenges and design
decisions when developing and deploying the system.

##  1 Introduction

The Purpose of Call as we define it is similar to a thesis statement in an
argument: it introduces the speaker’s intent, the broad theme or topic of a
conversation, any key entities, and relevant relationships between them. The
Purpose of Call statement might also include a linguistic signpost – an
indication to the listener that the utterance is intended to convey the
purpose of the speaker’s call.

For instance:

I’m calling because [signpost] I’m trying to open one of the programs [entity]
on my computer [entity] and it’s not opening [relation] so I’m hoping I can
get some assistance [intent] with that. 111In contrast, statements such as I’m
calling to ask a question are not considered Purpose of Call expressions even
though they contain relevant signposting language because there are no
entities an agent or a customer can relate to.

Purpose of Call statements in a contact centre setting are usually uttered by
the customer in inbound calls, and by the agent in outbound calls. The Purpose
of Call is typically stated near the beginning of the call, is often stated in
a single utterance, and does not contain extra information. Atypically, we may
sometimes see the Purpose of Call occurring in the middle of a conversation,
occurring across several utterances, being implicit, or being uttered by a
call recipient rather than a call initiator.

The models described below have been implemented in the Dialpad Contact Center
product and are running in production. The Purpose of Call is extracted from
an automatic speech recognition (ASR) generated transcript in near-realtime
and displayed in a dashboard used by call center supervisors to monitor calls
taking place. The dashboard shows information about the caller and the agent,
the duration of the call, the Purpose of Call, and customer sentiment. A
separate analytics component clusters the Purpose of Call from all calls in a
call center during a time period to provide insights about trends and
anomalies, customer pain points, and common problems and knowledge gaps among
agents. Additional use-cases include showing the Purpose of Call in a summary
of prior calls with a customer, and including the Purpose of Call in summaries
of the conversation. The utterance segment containing the Purpose of Call is
highlighted in the call transcript and the call recording to be easily
accessible to agents and call center supervisors. These use-cases are
summarized in Figure 1. The Purpose of Call feature is used to help call
center managers to navigate to relevant sections of conversations to identify
areas to coach sales and support agents and sample relevant calls. Through
customer education, we emphasize that the feature should not be used for
automated evaluation of agent performance.

Figure 1: An illustration of the applications of Purpose of Call

There are a few challenges that arise when building an automatic system to
detect Purpose of Call.

Diversity of Purpose of Call statements. This type of detection system should
be an open-class system, since call purposes vary across different domains,
industries, and types of calls.

Robustness to noise. There are challenges related to the fact that Purpose of
Call extraction relies on the output of an ASR system. There are two sources
of noise in the ASR transcripts: language production issues such as false
starts, dysfluencies, filled pauses, inconsistency in conversational turn-
taking Cailliau and Cavet (2013); Dutrey et al. (2014); Żelasko et al. (2019);
Clavel et al. (2013) as well as their representation in the ASR system and
recognition errors due to acoustic noise.

Limitations in training data. Existing intent detection datasets do not
reflect real world settings (e.g. they do not distinguish the Purpose of Call
from other intent-like statements, and are limited to a subset of domains).
Manually annotating data (e.g. using crowd-sourced annotators) raises privacy
concerns since annotators must have access to a full conversation transcript
in order to find the best Purpose of Call. Annotation is a complex task that
requires highly trained annotators, and is expensive and time-consuming
because annotators must consider the larger context of the conversation to
make a judgment.

Computational efficiency. The need for the system to extract the Purpose of
Call statement in real time imposes constraints on memory consumption,
latency, and inference speed.

This paper describes an end-to-end system to extract a Purpose of Call
statement from the transcript of a business telephone call. Our contributions
are three-fold:

1\. Data analysis: we present a detailed analysis of language patterns and
other features involved in call purpose detection;

2\. Data: we describe a process to overcome a lack of training data by
bootstrapping a deep learning model from a knowledge-engineered model and
discuss the heuristics for developing such a model;

3\. System: we describe optimizations that were done in an online commercial
system to identify Purpose of Call statements in near-realtime (within three
seconds of an utterance being transcribed). To evaluate the effectiveness of
our approach, we examine the actual output of our production system.

##  2 Related work

The concept of a Purpose of Call statement has its roots in the Conversation
Analysis framework Schegloff and Sacks (1973); Sacks et al. (1974). Within
this framework, which combines perspectives from Linguistics and Sociology, a
conversation is understood to be composed of turn-taking utterances, with each
“turn” being indicated via linguistic and paralinguistic cues. Conversational
turns often form adjacency pairs such as question-answer pairs or offer-
acceptance/refusal pairs. There are other key aspects of a conversation as
well. For instance, a conversation is likely to end after a linguistic cue
known as a "closing" is given; likewise, there is usually a linguistic
indicator that a conversation is being initiated: an opening Schegloff and
Sacks (1973); Sacks et al. (1974); Pomerantz and Fehr (2011). Most work
analyzing telephone conversation openings within the framework of Conversation
Analysis has been conducted on English, but similar patterns have been
observed in other languages, including German and Farsi Taleghani-Nikazm
(2002).

Within a contact center environment, the Purpose of Call is, like openings and
closings, an integral aspect of the conversation (i.e. call) between customer
and support agent. We propose that a Purpose of Call is a particular
conversational feature that is necessary in contact center calls, and is
distinct from the call opening, the body of the call, and the call closing.

The first 120 seconds of a customer support call are predictive of that call’s
outcome Takeuchi et al. (2007); Hall et al. (2014). The Purpose of Call
statement typically occurs within this timeframe, so highlighting a Purpose of
Call in real time could provide agents with additional support in meeting
customer needs.

##  3 Methodology

We formulate the Purpose of Call detection task as a binary classification
problem. Each call, after being transcribed by the ASR system, is represented
as a sequence of utterances, which may consist of one or more sentences. The
division into utterances is based on acoustic features such as silent pauses
and the length of a speech fragment.

For a given utterance, we determine the probability that the utterance is the
Purpose of Call statement for that particular call. We impose the following
constraints on this task: (i) For a given call, there is only one most
probable Purpose of Call. (ii) Only calls with two call sides (agent and
customer) are considered, which excludes multiparty business conversations.
(iii) The model should make a prediction as the call is ongoing and therefore
will not have access to the full conversation.

Due to the lack of available annotated data representing the concept of
Purpose of Call, we followed an iterative approach to develop the model,
consisting of three steps: (1) Computational Linguists on-staff conducted
extensive linguistic analysis of transcripts to identify the characteristics
of Purpose of Call statements. (2) We then implemented a knowledge-engineered
approach to identify these Purpose of Call statements. (3) We bootstrapped
from the knowledge-engineered solution, using it to label training data for a
transformer-based approach. We select a transformer-based model as it is the
current state-of-the-art in sequence classification and is known to have
better generalization power than rule-based models.

We evaluate the performance using F1, Precision, and Hit rate, i.e. the number
of calls in which a Purpose of Call was detected out of all available calls.
We measure Hit rate in calls at least 30 seconds long, based on the
observation that shorter calls may not include any content (e.g. because the
caller hung up before starting the conversation). The model is tested on an
automatically obtained validation set that represents 10% (18K utterances) of
the training data, a manually annotated gold test set of 13215 utterances from
909 calls, and unlabeled samples from 600 real-life calls.

###  3.1 System Overview

Figure 2: Purpose of Call Detection System Architecture

The production system to detect a Purpose of Call utterance is a hybrid model
consisting of three parts (see Figure 2).

The Selection model, or outer model, inputs an utterance, the previous context
of the conversation, and the probabilities of previously detected Purpose of
Call events. It consists of two sets of rules: (1) empirically derived filters
that determine whether an incoming utterance is a candidate for a Purpose of
Call and should be processed by the inner model, a successful candidate is
within 180 seconds and 30 utterances in the call, and is between 4 and 150
tokens long; (2) rules that combine and compare scores from the inner model
and set various thresholds for different linguistic types of call purpose
statements (i.e. the utterances containing signposting language typically
receive higher scores than other types and need a higher bar). Every time a
new utterance qualifies to be a Purpose of Call, it is dynamically updated in
the user interface. (See Appendix B for an example of a Selection rule.)

The Scoring model, or inner model, is implemented as a multiclass
classification model which performs inference on a single utterance. We fine-
tuned a transformer-based model for classification on proprietary labeled
data. The model assigns to an utterance probabilities of it being a call
purpose, question, or negative (not a call purpose or a question). The
question class represents question_response pattern (see Table 1) and is used
to boost probabilities of utterances that would otherwise be of the negative
class.

The Simplification model. The utterance with the highest score is stripped of
information that is irrelevant to the purpose of the call (e.g. greetings,
pleasantries, introductions, technical problems). It consists of a small set
of common expressions (many of which are reused from the knowledge-engineered
model) to exclude from utterances and reduces the length of Purpose of Call
utterances by 7.8% on average. 49% of utterances undergo simplification with
precision of 96%.

###  3.2 Model Development

The model development process consists of three main stages: data analysis and
feature selection, designing a knowledge-engineered model informed by the
insights from the data, and bootstrapping a transformer based model from the
rule-based system. In this section, we discuss these stages.

####  3.2.1 Data Research and Feature Selection

We manually analyzed a sample of 2000 call transcripts across several
dimensions, which are outlined below.

  1. 1.

Inbound vs. outbound calls:
In outbound call center calls (40.8%), the call initiator and the side that
utters the Purpose of Call is usually an agent; in inbound calls (59.2%), it
is a customer, with some exceptions. 55.3% of all Purpose of Call statements
are uttered by the customer.

  2. 2.

Place: the Purpose of Call is uttered within the first 40 seconds in a
majority of calls (73.8%), in the middle in a minority of calls (10.7%), and
towards the end in a handful of (mainly short) calls. The mean time of
occurrence is 29.9 seconds, std=19.1, median=25.5 seconds. The maximum time is
180 seconds.

  3. 3.

Speaker role:
The call initiator utters the Purpose of Call in the vast majority of cases in
the form of a statement; in returned or scheduled calls, the Purpose of Call
can be uttered by a call recipient in the form of a guess, assumption or
inquiry.

  4. 4.

Domain: There are three main types of call center calls:

Sales calls: commonly characterized by the Purpose of Call not being stated
explicitly in one utterance, but gradually being revealed during the course of
the call. Agents often spend a longer time building rapport, so utter the
purpose later in the call compared to support calls. 74% of Purpose of Call
statements still occur within the first 40 seconds. Outbound calls are
prevalent. 48% of Purpose of Call statements are uttered by customers.
Support calls: inbound calls are prevalent, the Purpose of Call is introduced
early in the conversation. In fact, 56% of Purpose of Call statements are in
the very first utterance, accompanied by a greeting, and 63% of the statements
occur within the first 50 seconds. 62% of Purpose of Call statements are
uttered by customers.
General business calls: may include support and sales calls as well as other
communications, both formal and informal. The Purpose of Call is often implied
(e.g. a conversation between colleagues, transfers from a chat to a call, with
the purpose of the conversation being known by both parties). 84% occur within
the first 45 seconds, and 59% are uttered by customers.

  5. 5.

Length distribution: Purpose of Call utterances range in length from 4 to 224
tokens, with the mean=45.5, std=29.9, median=37, and 75% being under 59
tokens.

  6. 6.

Language patterns: We identified several language markers associated with
Purpose of Call statements in Table 1.

Pattern |  % |  Description |  Example
---|---|---|---
call_purpose_phrases |  32.7 |  explicit declarations of the Purpose of Call typically signposted with lexical cues containing purpose and call and their synonyms |  The reason for my call is I moved to a new address, so I need to change it on my profile.
desire_phrases |  31.7 |  expressions of volition, desire or need |  Hi, I need a refund for my order.
question_response |  15.8 |  responses to an agent’s prompt |  \- How can I help you? / - I received a message that my order has been delayed.
greetings |  9.1 |  long statements of at least 30 word tokens that follow a greeting and occur within the first 6 utterances in the conversation |  Hey, this is Christine. There is a police report, it was next to you guys why you heard it <…>
problem_phrases |  4.4 |  express problems and concerns |  I’m having an issue with the delivery.
update |  5.8 |  updates and announcements |  I have an update on your passport status.
continuation |  0.4 |  questions preceded by a signpost in the same utterance or a subsequent one from the same speaker |  Hi, I’m calling because I have a question. / Do you accept new patients?
Table 1:  Language patterns in Purpose of Call statements

Approximately 7% of calls in this sample do not contain an explicit Purpose of
Call statement. Instead, the participants in the call appear to already have
the context necessary to understand the call purpose.

####  3.2.2 Knowledge-Engineered Model

As outlined in Section 1, collecting labeled data for Purpose of Call
extraction is a challenging task. Therefore, to obtain a representative sample
of training data, we first implemented a knowledge-based model that takes into
account the following parameters: utterance length in tokens, the order of an
utterance in the conversation, the history including several preceding
utterances, and the presence or absence of language patterns summarized in
Table 1 and implemented using regular expressions syntax (see Appendix A for
an example). In total, stemming from the analysis in Section 3.2.1, 8 rules
(56 regex patterns) to detect call purpose candidates and 6 rules (55 regex
patterns) to filter out negative statements were developed. The model reached
a precision of 90.8% and hit rate of 77% on average across three domains (see
Table 4).

Further, we conducted error analysis by manually labelling the output of the
production system on a random sample of 1000 calls. After human review, we
determined that 3% of calls did not contain an identifiable Purpose of Call
and could be considered true negatives, while 20% were false negatives. 40% of
these false negatives can be attributed to ASR errors. 27.6% of false
negatives include cases with the Purpose of Call being known prior to the
conversation (e.g. from shared knowledge, logged information, or in return
calls) and therefore not considered by the model, 9.1% correspond to specific
industries (e.g. transportation) underrepresented in the data used in the
analysis, and 44.7% were caused by the limitations of the rules (note that
these groups of false negatives intersect, hence the percentages do not add up
to 100%). False positives were mainly related to the lack of morphological
flexibility in the rules and speech dysfluencies. In 6.2% of calls, several
utterances were legitimate Purpose of Call statements and the one selected by
the model was not the best one. These findings motivated the need for a
transformer-based model that was more forgiving of ASR noise, had better
generalization power, and was more responsive to changes in the data.

####  3.2.3 Transformer-Based Model

Training Data Collection. Since the knowledge-engineered model achieved high
precision, we could rely on its output to train a deep learning model. The
dataset consists of English language utterances obtained from business calls
in a variety of industries, with accompanying metadata such as timestamps for
each token, call side, and call id. See Appendix C and 3.2.1 for detailed
statistics. We randomly sampled one million utterances between February 6,
2020 and February 22, 2021, allowing only those that meet the requirements for
a Purpose of Call candidate in 3.1. The utterances were divided into two sets:
(1) those from the calls with a Purpose of Call hit (likely to be a true
positive), (2) utterances from calls with no hit (may contain false
negatives). With a series of patterns, we filtered out utterances that are
likely to be false positives based on error analysis in 3.2.2. Further, we
sampled several datasets of 180K utterances with varying label and language
pattern distributions in order to experimentally find the best configuration
(see Appendix C). A train, development, and validation split of 80/10/10% of
data was used in each experiment. In addition, we created a golden dataset of
909 manually labeled calls, with the utterances organized chronologically
within the call and limited to up to 30 utterances per call. This sample
comprises 13215 utterances.

Training Details. We employ the DistilBERT Sanh et al. (2019) model, trained
for classification with multimodal features. We combine text features with
numerical and binary features, utterance start time and call side
respectively, which have proven to be useful in the knowledge-engineered
model, and pass on a gated summation of the transformer output with these
features to the classification layer, following the approach in Gu and Budhkar
(2021). This configuration outperformed other base models
222https://huggingface.co/microsoft/DialoGPT-small
333https://huggingface.co/DeepPavlov/bert-base-cased-conversational,
combinations of multimodal features, and combining mechanisms outlined in Gu
and Budhkar (2021). The model architecture is shown in Figure 3. We implement
a data-driven iterative fine-tuning process with extensive error analysis and
data resampling. See Appendix D for details.

Figure 3: Multimodal Transformer based scoring model

###  3.3 Model Deployment

Since the model was to operate in a near-realtime environment as a call is
ongoing, optimising for inference time was a dominant consideration during
model design. The model would perform inference on one CPU core. The model
would need to accommodate the time taken to transcribe voice to text and
properly format and punctuate the transcription, many of these tasks being
accomplished by other deep learning models.

Optimizations include: (i) Having the Selection model that uses input features
to filter utterances, thereby reducing the number of utterances that were
attended to by the transformer model. These features include utterance count
number and utterance start time, both of which should be below a threshold
determined by experimenting with different parameter values in the knowledge-
engineered model. (ii) Incorporating numerical and binary features into the
deep learning model - adding signals beyond lexical features allowed us to use
a lower capacity BERT variant with faster inference time. (iii) Capping input
length to an empirically derived ceiling further reduced memory consumption
and inference time.

The system was deployed in containers444https://cloud.google.com/kubernetes-
engine with 1 CPU and maximum 1GB memory per instance. The average inference
time at the 95th percentile is 0.51 seconds, which meets the requirements of
our production system for near-realtime deployments to complete inference in
under 3 seconds.

##  4 Evaluation

Table 2 shows full results of the comparative evaluation of the knowledge-
engineered and the hybrid models on business calls in three domains.

Domain | Model | Precision | Hit Rate | F1
---|---|---|---|---
Support | rules | 93.5 | 80.0 | 86.2
| hybrid | 91.0 | 90.4 | 90.7
General | rules | 90.0 | 74.2 | 81.3
| hybrid | 89.0 | 85.6 | 87.3
Sales | rules | 88.5 | 78.7 | 83.5
| hybrid | 87.0 | 88.9 | 87.9
Avg | rules | 90.6 | 77.6 | 83.6
| hybrid | 89.6 | 88.3 | 88.6
Table 2:  Comparative evaluation of knowledge-engineered (here rules) and
hybrid models for Purpose of Call detection.

Qualitative analysis was conducted on the gold set: real life user data of 600
samples, and 200 calls with missed hits. False positives mainly correspond to
signposting language without mention of the actual Purpose of Call, and
indicate the model’s over-reliance on lexical features. The model was found to
be accurate in assigning utterances to classes, but not always sensitive to
the difference between a valid and the best Purpose of Call. This can be
addressed by introducing more contrastive examples in training data. Missed
hits include cases initially excluded from the sample such as Purpose of Call
stated across several utterances, and multiple Purpose of Call statements of
equal importance. A synthesis of several utterances instead of selecting only
one of them might be useful in such cases.

##  5 Conclusion

This paper discusses the development and deployment of a hybrid system to
detect a Purpose of Call statement in business call transcripts for the
English language in near-realtime settings. We introduce the concept of the
Purpose of Call, provide in-depth analysis of real life data, and discuss
overcoming the absence of available training data by bootstrapping from a
knowledge-engineered model to a deep learning one. Both the knowledge-
engineered and hybrid models demonstrate high precision and hit rate, with the
hybrid model showing better performance while maintaining computational
efficiency.

##  6 Ethics Statement

Data. The conversational data is presented in the form of individual
utterances with sensitive data such as personal identifiable information
removed. No crowdsourced annotation has been conducted, and access to the data
was available only to a small number of in-house Scientists.

Use. The Purpose of Call feature is used by call center managers to identify
areas to coach sales and support agents. It is recommended to not use this
feature for automated evaluation of agent performance. Incorrect Purpose of
Call prediction may provide unsatisfactory user experience for the managers as
they sample calls but does not present any risk of negative impact for the
agents.

Licensing. We follow the licensing requirements accordingly while using
external tools such as HuggingFace 555https://huggingface.co/ and Multimodal-
Toolkit Gu and Budhkar (2021) libraries.

## References

  * Cailliau and Cavet (2013) Frédérik Cailliau and Ariane Cavet. 2013.  Mining automatic speech transcripts for the retrieval of problematic calls.  In _CICLing_.
  * Clavel et al. (2013) Chloé Clavel, Gilles Adda, Frédérik Cailliau, Martine Garnier-Rizet, Ariane Cavet, Géraldine Chapuis, Sandrine Courcinous, Charlotte Danesi, Anne-Laure Daquo, Myrtille Deldossi, Sylvie Guillemin-Lanne, Marjorie Seizou, and Philippe Suignard. 2013.  Spontaneous speech and opinion detection: mining call-centre transcripts.  _Language Resources and Evaluation_ , 47:1089–1125.
  * Dutrey et al. (2014) Camille Dutrey, Chloé Clavel, Sophie Rosset, Ioana Vasilescu, and Martine Adda-Decker. 2014.  A CRF-Based Approach to Automatic Disfluency Detection in a French Call-Centre Corpus.  In _15th Annual Conference of the International Speech Communication Association (Interspeech’14)_ , pages 2897–2901, Singapour, Singapore. International Speech Communication Association (ISCA).
  * Gu and Budhkar (2021) Ken Gu and Akshay Budhkar. 2021.  A package for learning on tabular and text data with transformers.  In _Proceedings of the Third Workshop on Multimodal Artificial Intelligence_ , pages 69–73, Mexico City, Mexico. Association for Computational Linguistics.
  * Hall et al. (2014) Judith Hall, Phil Verghis, William Stockton, and Jin Goh. 2014.  It takes just 120 seconds: Predicting satisfaction in technical support calls.  _Psychology and Marketing_ , 31.
  * Pomerantz and Fehr (2011) Anita Pomerantz and B.J. Fehr. 2011.  Conversation analysis: An approach to the analysis of social interaction.  _Discourse studies: A multidisciplinary introduction_ , pages 165–190.
  * Sacks et al. (1974) Harvey Sacks, Emanuel A. Schegloff, and Gail Jefferson. 1974.  A simplest systematics for the organization of turn-taking for conversation.  _Language_ , 50(4):696–735.
  * Sanh et al. (2019) Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019.  Distilbert, a distilled version of BERT: smaller, faster, cheaper and lighter.  _CoRR_ , abs/1910.01108.
  * Schegloff and Sacks (1973) Emanuel Schegloff and Harvey Sacks. 1973.  Opening up closings.  _Semiotica_ , 8:289–327.
  * Takeuchi et al. (2007) Hironori Takeuchi, L Venkata Subramaniam, Tetsuya Nasukawa, and Shourya Roy. 2007\.  Automatic identification of important segments and expressions for mining of business-oriented conversations at contact centers.  In _Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL)_ , pages 458–467, Prague, Czech Republic. Association for Computational Linguistics.
  * Taleghani-Nikazm (2002) Carmen Taleghani-Nikazm. 2002.  A conversation analytical study of telephone conversation openings between native and nonnative speakers.  _Journal of Pragmatics_ , 34:1807–1832.
  * Żelasko et al. (2019) Piotr Żelasko, Jan Mizgajski, Mikołaj Morzy, Adrian Szymczak, Piotr Szymański, Łukasz Augustyniak, and Yishay Carmiel. 2019.  Towards better understanding of spontaneous conversations: Overcoming automatic speech recognition errors with intent recognition.

##  Appendix A Appendix: Example Rule in a Knowledge-Engineered Model

An utterance is a Purpose of Call if:

  * •

It contains signposting phrases expressing a problem such as I’m having a
problem, There is an issue, I’m having a hard time, I’m trying … and it’s not
working,

  * •

It occurs within the first 10 utterances

  * •

It is at least 12 tokens long

Example: I got a really big problem here. When I log in, it asks for some pin,
and I really, I can’t use it. So there’s obviously an issue here and can you
help me with it?.

##  Appendix B Appendix: Example Selection model heuristics

Combine the positive score for an utterance with the maximum question score of
the two preceding utterances in another call side. If it passes a threshold
and is the biggest score so far, this utterance is a Purpose of Call.

##  Appendix C Appendix: Data Statistics

Total number of calls: 86310
Total number of utterances: 180 000
Industry distribution: see Table 3.

Industry |  %
---|---
Technology |  25.1
IT, Consulting |  15.5
Professional, Business Support Services |  14.1
Travel |  11.4
Health and Wellness |  5.6
Real Estate |  5.1
Table 3: Industry distribution in training data: top 6 types

Label distribution: A key factor in training the model was determining the
right distribution of labels and language patterns. The classes in our problem
are naturally imbalanced: since only one utterance per call is a valid Purpose
of Call, the vast majority of utterances are of the negative class. In a
random sample, only 4.7% utterances are positive hits, and only 1.9% are
questions. If the data is sampled randomly, the model is likely to overfit to
the negative class. Sampling uniformly may reduce the number of complex
instances in favor of the ones easier for the model to learn. A set of
experiments were conducted to determine the distribution of classes with the
goal of optimizing accuracy of the Purpose of Call class predictions. We
determined the optimal distribution of classes as follows: 42.5% positive,
42.5% negative, 15% question utterances (corresponds to the share of this
pattern in real data). All utterances came from calls with a positive hit,
which minimized the chance of false negatives in the training data.

Language patterns distribution: From the error analysis and experiments, we
determined the optimal distribution of language patterns within the positive
class:

  * •

30% call_purpose_phrases

  * •

30% desire_phrases

  * •

20% problem_phrases

  * •

20% other patterns

Other aspects of the data are the same as described in 3.2.1.

##  Appendix D Appendix: Training details

Parameters: The pretrained distilbert-base-cased model we use has 6 layers,
768 hidden units, 12 attention heads and 65M parameters and is available
through Multimodal-Toolkit Gu and Budhkar (2021). We run all fine-tuning
experiments on a Google Cloud VM n1-standard-8 instance with 496GB disk size
and 1 NVIDIA Tesla K80 GPU. The maximum time for a single experiment was 8 GPU
hours. We truncate text input to a maximum 150 tokens since most relevant
statements fall into this category. We set the train and eval batch size to 32
and 64 respectively, and use AdamW optimizer with default parameters. We fine-
tune the model for 4 epochs with a learning rate of 5e-05, weight decay of
0.01 and 500 warm up steps. The hyperparameters were obtained from experiments
using an in-house tuning tool implementing grid search algorithm. For fine-
tuning on small subsets (4K) of data collected through error analysis, we
repeat the training process for 12 epochs and a learning rate of 9e-05.

Relevant features: Besides the text features, we experimented with two extra
features that have proven to be useful in the knowledge-engineered model: the
start time of the utterance and the call side. We also experimented with
several mechanisms to combine the numerical and categorical features with
textual data using Multimodal-Toolkit Gu and Budhkar (2021). The results are
presented in Table 4.

Feature |  P |  HR |  F1 |  PP
---|---|---|---|---
text only |  0.891 |  0.891 |  0.891 |  0.894
text + start time |  0.948 |  0.948 |  0.948 |  0.944
text + call side |  0.948 |  0.948 |  0.948 |  0.946
all |  0.949 |  0.949 |  0.949 |  0.957
Table 4:  Comparing model performance using tabular features start time and
call side. P-Precision, HR-Hit rate, PP - precision in positive class. The
results are reported for a single run using concatenation to combine features.