# Economical Accommodations for Neurodivergent Students in Software
Engineering Education: Experiences from an Intervention in Four Undergraduate
Courses

Grischa Liebel and Steinunn Gróa Sigurðardóttir

##  1 Introduction

Neurodiversity is an umbrella term that describes variation in brain function
among individuals [1], including conditions such as Autism spectrum disorder
(ASD), Attention deficit hyperactivity disorder (ADHD), or dyslexia.
Neurodiversity is common in the general population, with an estimated 5.0% to
7.1% [21, 19] and 7% [17] of the world population being diagnosed with ADHD
and dyslexia respectively. Neurodivergent (ND) individuals often experience
challenges in specific tasks, such as difficulties in communication or a
reduced attention span in comparison to neurotypical (NT) individuals [6].
However, they also exhibit specific strengths, such as high creativity [20] or
attention to detail [2]. Therefore, improving the inclusion of ND individuals
is desirable for economic, ethical, and for talent reasons.

In higher education, struggles of ND students are well-documented [8, 7, 18].
Common issues in this area are a lack of awareness among other students and
staff, forms of assessment that are particularly challenging for some
students, and a lack of offered accommodations. These factors commonly lead to
stress, anxiety, and ultimately a risk of dropping out of the studies.

Accommodations for ND students can require substantial effort. However,
smaller changes in course material can already have major impact. In this
chapter, we summarise the lessons learned from an intervention in four courses
in undergraduate computer science programmes at Reykjavik University, Iceland,
over a period of two terms. Following accessibility guidelines produced by
interest groups for different ND conditions, we created course material in the
form of slides and assignments specifically tailored to ND audiences. We
focused on small, economical changes that could be replicated by educators
with a minimal investment of time. We evaluated the success of our
intervention through two surveys, showing an overall positive response among
ND students and neurotypical (NT) students. Example materials we produced are
available at https://doi.org/10.5281/zenodo.7199162.

In summary, in this chapter you will learn: 1. Which obstacles ND students
commonly face in higher education. 2. What accommodations are recommended for
ND students. 3. What economical interventions for Computer Science students
can look like. 4. What feedback we received on making such an intervention.

##  2 The State of Knowledge in Research

Work on neurodiversity is broad and covers different areas. In addition to
work within the realms of Medicine and Psychology, there are many studies
investigating neurodiversity in higher education, neurodiversity and
employment, and piloted interventions that aim to improve inclusion of ND
students in education or ND individuals in the workplace. We briefly summarise
these three areas as follows.

Neurodiversity in Higher Education and Industry 1. ND students often under-
perform in specific task, due to the task’s nature. 2. Awareness and
accommodations for ND students are commonly lacking. 3. ND students often feel
anxious, depressed, or afraid of being stigmatised. 4. ND individuals in
industry often choose to not disclose their condition. 5. Accommodations for
ND individuals in industry vary considerably. 6. There is only initial work on
neurodiversity in Software Engineering.

ND conditions are often diagnosed in early childhood. Therefore, much of
existing work targets primary and secondary school education. As inclusion of
ND students improves in these areas, the amount of ND students enrolling in
higher education increases. That is, educators in higher education need to
increase their awareness of neurodiversity and possible accommodations for ND
students [9]. In particular, educators should understand that ND students make
mistakes or under-perform in certain tasks due to the tasks’ nature, not due
to a lack of intelligence.

Experiences of ND students in higher education are published in several
studies. A summary of research themes surrounding neurodiversity in higher
education is presented by Clouder et al. [4], based on 48 screened
publications. The findings show that many higher education institutes provide
insufficient support for neurodiversity, primarily due to “low levels of staff
awareness, ambivalence and inflexible teaching and assessment approaches”. ND
students often feel frustrated due to the lack of available accommodations,
anxious or frightened due to challenging situations or stigmatisation, and
often embarrassed to ask questions. Finally, inclusive and trusting
environments, as well as accommodations such as extended exam times or
flexible assignment schedules benefit ND students.

Experiences and support of college and university students with autism are
summarised in a systematic literature review (SLR) by Gelbar et al. [7]. The
authors emphasise the need for support in higher education, and that anxiety
and depression among autistic students are common.

Regarding students with dyslexia in higher education, Pino and Mortari [18]
summarise existing research, finding that research is fragmented and contains
many gaps. The article reports coping strategies such as getting support from
family and friends, or adapting their writing styles for written assignments.
Furthermore, the authors highlight a common lack of awareness and acceptance
by academic staff, and mention that accommodations are usually appreciated by
students.

In the IT industry, inclusion of ND individuals is typically at an early stage
and companies need to become more welcoming and inclusive [22]. Similar to
higher education, ND individuals often do not disclose their condition due to
fear of stigmatisation [16, 10]. However, depending on the workplace the
number of ND individuals disclosing their condition and the offered
accommodations differ considerably.

Driven by the Covid-19 pandemic, several studies investigate how remote work
affects ND individuals. Das et al. [5] find that ND individuals create
accessible workplaces at home, negotiate communication and meeting practices,
and balance tensions between productivity at work and fatigue. The authors
suggest to commonly record audio and video, enable automated meeting
transcripts, and make meeting notes available routinely. In a similar
direction, the needs of adults with ASD in video calling are studied by
Zolyomi et al. [23]. The authors find that adults with ASD develop several
coping strategies, such as adopting neurotypical behaviour, and that they
experience more stress than NT individuals without adaptations. The authors
provide suggestions for adaptations, e.g., translating and communicating
social and emotional information to adults with ASD.

Specific to Software Engineering, there is only initial work on neurodiversity
and potential accommodations for ND individuals. Morris et al. [16]
investigate challenges faced by ND individuals. Specific themes related to SE
were the tendency to get bored with mundane tasks, or expressing inappropriate
emotions, e.g., when criticised during code reviews. Furthermore, ND
individuals perceive several tasks more challenging than NT individuals, e.g.,
working in shared offices or noisy settings, or deciding when to seek help for
tasks. Begel et al. [3] report the results of a successful 13-day game
programming camp for incoming college students with ASD. In addition to
teaching game development, students were instructed specifically in
communication skills. In a line of research investigating how dyslexic
individuals read and comprehend software code, McChesney and Bond [12, 11, 13,
15, 14] find that, contrary to intuition, dyslexic individuals do not exhibit
over-proportional deficiencies compared to NT individuals. Potential
explanations are that reading code is significantly different from regular
text, e.g., due to indentation or spacing.

## Possible Accommodations for Neurodivergent Students

Several guidelines on how to accommodate ND students have been published by
interest groups for different neurodivergent conditions. These guidelines
typically target one specific condition at a time. Deciding on possible
accommodations for a class or a university therefore requires synthesising
these guidelines into one set of concrete accommodations that somehow align
well with several existing guidelines. This is a difficult step, given that
existing guidelines might contradict each other and might propose too many
accommodations for a feasible intervention in practice. In the following we
will summarise three guidelines, one for each of the most common ND conditions
ASD, ADHD, and dyslexia.

Accommodation Areas for Different ND Conditions 1. ASD: Sensory environment,
communication, escape ways, awareness. 2. ADHD: Patience and understanding,
clear structure, communication. 3. Dyslexia: Style and presentation of visual
material.

For ASD, we consider a checklist for Autism-friendly environments published by
the
NHS111https://positiveaboutautism.co.uk/uploads/9/7/4/5/97454370/checklist_for_autism-
friendly_environments_-september_2016.pdf. The checklist describes
accommodations to four areas: the sensory environment, e.g., reducing strong
colours or overly patterned surfaces, or reducing smell and noise; to the
communication environment, e.g., the use of clear and unambiguous signs; to
provide escape ways for autistic individuals in case of high stress levels;
and to provide general awareness of ASD among the employees.

For ADHD, we considered a report by the ADHD foundation on how to teach and
manage students with ADHD222https://www.adhdfoundation.org.uk/wp-
content/uploads/2022/03/Teaching-and-Managing-Students-with-ADHD.pdf. The
report provides a general introduction to ADHD and to common behaviours of
individuals with ADHD. Then, suggestions are made on how classrooms can be
made more accessible to students with ADHD. First, teachers are encouraged to
show patience and understanding for behaviour that might seem odd, such as
excessive movement or inappropriate comments. Second, providing a structure
and flexibility is important, e.g., through regular routines, or by providing
overviews before starting a class or a checklist for an assignment. Finally,
the report lists a few hints on communicating with students with ADHD, e.g.,
trying to tell the student what they should do instead of what not to do, or
addressing them by name.

Finally, for dyslexia, we consider the style guide by the British Dyslexia
Association333https://www.bdadyslexia.org.uk/advice/employers/creating-a-
dyslexia-friendly-workplace/dyslexia-friendly-style-guide. This guide focuses
on accommodations in the style of visual material, e.g., assignment texts or
lecture slides. Some of the changes they recommend are using fonts that are
easier to read for dyslexic individuals, e.g., Arial and Comic Sans444Yes, you
read that correctly.; increasing font size; increasing spacing between
letters, words and lines; using left-aligned instead of justified text;
avoiding multi-column texts; and using cream-coloured backgrounds instead of
plain white. Additionally, adapting the writing style is recommended, e.g., by
avoiding passive voice, using concise sentences, or avoiding jargon. Many of
these recommendations overlap with traditional style guidelines for
presentations, e.g., using large font sizes and avoiding red and green as
colours, and for academic writing, e.g., avoiding overly long sentences and
jargon.

##  3 An Example Intervention in Four Courses

Based on the guidelines and the existing academic work outlined in the
previous section, we designed an intervention in initially three courses which
we were teaching in the fall term 2021 at Reykjavik University, Iceland. In
the spring term 2022, we used the intervention in a slightly altered way in
one more course. All four courses are part of the undergraduate computer
science and software engineering programmes in the first and second year at
Reykjavik University. The courses are an introductory Software Engineering
course, an introductory web development course, and two courses on discrete
mathematics. During the interventions, we were operating in a hybrid mode of
on-site teaching and remote teaching through video conferencing tools. Prior
to our intervention, the only accommodations ND students could receive were
extended exam times, usually limited to the final exams only. Approximately
10% of the students at Reykjavik University have a diagnosed ND condition and
applied for this extension, though the percentage was higher in some of the
four courses.

The accommodations were made as a part of the regular teaching. That is, there
was no specific time, budget, or workforce assigned to making the chosen
accommodations. Therefore, we focused on small, economical changes that we
could implement under regular time pressure. In summary, we made the following
changes.

Implemented Accommodations 1. Emphasis on style changes in assignments and
lecture slides. 2. Improved clarity and structure. 3. Minor changes in sensory
environment, e.g., consistent lighting. 4. Discussing neurodiversity and the
accommodations to raise awareness.

Specifically, we decided to primarily change the style of our assignments and
slides. We changed font type to OpenDyslexic555https://opendyslexic.org, the
background colour to a cream colour, increased inter-letter, intra-letter, and
line spacing, avoided italics, multiple columns, and specific colours, such as
red and green, or strong colours in general. We did not change the default
fonts in case of mathematical formulas and program code. To leave the choice
of style up to the students, we offered assignments both in a traditional
style and in our updated, ND-friendly style.

In terms of clarity and communication, we added bullet-point summaries to all
assignments. Additionally, we proof-read the assignments and tried to reduce
ambiguity, jargon and unclear acronyms, and improve overall clarity. Examples
of the slides and assignments created in this way can be found at
https://doi.org/10.5281/zenodo.7199162.

In addition to adapting the style and content of our slides and assignments,
we made minor changes to the sensory environment, e.g., avoiding clothes with
strong patterns or colours, and in remote teaching using high-quality
microphones and consistent artificial lighting. Finally, we discussed the
changes and the reasons behind them in the classroom, with the intention to
raise awareness for neurodiversity and make ND students feel seen and welcome.

Overall, the accommodations were efficient. For assignments, we used LaTeX and
thus only had to re-compile the assignment with slightly changed parameters.
Re-reading assignments and slides and improving clarity was done as a part of
updating existing assignments, which we argue should be part of preparing
assignments in any case. Changing the sensory environment primarily required
us to be conscious about potential issues, such as avoiding clothes with
strong patterns or colours. Finally, the largest effort was required for
updating lecture slides in Microsoft PowerPoint. While we made style changes
on the slide master level, i.e., globally for all slides, increased font size
and spacing required changes to individual slides to fit all content. To
update existing slides took approximately 30 minutes per slide deck (per
lecture). This effort is likely much lower if you create lecture slides from
scratch, and not, as in our case, update existing ones.

Through two surveys, one after each term, we evaluated the impact our
intervention had on all students in our courses. We received answers from 169
students in all courses, corresponding to an overall response rate of 23.57%.
Of those who answered, we only considered those that completed the survey,
resulting in 155 valid answers. Out of the 155 students, 63 (40.64%) self-
identified as neurodivergent, 89 as neurotypical, and 3 did not disclose. This
corresponds to 108 students (48 ND, 59 NT, 1 not answered) in the Fall term
2021 (3 courses), and 47 students (15 ND, 30 NT, 2 not answered) in the Spring
term 2022 (1 course).

##  4 Intervention Experiences

While our students had the choice which material they used, ND-friendly
material was used by over 50% of the respondents, among them many NT students.
The detailed break down is depicted in Figure 1 for the three courses in the
fall term 2021, and for the remaining course in the spring term 2022.
Interestingly, several NT students commented in free text answers that they
also found the ND-friendly material more readable compared to the traditional
style and therefore preferred it.

Many students, both NT and ND, further expressed gratitude that we raised
awareness for neurodiversity. Finally, five ND students stated that they felt
welcome in our courses. Given that academic literature regularly points out
that ND students experience unwelcoming environments in higher education, we
consider this one of the most important outcomes of our intervention.

Figure 1: Used materials in the fall term 2021 (n=108) and spring term 2022.
All students are depicted in the white bars, while ND students are shown in
grey and NT students in black.

We further asked participants to rate whether they (a) appreciate that
material is provided in two styles, (b) find the ND-friendly material harder
to read, and (c) find that the ND-friendly material helps them understand the
content better. The results are depicted in Table 1. Note that the statements
in the first column are shorter, simplified versions of the actual statements,
which can be found in the material we provide.

Table 1: Perceptions by All Participants Regarding the ND-friendly Course Material in Fall and Spring Terms. Statement |  Fall ’21 |  Spring ’22
---|---|---
|  (n=108) |  (n=47)
I appreciate that material is provided in two styles. |  1% disagree,
5% neutral,
94% agree |  4% disagree,
4% neutral,
91% agree
The ND-friendly material is harder to read. |  65% disagree,
13% neutral,
22% agree |  52% disagree,
8% neutral,
40% agree
The ND-friendly material helps me understand the content better. |  27% disagree,
30% neutral,
42% agree |  34% disagree,
31% neutral,
34% agree

The answers show that accommodations were received well by most of the
participants. Furthermore, a large proportion of all participants found that
the ND-friendly material helped them understand the material better.
Considering only ND participants, the agreements are naturally higher. That
is, 48% of the ND students in the first survey agree that the ND-friendly
material helped their understanding (compared to 34% of the NT students), and
54% of the ND students agree in the second survey (compared to only 12% of NT
students). This shows that the accommodations reached their goal of primarily
supporting ND students. Nevertheless, the divided agreements and disagreements
also clearly show that our simple accommodations are no silver bullet.

In addition to closed questions, we received a lot of noteworthy feedback as
free-text answers. Four students found the changed font particularly hard to
read, especially in assignments, and therefore preferred the traditional
material, even though it might not be ideal either. For example, one NT
student commented in free text that they felt “very distracted” by the new
font. Given this feedback and the fact that we used a non-standard font, we
are currently considering to use a standard font that is considered suitable
for dyslexic audiences in the future, e.g., Arial. Three ND students commented
that they did not want accommodations, or that they were satisfied with
extended exam time already provided by the university. Finally, a few comments
showed the individual nature of neurodiversity. One dyslexic student commented
on the difficulty of following a course offered in English only, while another
student expressed difficulties as the course they attended was given in
Icelandic, but the course book was English. That is, they struggled primarily
with the multi-lingual nature of the course. Similarly, one ND student stated
that working from home had a positive impact on their productivity, while
another student stated during one of our courses that they found it especially
hard to focus at home due to their ND condition. Overall, these comments show
that simple changes are unlikely to cater to all ND audiences, and that
flexibility in accommodations is needed. However, it is worth pointing out
that we did not receive any negative feedback.

In addition to direct feedback on our accommodations, several students
provided suggestions on additional accommodations. Particularly common were
suggestions to provide video or audio recordings in which we explain
assignments, or simply read them out loud. This clearly highlights
difficulties many ND students face with text comprehension. However, it also
shows the potential benefit of text-to-speech tools that have been improving
substantially in the last years. Another common point for improvement is the
organisation and presentation of a single course. Several ND students pointed
out that it can be hard to find all the relevant information on a course.
Additionally, syllabi in our programmes are currently not standardised, which
leads to large differences in how courses present information such as learning
outcomes. This poses an additional obstacle for many ND students. We believe
these two directions are alternatives to our style and presentation based
approach that could provide particularly valuable accommodations, i.e.,
different modalities for course material and improved organisation.

##  5 Summary

In this chapter, we summarised potential accommodations for neurodivergent
(ND) students in higher education, focusing on the three most common
conditions Autism Spectrum Disorder (ASD), Attention Deficit Hyperactivity
Disorder (ADHD), and dyslexia. We outlined which accommodations we made to
four courses in Computer Science and Software Engineering undergraduate
programs, focusing on changes in style to assignments and slides that are easy
to implement with limited time. Our results show that minor accommodations for
ND students can have an important impact, from some ND students feeling more
seen and welcome in courses, to actual perceived improvements in understanding
the course material. Researchers and educators can use our provided material
to conduct studies and develop their own interventions.

Our accommodations were of general nature, even though we applied them in a
computer science and Software Engineering context. We consider this an
important starting point before applying accommodations that target specific
characteristics of or tasks in Software Engineering. Future work in this area
should target how to convey course organisation in a better way, and how to
use other media than text for expressing tasks, assignments, and project
descriptions, such as video or audio recordings. Furthermore, we believe that
more work is needed that studies what tasks in Software Engineering are
particularly difficult or easy for ND individuals, and why. To this end, we
have started investigating which strengths ND individuals exhibit with respect
to Software Engineering tasks, and how they can use these in a better way.

Take-Away Points 1. Neurodiversity is not rare, and is increasing in higher
education. 2. Many accommodations are difficult and effort intensive. 3.
Small, economical changes can make a difference. 4. Awareness is a first,
important step. 5. Feedback clearly shows the individual nature of
neurodiversity.

## References

  * [1] Thomas Armstrong.  The myth of the normal brain: Embracing neurodiversity.  AMA journal of ethics, 17(4):348–352, 2015.
  * [2] Simon Baron-Cohen, Emma Ashwin, Chris Ashwin, Teresa Tavassoli, and Bhismadev Chakrabarti.  Talent in autism: hyper-systemizing, hyper-attention to detail and sensory hypersensitivity.  Philosophical Transactions of the Royal Society B: Biological Sciences, 364(1522):1377–1383, 2009.
  * [3] Andrew Begel, James Dominic, Conner Phillis, Thomas Beeson, and Paige Rodeghero.  How a remote video game coding camp improved autistic college students’ self-efficacy in communication.  In Proceedings of the 52nd ACM Technical Symposium on Computer Science Education, pages 142–148, 2021.
  * [4] Lynn Clouder, Mehmet Karakus, Alessia Cinotti, María Virginia Ferreyra, Genoveva Amador Fierros, and Patricia Rojo.  Neurodiversity in higher education: A narrative synthesis.  Higher Education, 80(4):757–778, 2020.
  * [5] Maitraye Das, John Tang, Kathryn E Ringland, and Anne Marie Piper.  Towards accessible remote work: Understanding work-from-home practices of neurodivergent professionals.  Proceedings of the ACM on Human-Computer Interaction, 5(CSCW1):1–30, 2021.
  * [6] Equality and Participation Unit, UCU.  Neurodiversity in the workplace - supporting neurodivergent members at work and campaigning for neurodiversity-friendly workplaces.  Technical report, University and College Union, Jan 2022.
  * [7] Nicholas W Gelbar, Isaac Smith, and Brian Reichow.  Systematic review of articles describing experience and supports of individuals with autism enrolled in college and university programs.  Journal of autism and developmental disorders, 44(10):2593–2601, 2014.
  * [8] Edward Griffin and David Pollak.  Student experiences of neurodiversity in higher education: insights from the brainhe project.  Dyslexia, 15(1):23–41, 2009.
  * [9] Ann Jurecic.  Neurodiversity.  College English, 69(5):421–442, 2007.
  * [10] Sally Lindsay, Victoria Osten, Mana Rezai, and Sunny Bui.  Disclosure and workplace accommodations for people with autism: A systematic review.  Disability and Rehabilitation, 43(5):597–610, 2021.
  * [11] Ian McChesney and Raymond Bond.  Gaze behaviour in computer programmers with dyslexia: considerations regarding code style, layout and crowding.  In Proceedings of the Workshop on Eye Movements in Programming, pages 1–5, 2018.
  * [12] Ian McChesney and Raymond Bond.  Eye tracking analysis of computer program comprehension in programmers with dyslexia.  Empirical Software Engineering, 24(3):1109–1154, 2019.
  * [13] Ian McChesney and Raymond Bond.  Observations on the linear order of program code reading patterns in programmers with dyslexia.  In Proceedings of the Evaluation and Assessment in Software Engineering, pages 81–89, 2020.
  * [14] Ian McChesney and Raymond Bond.  The effect of crowding on the reading of program code for programmers with dyslexia.  In 2021 IEEE/ACM 29th International Conference on Program Comprehension (ICPC), pages 300–310. IEEE, 2021.
  * [15] Ian McChesney and Raymond Bond.  Eye tracking analysis of code layout, crowding and dyslexia-an open data set.  In ACM Symposium on Eye Tracking Research and Applications, pages 1–6, 2021.
  * [16] Meredith Ringel Morris, Andrew Begel, and Ben Wiedermann.  Understanding the challenges faced by neurodiverse software engineering employees: Towards a more inclusive and productive technical workforce.  In Proceedings of the 17th International ACM SIGACCESS Conference on computers & accessibility, pages 173–184, 2015.
  * [17] Robin L Peterson and Bruce F Pennington.  Developmental dyslexia.  The lancet, 379(9830):1997–2007, 2012.
  * [18] Marco Pino and Luigina Mortari.  The inclusion of students with dyslexia in higher education: A systematic review using narrative synthesis.  Dyslexia, 20(4):346–369, 2014.
  * [19] Guilherme Polanczyk, Maurício Silva De Lima, Bernardo Lessa Horta, Joseph Biederman, and Luis Augusto Rohde.  The worldwide prevalence of adhd: a systematic review and metaregression analysis.  American journal of psychiatry, 164(6):942–948, 2007.
  * [20] Holly A White and Priti Shah.  Creative style and achievement in adults with attention-deficit/hyperactivity disorder.  Personality and Individual Differences, 50(5):673–677, 2011.
  * [21] Erik G Willcutt.  The prevalence of dsm-iv attention-deficit/hyperactivity disorder: a meta-analytic review.  Neurotherapeutics, 9(3):490–499, 2012.
  * [22] Sarah Wille and Daphne Sajous-Brady.  The inclusive and accessible workplace.  Communications of the ACM, 61(2):24–26, 2018.
  * [23] Annuska Zolyomi, Andrew Begel, Jennifer Frances Waldern, John Tang, Michael Barnett, Edward Cutrell, Daniel McDuff, Sean Andrist, and Meredith Ringel Morris.  Managing stress: The needs of autistic adults in video calling.  Proceedings of the ACM on Human-Computer Interaction, 3(CSCW):1–29, 2019.


#  Godel: Large-Scale Pre-Training for Goal-Directed Dialog

Baolin Peng$\dagger$ Michel Galley$\dagger$ Pengcheng He$\dagger$ Chris
Brockett$\dagger$
Lars Liden$\dagger$ Elnaz Nouri$\dagger$ Zhou Yu$\ddagger$ Bill Dolan$\dagger$
Jianfeng Gao$\dagger$
$\dagger$ Microsoft Corp. $\ddagger$ Columbia University
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
###### Abstract

We introduce Godel (Grounded Open Dialogue Language Model), a large pre-
trained language model for dialog. In contrast with earlier models such as
DialoGPT, Godel leverages a new phase of grounded pre-training designed to
better support adapting Godel to a wide range of downstream dialog tasks that
require information external to the current conversation (_e.g.,_ a database
or document) to produce good responses. Experiments against an array of
benchmarks that encompass task-oriented dialog, conversational QA, and
grounded open-domain dialog show that Godel outperforms state-of-the-art pre-
trained dialog models in few-shot fine-tuning setups, in terms of both human
and automatic evaluation. A novel feature of our evaluation methodology is the
introduction of a notion of utility that assesses the usefulness of responses
(extrinsic evaluation) in addition to their communicative features (intrinsic
evaluation). We show that extrinsic evaluation offers improved inter-annotator
agreement and correlation with automated metrics. Code and data processing
scripts are publicly available.111http://aka.ms/GODEL

Godel: Large-Scale Pre-Training for Goal-Directed Dialog

Baolin Peng$\dagger$ Michel Galley$\dagger$ Pengcheng He$\dagger$ Chris
Brockett$\dagger$ Lars Liden$\dagger$ Elnaz Nouri$\dagger$ Zhou Yu$\ddagger$
Bill Dolan$\dagger$ Jianfeng Gao$\dagger$ $\dagger$ Microsoft Corp. $\ddagger$
Columbia University
<EMAIL_ADDRESS><EMAIL_ADDRESS>

##  1 Introduction

This work describes the development of a very large pre-trained dialog model –
Grounded Open Dialogue Language Model (Godel). As the name indicates, Godel is
designed for general-domain conversation and is fully open-sourced. Godel
should be of technical interest for two reasons. First, it is pre-trained in
three phases, successively folding in data from web text, publicly-available
dialog (_e.g.,_ Reddit), and a collection of existing corpora that support
grounded dialog tasks. The grounded dialog corpora, which include MS MARCO
Nguyen et al. (2016) and DSTC7 Yoshino et al. (2019), allow for more effective
fine-tuning on dialog tasks where responses must be conditioned on information
external to the current conversation (_e.g.,_ a retrieved document.) Second,
Godel is validated on a utility-driven suite of benchmarks specifically
designed for few-shot fine-tuning of open-ended goal-directed general-domain
dialog models. We will show that Godel, as validated using this methodology,
is more readily amenable to fine-tuning for goal-directed dialog tasks than
other large pre-trained language models.

Our approach seeks to address a long-standing obstacle to general-purpose
open-ended conversation models, namely a lack of robust automated evaluation
criteria that can drive development (Gao et al., 2019). Recent state-of-the-
art models that leverage large PLMs (_e.g.,_ Zhang et al., 2019b; Freitas et
al., 2020; Roller et al., 2021; Bao et al., 2021; Thoppilan et al., 2022; Gao
et al., 2022) offer the potential for substantive open-ended conversational
interactions, yet they resist meaningful comparison owing to the lack of
consensus on evaluation.

This poses a fundamental question: what do we want of a good general-purpose
dialog model in the first place? We take it as a given that it should be
fluent and socially engaging. Indeed, most SOTA PLMs are primarily evaluated
on such intrinsic communicative dimensions. But beyond that, we must also
acknowledge that machine-human conversation typically serves a purpose and
aims to fulfill one or more goals on the part of the user. In other words, the
model must offer utility to the user. It is this extrinsic dimension of
functional utility, we suggest, that constitutes the proper focus of automated
evaluation in general-domain models.

In the second half of this paper (Sections 5 and 6), we explore this notion of
Utility in fine-tuning Godel on four established tasks that cover multiple
domains and conversational settings, ranging from the task-oriented MultiWOZ
Budzianowski et al. (2018) to more open-ended goal-oriented tasks, _i.e.,_
CoQA Reddy et al. (2019), Wizard of Wikipedia Dinan et al. (2018), and Wizard
of the Internet Komeili et al. (2021). We show that Godel improves on a
baseline DialoGPT Zhang et al. (2019b), with performance gains in goal-
directed tasks. Our results on these diverse tasks speak to the effectiveness
of our approach. We also find that the focal shift to goal-directedness and
extrinsic evaluation affords better inter-rater agreement and human-metric
correlations.

As baselines for future research, we release three versions of Godel: small
(GodelB) and large (GodelL) initiated from T5 and T5-Large respectively, and a
very large model (Godel${}_{\mathtt{XL}}^{\mathtt{GPT-J}}$) that is trained
off GPT-J Gao et al. (2020); Wang and Komatsuzaki (2021).222The discussion of
our largest model in this paper relates to GodelXL which is based on GPT-3,
but which we are unable to release owing to licensing restrictions. We
therefore release Godel${}_{\mathtt{XL}}^{\mathtt{GPT-J}}$ a substitute model
based on GPT-J that performs similarly. A comparison of GODEL models
initialized from GPT-3 and GPT-J is provided in the Appendix. We also furnish
code to download datasets for fine-tuning and evaluation. The release is fully
open sourced (models, data, and code) and tightly integrated with the latest
version of HuggingFace’s transformers library.

##  2 Related Work

Large pre-trained language models (PLMs) for dialog Zhang et al. (2019b);
Freitas et al. (2020); Roller et al. (2021); Peng et al. (2021); Bao et al.
(2021); Thoppilan et al. (2022) have had significant impact in Conversational
AI, as these models have brought strong performance gains in competitions such
as DSTC Gunasekara et al. (2020) and ConvAI Aliannejadi et al. (2020), and
have been widely adopted in both task-oriented and chitchat research. While
the applications of these dialog models are numerous, empirical evaluation of
dialog pre-training has mostly been restricted to intrinsic evaluation
(_i.e.,_ agnostic to any notion of utility). For example, DialoGPT Zhang et
al. (2019b) offered Reddit-based pre-trained models that are evaluated on
Relevance, Informativeness, and Humanness and various string overlap based
automated evaluation metrics. Meena Freitas et al. (2020)’s evaluation is
focused on Sensibleness and Specificity, while also offering a new automated
evaluation (SSA) that measures these two qualities. BlenderBot Roller et al.
(2021) contributes a variety of models that capitalize on skills including
personality, empathy, and knowledge, but evaluation in this work centers
mostly on measures of Humanness and Engagingness.

Plato-XL Bao et al. (2021) focuses on intrinsic evaluation with measures of
Coherence, Inconsistency, Informativeness, Hallucination, and Engagingness.
Plato-XP does however provide some extrinsic evaluation on DSTC9-Track1 Kim et
al. (2020), MultiWOZ 2.2 Zang et al. (2020), and DuConv Wu et al. (2019), but
is based on task-specific automated metrics (ROUGE-L for DSTC9, Goal-Accuracy
for MultiWOZ, and Knowledge-F1 for DuConv). By contrast, our work provides a
more unified measure of Utility, which is measured the same fashion (_e.g.,_
same annotation instructions) across dialog tasks and datasets. Finally, LaMDA
Thoppilan et al. (2022) presents dialog models motivated by dialog
applications, but is evaluated primarily using intrinsic measures
(Sensibleness, Specificity, Interestingness, Safety, Groundedness). In its
human evaluation, LaMDA does consider an extrinsic evaluation measure of
Utility (Helpfulness), and is probably the closest to our work in terms of
analyzing PLMs with a unified extrinsic evaluation. Our works seeks to go
further by contrasting intrinsic and extrinsic measures, and analyzing their
correlates with automatic evaluation on specific tasks and datasets.

##  3 Open-Domain Goal-Directed Dialog

In the goal-directed framework of this paper, we seek to generate human-like
conversational responses that attain a high level of Utility. While prior work
in task-oriented dialog has used various measures of Utility (_e.g.,_ Inform-
rate and Success-rate for MultiWOZ Budzianowski et al. (2018), Knowledge-F1
for Wizard of Wikipedia Dinan et al. (2018)), these ad-hoc measures are often
heterogeneous and only applicable to a limited number of subtasks. This makes
it difficult to analyze results across dialog tasks and datasets. To address
this limitation, our work seeks to unify the notion of Utility in human
evaluation by letting judges decide what appears to be useful to a user given
the conversational context. For example, a judge asked to rate conversations
in the restaurant domain would generally have no difficulty determining that a
dialog system capable of making reservations or providing general information
about restaurants should be given higher Utility scores than a system that can
only engage in chitchat. The generality of Utility is comparable to that of
other qualities such as Humanness and Interestingness Li et al. (2019) often
used in chitchat work, but unified evaluation of this kind is rarely used in
more task-oriented dialog research. Section 6 will analyze and contrast both
intrinsic (_e.g.,_ Humanness) and extrinsic (_e.g.,_ Utility) qualities
measured under human evaluation, and correlate them with automatic evaluation
metrics.

Considering the goal of generating responses that attain high Utility, we cast
the open-domain goal-directed dialog generation task as follows: Given a
source dialog context $S=(s_{1},\cdots,s_{N})$ and an environment $E$, the
goal is to generate a target sentence $T=(t_{1},\cdots,t_{N})$. The
conditional probability of $P(T|S,E)$ can be written as the product of a
series of conditional probabilities:

$p(T|S,E)=\prod_{n=1}^{N}p(t_{n}|t_{1},\cdots,t_{n-1},S,E)$ (1)

The environment $E$ represents the state of the world and external knowledge
(_e.g.,_ a database or results of a search engine) that are needed to go
beyond chitchat and satisfy a need for Utility. In the pre-training stage, $E$
is generally absent as $E$ tends to be task specific, but we find it effective
to substitute $E$ with a grounding text during the third phase of pre-
training, _i.e.,_ grounded dialog pre-training.

##  4 Models

Figure 1: Godel pre-training and fine-tuning with a Transformer-based encoder-
decoder model, taking dialog context and environment (world state or external
knowledge) as an input represented as a string.

At its core, Godel is a standard pre-trained LM, trained on web text data. We
utilize a sequence-to-sequence Transformer model, as shown in Figure 1, to
generate responses given dialog history and environment. The dialog context
$S$ and environment $E$ are concatenated as a long sequence, which is the
input to the model. A sample training example is shown in Figure 2.

User : I would like to find an expensive restaurant that serves Chinese food.
System : sure, which area do you prefer ? User : Bellevue downtown.
<|environment|> Multiple expensive Chinese restaurants in Bellevue downtown.
=> There are multiple restaurants meet your requirement. peony kitchen is a
great Chinese Restaurant. Would you like to book a table there? Figure 2:
Sample training instance, with conversation history in red, grounding in blue,
and response in green.

Godel is pre-trained in three phases: (1) Linguistic pre-training on public
web documents to provide a basic capability for text generation. (2) Dialog
pre-training on public dialog data to improve the models’ handling of general
conversational behavior. (3) Grounded dialog pre-training to enable grounded
response generation.

For general-domain dialog pre-training, we use a dataset extracted from Reddit
comment chains for DialoGPT Zhang et al. (2019b), which consists of 147M
dialog sessions for a total of 6B tokens. For the final grounded dialog pre-
training phase, we use a corpus constructed by leveraging existing datasets
that support knowledge grounded response generation, conversational question-
answering, and task-oriented dialog, as follows:

  * •

The DSTC7 Task 2 corpus Galley et al. (2019) is designed for end-to-end
conversational modeling that goes beyond chitchat by injecting informational
responses grounded in external knowledge. This has 2 million examples for
training.

  * •

MS MARCO Nguyen et al. (2016) is a large-scale question-answering dataset.
Each question is associated with multiple passages retrieved by the Bing
search engine. Each passage is annotated as to whether it is selected to
compose the final answer. The selected passages serve as external knowledge
for grounded training.

  * •

UnifiedQA Khashabi et al. (2020) is a large-scale question-answering corpus
that unifies different QA datasets into a plain text-to-text format.
Similarly, context paragraphs are treated as the external knowledge on which
the model is trained to generate answers given questions.

  * •

The Schema-Guided Dialog Chen et al. (2020) is a multi-domain, task-oriented
conversation dataset that contains approximately 20k annotated examples. Each
conversation is annotated with belief states that track users’ goals. The
belief states in plain text are used as the external knowledge for grounded
training.

We consider three sizes of model: a 220M parameter base version (GodelB), a
770M parameter version (GodelL), and a 175B parameter version (GodelXL). The
220M parameter base model (GodelB) has a 12 layer encoder and 12 layer decoder
with 768 dimensional embeddings. The 770M parameter model (GodelL) doubles the
number of layers for encoder and decoder and has 1024 dimensional embeddings.
These two models, GodelB and GodelL, are initiated from T5 and T5-Large,
respectively, and are based on the HuggingFace repository versions. We use
Byte-Level BPE tokenization as implemented in HuggingFace.

GodelXL is initiated from GPT-3 and is based on the implementation by Brown et
al. (2020). We chose GPT-3 for our initialization of GodelXL, despite being
aware that we would not be able to release a model due to licensing
restrictions, because we wanted to test our grounded fine-tuning recipe on one
of the best pre-trained language models available at the time of the writing.
As a proxy for the GPT-3-initialized GodelXL, we release a version pre-trained
from GPT-J. Table 15 in the Appendix shows that this substitute model is
competitive with the GPT-3-based version.

GodelB and GodelL were trained on 16 Nvidia V100 machines, and GodelXL was
trained with 128 Nvidia V100 GPUs. Each model is trained until there is no
significant loss reduction on the validation data set. The models are trained
for at most 10 epochs, and we select the best versions on the validation set.
We use beam search for inference using a beam size of 5.

##  5 Experiments

###  5.1 Experimental Setup

We evaluate fine-tuned models in two different setups: few-shot and full. Our
evaluation focuses primarily on few-shot finetuning, for the following
reasons. First, datasets of goal-directed dialogs (_e.g.,_ MultiWOZ) are more
expensive to construct and tend to be much smaller than chitchat datasets,
hence we want to assess models’ ability to perform well even if fine-tuning
examples are relatively few. Second, this work focuses on language model pre-
training for dialog, and evaluation in few-shot setups more directly evaluates
the effectiveness of our released pre-trained models. Third, few-shot fine-
tuning is a realistic approach in application scenarios where it can
facilitate fast turnaround of updated models and greater developer control
over model characteristics.

Godel can be used as an initial model to fine-tune for any open-domain goal-
directed dialog tasks with a handful of annotated examples. We evaluate Godel
in terms of its success in fine-tuning three types of goal-directed dialog,
_i.e.,_ knowledge-grounded response generation, task-oriented dialog, and
conversational QA:

#### Evaluation datasets:

  * •

Wizard of Wikipedia Dinan et al. (2018) is a dataset of multi-turn knowledge
grounded dialogs between an apprentice and a wizard, who has access to
Wikipedia sentences and labels the sentences utilized for each utterance they
made. To focus on grounded response generation, we use the gold Wikipedia
sentences provided at each turn of the dialog.

  * •

Wizard of Internet Komeili et al. (2021) is an open-domain dialog dataset
grounded on internet retrieved sentences. At each turn, the wizard can issue a
free text web search and replies to the apprentice grounding the response on
the retrieved sentences. Similarly, we use the gold retrieved sentences
provided at each turn of the dialog.

  * •

MultiWOZ Budzianowski et al. (2018) is a multi-domain task-oriented dialog
dataset that contains 8438/1000/1000 dialogs for training/validation/testing,
respectively. Each dialog session spans multiple domains, such as Attraction,
Hotel, Hospital, Police, Restaurant, Train, and Taxi. This dataset is
challenging owing to its multi-domain setting and diverse language styles. For
this dataset, we consider the task of context-to-response, _i.e.,_ the model
generates responses given the golden belief state and database status in plain
text.

  * •

CoQAReddy et al. (2019) is a conversational question answering dataset that
contains 127,000 questions and answers collected from seven diverse domains.
For each conversation, two workers chat in the form of conversational
questions and free-form text-based answers grounding on a provided passage.

These existing goal-oriented dialog corpora contain thousands of dialog
instances. For few-shot fine-tuning, we sample randomly from the corpora 50
dialogs for each task for fine-tuning and use their original test sets for
evaluation. We fine-tune Godel on each task with the same setup as in pre-
training, using the training objective shown in Equation 1. The best model is
selected based on the perplexity score on the validation set.

#### Baseline pretrained models

We compare Godel against established pretrained models: T5 Raffel et al.
(2020), BART Lewis et al. (2020), DialoGPT Zhang et al. (2019b), and
BlenderBot Roller et al. (2021). For T5, we fine-tune from both T5-base (T5B)
and T5-large (T5L). For BART, we fine-tune from both BART (BARTB) and BART-
large (BARTL). For BlenderBot, we fine-tune from Blenderbot400M, which is
distilled from a 3B model.333We used version 1.0 of BlenderBot. BlenderBot 2.0
is not evaluated in this paper as it is specifically designed for internet
augmented generation and uses a paradigm that is different from that of task-
oriented dialog (_e.g.,_ MultiWOZ).

#### Automatic evaluation metrics

As surrogates for the broad notion of Utility introduced in Section 3, we use
the following functions:

  * •

$\mathtt{F_{1}^{R}}$ is a metric measuring the average overlap between the
prediction and ground truth response or answer.

  * •

$\mathtt{F_{1}^{K}}$, _i.e.,_ knowledge F1, measures the overlap between the
model’s response and the knowledge sentences on which the responses are
grounded during dataset collection.

  * •

Inform indicates whether the model provides sufficient information to satisfy
users’ information seeking needs, as defined for MultiWOZ Budzianowski et al.
(2018).

  * •

Success measures whether the model provides appropriate information and
answers contain all required attributes, as defined for MultiWOZ Budzianowski
et al. (2018).

  * •

The Combined score Mehri et al. (2019) defined as (Inform \+ Success) * 0.5 +
BLEU is used as an overall quality measure.

Besides these utility functions (extrinsic), we report intrinsic evaluation
results using BLEU, BLEURT Sellam et al. (2020), BERTScore Zhang et al.
(2019a), and chrF Popović (2015). All BLEU scores in this paper are based on
corpus-level BLEU-4. In tables 1 through 5, significance is computed using a
paired 2-sided t-test against the best competitor.

#### Human evaluation setup

We perform a turn-level human evaluation to investigate whether Godel
generates responses that are 1) useful, 2) human-like and 3) safe. Using
Amazon Mechanical Turk, we hired master-level workers with lifetime HIT
acceptance % $>$ 95\. For each example, we presented workers with dialog
history, related knowledge, and two responses from different systems. The
order of the system responses displayed in the interface is randomized. We
consider three questions:

  * •

Extrinsic evaluation: Which response sounds more useful? (_i.e.,_ contributes
to making the conversation productive, especially towards achieving any stated
goals)

  * •

Intrinsic evaluation: Which speaker sounds more human? (_e.g.,_ coherent,
fluent, and natural)

  * •

Safety evaluation: Which response is socially safer? (_e.g.,_ friendly,
polite, and empathetic, as opposed to harmful, biased, misinformative, or
incomprehensible)

These were judged on a 5-point Likert scale. A snapshot of the human
evaluation interface is provided in Appendix Figure 3.

###  5.2 Automatic Evaluation Results

Model | #params | Bleu | $\mathtt{F_{1}^{R}}$ | Bert | Bleurt | chrF
---|---|---|---|---|---|---
BARTB | 139M | 7.84 | 34.23 | 86.93 | 40.94 | 27.03
T5B | 220M | 9.13 | 34.75 | 86.32 | 41.30 | 27.42
DGPTB | 345M | 0.39 | 5.56 | 56.86 | 16.76 | 6.28
BlenderBot | 400M | 2.50 | 12.98 | 83.77 | 27.31 | 18.70
BARTL | 406M | 9.68 | 37.29 | 87.33 | 44.43 | 30.66
DGPTL | 762M | 5.19 | 23.28 | 86.22 | 33.09 | 21.73
T5L | 770M | 12.84 | 44.47 | 90.02 | 47.86 | 34.99
GodelB | 220M | 12.77 | 42.80 | 89.59 | 47.79 | 34.42
GodelL | 770M | 13.47[**] | 45.82[**] | 90.27[*] | 49.37[**] | 35.87[**]

Table 1: Automated evaluation on all tasks in the few-shot fine-tuning
setting. ‘Bert’ is BERTScore. All differences between GodelL and the closest
competitor are statistically significant. (*$p<0.05$, **$p<0.001$).

Model | #params | Bleu | $\mathtt{F_{1}^{R}}$ | Bert | Bleurt | chrF
---|---|---|---|---|---|---
BARTB | 139M | 12.75 | 46.61 | 88.64 | 50.78 | 34.43
T5B | 220M | 16.06 | 48.38 | 90.63 | 51.13 | 38.62
DGPTB | 345M | 10.15 | 30.03 | 87.47 | 37.64 | 28.98
GodelB | 220M | 16.30[*] | 48.87[*] | 90.74[*] | 51.73[*] | 39.14[*]

Table 2: Automated evaluation on all tasks in the full fine-tuning setting.
‘Bert’ is BERTScore. All differences between GodelB and its closest competitor
are statistically significant (*$p<0.05$).

Model | Bert | Bleu | $\mathtt{F_{1}^{R}}$ | $\mathtt{F_{1}^{K}}$
---|---|---|---|---
T5B | 86.16 | 11.38 | 30.53 | 62.34
GodelB | 86.77[**] | 12.49[**] | 32.92[**] | 62.74
T5L | 86.26 | 12.21 | 31.04 | 60.52
GodelL | 86.79[*] | 12.38 | 32.89[*] | 60.88

Table 3: Automated evaluation on grounded generation tasks in the few-shot
fine-tuning setting. Bold is best in class. Statistically significant
differences are indicated with asterisks (*$p<0.05$, **$p<0.001$).

Model | Bert | Bleu | Inform | Success | Combined
---|---|---|---|---|---
T5B | 79.21 | 4.31 | 60.60 | 22.50 | 45.86
GodelB | 87.58[**] | 12.81[**] | 67.60[**] | 46.10[**] | 69.72[**]
T5L | 88.14 | 12.69 | 71.50 | 56.20 | 76.54
GodelL | 88.34[*] | 14.07[**] | 81.60[**] | 62.10[**] | 85.90[**]

Table 4: Automated evaluation on MultiWOZ dataset in the few-shot fine-tuning
setting. Bold is best in class. Statistically significant differences are
indicated with asterisks (*$p<0.05$, **$p<0.001$).

Model | Bert | Bleu | $\mathtt{F_{1}^{R}}$
---|---|---|---
T5B | 93.04 | 43.78 | 64.60
GodelB | 94.03[**] | 43.17 | 72.00[**]
T5L | 95.19 | 66.94[*] | 79.10
GodelL | 95.24 | 64.04 | 79.60

Table 5: Automated evaluation on CoQA in the few-shot fine-tuning setting.
Bold is best in class. Statistically significant differences are indicated
with asterisks ( *$p<0.05$, **$p<0.001$).

| MultiWOZ
---|---
| $\mathtt{Inform}$ | $\mathtt{Success}$ | Bleu | $\mathtt{Combd}$
GPT-3 | 58.90 | 9.10 | 4.60 | 38.61
GodelXL | 68.80 | 19.90 | 6.72 | 51.06
| Wizard of Wikipedia
| Bleu | $\mathtt{F_{1}^{R}}$ | $\mathtt{F_{1}^{K}}$
GPT-3 | 3.45 | 20.86 | 21.98
GodelXL | 9.46 | 31.04 | 41.08
| Wizard of Internet
| Bleu | $\mathtt{F_{1}^{R}}$ | $\mathtt{F_{1}^{K}}$
GPT-3 | 1.43 | 17.01 | 11.49
GodelXL | 2.85 | 20.68 | 20.97
| CoQA
| Bleu | $\mathtt{F_{1}^{R}}$
GPT-3 | 61.49 | 74.00
GodelXL | 66.90 | 78.30

Table 6: Automated evaluation of GPT-3 and GodelXL, our best performing model
(fine-tuned from GPT3). All differences are significant ($p<0.05$).

Model | Bleu | $\mathtt{F_{1}^{R}}$ | Bert | Bleurt | chrF
---|---|---|---|---|---
T5B | 9.13 | 34.75 | 86.32 | 41.30 | 27.42
\+ Dialog PT | 8.00 | 31.92 | 84.38 | 39.54 | 24.40
\+ Grounded PT | 12.77[*] | 42.80[*] | 89.59[*] | 47.79[*] | 34.42[*]

Table 7: Ablation studies of different pre-training stages. Results are
aggregated on all tasks for the few-shot fine-tuning setting. ‘BERT’ is
BERTScore. All differences between Grounded PT and the closest competitor are
statistically significant (*$p<0.001$).

Tables 1 and 2 show the aggregated results for all the tasks in few-shot and
full fine-tuning settings, respectively. We found that Godel shows
improvements over T5 and DialoGPT (DGPT) on all metrics, although differences
are smaller in the full fine-tuning setting. This highlights the need to focus
on the few-shot fine-tuning setup, as models of the same structure (_e.g.,_
T5B and GodelB) have a tendency to reach similar levels of performance if
fine-tuned until convergence. We also observed that open-domain dialog models
such as DialoGPT and Blenderbot exhibit much worse results–to be expected
given that these models are trained only on chitchat corpora.444Madotto et al.
(2021) makes a similar observation.

Additionally, Godel shows improvement in intrinsic evaluations on knowledge-
grounded generation tasks including WoW and WoI and in extrinsic evaluations
on MultiWOZ and CoQA. Table 3 shows the aggregated results on WoW and WoI. Our
results show that in the few-shot fine-tuning setting, Godel achieves better
performance in terms of BLEU and maintains a similar level of utility
($\mathtt{F_{1}^{K}}$) to that of baseline methods. $\mathtt{Success}$ is the
utility function for this task. As shown in Table 4, Godel outperforms T5 by
23.6 utility scores and 8.5 BLEU points. Table 5 lists the results on CoQA,
which focuses on response correctness and hence $\mathtt{F_{1}^{R}}$ is
utilized as the utility function. Again, the results show that Godel achieves
a much better utility score in the few-shot learning setting. However, the
BLEU score fails to improve, which is probably not surprising as the responses
of CoQA are usually short. In general, scaling up the model improves both
utility and BLEU scores.

Our recipe for building goal-directed dialog models generalizes to different
PLMs _e.g.,_ T5 and GPT-3. In Table 6, we compare GodelXL to 175B GPT-3
baselines fine-tuned on each task.555The original davinci version by Brown et
al. (2020). We observe that GodelXL performs better than GPT-3 by a large
margin. This indicates that grounded pre-training is beneficial in using GPT-3
on goal-directed dialogs.

We also conducted an ablation study of pre-training stages of Godel. Automatic
evaluation results in the few-shot fine-tuning setting are shown in Table 7.
We observe that dialog pre-training alone hurts performance. This is likely
because the model primarily learns to be conversational in this setting while
the downstream tasks all require the capacity for knowledge-grounded
generation. Godel, which incorporates grounded pre-training, achieves the best
performance.

Model | Extrinsic | Intrinsic | Safety
---|---|---|---
| MultiWOZ
T5B | 20.36 | 16.17 | 10.13
GodelB | 62.84 | 55.42 | 37.41
| Wizard of Wikipedia
T5B | 29.36 | 21.99 | 13.54
GodelB | 37.26 | 40.62 | 14.62
| Wizard of the Internet
T5B | 31.10 | 22.33 | 15.45
GodelB | 44.29 | 30.25 | 23.27
| CoQA
T5B | 39.12 | 28.14 | 27.64
GodelB | 44.01 | 35.62 | 32.14
| Overall
T5B | 29.69 | 22.20 | 16.51
GodelB | 47.80 | 40.84 | 27.40

Table 8: Human evaluation of Godel and T5 fine-tuned in the few-shot setting.
All differences are significant ($p<0.05$).

###  5.3 Human Evaluation Results

We compare GODEL against T5, since this model family performed best in terms
of automatic evaluation. Furthermore, Godel is initialized from T5, and
comparing T5 and Godel directly measures the impact of our grounded pre-
training recipe. Human raters significantly prefer Godel over T5 both in terms
of Extrinsic and Intrinsic evaluations on all the tasks. We evaluated a total
of 4,137 randomly sampled examples from all the tasks. Table 8 shows the
winning percentage of the total. Evaluation results are converted from a
5-scale Likert-like scale to a win/tie/loss scale for reporting. We observed a
strong preference for Godel over T5 in terms of both Extrinsic and Intrinsic
evaluations on all tasks. It is interesting to note that Godel performs
similarly to T5 on CoQA in terms of utility score but has significantly higher
Extrinsic and Intrinsic scores in human evaluation on this task, presumably
because the goal of CoQA is more explicit than other tasks.

##  6 Analysis and Discussion

Dataset | Extrinsic | Intrinsic | Safety
---|---|---|---
MultiWOZ | 0.438 | 0.331 | 0.225
WOW | 0.201 | 0.244 | 0.070
WOI | 0.243 | 0.046 | 0.129
CoQA | 0.081 | 0.067 | 0.076
Overall | 0.263 | 0.196 | 0.140

Table 9: Inter-annotator agreement according to Krippendorff’s alpha (interval
metric).

Inter-annotator agreement is shown in Table 9. This is computed using
Krippendorff’s alpha Krippendorff (2011) which is suitable for Likert-scale
crowd-sourced evaluation as it is applicable to any number of coders, to
missing data, and to ordinal ratings. Inter-rater agreement is generally
higher for Extrinsic evaluation compared to Intrinsic and Safety. This
suggests that Utility is more reliably evaluated than Humanness–an interesting
finding given that the latter is more commonly evaluated in open-ended dialog
research. We note that on WoW agreement on Intrinsic is slightly higher than
Extrinsic. Conversations in WoW lean more towards chitchat and there may not
always be an overarching sense of what is useful in a given conversational
context. Finally, we suggest that low agreement on CoQA data is due to the
prevalence of short, factual responses (_e.g.,_ a name) in this dataset, which
may make it harder for judges to tease system responses apart.

Metric | Extrinsic | Intrinsic | Safety
---|---|---|---
| MultiWOZ
Bleurt | 0.252 | 0.212 | 0.276
Bert | 0.224 | 0.265 | 0.257
Bleu | 0.217 | 0.142 | 0.132
chrF | 0.359 | 0.233 | 0.312
| Wizard of Wikipedia
Bleurt | 0.175 | 0.081 | 0.091
Bert | 0.135 | 0.129 | 0.122
Bleu | 0.072 | $-0.022$ | 0.020
chrF | 0.200 | 0.059 | 0.093
| Overall (4 datasets)
Bleurt | 0.217 | 0.186 | 0.222
Bert | 0.186 | 0.230 | 0.224
Bleu | 0.169 | 0.127 | 0.148
chrF | 0.321 | 0.213 | 0.267

Table 10: Metric-human Spearman’s rank correlation coefficients for extrinsic
evaluation (utility), intrinsic (humanness), and safety (e.g., friendly,
unoffensive). In addition to Overall results, we also single out MultiWOZ and
WoW as the former is task-oriented while the latter is more chitchatty.

Table 10 analyzes the correlation between human evaluation results of Section
5 and several common automated evaluation metrics for generation. Here, we
consider BLEU Papineni et al. (2002), BLEURT Sellam et al. (2020), BERTScore
Zhang et al. (2019a), and chrF Popović (2015). The chrF metric is a lexical-
match metric similar to BLEU, but is character-based rather than word-based
and but has been found to be more robust than other surface-level metrics
Kocmi et al. (2021). As the distinction between Intrinsic and Extrinsic
measures of quality for open-ended response generation is relatively new, we
sought to determine whether some metrics are better suited than others for
measuring these different traits.

First, we observe that these metrics tend to correlate better with Extrinsic
than with Intrinsic. This is perhaps not surprising considering the higher
level of inter-annotator agreement on Extrinsic. We also observe that levels
of correlation involving LM-based metrics (BERTScore and BLEURT) are inferior
to those of lexical metrics (_e.g.,_ chrF), while LM-based metrics tend to
better correlate with intrinsic evaluation. We attribute this to the fact that
their focus on language modeling naturally primes them to measure Humanness
(_e.g.,_ fluency and well-formedness), and that may sometimes hamper their
ability to recognize Utility. We are not exactly sure why chrF performs so
well on Extrinsic relative to metrics generally regarded as superior (_e.g.,_
BLEURT), but we hypothesize that usefulness in tasks such as QA and knowledge-
grounding requires getting factual information exactly right (_e.g.,_ a year
or a person’s name) with little need or room for paraphrasing. Finally, we
find it interesting that even Safety enjoys decent correlation with automated
metrics. On task-oriented dialog (MultiWOZ), Safety correlates best with chrF,
but with BERTScore on a more chitchatty dataset (WoW). We hypothesize this
increased correlation is due to references being generally quite safe, so that
unsafe or harmful statements are less likely to be favored by reference-based
metric.

These inter-annotator and correlation results argue for adopting extrinsic
evaluation for open-ended dialog systems. Whether or not the dialog system is
associated with a specific task, conversations tend to have one or multiple
goals, and judging the utility of responses as steps towards these goals is
both natural and desirable. The results shown here also suggest that a shift
towards more extrinsic qualities could make automatic dialog evaluation less
challenging. The assessment of Humanness (Intrinsic evaluation) still has a
purpose, but mainly to ensure that dialog systems do not sacrifice either
Humanness or Safety in order to be become more useful.

##  7 Conclusion

In this paper, we presented a new large-scale pre-trained model for dialogue
(Godel) that improves on Zhang et al. (2019b) in two main ways. First, it is
textually grounded and therefore suitable for downstream grounded text
generation tasks that rely on external knowledge, _e.g.,_ provided by the
dialog’s environment. Second, it is designed for open-ended goal-directed
dialog and evaluated on a variety of tasks ranging from traditional task-
oriented dialog (MultiWOZ) and scenarios that mainly involve informing the
user (_e.g.,_ Wizard of Wikipedia). In our evaluation setup we have been able
to disentangle intrinsic and extrinsic evaluation, which respectively capture
Utility and Humanness. Considering that most conversations are aimed at being
useful, we advocate greater use of extrinsic evaluation in research on dialog
pre-training. The evidence that extrinsic evaluation enjoys greater inter-
annotator agreement and correlation with automated evaluation metrics further
supports this proposition. Our experiments show that Godel is quite
competitive when evaluated against state-of-the-art pre-trained models for
dialog and generation (T5, BART, and BlenderBot), and outperforms these models
by a large margin in few-shot fine-tuning setups appropriate to goal-directed
datasets that are often small. We make all of Godel’s code, models, and
dataset extraction scripts publicly available.

## Ethics Statement

Ensuring that responses generated by dialog systems are safe and inoffensive
is a long-standing problem in Conversational AI. We performed extensive
filtering of the social-media data (Reddit) used to train Godel. This involved
block-word filtering, and excluding troll responses (_e.g.,_ large number of
downvotes) and entire subreddits marked as not-safe-for-work or that have been
banned by Reddit. This extensive filtering reduces dataset size to 25% of that
prior to filtering, with the result that our data is relatively small relative
to that of other Reddit-based models (_e.g.,_ Plato-XL). Nonetheless, harder-
to-detect forms of social bias and other toxicity that do not involve overtly
offensive terms persist the training data, and we acknowledge that Godel may
generate harmful responses, especially when challenged with adversarial
inputs. Further mitigation constitutes a major area for future research, and
it is in part to this end that we make our models and dataset extraction code
available. We have noted that dialogs geared towards utility appear to tend to
be safer. This suggests that future work aimed at optimizing utility might
offer the added benefit of increasing model safety.

## Acknowledgements

We thank Clarisse Simoes Ribeiro, Chunyuan Li, Zhu Zhang, Hao Cheng, and other
members of the Microsoft Research Deep Learning team for valuable discussions
and comments.

## References

  * Aliannejadi et al. (2020) Mohammad Aliannejadi, Julia Kiseleva, Aleksandr Chuklin, Jeff Dalton, and Mikhail S. Burtsev. 2020.  ConvAI3: Generating clarifying questions for open-domain dialogue systems (ClariQ).  _CoRR_ , abs/2009.11352.
  * Bao et al. (2021) Siqi Bao, H. He, Fan Wang, Hua Wu, Haifeng Wang, Wenquan Wu, Zhihua Wu, Zhen Guo, Hua Lu, Xinxian Huang, Xin Tian, Xinchao Xu, Yingzhan Lin, and Zhengyu Niu. 2021.  PLATO-XL: Exploring the large-scale pre-training of dialogue generation.  _ArXiv_ , abs/2109.09519.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language models are few-shot learners.  In _Advances in Neural Information Processing Systems_ , volume 33, pages 1877–1901.
  * Budzianowski et al. (2018) Pawel Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Iñigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gasic. 2018.  MultiWOZ – a large-scale multi-domain Wizard-of-Oz dataset for task-oriented dialogue modelling.  In _EMNLP_.
  * Chen et al. (2020) Lu Chen, Boer Lv, Chi Wang, Su Zhu, Bowen Tan, and Kai Yu. 2020.  Schema-guided multi-domain dialogue state tracking with graph attention neural networks.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , pages 7521–7528.
  * Dinan et al. (2018) Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. 2018.  Wizard of Wikipedia: Knowledge-powered conversational agents.  In _International Conference on Learning Representations_.
  * Freitas et al. (2020) Daniel De Freitas, Minh-Thang Luong, David R. So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, and Quoc V. Le. 2020.  Towards a human-like open-domain chatbot.  _ArXiv_ , abs/2001.09977.
  * Galley et al. (2019) Michel Galley, Chris Brockett, Xiang Gao, Jianfeng Gao, and Bill Dolan. 2019.  Grounded response generation task at DSTC7.  In _AAAI DSTC7 workshop_.
  * Gao et al. (2019) Jianfeng Gao, Michel Galley, and Lihong Li. 2019.  Neural approaches to conversational AI.  _Foundations and Trends in Information Retrieval_ , 13(2-3):127–298.
  * Gao et al. (2022) Jianfeng Gao, Chenyan Xiong, Paul Bennett, and Nick Craswell. 2022.  Neural approaches to conversational information retrieval.  _arXiv preprint arXiv:2201.05176_.
  * Gao et al. (2020) Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. 2020.  The Pile: An 800GB dataset of diverse text for language modeling.  _arXiv preprint arXiv:2101.00027_.
  * Gunasekara et al. (2020) Chulaka Gunasekara, Seokhwan Kim, L. F. D’Haro, Abhinav Rastogi, Yun-Nung Chen, Mihail Eric, Behnam Hedayatnia, Karthik Gopalakrishnan, Yang Liu, Chao-Wei Huang, Dilek Z. Hakkani-Tür, Jinchao Li, Qi Zhu, Lingxiao Luo, Lars Lidén, Kaili Huang, Shahin Shayandeh, Runze Liang, Baolin Peng, Zheng Zhang, Swadheen Shukla, Minlie Huang, Jianfeng Gao, Shikib Mehri, Yulan Feng, Carla Gordon, Seyed Hossein Alavi, David R. Traum, Maxine Eskénazi, Ahmad Beirami, Eunjoon Cho, Paul A. Crook, Ankita De, Alborz Geramifard, Satwik Kottur, Seungwhan Moon, Shivani Poddar, and Rajen Subba. 2020.  Overview of the ninth dialog system technology challenge: DSTC9.  _ArXiv_ , abs/2011.06486.
  * Khashabi et al. (2020) Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Hannaneh Hajishirzi. 2020.  UnifiedQA: Crossing format boundaries with a single qa system.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 1896–1907.
  * Kim et al. (2020) Seokhwan Kim, Mihail Eric, Karthik Gopalakrishnan, Behnam Hedayatnia, Yang Liu, and Dilek Hakkani-Tur. 2020.  Beyond domain APIs: Task-oriented conversational modeling with unstructured knowledge access.  In _Proceedings of the 21th Annual Meeting of the Special Interest Group on Discourse and Dialogue_ , pages 278–289.
  * Kocmi et al. (2021) Tom Kocmi, Christian Federmann, Roman Grundkiewicz, Marcin Junczys-Dowmunt, Hitokazu Matsushita, and Arul Menezes. 2021.  To ship or not to ship: An extensive evaluation of automatic metrics for machine translation.  In _Proceedings of the Sixth Conference on Machine Translation_ , pages 478–494, Online. Association for Computational Linguistics.
  * Komeili et al. (2021) Mojtaba Komeili, Kurt Shuster, and Jason Weston. 2021.  Internet-augmented dialogue generation.  _arXiv preprint arXiv:2107.07566_.
  * Krippendorff (2011) Klaus Krippendorff. 2011.  Computing krippendorff’s alpha-reliability.  Technical report, University of Pennsylvania.
  * Lewis et al. (2020) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020.  BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7871–7880, Online. Association for Computational Linguistics.
  * Li et al. (2019) Margaret Li, Jason Weston, and Stephen Roller. 2019.  ACUTE-EVAL: Improved dialogue evaluation with optimized questions and multi-turn comparisons.  _ArXiv_ , abs/1909.03087.
  * Madotto et al. (2021) Andrea Madotto, Zhaojiang Lin, Genta Indra Winata, and Pascale Fung. 2021.  Few-shot bot: Prompt-based learning for dialogue systems.  _arXiv preprint arXiv:2110.08118_.
  * Mehri et al. (2019) Shikib Mehri, Tejas Srinivasan, and Maxine Eskenazi. 2019.  Structured fusion networks for dialog.  In _20th Annual Meeting of the Special Interest Group on Discourse and Dialogue_ , page 165.
  * Nguyen et al. (2016) Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016.  MS MARCO: A human generated machine reading comprehension dataset.  _arXiv preprint arXiv:1611.09268_.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  BLEU: a method for automatic evaluation of machine translation.  In _Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics_ , pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
  * Peng et al. (2021) Baolin Peng, Chunyuan Li, Jinchao Li, Shahin Shayandeh, Lars Lidén, and Jianfeng Gao. 2021.  SOLOIST: Building task bots at scale with transfer learning and machine teaching.  _Transactions of the Association for Computational Linguistics_ , 9:807–824.
  * Popović (2015) Maja Popović. 2015.  chrF: character n-gram F-score for automatic MT evaluation.  In _Proceedings of the Tenth Workshop on Statistical Machine Translation_ , pages 392–395, Lisbon, Portugal. Association for Computational Linguistics.
  * Raffel et al. (2020) Colin Raffel, Noam M. Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _ArXiv_ , abs/1910.10683.
  * Reddy et al. (2019) Siva Reddy, Danqi Chen, and Christopher D Manning. 2019.  CoQA: A conversational question answering challenge.  _Transactions of the Association for Computational Linguistics_ , 7:249–266.
  * Roller et al. (2021) Stephen Roller, Emily Dinan, Naman Goyal, Da Ju, Mary Williamson, Yinhan Liu, Jing Xu, Myle Ott, Kurt Shuster, Eric Michael Smith, Y.-Lan Boureau, and Jason Weston. 2021.  Recipes for building an open-domain chatbot.  In _Proc. of EACL_.
  * Sellam et al. (2020) Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020.  BLEURT: Learning robust metrics for text generation.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7881–7892, Online. Association for Computational Linguistics.
  * Thoppilan et al. (2022) Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam M. Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, YaGuang Li, Hongrae Lee, Huaixiu Zheng, Amin Ghafouri, Marcelo Menegali, Yanping Huang, Maxim Krikun, Dmitry Lepikhin, James Qin, Dehao Chen, Yuanzhong Xu, Zhifeng Chen, Adam Roberts, Maarten Bosma, Yanqi Zhou, Chung-Ching Chang, I. A. Krivokon, Willard James Rusch, Marc Pickett, Kathleen S. Meier-Hellstern, Meredith Ringel Morris, Tulsee Doshi, Renelito Delos Santos, Toju Duke, Johnny Hartz Søraker, Ben Zevenbergen, Vinodkumar Prabhakaran, Mark Diaz, Ben Hutchinson, Kristen Olson, Alejandra Molina, Erin Hoffman-John, Josh Lee, Lora Aroyo, Ravindran Rajakumar, Alena Butryna, Matthew Lamm, V. O. Kuzmina, Joseph Fenton, Aaron Cohen, Rachel Bernstein, Ray Kurzweil, Blaise Aguera-Arcas, Claire Cui, Marian Croak, Ed Chi, and Quoc Le. 2022.  LaMDA: Language models for dialog applications.  _ArXiv_ , abs/2201.08239.
  * Wang and Komatsuzaki (2021) Ben Wang and Aran Komatsuzaki. 2021.  GPT-J-6B: A 6 billion parameter autoregressive language model.  https://github.com/kingoflolz/mesh-transformer-jax.
  * Wu et al. (2019) Wenquan Wu, Zhen Guo, Xiangyang Zhou, Hua Wu, Xiyuan Zhang, Rongzhong Lian, and Haifeng Wang. 2019.  Proactive human-machine conversation with explicit conversation goal.  _ArXiv_ , abs/1906.05572.
  * Yoshino et al. (2019) Koichiro Yoshino, Chiori Hori, Julien Perez, L. F. D’Haro, Lazaros Polymenakos, R. Chulaka Gunasekara, Walter S. Lasecki, Jonathan K. Kummerfeld, Michel Galley, Chris Brockett, Jianfeng Gao, William B. Dolan, Xiang Gao, Huda AlAmri, Tim K. Marks, Devi Parikh, and Dhruv Batra. 2019.  Dialog system technology challenge 7.  _ArXiv_ , abs/1901.03461.
  * Zang et al. (2020) Xiaoxue Zang, Abhinav Rastogi, Srinivas Sunkara, Raghav Gupta, Jianguo Zhang, and Jindong Chen. 2020.  MultiWOZ 2.2 : A dialogue dataset with additional annotation corrections and state tracking baselines.  In _Proceedings of the 2nd Workshop on Natural Language Processing for Conversational AI_ , pages 109–117, Online. Association for Computational Linguistics.
  * Zhang et al. (2019a) Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2019a.  BERTScore: Evaluating text generation with BERT.  _CoRR_ , abs/1904.09675.
  * Zhang et al. (2019b) Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and Bill Dolan. 2019b.  DialoGPT: Large-scale generative pre-training for conversational response generation.  _arXiv preprint arXiv:1911.00536_.

Model | Few-shot | Full
---|---|---
$\mathtt{Inform}$ | $\mathtt{Success}$ | Bleu | $\mathtt{Combined}$ | $\mathtt{Inform}$ | $\mathtt{Success}$ | Bleu | $\mathtt{Combined}$
T5B | 60.60 | 22.50 | 4.31 | 45.86 | 81.90 | 74.50 | 16.77 | 94.90
DGPTB | 38.70 | 3.00 | 0.20 | 21.05 | 60.90 | 39.60 | 12.90 | 63.15
GodelB | 67.60 | 46.10 | 12.81 | 69.72 | 89.10 | 83.00 | 17.73 | 103.78
DGPTL | 62.40 | 34.70 | 10.52 | 59.06 | - | - | - | -
T5L | 71.50 | 56.20 | 12.69 | 76.54 | - | - | - | -
GodelL | 81.60 | 62.10 | 14.07 | 85.90 | - | - | - | -

Table 11: Automated evaluation on the MultiWOZ dataset in few-shot and full
fine-tuning settings.

Model | Few-shot | Full
---|---|---
Bleu | $\mathtt{F_{1}^{R}}$ | $\mathtt{F_{1}^{K}}$ | Bleu | $\mathtt{F_{1}^{R}}$ | $\mathtt{F_{1}^{K}}$
T5B | 14.10 | 34.73 | 77.34 | 18.12 | 40.11 | 72.15
DGPTB | 0.71 | 13.23 | 10.48 | 14.41 | 36.34 | 57.13
GodelB | 15.95 | 37.08 | 77.54 | 18.10 | 40.13 | 72.43
T5L | 14.87 | 35.89 | 79.48 | - | - | -
DGPTL | 1.41 | 16.46 | 11.87 | - | - | -
GodelL | 15.99 | 38.46 | 76.90 | - | - | -

Table 12: Automated evaluation on the Wizard of Wikipedia dataset in few-shot
and full fine-tuning settings.

Model | Few-shot | Full
---|---|---
Bleu | $\mathtt{F_{1}^{R}}$ | $\mathtt{F_{1}^{K}}$ | Bleu | $\mathtt{F_{1}^{R}}$ | $\mathtt{F_{1}^{K}}$
T5B | 4.15 | 21.11 | 32.63 | 5.11 | 24.81 | 28.89
DGPTB | 0.99 | 15.33 | 6.93 | 2.08 | 19.95 | 17.74
GodelB | 5.05 | 22.96 | 33.28 | 5.35 | 24.93 | 29.67
T5L | 4.29 | 23.39 | 30.98 | - | - | -
DGPTL | 1.78 | 17.64 | 16.48 | - | - | -
GodelL | 5.08 | 24.11 | 35.88 | - | - | -

Table 13: Automated evaluation on the Wizard of the Internet dataset in few-
shot and full fine-tuning settings.

Model | Few-shot | Full
---|---|---
Bleu | $\mathtt{F_{1}^{R}}$ | Bleu | $\mathtt{F_{1}^{R}}$
T5B | 43.78 | 64.60 | 58.06 | 80.50
DGPTB | 3.10 | 1.14 | 28.90 | 31.70
GodelB | 43.17 | 72.00 | 55.52 | 80.70
T5L | 66.94 | 79.10 | - | -
DGPTL | 23.75 | 28.70 | - | -
GodelL | 64.04 | 79.60 | - | -

Table 14: Automated evaluation on the CoQA dataset in few-shot and full fine-
tuning settings.

##  Appendix A Detailed Results

We first provide detailed automatic evaluation for individual datasets.
Results for MultiWOZ are shown in Table 11. GodelB achieves the best results
on Combined score compared to the competitors in both few-shot and full fine-
tuning settings. Results for Wizard of Wikipedia and Wizard of Internet are
listed in Table 12 and 13, respectively. Both GodelB and GodelL obtain better
BLEU and $\mathtt{F_{1}^{R}}$ than competitors of the same model size in the
few-shot fine-tuning setting, but this gap is not observable in the full fine-
tuning setting. In addition, T5 and Godel achieve high $\mathtt{F_{1}^{K}}$ as
they tend to copy content from the given knowledge into responses. Whether
$\mathtt{F_{1}^{K}}$ is an appropriate metric for knowledge grounded
generation tasks is still an open problem. Table 14 list the results on CoQA.

These results also show that GodelXL has significant lower BLEU scores than
Godel and GodelL. We speculate that this is due to GodelXL having a decoder-
only transformer architecture, which tends it be more abstractive and
therefore to be less favored by exact-match metrics such as BLEU. Encoder-
decoder transformer models such Godel and GodelL, on the other hand, are more
likely to copy words or phrases directly from the grounding, which yields more
a extractive behavior favored by metrics such as BLEU.

| MultiWOZ
---|---
| $\mathtt{Inform}$ | $\mathtt{Success}$ | Bleu | $\mathtt{Combd}$
Godel${}_{\mathtt{XL}}^{\mathtt{GPT-J}}$ | 60.50 | 21.00 | 6.27 | 47.01
Godel${}_{\mathtt{XL}}^{\mathtt{GPT-3}}$ | 68.80 | 19.90 | 6.72 | 51.06
| Wizard of Wikipedia
| Bleu | $\mathtt{F_{1}^{R}}$ | $\mathtt{F_{1}^{K}}$
Godel${}_{\mathtt{XL}}^{\mathtt{GPT-J}}$ | 5.72 | 24.31 | 26.89
Godel${}_{\mathtt{XL}}^{\mathtt{GPT-3}}$ | 9.46 | 31.04 | 41.08
| Wizard of Internet
| Bleu | $\mathtt{F_{1}^{R}}$ | $\mathtt{F_{1}^{K}}$
Godel${}_{\mathtt{XL}}^{\mathtt{GPT-J}}$ | 1.95 | 18.71 | 11.37
Godel${}_{\mathtt{XL}}^{\mathtt{GPT-3}}$ | 2.85 | 20.68 | 20.97
| CoQA
| Bleu | $\mathtt{F_{1}^{R}}$
Godel${}_{\mathtt{XL}}^{\mathtt{GPT-J}}$ | 41.44 | 64.20
Godel${}_{\mathtt{XL}}^{\mathtt{GPT-3}}$ | 66.90 | 78.30

Table 15: Automated evaluation of Godel${}_{\mathtt{XL}}^{\mathtt{GPT-3}}$ and
Godel${}_{\mathtt{XL}}^{\mathtt{GPT-J}}$.

As we are unable to distribute the GPT-3 version of GodelXL
(Godel${}_{\mathtt{XL}}^{\mathtt{GPT-3}}$ below), we instead release GodelXL
based on GPT-J Gao et al. (2020); Wang and Komatsuzaki (2021)
(Godel${}_{\mathtt{XL}}^{\mathtt{GPT-J}}$) as a proxy for GodelXL. Table 15
compares the results of Godel${}_{\mathtt{XL}}^{\mathtt{GPT-3}}$ and
Godel${}_{\mathtt{XL}}^{\mathtt{GPT-J}}$ on all tasks in the few-shot setting.

Table 16: Sample responses of models pretrained from BART, T5, BlenderBot, and
Godel:

Example 1
---
User: What is good in the south part of town?
Agent: There are nine restaurants in the south area. Is there a price range
you would like in particular?
User: Cheap, please
Agent: There are two available in the south: Nandos, which serves Portuguese,
and The Lucky Star, which servers Chinese. Do you have a preference?
User: How is the ambiance at Nandos?
Grounding: The ambiance was very friendly, well-lit and service was great. As
this was our first time here we both ordered their signature 1/2 chicken with
corn on the cob and the Portuguese rice.
BARTL: There are two available in the south: Nandos, which serves Portuguese,
and The Lucky Star. Do you have a preference?
T5L: The ambiance is very good. It is located in a quiet area and the service
is good
BlenderBot: I think it is pretty good. It is located in Nashville, Tennessee,
which is one of the most populous cities in the United States.
GodelL: The ambiance is very friendly, well-lit and service is great. I would
recommend the Portuguese rice, which comes with half chicken with corn on the
cob, and a side of rice.
GodelXL: It is very friendly. I would recommend it. The food is good and the
price is very reasonable. Have you tried their signature dish? A 1/2 Chicken
with corn on the cob and Portuguese rice.
Example 2
User: What is good in the south part of town?
Agent: There are nine restaurants in the south area. Is there a price range
you would like in particular?
User: Cheap, please
Agent: There are two available in the south: Nandos, which serves Portuguese,
and The Lucky Star, which servers Chinese. Do you have a preference?
User: Nandos. By the way, how many tornadoes did we get in Earlsboro recently?
Grounding: Most tornadoes in the United States move from southwest to
northeast, but once in a while, storms can do something funky. That was the
case Wednesday night about 45 miles east-southeast of Oklahoma City in
Pottawatomie County, Okla. Earlsboro — a community that may have been hit by
three tornadoes in three days — hosted a meteorological marvel.
BARTL: Did you hear there was a tornado in Earlsboro?
T5L: No, The Lucky Star is located in the north part of town. There has been a
tornado in Earlsboro recently.
BlenderBot: I’m not sure, but I do know that there have been a lot of
earthquakes in the past.
GodelL: Three tornadoes in three days. Is there anything else I can help you
with?
GodelXL: Just one, but it was violent. I suggest the Nandos if you want to eat
there. They serve Portuguese food. Have you ever been there?

Table 16 provides examples of baselines, GodelL, and GodelXL in given
contexts. We observe that GodelL based on T5 is more extractive while GodelXL
based on GPT-J is more abstractive and tries to introduce more factual
content.

Figure 3: Human evaluation task design.