# Unsupervised Learning of Hierarchical Conversation Structure

Bo-Ru Lu♠ Yushi Hu♠ Hao Cheng♡ Noah A. Smith♠♢ Mari Ostendorf♠
♠University of Washington ♡Microsoft Research ♢Allen Institute for AI
<EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Human conversations can evolve in many different ways, creating challenges for
automatic understanding and summarization. Goal-oriented conversations often
have meaningful sub-dialogue structure, but it can be highly domain-dependent.
This work introduces an unsupervised approach to learning hierarchical
conversation structure, including turn and sub-dialogue segment labels,
corresponding roughly to dialogue acts and sub-tasks, respectively. The
decoded structure is shown to be useful in enhancing neural models of language
for three conversation-level understanding tasks. Further, the learned finite-
state sub-dialogue network is made interpretable through automatic
summarization. ††We release our code for experiments at
https://github.com/boru-roylu/THETA.

##  1 Introduction

Increasingly, language understanding applications involve conversational
speech and text. Much attention has recently been directed at human-agent
dialogue systems, including virtual assistants, interactive problem solving,
and information seeking tasks (e.g., conversational question answering).
However, automatic understanding of human-human conversations is also of
interest for problems such as call-center analytics, conversation outcome
prediction, meeting summarization, and human-agent interaction involving
multiple people. The focus of this paper is on human-human conversation
understanding.

Like written documents, goal-oriented conversations tend to have structure
(openings, context setting, problem solving, etc.). However, in human-human
conversations (both text and speech), participant roles factor into the
structure, and the structure is less rigid due to the need to accommodate
miscommunications and varying objectives. Yet, most work on conversational
systems treats dialogues like written text, i.e., the dialogue history is a
linear sequence of text. In this paper, we explore unsupervised learning
strategies for adding structural information to a state-of-the-art
hierarchical transformer-based model of text.

Linguistic analysis of conversations often involves associating speaker
utterances with dialogue acts (DAs), e.g., question, statement, backchannel,
clarification, etc. Jurafsky et al. (1997); Core and Allen (1997), and
segmenting the conversation into nested subsequences of participant turns that
reflect a common topic or conversational goal Grosz and Sidner (1986). Past
studies have explored using such structure, particularly DAs, to improve
automated human-agent dialogues. Here, we use hierarchical structure (both
turn-level DA labels and sub-dialogue states) to improve classification of
human-human conversations. Specifically, we introduce Three-stream
Hierarchical Transformer (Theta), which integrates transformer representations
of the DA and sub-dialogue state sequences into a hierarchical transformer
(ht) Santra et al. (2021); Pappagari et al. (2019) operating on the original
text. In addition to improving performance, the use of discrete structural
cues in classification can support conversation analysis. For example, we can
identify seller strategies that are more likely to lead to a successful
outcome or use the sub-dialogue state sequence to summarize frequently visited
states in unsuccessful interactions.

Since hand-annotation of structure can be costly and inventories vary across
tasks, there is substantial interest in unsupervised learning of structure for
specific task domains. Here, the approach to structure learning involves two
steps. First, we use a clustering algorithm to learn a mapping of utterance
embeddings to discrete categories, which serve as an unsupervised version of
DAs. Each conversation is then represented by the discrete sequence of cluster
identifiers (IDs) associated with the sequence of utterances. Using the
collection of discretized conversations, we automatically learn the topology
of a latent finite-state model over these sequences, i.e., a hidden Markov
model (HMM), using a greedy state-splitting algorithm that maximizes the
likelihood of the sequence data without requiring any annotations. The states
of the HMM correspond to different sub-dialogues that may be associated with
specific topics, strategies or sub-tasks. The sub-dialogue structure of a new
conversation is identified by finding the most likely state sequence given
that discretized utterance sequence.

The learned structure is assessed in experiments on three conversation-level
classification tasks: buyer/seller negotiation outcomes on CraigslistBargain
He et al. (2018), conversation category in the Action-based Conversations
Dataset (Abcd) Chen et al. (2021), and client callback prediction in a private
call center corpus. In each task, we find that a combination of both
utterance-level category and sub-dialogue state information lead to improved
performance. Further, we use automatically generated descriptions of the
clusters and sub-dialogue states to provide an interpretable view of the
finite-state topology and a summarized view of a conversation. Anecdotally, we
find that this structure lends insights into how participant strategies (state
paths) are associated with different conversation outcomes.

The contributions of this work are as follows. First, we introduce a simple
unsupervised approach to learn a hierarchical representation of conversation
structure that includes turn-level labels and sub-dialogue segmentation,
accounting for participant role. Using three conversation-level classification
tasks, we demonstrate that integrating the structural information into a
state-of-the-art hierarchical transformer consistently improves performance.
Lastly, we show how the discrete representation of structure combined with
automatic summarization can provide a mechanism for interpreting what the
model is learning or for conversation summarization and analytics.

Figure 1: Overview of Theta conversation encoding. The text of each utterance
text is encoded by BERT, and a 1-layer transformer further contextualizes
utterance embeddings to generate the text vector $\mathbf{U}$. For structure,
utterances are mapped to K-means dialogue acts (DAs), which are input to an
HMM to decode sub-dialogue states. 1-layer transformers are applied to
sequences of DAs and sub-dialogue states, yielding cluster vector $\mathbf{C}$
and state vector $\mathbf{S}$. The concatenation of $\mathbf{U}$, $\mathbf{C}$
and $\mathbf{S}$ is fed into a linear layer to obtain the structure-enhanced
vector for the predictive task. For simplicity, Emb. and Trans. stand for
embedding and transformer, respectively.

##  2 Method

As shown in Figure 1, Theta represents the sequence of turns in a conversation
using: i) a hierarchical transformer (ht) operating on a turn-segmented word
sequence, ii) a transformer operating on a sequence of turn-level DAs, and
iii) a separate transformer operating on a sequence of sub-dialogue states
derived from the DAs. The conversation-level vectors produced by the three
transformers are concatenated and used in a final task-specific layer for
conversation classification tasks. The ht alone is the state-of-the-art model
for conversation-level tasks. The DA and sub-dialogue states comprise the
structural information that enhances the ht for improving performance of the
end task. In addition, the discrete nature of the structure representation
provides a mechanism for analyzing the conversation classes via summarization
of utterances associated with the DA labels or sub-dialogue states.

###  2.1 Model Components

#### Definitions

More formally, each dialogue consists of a sequence of words (or tokens)
$X=[x_{1},\ldots,x_{T}]$ associated with $T$ customer/agent (or seller/buyer)
utterances, where $x_{t}$ is the subsequence of words associated with the
$t$th utterance.111We use the term ”utterance” although some conversations
involve text-based interactions. The word sequence is decorated with three
special tokens: [CLS], [PTY] and [SEP], where [PTY] indicates the utterance
speaker role ([AGT] for agent/seller and [USR] for customer/buyer). The word
sequence $X$ is mapped to two sequences of utterance-level embeddings
$U^{v}=[u_{1}^{v},\ldots,u_{T}^{v}]$, $v\in\\{\textsc{ht},\textsc{da}\\}$. The
vector $u_{t}^{\textsc{ht}}$ is output from the last layer of the ht that is
used to derive the text-based conversation-level vector $\mathbf{U}$. The
vector $u_{t}^{\textsc{da}}$ is the output of a separate transformer, which is
then mapped to a DA category $c_{t}$ to produce the sequence
$C=[c_{1},\ldots,c_{T}]$. The sequence $C$ is associated with a hidden
subdialogue sequence that is represented using the HMM state sequence
$S=[s_{1},\ldots,s_{T}]$. Additional transformers derive conversation-level
vectors $\mathbf{C}$ and $\mathbf{S}$ from $C$ and $S$, respectively. Theta
enhances the conversation representation by concatenating $\mathbf{U}$,
$\mathbf{C}$ and $\mathbf{S}$ together for input to a task-specific layer.

#### Hierarchical Transformer

The hierarchical transformer Pappagari et al. (2019) has been shown to be
useful for classifying long documents (like customer support conversations),
which exceed the length limits placed on transformer-based models due to the
quadratic complexity of the self-attention module. At a high level, two
transformer blocks, a lower utterance transformer and an upper conversation
transformer are stacked together for encoding dialogues. Here, the utterance-
level transformer first encodes utterances into utterance embeddings, one for
each utterance. In this case, the first contextualized token embedding as the
utterance embedding, which corresponds to the sentence-level [CLS] token. The
sequence of utterance embeddings augmented with a conversation-level [CLS]
token are then fed as inputs to another one-layer conversation-level
transformer to further contextualize the vector sequence. We use the output
vector associated with the conversation-level [CLS] token as the conversation
representation.

#### Dialogue Act Sequence Module

To obtain the DA labels, we first derive an utterance embedding
$u^{\textsc{da}}_{t}$ by mean pooling the final layer of the BERT
transformer.222We also experimented with using the [CLS] token, but mean
pooling gave better results. The resulting embedding is mapped to a DA class
$c_{t}$ using a vector quantization (VQ) approach: K-means clustering is used
to learn the classes, and vectors are labeled at inference time by minimizing
the Euclidean distance to cluster means. The number of clusters is treated as
a hyperparameter of the overall model. We apply K-means clustering separately
for utterances from the two different participant roles, so the DA index
reflects the role. This simple approach is motivated by prior work on
unsupervised learning of DA categories Brychcin and Kral (2017), which showed
that K-means clustering gives a performance that is only slightly worse than
HMM-based learning.

In linguistic analyses, a turn can contain a sequence of DAs. Our work assigns
a single DA to a user turn, as in other work using unsupervised learning as
well as the negotiation data set that we report results on. Since the prior
work often uses “dialogue act” for turn-level labels, we have chosen to use
the DA term here, acknowledging the abuse of terminology. For complex tasks
like the call center data (and other data with real users), the turns will
involve multiple dialogue acts, in which case a large number of clusters is
useful.

#### Sub-Dialogue Sequence Module

The DA sequence $C$ is input to a hidden Markov model (HMM) to derive the sub-
dialogue structure. An HMM is a statistical model that characterizes an
observation sequence $C$ in terms of a discrete, latent (hidden) Markov state
sequence $S$,

$\begin{split}P(C)&=\sum_{\text{all}\ S}P(C,S)\\\ &=\sum_{\text{all}\
S}\pi(s_{1})\prod_{t=1}^{T}\eta(c_{t}|s_{t})\gamma(s_{t+1}|s_{t}),\end{split}$

where $\pi$, $\eta$, and $\gamma$ are start-state, observation, and transition
distributions, respectively. $s_{T+1}$ is a dummy stopping state. The HMM is
used to decode the hidden sub-dialogue state sequence $S$, which provides a
segmentation of the conversation into different stages or sub-tasks in problem
solving or negotiation. The HMM topology and parameters are derived using
unsupervised learning as described in the next section.

###  2.2 Sub-Dialogue Structure Learning

((a)) Before split.

((b)) Temporal split.

((c)) Contextual split.

Figure 2: The design of two split methods. The dark-blue state is chosen to be
split. The light-blue state is the new state after split. Transitions to other
states are omitted for simplicity.

Given a specified topology, inference and training algorithms for HMMs are
well established Murphy (2012); the Viterbi algorithm gives the most likely
state sequence, and the Expectation-Maximization (EM) algorithm is used for
parameter estimation. To automatically learn the HMM topology, we apply a
greedy state splitting algorithm Ostendorf and Singer (1997), which learns a
left-to-right topology by constraining states to inherit the transition
constraints of their parent. The standard objective is maximum likelihood of
the DA sequence, which is unsupervised with respect to the conversation-level
task.

Topology learning is outlined in Algorithm 1. The initial model has a 3-state
left-to-right topology, initialized (assuming 70% of the conversation is
associated with the middle state) and then iteratively trained until the
improvement is lower than a fixed threshold or the iteration count exceeds
some number. At each iteration, the state with the highest entropy of emission
probability is chosen to be split. The topology can change into two new
configurations corresponding to temporal and contextual splits (Figure 2). The
EM algorithm is applied again on each configuration and the topology that
leads to the higher likelihood is chosen. We iteratively conduct the splitting
until the total number of states reaches the desired value (a hyperparameter).
The HMMs of Ostendorf and Singer (1997) used continuous observation
distributions. The splitting approach described below was designed for
discrete distributions.

1:$n$: number of split. $\tau_{i}$: topology after $i$ split.

2:Initialization: Run the EM algorithm on 3-state initial topology $\tau_{0}$.

3:for $i=1$ to $n$ do

4: begin

5: Select state $s\in\tau_{i-1}$ to split based on max entropy of observation
distribution $\eta_{i-1}(c|s)$.

6: Apply temporal split and get new topology $\tau_{i,t}$.

7: Apply contextual split and get new topology $\tau_{i,c}$.

8: Run the EM algorithm on $\tau_{i,t}$ and $\tau_{i,c}$.

9: Select the topology with higher likelihood as $\tau_{i}$.

10: end

Algorithm 1 Topology Learning Algorithm

#### Temporal split

The temporal split provides more detailed sequential structure along a path.
Figure 2(b) shows the result of a temporal split on the selected state (dark-
blue) in Figure 2(a). The light-blue node is the new child state that inherits
all incoming and outgoing edges and the transition probabilities of the dark-
blue state except $y$ and $z$. Edges $y$ and $z$ are initialized to $p_{x}/2$
where $p_{x}$ is the probability of the original edge $x$ of dark-blue state.
The old incoming edges of dark-blue state are removed and outgoing edges are
preserved.

#### Contextual split

The contextual split allows for alternate sub-dialogue paths. 2(c) illustrates
the contextual split applied on the dark-blue state. The light-blue state
inherits everything but the observation distribution of dark-blue state. With
the aim of modeling different types of paths, when copying the observation
probabilities to the light-blue state, we omit the top emission probability of
the dark-blue node and set it to 0 and normalize the rest of probabilities. In
terms of the transition probabilities, the light-blue state inherits all from
the dark-blue one; $p_{x}=p_{z}$ where $p_{x}$ and $p_{z}$ are the transition
probabilities of edges $x$ and $z$.

###  2.3 Pre-Training and End-Task Training

Both for initializing the ht and for deriving the DAs, we use the transformer-
based BERT model Devlin et al. (2019) for encoding individual utterances
$u_{t}$, pre-trained using masked language modeling and next-sentence
prediction. Due to the style differences of dialogue data vs. written text, we
apply domain-adaptive pretraining (DAPT) Gururangan et al. (2020) to adapt
BERT for dialogue applications. As shown later (section 3), adapting BERT with
DAPT provides substantial improvement in terms of predictive power as well as
optimization stability.

For the ht alone, supervised training involves learning the weights of the
final task-level linear layer, the utterance-level transformer, and the word-
level transformer.

For Theta, supervised training involves learning the weights of the cluster-
and state-level transformers, in addition to all updates associated with the
ht component described above. The cluster sequences are obtained using the
word-level transformer with DAPT and the associated cluster mapping obtained
from unsupervised learning, i.e., without task-level finetuning. Similarly,
there are no task-level supervision updates to the parameters associated with
the HMM that is used to derive the state sequence.

##  3 Experiment

###  3.1 Datasets and Evaluation Metrics.

We use three datasets with conversation-level classification tasks to evaluate
our model. The detailed statistics of the datasets are shown in Appendix B.

CraigslistBargain He et al. (2018) is a public negotiation dataset where
buyers and sellers negotiate the prices of items on sale. In each
conversation, the buyer has a target price in their mind and attempts to reach
an agreement with the seller. Following previous work Zhou et al. (2020);
Joshi et al. (2021), we use the same list of 14 handcrafted utterance DAs and
the 5-class sale-to-list price ratio labels provided in their code base. The
14 handcrafted utterance DAs are used as comparison to evaluate if our
unsupervised version of DAs is learning good representations. Classification
of sale-to-list price ratio is used as the downstream task, with accuracy as
the evaluation criterion.

Abcd Chen et al. (2021) is a public customer support dataset that is
introduced to study customer service dialogues. In each conversation, an agent
follows guidelines to help a customer solve their issue. Conversations are
categorized with flows and subflows. Flows are broad categories, such as
shipping issue, account access, or purchase dispute. Subflows comprise 96
fine-grained labels, for example, shipping status question, recover password,
or invalid promotion code. Each conversation is annotated with a flow and a
subflow. We use classification of the subflows as our conversation-level task.
Macro and micro F1 scores are used to reflect the performance of imbalanced
subflow classes.

Call Center is a private collection of customer service conversations. Phone
calls are automatically transcribed and private user information is
anonymized. Conversations are annotated with a binary indicator as to whether
or not there will be a callback within two days. (Such callbacks are an
indicator that the problem was not solved in the call.) For the task of
callback prediction, we measure area under the ROC curve (ROC AUC).

###  3.2 Implementation Details

#### Experimental Setup.

We develop our K-means and HMMs using the packages Faiss Johnson et al. (2019)
and Pomegranate Schreiber (2018). The number of DAs and the size of the HMM
state space are chosen separately for each dataset based on development set
performance. We initialize and finetune our experiments based on uncased base
model of BERT downloaded from HuggingFace Wolf et al. (2020). We DAPT with
dynamic whole-word masking (WWM) on 128-token segments for each dataset.
During finetuning, the learning rate and warm-up steps are $1\times 10^{-5}$
and $0.1$ epoch, respectively. Models are selected by the best score on the
development set for each dataset. Further hyperparameter details are in
Appendix A.

Model | % Acc.
---|---
FeHED | 42.3
HED + RNN | 47.9
HED + transformer | 53.7
DialoGraph | 53.1
ht | 54.1$\pm$ 2.4
Theta | 66.1$\pm$ 1.0

Table 1: Results on the test set of CraigslistBargain in accuracy. For models
studied in this paper (lower part), the median number is reported with
standard deviation calculated based on 15 random runs.

| Abcd | Call Center
---|---|---
| F1 |
Model | Micro | Macro | Weighted | ROC AUC
ht | 52.2 | 25.4 | 45.7 | 69.6
Theta | 62.8 | 39.1 | 59.9 | 71.3

Table 2: Results on the test sets of Abcd and Call Center datasets.

| CraigslistBargain | Abcd | Call Center
---|---|---|---
|  | F1 |
Model | Accuracy | Micro | Macro | Weighted | ROC AUC
ht w/o DAPT | 48.0 | 15.4 | 4.2 | 9.4 | 68.4
ht | 50.3 | 52.2 | 26.9 | 46.3 | 71.2
Theta (cluster only) | 60.2 | 59.8 | 35.3 | 55.7 | 72.2
Theta (state only) | 51.7 | 58.8 | 32.8 | 54.1 | 72.1
Theta | 61.3 | 62.6 | 38.6 | 59.5 | 72.8

Table 3: Ablation on the development sets of CraigslistBargain, Abcd and Call
Center datasets. All models with structure are statistically better than ht.
Theta is better ($p<0.01$) than the cluster-only alternative except for the
Call Center.

###  3.3 Comparison Systems

We use the hierarchical transformer (ht) as a baseline for all datasets in
comparison to Theta. For CraigslistBargain, we also include three additional
baselines from two works Zhou et al. (2020); Joshi et al. (2021) that employ
the DAs extracted by heuristic methods; our systems use K-means to obtain
primitive DAs.

#### FST-enhanced hierarchical encoder-decoder model (FeHED).

FeHED Zhou et al. (2020) uses an RNN-based sequence-to-sequence model with
finite-state transducers for encoding sequences of strategies and DAs.

#### Hierarchical encoder-decoder (HED) + RNN or transformer.

HED encodes dialogue utterances with a transformer (initialized from
pretrained BERT), and the decoder generates the next response. An RNN or
transformer encodes strategies and DAs. HED + RNN is based on the dialogue
manager of He et al. (2018); Joshi et al. (2021) replace the RNN with a
transformer.

####  DialoGraph.

Joshi et al. (2021). The state-of-the-art HED-based model on CraigslistBargain
dataset leverages graph attention networks (GAT; Veličković et al., 2018) to
encode strategies and DAs.

###  3.4 Prediction Results

#### Performance on Negotiation Dialogues.

Table 1 reports the results of different systems on the test set of
CraigslistBargain dataset. All models are based on the BERT-base model. ht
with only text outperforms the state-of-the-art DialoGraph which leverages a
graph-based representation of conversation structure. This verifies our
hypothesis that DAPT with target data indeed improves BERT for dialogue tasks.
Compared with ht, Theta achieves better prediction accuracy and smaller
variance, which suggests that integrating the structure view helps stabilize
training with different random seeds. Theta provides a 24.5% relative gain in
accuracy over DialoGraph, setting a new state of the art. This further
validates the advantage of our learned conversation structure for a predictive
task.

#### Performance on Customer Support Domain.

Similar to the results on the negotiation dialogue domain, Table 2 shows that
conversation structure effectively enhances the performance in the customer
service domain, Abcd and Call Center.

#### Ablation.

Table 3 reports the results of ablating different components of Theta on the
validation sets of all datasets. The first rows show that DAPT is useful on
all tasks particularly for Abcd with its skewed class distribution. We also
observe that Theta consistently achieves the best performance over all tasks.
The cluster-based DA sequence provides more information than the sub-dialogue
states, but incorporating all three views together leads to the best
performance. Statistical significance is tested using bootstrap resampling
Efron and Tibshirani (1993); Berg-Kirkpatrick et al. (2012).

Prior work Zhou et al. (2020); Joshi et al. (2021) on CraigslistBargain use
domain knowledge in rule-based annotation of DAs. To assess the use of K-means
clusters for learning DAs, we also trained an HMM using the provided DAs. The
resulting model obtained 66.5% accuracy on the test data, which is not
significantly different the 66.1% results obtained using K-means (cf. Table
1).

##  4 Interpretation and Analysis

In this section, we leverage automatic summarization of clusters and states to
derive insights into the learned conversation structure, both for
interpretability of the model and for applications such as conversation
analytics and summarization. As an example, we analyze fine-grained components
from the learned topology, i.e., most frequent paths and individual state
n-grams, to investigate their associations with different dialogue
characteristics.

We apply graph-based unsupervised summarization Boudin and Morin (2013); Shang
et al. (2018) over utterances in each state (decoupling participant roles) and
in each cluster. On CraigslistBargain and Abcd, this leads to more than
3$\times$ reduction in conversation length.

Figure 3: The 8-state topology on CraigslistBargain dataset. The thicker edges
indicate higher levels of negotiation success; in contrast, the thinner edges
represent lower levels. Due to space limitations, only 6 state summaries are
shown. The detailed topology with all cluster and state summaries is in
Appendix D.

Figure 3 shows the 8-state topology of CraigslistBargain with selected state
summaries. Based on the summaries, it is easy to see that S1 and S8 capture
opening and closing DAs, respectively, while S5 and S6 correspond to different
negotiation strategies. We also find that conversations with shorter paths are
likely to involve a less experienced seller or lower buyer interest, e.g.,
$92\%$ conversations with path S1-S2-S8 lead to under listing sells. On the
other hand, sellers that say offers are too low are more likely get better
prices, e.g., $91\%$ conversations with path S1-S2-S3-S5-S7-S8.

Party | Utterance | Cluster Summary | State Summary
---|---|---|---
Agent | Welcome to AcmeBrands! How can I help you? | How can I help you? |  A: How can I help you today? C: I want to check my order.
Customer | Hello, I would like to change my shipping details | I want to check my shipping
| as they have changed recently due to a move
Agent | I would be happy to help you with that | I can help you with that |  A: Can I have your account/order? C: My account/order is _______
Agent | Is there an outstanding order? | How long have you been waiting?
Agent | Or is this just an update to your account? | I have pulled up your account.
Customer | Yes my order id is 4870952797 | My order/account ID is _______
Agent | What is your name please? | What is your name?
Customer | Crystal Minh | <name>
Agent | What is the shipping status of the order? | What is the shipping status?
Customer | In Transit | In store/ In transit
Agent | Next I need to validate your purchase. | I need your name
| I will need your username and email.
Customer | cminh948<EMAIL_ADDRESS>| <email>
Agent | Thank you | Thank you
Agent | and the new address please? | Can you tell me ______? |  A: Can I have the address? C: My address is _______
Customer | 9756 Primrose Street Newark, MI 85971 | <address>
Agent | All taken care of! | Your order has been updated
Agent | Is there anything else today? | Anything else? |  A: Anything else I can help? C: That’s all. Thank you.
Customer | Thank you that is all | That’s all. Thank you.
Agent | Have a great one! | Have a good one!

Table 4: An example of Abcd with cluster and state summaries. A and C stand
for agent and customer, respectively.

####  Abcd

Table 4 shows an example with both cluster and state summaries. Based on the
cluster summaries, we see that K-means learns typical DAs associated with
customer service, e.g., information requests from the agent and the
corresponding customer replies (<name>, <email>, <address>). States correspond
to sub-dialogues where the agent follows certain protocols in resolving a sub-
task (e.g., verifying account information). Alignment of flow labels with the
most frequent paths through the HMM topology shows that paths are highly
indicative of the corresponding dialogue flow. The high confusions are among
certain flows, such as storewide_query and single_item_query, which one would
expect to have have similar DAs.

##  5 Related Work

HMMs have been leveraged for learning structure in language for many years,
such as in early work on inducing word-level part-of-speech tags (Merialdo,
1994). Accordingly, most work on unsupervised learning of both DAs and
conversation structure leverages HMMs.

#### Unsupervised Learning of Dialogue Acts.

Since dialogue act recognition can be thought of as a sentence-level tagging
task, initial work on unsupervised learning of DAs was similar to word
tagging, involving some use of language models or fully-connected HMMs to
account for sequential dependency of labels. Ritter et al. (2010) use an HMM
with a factored state space with a topic model to decouple speech act from
topic characteristics. The observation model $\eta$ in the HMM is a bag of
words (unigram) model. The approach was later extended by incorporating
speaker information Joty et al. (2011); Paul (2012). Brychcin and Kral (2017)
further extend this work with a Gaussian mixture observation model (GMM) where
the utterance representation is the average of GloVe word embeddings. They
compare the results to a simple K-means clustering, which is not as effective
as the HMM but gives similar results to the method proposed by Ritter et al.
(2010) when applied to the Switchboard corpus. Hierarchical clustering of
delexicalized utterance embeddings is used by Gunasekara et al. (2019), who
use domain knowledge in preprocessing to identify phrases such as “Indian
food” as “CUISINE_TYPE,” for example. Our work on utterance categorization is
similar to the K-means approach in Brychcin and Kral (2017), but we use more
recent transformer-based utterance embeddings.

#### Unsupervised Learning of Dialogue Structure.

Task- or goal-oriented conversations typically have structure above the level
of the sentence in that a sequence of turns are associated with a common
function. In more complex conversations, the structure can be hierarchical,
with tasks and sub-tasks. Bangalore et al. (2008) used a parsing model to
automatically recognize dialogue acts and segment a conversation into sub-
tasks, leveraging hand-annotations of both DAs and sub-tasks. Since sub-task
structure varies depending on the task and there is little hand-annotated
data, most work has focused on unsupervised approaches with a flat
segmentation. Note that the problem of unsupervised learning here involves
jointly recognizing sub-dialogue segment boundaries, learning an inventory of
sub-dialogue types, and learning (or constraining) the sequential structure of
these types.

Early work on unsupervised learning used fully-connected HMMs to identify
structure in documents Barzilay and Lee (2004) for extractive summarization
and information ordering. The observation model was based on word bigrams with
the aim of capturing topic coherent segment. A similar idea is applied to
task-oriented dialogues using latent Dirichlet allocation for the observation
model Zhai and Williams (2014).

Studies that leverage constrained left-to-right HMM technologies include
Althoff et al. (2016), which aimed to learn stages/strategies of counselors in
mental health counseling, and Ramanath et al. (2014), which used a hidden
semi-Markov model for unsupervised alignment of privacy policy documents. Both
used unigram observation models. HMM-based conversation stages are combined
with a topic-based segmentation by Chen and Yang (2020) for dialogue
summarization. The use of unigram and bigram word models emphasizes topic in
segmenting conversations. Our work differs in that the automatically learned
speech acts are observations of the HMM, since word distributions are captured
by the ht.

Most similar to our work is Zhou et al. (2020), which uses two finite state
transducers (FSTs) to map a sequence of dialogue acts (or strategies) to a
sequence of state embeddings, which are then integrated into a hierarchical
encoder-decoder model for prediction of the next strategy in a negotiation
dialogue. The FSTs are analogous to our HMM, but the inputs are based on
learning from hand-labeled strategies and rule-based dialogue acts.

There are other approaches to modeling conversation structure that do not rely
on HMMs. DialoGraph Joshi et al. (2021) uses a graph attention network to
encode discrete DA and strategy label sequences. A variational recurrent
neural network is used to model structure by Shi et al. (2019). These
approaches are less amenable to the interpretation methods used in our work.

Two key differences in our approach compared to all these studies are: i) the
use of HMM topology learning via successive state splitting, and ii) the
integration of structural information using a multi-stream neural sequence
model.

##  6 Conclusion

In summary, this work combines two simple approaches for unsupervised learning
on top of embedded utterance representations (K-means clustering and HMM
topology design) to derive a hierarchical representation of conversation
structure, which is useful to enhance a hierarchical transformer in three
conversation-level classification tasks. The K-means clusters are intended to
approximate DAs and the HMM is intended to learn sub-dialogue structure.
Unlike prior work in this area, the sub-dialogues build on DA sequences rather
than unigram/bigram statistics, and the HMM incorporates forward-moving
dialogue flow constraints in topology learning, with the goal of capturing
sub-dialogue function.

## Acknowledgments

We thank Mourad Heddaya for exploring preliminary experiments when he was at
the University of Washington, and Chih-Ning (Sonia) Ho for help with
experiments and technical discussions. We also thank all members of the TIAL
lab and NLP groups at the University of Washington who provided suggestions
and insights into this work.

## Limitations

First, our experiments explore only two types of dialogues (negotiation and
customer support) with conversation-level tasks (identifying the topic or
assessing some measure of conversation success). Although Theta shows
promising results, it requires further exploration with other types of
conversations (e.g. information gathering, tutoring), including more examples
of spoken interactions, as well as extending Theta to multi-party discussions.
In addition, it would be of interest to assess the utility of automatically
learned structure for other types of tasks, such as call center analytics or
state tracking to support dialogue management or online agent support.

Second, we use K-means and HMMs for deriving the conversation structure, both
of which require dataset-specific hyperparameters that are unlikely to
transfer well to new datasets. Additionally, we only study a late fusion
strategy for combining discrete structure and text-based representations. A
more tightly integrated approach might be more effective. For example, our
K-means DA is based on a single utterance; however, sequence models have been
important for past work on unsupervised learning of DAs. Future work could
leverage sequential DA dependencies in joint DA and sub-dialogue structure
learning or explore continuous DA-like representations, as in Cheng et al.
(2019).

## Ethical Considerations

The automatic learning of conversation structure is dependent on having data
that is matched to the task of interest. A potential challenge is that biases
in the data could result in some conversation strategies not being well
represented. The summarization approach provides interpretability of the
model, but imperfect summarizations could lead to incorrect interpretations.

## References

  * Althoff et al. (2016) Tim Althoff, Kevin Clark, and Jure Leskovec. 2016.  Large-scale analysis of counseling conversations: An application of natural language processing to mental health.  _Transactions of the Association for Computational Linguistics_ , 4:463–476.
  * Bangalore et al. (2008) Srinivas Bangalore, Giuseppe Di Fabbrizio, and Amanda Stent. 2008.  Learning the structure of task-driven human–human dialogs.  _IEEE Transactions on Audio, Speech, and Language Processing_ , 16(7):1249–1259.
  * Barzilay and Lee (2004) Regina Barzilay and Lillian Lee. 2004.  Catching the drift: Probabilistic content models, with applications to generation and summarization.  In _Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics: HLT-NAACL 2004_ , pages 113–120, Boston, Massachusetts, USA. Association for Computational Linguistics.
  * Berg-Kirkpatrick et al. (2012) Taylor Berg-Kirkpatrick, David Burkett, and Dan Klein. 2012.  An empirical investigation of statistical significance in NLP.  In _Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning_ , pages 995–1005, Jeju Island, Korea. Association for Computational Linguistics.
  * Boudin and Morin (2013) Florian Boudin and Emmanuel Morin. 2013.  Keyphrase extraction for n-best reranking in multi-sentence compression.  In _Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 298–305, Atlanta, Georgia. Association for Computational Linguistics.
  * Brychcin and Kral (2017) T. Brychcin and P. Kral. 2017.  Unsupervised dialogue act induction using gaussian mixtures.  In _Prof. EMNLP_ , volume 2, page 485–490.
  * Chen et al. (2021) Derek Chen, Howard Chen, Yi Yang, Alexander Lin, and Zhou Yu. 2021.  Action-based conversations dataset: A corpus for building more in-depth task-oriented dialogue systems.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 3002–3017, Online. Association for Computational Linguistics.
  * Chen and Yang (2020) Jiaao Chen and Diyi Yang. 2020.  Multi-view sequence-to-sequence models with conversational structure for abstractive dialogue summarization.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 4106–4118, Online. Association for Computational Linguistics.
  * Chen and Guestrin (2016) Tianqi Chen and Carlos Guestrin. 2016.  Xgboost: A scalable tree boosting system.  In _Proceedings of the 22nd ACM sigkdd international conference on knowledge discovery and data mining_ , pages 785–794.
  * Cheng et al. (2019) Hao Cheng, Hao Fang, and Mari Ostendorf. 2019.  A dynamic speaker model for conversational interactions.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 2772–2785, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Core and Allen (1997) Mark G Core and James F Allen. 1997.  Coding dialogs with the damsl annotation scheme.  In _in Proc. Working Notes AAAI Fall Symp. Commun. Action in Humans_.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Efron and Tibshirani (1993) Bradley Efron and Robert Tibshirani. 1993.  An introduction to the bootstrap.  In _Chapman & Hall/CRC Monographs on Statistics & Applied Probability._ Taylor & Francis.
  * Grosz and Sidner (1986) Barbara J. Grosz and Candace L. Sidner. 1986.  Attention, intentions, and the structure of discourse.  _Computational Linguistics_ , 12(3):175–204.
  * Gunasekara et al. (2019) R. Chulaka Gunasekara, David Nahamoo, Lazaros C. Polymenakos, David Echeverría Ciaurri, Jatin Ganhotra, and Kshitij P. Fadnis. 2019.  Quantized dialog – a general approach for conversational systems.  _Computer Speech and Language_ , 54:17–30.
  * Gururangan et al. (2020) Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020.  Don’t stop pretraining: Adapt language models to domains and tasks.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 8342–8360, Online. Association for Computational Linguistics.
  * He et al. (2018) He He, Derek Chen, Anusha Balakrishnan, and Percy Liang. 2018.  Decoupling strategy and generation in negotiation dialogues.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 2333–2343, Brussels, Belgium. Association for Computational Linguistics.
  * Johnson et al. (2019) Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2019.  Billion-scale similarity search with GPUs.  _IEEE Transactions on Big Data_ , 7(3):535–547.
  * Joshi et al. (2021) Rishabh Joshi, Vidhisha Balachandran, Shikhar Vashishth, Alan Black, and Yulia Tsvetkov. 2021.  Dialograph: Incorporating interpretable strategy-graph networks into negotiation dialogues.  In _International Conference on Learning Representations_.
  * Joty et al. (2011) Shafiq Joty, Giuseppe Carenini, and Chin-Yew Lin. 2011.  Unsupervised modeling of dialog acts in asynchronous conversations.  In _Proc. International Joint Conference on Artificial Intelligence_ , pages 1807–1813.
  * Jurafsky et al. (1997) Dan Jurafsky, Elizabeth Shriberg, , and Debra Biasca. 1997.  Switchboard swbd-damsl shallow-discourse-function annotation coders manual, draft 13.  Technical report, University of Colorado, Boulder.
  * Lundberg and Lee (2017) Scott M Lundberg and Su-In Lee. 2017.  A unified approach to interpreting model predictions.  In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, _Advances in Neural Information Processing Systems 30_ , pages 4765–4774. Curran Associates, Inc.
  * Merialdo (1994) Bernard Merialdo. 1994.  Tagging English text with a probabilistic model.  _Computational Linguistics_ , 20(2):155–171.
  * Murphy (2012) Kevin Murphy. 2012.  _Machine Learning: A Probabilistic Perspective_.  MIT Press.
  * Ostendorf and Singer (1997) Mari Ostendorf and Harald Singer. 1997.  Hmm topology design using maximum likelihood successive state splitting.  _Computer Speech & Language_, 11(1):17–41.
  * Pappagari et al. (2019) Raghavendra Pappagari, Piotr Zelasko, Jesús Villalba, Yishay Carmiel, and Najim Dehak. 2019.  Hierarchical transformers for long document classification.  In _2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)_ , pages 838–844. IEEE.
  * Paszke et al. (2019) Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019.  Pytorch: An imperative style, high-performance deep learning library.  In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, _Advances in Neural Information Processing Systems 32_ , pages 8024–8035. Curran Associates, Inc.
  * Paul (2012) Michael J. Paul. 2012.  Mixed membership Markov models for unsupervised conversation modeling.  In _Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning_ , pages 94–104, Jeju Island, Korea. Association for Computational Linguistics.
  * Ramanath et al. (2014) Rohan Ramanath, Fei Liu, Norman Sadeh, and Noah A. Smith. 2014.  Unsupervised alignment of privacy policies using hidden Markov models.  In _Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 605–610, Baltimore, Maryland. Association for Computational Linguistics.
  * Rasley et al. (2020) Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. 2020.  Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters.  In _Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, pages 3505–3506.
  * Ritter et al. (2010) Alan Ritter, Colin Cherry, and Bill Dolan. 2010.  Unsupervised modeling of Twitter conversations.  In _Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics_ , pages 172–180, Los Angeles, California. Association for Computational Linguistics.
  * Santra et al. (2021) Bishal Santra, Potnuru Anusha, and Pawan Goyal. 2021.  Hierarchical transformer for task oriented dialog systems.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 5649–5658, Online. Association for Computational Linguistics.
  * Schreiber (2018) Jacob Schreiber. 2018.  Pomegranate: fast and flexible probabilistic modeling in python.  _Journal of Machine Learning Research_ , 18(164):1–6.
  * Shang et al. (2018) Guokan Shang, Wensi Ding, Zekun Zhang, Antoine Tixier, Polykarpos Meladianos, Michalis Vazirgiannis, and Jean-Pierre Lorré. 2018.  Unsupervised abstractive meeting summarization with multi-sentence compression and budgeted submodular maximization.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 664–674, Melbourne, Australia. Association for Computational Linguistics.
  * Shi et al. (2019) Weiyan Shi, Tiancheng Zhao, and Zhou Yu. 2019.  Unsupervised dialog structure learning.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 1797–1807, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Veličković et al. (2018) Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. 2018.  Graph attention networks.  In _International Conference on Learning Representations_.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020.  Transformers: State-of-the-art natural language processing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 38–45, Online. Association for Computational Linguistics.
  * Zhai and Williams (2014) Ke Zhai and Jason D. Williams. 2014.  Discovering latent structure in task-oriented dialogues.  In _Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 36–46, Baltimore, Maryland. Association for Computational Linguistics.
  * Zhou et al. (2020) Yiheng Zhou, Yulia Tsvetkov, Alan W Black, and Zhou Yu. 2020.  Augmenting non-collaborative dialog systems with explicit semantic and strategic dialog history.  In _International Conference on Learning Representations_.

##  Appendix A Experimental Setup Details

We pretrain and finetune on BERT Devlin et al. (2019) downloaded from
Huggingface Transformers Wolf et al.
(2020)333https://github.com/huggingface/transformers and use the uncased base
model of BERT in most of our experiments. To feed lengthy conversations to the
model, we employ gradient checkpoint and DeepSpeed Rasley et al. (2020), a
deep learning optimization library, to reduce GPU memory usage and accelerate
the training process.

The details of the model hyperparameters are as follows. 1-layer and 2-head
transformers with 300 hidden size are applied to encode sequences of
utterance-level embeddings in text view and sequences of clusters and states
in structure view. Thus, the total number of parameters of our best system
Theta, including base model of BERT and 3 one-layer transformers, is about
113M. For in-domain adaptation pretraining (DAPT), we use $5\times 10^{-5}$ as
learning rate and 5000 steps for CraigslistBargain and Abcd and 30000 steps
for Call Center. $0.1$ epochs are used as warm-up steps with linear learning
rate decay. Gradient accumulation and PyTorch Paszke et al. (2019) distributed
data parallel GPU training are applied to achieve the equivalent training
batch size 4096. For finetuning, we set $1\times 10^{-5}$ as the learning
rates, 4 epochs in total and $0.1$ epochs for warm-up steps with linear decay.
The equivalent training batch size is 16 during finetuning. Besides, the
layer-wise learning rate decay is utilized to stabilize the training results;
the rates are from $0.7,0.8,0.9$ and the $0.9$ leads to the best performance.
For the rest of the training hyperparameters, we follow the default values in
HuggingFace’s training script.

For K-means, we use Faiss Johnson et al.
(2019)444https://github.com/facebookresearch/faiss with GPU to speed up
clustering process for large private corpus. For HMMs, we develop our
splitting algorithm via Pomegranate Schreiber
(2018),555https://github.com/jmschrei/pomegranate a Python package that
implements fast and flexible probabilistic models, to build our topology
learning algorithm. The predefined numbers of clusters vary for different
datasets. To compare with handcrafted DAs provided in CraigslistBargain, we
define number of clusters $k=14$ for each party. For customer service domain,
we set $k=60$ for Abcd and $k=120$ for Call Center. For all datasets, we try
the number of states from 5 to 20 and find the best numbers of states are 8,
12, and 12 for CraigslistBargain, Abcd, and Call Center, respectively. Each
training run takes at most 2 hours on 2 Nvidia GeForce RTX 2080Ti GPUs for
CraigslistBargain and Abcd and 54 hours on 8 GPUs on Call Center. All models
are saved based on the best performance on the development sets. For each
experiment on CraigslistBargain and Abcd, we conduct $15$ random runs and
report the median and standard deviation. Due to the computation limitations
and the size of corpus, we only conduct a single run for Call Center for each
experiment setting. The total number of GPU hours for all experiments,
including different runs with random seeds, is 1536 hours approximately.

|  | Craigslist
---
Bargain
Abcd |  | Call
---
Center
# dialogues | 6682 | 10042 | 949410
# turns / dialogue | 9.2 | 22.1 | 71.6
# tokens / turn | 15.5 | 9.2 | 16.3
# tokens / dialogue | 142.6 | 202.5 | 1167.1

Table 5: Data statistics of the datasets.

|  | Craigslist
---
Bargain
Abcd |  | Call
---
Center
train set # dialogues | 4828 | 8034 | 711310
dev. set # dialogues | 561 | 1004 | 95540
test set # dialogues | 567 | 1004 | 142560

Table 6: Train/dev./test split of datasets

##  Appendix B Dataset Details

We follow all original data preprocessing scripts for CraigslistBargain
666https://github.com/rishabhjoshi/DialoGraph_ICLR21. and
Abcd.777https://github.com/asappresearch/abcd. For the private collection of
customer service conversations, Call Center, all private user information is
anonymized. The data statistics are summarized in Table 5 and Table 6.

##  Appendix C Topology with Summaries

Figure 4 shows the detailed topology with both cluster and sub-dialogue state
summaries. For each sub-dialogue state, we add the cluster summaries with top
3 emission probabilites and the sub-dialogue state summaries for the buyer and
the seller. The thickness of edges indicates the levels of negotiation success
and the edges with probabilities lower than $0.01$ are pruned for simplicity.

##  Appendix D License of Artifacts

The license of code for Wolf et al. (2020) and Schreiber (2018) are Apache
license version 2.0. The license of code for Joshi et al. (2021), Rasley et
al. (2020), and Chen et al. (2021) are MIT License. The terms for use of our
artifacts will be included in our released package.

Figure 4: The 8-state full topology with cluster and sub-dialogue state
summaries on CraigslistBargain dataset. The thicker edges represent higher
levels of negotiation success.


* (42) K. G. Chetyrkin, Combinatorics of $R$-, $R^{-1}$-, and $R^{*}$-operations and asymptotic expansions of Feynman integrals in the limit of large momenta and masses, tech. rep., Max-Planck-Institut für Physik, Werner-Heisenberg-Institut, Munich, Germany, Mar., 1991, arXiv:1701.08627 [hep-th].
  * (43) K. G. Chetyrkin, S. G. Gorishny, S. A. Larin and F. V. Tkachov, Five-loop renormalization group calculations in the $g\varphi^{4}$ theory, _Physics Letters B_ 132 (Dec., 1983) pp. 351–354.
  * (44) K. G. Chetyrkin, S. G. Gorishny, S. A. Larin and F. V. Tkachov, “Analiticheskoe vychislenie pyatipetlevykh priblizheniĭ renormgruppovykh funktsiĭ modeli $g\varphi^{4}_{(4)}$ v $\mathrm{MS}$-skheme: podiagrammnyĭ analiz.” preprint INR P-0453, Moscow, in Russian, translated title: Analytic calculation of the five-loop approximations of the renormalization group functions of the $g\varphi^{4}_{(4)}$ model in the MS-scheme: a diagrammatic analysis, 1986.
  * (45) K. G. Chetyrkin, A. L. Kataev and F. V. Tkachov, New approach to evaluation of multiloop Feynman integrals: The Gegenbauer polynomial $x$-space technique, _Nuclear Physics B_ 174 (Nov., 1980) pp. 345–377.
  * (46) K. G. Chetyrkin, A. L. Kataev and F. V. Tkachov, Errata, _Physics Letters B_ 101 (1981), no. 6 pp. 457–458.
  * (47) K. G. Chetyrkin, A. L. Kataev and F. V. Tkachov, Five-loop calculations in the $g\varphi^{4}$ model and the critical index $\eta$, _Physics Letters B_ 99 (Feb., 1981) pp. 147–150. Erratum: ChetyrkinKataevTkachov:5loopPhi4Erratum .
  * (48) K. G. Chetyrkin and V. A. Smirnov, $R^{*}$-operation corrected, _Physics Letters B_ 144 (Sept., 1984) pp. 419–424.
  * (49) K. G. Chetyrkin and F. V. Tkachov, Integration by parts: The algorithm to calculate $\beta$-functions in 4 loops, _Nucl. Phys. B_ 192 (Nov., 1981) pp. 159–204.
  * (50) K. G. Chetyrkin and F. V. Tkachov, Infrared $R$-operation and ultraviolet counterterms in the MS-scheme, _Physics Letters B_ 114 (Aug., 1982) pp. 340–344.
  * (51) F. Chishtie, V. Elias and T. G. Steele, Asymptotic Padé-approximant predictions for renormalization-group functions of massive $\phi^{4}$ scalar field theory, _Physics Letters B_ 446 (Jan., 1999) pp. 267–271, arXiv:hep-ph/9809538.
  * (52) N. Clisby, Accurate estimate of the critical exponent $\nu$ for self-avoiding walks via a fast implementation of the pivot algorithm, _Phys. Rev. Lett._ 104 (Feb., 2010) p. 055702, arXiv:1002.0494 [cond-mat.stat-mech].
  * (53) N. Clisby, Scale-free Monte Carlo method for calculating the critical exponent $\gamma$ of self-avoiding walks, _Journal of Physics A: Mathematical General_ 50 (June, 2017) p. 264003, arXiv:1701.08415 [cond-mat.stat-mech].
  * (54) N. Clisby and B. Dünweg, High-precision estimate of the hydrodynamic radius for self-avoiding walks, _Phys. Rev. E_ 94 (Nov., 2016) p. 052102.
  * (55) N. Clisby, R. Liang and G. Slade, Self-avoiding walk enumeration via the lace expansion, _Journal of Physics A: Mathematical and Theoretical_ 40 (Sept., 2007) p. 10973.
  * (56) J. C. Collins, Structure of counterterms in dimensional regularization, _Nuclear Physics B_ 80 (Oct., 1974) pp. 341–348.
  * (57) A. Connes and D. Kreimer, Renormalization in Quantum Field Theory and the Riemann-Hilbert Problem I: The Hopf Algebra Structure of Graphs and the Main Theorem, _Commun. Math. Phys._ 210 (2000), no. 1 pp. 249–273, arXiv:hep-th/9912092.
  * (58) P.-G. de Gennes, _Scaling concepts in polymer physics_.  Cornell University Press, London, 1979.
  * (59) B. Delamotte, M. Dudka, Y. Holovatch and D. Mouhanna, Relevance of the fixed dimension perturbative approach to frustrated magnets in two and three dimensions, _Phys. Rev. B_ 82 (Sept., 2010) p. 104432, arXiv:1009.1492 [cond-mat.stat-mech].
  * (60) Y. Deng, Bulk and surface phase transitions in the three-dimensional $O(4)$ spin model, _Phys. Rev. E_ 73 (May, 2006) p. 056116.
  * (61) F. M. Dittes, Y. A. Kubyshin and O. V. Tarasov, Four-loop approximation in the $\phi^{4}$ model, _Theoretical and Mathematical Physics_ 37 (1978), no. 1 pp. 879–884.
  * (62) D. Dorigoni, “An introduction to resurgence, trans-series and alien calculus.” preprint, Nov., 2014, arXiv:1411.3585 [hep-th].
  * (63) G. V. Dunne and M. Ünsal, Resurgence and trans-series in quantum field theory: the $\mathbb{CP}^{N-1}$ model, _JHEP_ 2012 (Nov., 2012) p. 170, arXiv:1210.2423 [hep-th].
  * (64) J.-P. Eckmann, J. Magnen and R. Sénéor, Decay properties and Borel summability for the Schwinger functions in $P(\Phi)_{2}$ theories, _Communications in Mathematical Physics_ 39 (1975), no. 4 pp. 251–271.
  * (65) S. El-Showk, M. F. Paulos, D. Poland, S. Rychkov, D. Simmons-Duffin and A. Vichi, Solving the 3d Ising model with the conformal bootstrap ii. $c$-minimization and precise critical exponents, _Journal of Statistical Physics_ 157 (2014), no. 4 pp. 869–914, arXiv:1403.4545 [hep-th].
  * (66) J. Ellis, M. Karliner and M. A. Samuel, A prediction for the 4-loop $\beta$ function in QCD, _Physics Letters B_ 400 (May, 1997) pp. 176–181, arXiv:hep-ph/9612202.
  * (67) W. A. A. Gaddah and I. S. Jwan, Borel resummation method with conformal mapping and the ground state energy of the quartic anharmonic oscillator, in _The 3rd National Conference of Basic Science 25–27/4/2009 Aljabal Algharbi University, Gharian - Libya_ , pp. 175–182, 2009.
  * (68) N. Ghosh, S. Rößler, U. K. Rößler, K. Nenkov, S. Elizabeth, H. L. Bhat, K. Dörr and K.-H. Müller, Heisenberg-like critical properties in ferromagnetic $\mathrm{Nd}_{1-x}\mathrm{Pb}_{x}\mathrm{MnO}_{3}$ single crystals, _Journal of Physics: Condensed Matter_ 18 (2006), no. 2 pp. 557–567.
  * (69) M. Golz, E. Panzer and O. Schnetz, Graphical functions in parametric space, _Letters in Mathematical Physics_ 107 (2017), no. 6 pp. 1177–1192, arXiv:1509.07296 [hep-th].
  * (70) S. G. Gorishny, S. A. Larin and F. V. Tkachov, $\varepsilon$-expansion for critical exponents: The $O(\varepsilon^{5})$ approximation, _Physics Letters A_ 101 (1984), no. 3 pp. 120–123.
  * (71) J. A. Gracey, Progress with large $N_{f}$ $\beta$-functions, in _New computing techniques in physics research V. Proceedings, 5th International Workshop, AIHENP ’96, Lausanne, Switzerland, September 2–6, 1996_ , vol. 389 of _Nucl. Instrum. Meth. A_ , pp. 361–364, 1997.  arXiv:hep-ph/9609409.
  * (72) J. A. Gracey, Four loop renormalization of $\phi^{3}$ theory in six dimensions, _Phys. Rev. D_ 92 (Jul, 2015) p. 025012, arXiv:1506.03357 [hep-th].
  * (73) J. A. Gracey, $F_{4}$ symmetric $\phi^{3}$ theory at four loops, _Phys. Rev. D_ 95 (Mar., 2017) p. 065030, arXiv:1703.03782 [hep-th].
  * (74) J. A. Gracey, T. Luthe and Y. Schröder, Four loop renormalization of the Gross-Neveu model, _Phys. Rev. D_ 94 (Dec., 2016) p. 125028, arXiv:1609.05071 [hep-th].
  * (75) G. Grunberg, Renormalization group improved perturbative QCD, _Physics Letters B_ 95 (Sept., 1980) pp. 70–74. [Erratum: Phys. Lett.110B,501(1982)].
  * (76) R. Guida and J. Zinn-Justin, Critical exponents of the $N$-vector model, _Journal of Physics A Mathematical General_ 31 (Oct., 1998) pp. 8103–8121, arXiv:cond-mat/9803240.
  * (77) M. Hasenbusch, Eliminating leading corrections to scaling in the three-dimensional $O(N)$ symmetric $\phi^{4}$ model: $N=3$ and $4$, _Journal of Physics A: Mathematical and General_ 34 (Oct., 2001) pp. 8221–8236, arXiv:cond-mat/0010463.
  * (78) M. Hasenbusch, Finite size scaling study of lattice models in the three-dimensional Ising universality class, _Phys. Rev. B_ 82 (Nov., 2010) p. 174433, arXiv:1004.4486 [cond-mat.stat-mech].
  * (79) M. Hasenbusch and E. Vicari, Anisotropic perturbations in three-dimensional $O(N)$-symmetric vector models, _Phys. Rev. B_ 84 (Sept., 2011) p. 125136, arXiv:1108.0491 [cond-mat.stat-mech].
  * (80) F. Herzog and B. Ruijl, The $R^{\ast}$-operation for Feynman graphs with generic numerators, _Journal of High Energy Physics_ 5 (May, 2017) p. 37, arXiv:1703.03776 [hep-th].
  * (81) F. Herzog, B. Ruijl, T. Ueda, J. A. M. Vermaseren and A. Vogt, The five-loop beta function of Yang-Mills theory with fermions, _JHEP_ 2017 (Feb., 2017) p. 090, arXiv:1701.01404 [hep-ph].
  * (82) S. Holtmann and T. Schulze, Critical behavior and scaling functions of the three-dimensional $O(6)$ model, _Phys. Rev. E_ 68 (Sept., 2003) p. 036111, arXiv:hep-lat/0305019.
  * (83) F. Jasch and H. Kleinert, Fast-convergent resummation algorithm and critical exponents of $\phi^{4}$-theory in three dimensions, _Journal of Mathematical Physics_ 42 (Jan., 2001) pp. 52–73, arXiv:cond-mat/9906246.
  * (84) U. D. Jentschura and G. Soff, Improved conformal mapping of the Borel plane, _J. Phys. A_ 34 (2001), no. 7 pp. 1451–1457, arXiv:hep-ph/0006089.
  * (85) G. A. Kalagov, M. V. Kompaniets and M. Y. Nalimov, Renormalization-group investigation of a superconducting $U(r)$-phase transition using five loops calculations, _Nuclear Physics B_ 905 (Apr., 2016) pp. 16–44, arXiv:1505.07360 [cond-mat.stat-mech].
  * (86) D. I. Kazakov, The method of uniqueness, a new powerful technique for multiloop calculations, _Physics Letters B_ 133 (Dec., 1983) pp. 406–410.
  * (87) D. I. Kazakov and V. S. Popov, On the summation of divergent perturbation series in quantum mechanics and field theory, _Journal of Experimental and Theoretical Physics_ 95 (2002), no. 4 pp. 581–600.
  * (88) D. I. Kazakov and V. S. Popov, Asymptotic behavior of the Gell-Mann-Low function in quantum field theory, _JETP Letters_ 77 (May, 2003) pp. 453–457.
  * (89) D. I. Kazakov, O. V. Tarasov and D. V. Širkov, Analytic continuation of the results of perturbation theory for the model $g\phi^{4}$ to the region $g\gtrsim 1$, _Theoretical and Mathematical Physics_ 38 (1979), no. 1 pp. 9–16.
  * (90) H. Kleinert, Critical exponents from seven-loop strong-coupling $\varphi^{4}$ theory in three dimensions, _Phys. Rev. D_ 60 (Sept., 1999) p. 085001, arXiv:hep-th/9812197.
  * (91) H. Kleinert, Converting divergent weak-coupling into exponentially fast convergent strong-coupling expansions, _Electron. J. Theor. Phys._ 8 (2011), no. 25 pp. 15–56, arXiv:1006.2910 [math-ph].
  * (92) H. Kleinert, J. Neu, V. Schulte-Frohlinde, K. G. Chetyrkin and S. A. Larin, Five-loop renormalization group functions of $O(n)$-symmetric $\phi^{4}$-theory and $\epsilon$-expansions of critical exponents up to $\epsilon^{5}$, _Physics Letters B_ 272 (Nov., 1991) pp. 39–44, arXiv:hep-th/9503230. Erratum: KNFCL:5loopPhi4Erratum .
  * (93) H. Kleinert, J. Neu, V. Schulte-Frohlinde, K. G. Chetyrkin and S. A. Larin, Five-loop renormalization group functions of $O(n)$-symmetric $\phi^{4}$-theory and $\epsilon$-expansions of critical exponents up to $\epsilon^{5}$, _Physics Letters B_ 319 (Dec., 1993) p. 545. Erratum to KNFCL:5loopPhi4 .
  * (94) H. Kleinert and V. Schulte-Frohlinde, Exact five-loop renormalization group functions of $\phi^{4}$-theory with $O(N)$-symmetric and cubic interactions. critical exponents up to $\epsilon^{5}$, _Physics Letters B_ 342 (1995), no. 1–4 pp. 284–296, arXiv:cond-mat/9503038.
  * (95) H. Kleinert and V. Schulte-Frohlinde, Critical exponents from five-loop strong-coupling $\phi^{4}$-theory in $4-\varepsilon$ dimensions, _Journal of Physics A: Mathematical and General_ 34 (2001), no. 5 pp. 1037–1049, arXiv:cond-mat/9907214.
  * (96) H. Kleinert and V. Schulte-Frohlinde, _Critical Properties of $\varphi^{4}$-theories_.  World Scientific, 2001.
  * (97) M. V. Komarova and M. Y. Nalimov, Large-order asymptotic terms in perturbation theory: The first $(4-\epsilon)$-expansion correction to renormalization constants in the $O(n)$-symmetric theory, _Theoretical and Mathematical Physics_ 143 (May, 2005) pp. 664–680.
  * (98) M. V. Komarova and M. Y. Nalimov, Asymptotic behavior of renormalization constants in higher orders of the perturbation expansion for the $(4-\varepsilon)$-dimensionally regularized $O(n)$-symmetric $\phi^{4}$ theory, _Theoretical and Mathematical Physics_ 126 (2001), no. 3 pp. 339–353.
  * (99) M. V. Kompaniets, Prediction of the higher-order terms based on Borel resummation with conformal mapping, in _Proceedings, 17th International Workshop on Advanced Computing and Analysis Techniques in Physics Research (ACAT 2016): Valparaiso, Chile, January 18–22, 2016_ , vol. 762 of _Journal of Physics: Conference Series_ , p. 012075, 2016.  arXiv:1604.04108 [hep-th].
  * (100) M. V. Kompaniets and E. Panzer, Renormalization group functions of $\phi^{4}$ theory in the MS-scheme to six loops, in _Loops and Legs in Quantum Field Theory Leipzig, Germany, April 24–29_ , Proceedings of Science, 2016.  arXiv:1606.09210 [hep-th], PoS(LL2016)038.
  * (101) F. Kos, D. Poland, D. Simmons-Duffin and A. Vichi, Precision islands in the Ising and $O(N)$ models, _JHEP_ 2016 (Aug., 2016) p. 36, arXiv:1603.04436 [hep-th].
  * (102) Y. A. Kubyshin, Corrections to the asymptotic expressions for the higher orders of perturbation theory, _Theor. Math. Phys._ 57 (1983), no. 3 pp. 1196–1202.
  * (103) J. C. Le Guillou and J. Zinn-Justin, Critical exponents for the $n$-vector model in three dimensions from field theory, _Phys. Rev. Lett._ 39 (July, 1977) pp. 95–98.
  * (104) J. C. Le Guillou and J. Zinn-Justin, Critical exponents from field theory, _Phys. Rev. B_ 21 (May, 1980) pp. 3976–3998.
  * (105) J. C. Le Guillou and J. Zinn-Justin, Accurate critical exponents from the $\varepsilon$-expansion, _J. Physique Lett._ 46 (1985), no. 4 pp. 137–141.
  * (106) R. N. Lee, A. V. Smirnov and V. A. Smirnov, Master integrals for four-loop massless propagators up to weight twelve, _Nucl. Phys. B_ 856 (Mar., 2012) pp. 95–110, arXiv:1108.0732 [hep-th].
  * (107) J. A. Lipa, J. A. Nissen, D. A. Stricker, D. R. Swanson and T. C. P. Chui, Specific heat of liquid helium in zero gravity very near the lambda point, _Phys. Rev. B_ 68 (Nov., 2003) p. 174518.
  * (108) L. N. Lipatov, Divergence of the perturbation-theory series and the quasi-classical theory, _Sov. Phys. JETP_ 45 (Feb., 1977) pp. 216–223. translation of Zh. Eksp. Teor. Fiz. 72, 411–427 (1977).
  * (109) D. A. Lobaskin and I. M. Suslov, Higher order corrections to the Lipatov asymptotics in the $\phi^{4}$ theory, _Journal of Experimental and Theoretical Physics_ 99 (Aug., 2004) pp. 234–253, arXiv:hep-ph/0509274. Russian version: Zh. Eksp. Teor. Fiz. 268, no.2, 268(2004).
  * (110) T. Luthe, A. Maier, P. Marquard and Y. Schröder, Complete renormalization of QCD at five loops, _JHEP_ 2017 (Mar., 2017) p. 020, arXiv:1701.07068 [hep-ph].
  * (111) T. Luthe, A. Maier, P. Marquard and Y. Schröder, Five-loop quark mass and field anomalous dimensions for a general gauge group, _JHEP_ 2017 (Jan., 2017) p. 81, arXiv:1612.05512 [hep-ph].
  * (112) T. Luthe and Y. Schröder, Five-loop massive tadpoles, in _Proceedings, 13th DESY Workshop on Elementary Particle Physics: Loops and Legs in Quantum Field Theory (LL2016): Leipzig, Germany, April 24–29, 2016_ , vol. LL2016, p. 074, 2016.  arXiv:1609.06786 [hep-ph], PoS(LL2016)074.
  * (113) A. Lytle and D. T. Jacobs, Turbidity determination of the critical exponent $\eta$ in the liquid-liquid mixture methanol and cyclohexane, _Journal of Chemical Physics_ 120 (Mar., 2004) pp. 5709–5716.
  * (114) J. Magnen and R. Sénéor, Phase space cell expansion and Borel summability for the Euclidean $\varphi^{4}_{3}$ theory, _Comm. Math. Phys._ 56 (1977), no. 3 pp. 237–276.
  * (115) J. Manuel Carmona, A. Pelissetto and E. Vicari, $N$-component Ginzburg-Landau Hamiltonian with cubic anisotropy: A six-loop study, _Phys. Rev. B_ 61 (June, 2000) pp. 15136–15151, arXiv:cond-mat/9912115.
  * (116) Maplesoft, a division of Waterloo Maple Inc., “Maple 16.”
  * (117) A. J. McKane and D. J. Wallace, Instanton calculations using dimensional regularisation, _Journal of Physics A Mathematical General_ 11 (Nov., 1978) pp. 2285–2304.
  * (118) A. J. McKane, D. J. Wallace and O. F. de Alcantara Bonfim, Non-perturbative renormalisation using dimensional regularisation: applications to the $\epsilon$ expansion, _Journal of Physics A: Mathematical and General_ 17 (1984), no. 9 pp. 1861–1876.
  * (119) B. D. McKay and A. Piperno, Practical graph isomorphism, II, _J. Symb. Comput._ 60 (Jan., 2014) pp. 94–112, arXiv:1301.1493 [cs.DM]. Program website: http://pallini.di.uniroma1.it/.
  * (120) H. Mera, T. G. Pedersen and B. K. Nikolić, Nonperturbative quantum physics from low-order perturbation theory, _Phys. Rev. Lett._ 115 (Sept., 2015) p. 143001, arXiv:1405.7956 [cond-mat.stat-mech].
  * (121) A. I. Mudrov and K. B. Varnashev, Modified Borel summation of divergent series and critical-exponent estimates for an $N$-vector cubic model in three dimensions from five-loop $\epsilon$ expansions, _Phys. Rev. E_ 58 (Nov., 1998) pp. 5371–5375, arXiv:cond-mat/9805081.
  * (122) B. G. Nickel, D. I. Meiron and G. A. Baker, “Compilation of 2-pt. and 4-pt. graphs for continuous spin.” unpublished University of Guelph report, available on http://users.physik.fu-berlin.de/~kleinert/nickel/, 1977.
  * (123) B. Nienhuis, Exact critical point and critical exponents of $O(n)$ models in two dimensions, _Phys. Rev. Lett._ 49 (Oct, 1982) pp. 1062–1065.
  * (124) M. A. Nikitina and A. I. Sokolov, Critical exponents in two dimensions and pseudo-$\epsilon$ expansion, _Phys. Rev. E_ 89 (Apr., 2014) p. 042146, arXiv:1312.1062 [cond-mat.stat-mech].
  * (125) L. Onsager, Crystal statistics. I. A two-dimensional model with an order-disorder transition, _Phys. Rev._ 65 (Feb., 1944) pp. 117–149.
  * (126) E. V. Orlov and A. I. Sokolov, Critical thermodynamics of two-dimensional systems in the five-loop renormalization-group approximation, _Physics of the Solid State_ 42 (2000), no. 11 pp. 2151–2158, arXiv:hep-th/0003140.
  * (127) E. Panzer, “The Hepp bound for Feynman integrals.” in preparation.
  * (128) E. Panzer, On the analytic computation of massless propagators in dimensional regularization, _Nucl. Phys. B_ 874 (Sept., 2013) pp. 567–593, arXiv:1305.2161 [hep-th].
  * (129) E. Panzer, _Feynman integrals and hyperlogarithms_.  PhD thesis, Humboldt-Universität zu Berlin, 2014.  arXiv:1506.07243 [math-ph].
  * (130) E. Panzer, Algorithms for the symbolic integration of hyperlogarithms with applications to Feynman integrals, _Computer Physics Communications_ 188 (Mar., 2015) pp. 148–166, arXiv:1403.3385 [hep-th].
  * (131) E. Panzer and O. Schnetz, “The Galois coaction on $\phi^{4}$ periods.” to appear in Communications in Number Theory and Physics, arXiv:1603.04289 [hep-th].
  * (132) A. Pelissetto and E. Vicari, Critical phenomena and renormalization-group theory, _Physics Reports_ 368 (Oct., 2002) pp. 549–727, arXiv:cond-mat/0012164.
  * (133) R. D. Pisarski and F. Wilczek, Remarks on the chiral phase transition in chromodynamics, _Phys. Rev. D_ 29 (Jan., 1984) pp. 338–341.
  * (134) P. V. Pobylitsa, “Superfast convergence effect in large orders of the perturbative and $\epsilon$ expansions for the $O(N)$ symmetric $\phi^{4}$ model.” preprint, July, 2008, arXiv:0807.5136 [hep-th].
  * (135) B. Ruijl, T. Ueda and J. A. M. Vermaseren, Forcer: a FORM program for 4-loop massless progagators, in _13th DESY Workshop on Elementary Particle Physics: Loops and Legs in Quantum Field Theory (LL2016) Leipzig, Germany, April 24–29, 2016_ , Proceedings of Science, May, 2016.  arXiv:1607.07318 [hep-ph], PoS(LL2016)070.
  * (136) B. Ruijl, T. Ueda, J. A. M. Vermaseren and A. Vogt, Four-loop QCD propagators and vertices with one vanishing external momentum, _Journal of High Energy Physics_ 6 (June, 2017) p. 40, arXiv:1703.08532 [hep-ph].
  * (137) O. Schnetz, “Calculation of the $\varphi^{4}$ 6-loop non-zeta transcendental.” preprint, Dec., 1999, arXiv:hep-th/9912149.
  * (138) O. Schnetz, Quantum periods: A Census of $\phi^{4}$-transcendentals, _Commun. Number Theory Phys._ 4 (2010), no. 1 pp. 1–47, arXiv:0801.2856 [hep-th].
  * (139) O. Schnetz, Graphical functions and single-valued multiple polylogarithms, _Communications in Number Theory and Physics_ 8 (2014), no. 4 pp. 589–675, arXiv:1302.6445 [math.NT].
  * (140) O. Schnetz, “Numbers and functions in quantum field theory.” preprint, June, 2016, arXiv:1606.08598 [hep-th].
  * (141) J. V. Sengers and J. G. Shanks, Experimental critical-exponent values for fluids, _Journal of Statistical Physics_ 137 (2009), no. 5 pp. 857–877.
  * (142) R. Seznec and J. Zinn-Justin, Summation of divergent series by order dependent mappings: Application to the anharmonic oscillator and critical exponents in field theory, _J. Math. Phys._ 20 (1979), no. 7 pp. 1398–1408.
  * (143) R. Shrock, Question of an ultraviolet zero of the beta function of the $\lambda(\vec{\phi}^{2})^{2}_{4}$ theory, _Phys. Rev. D_ 90 (Sept., 2014) p. 065023, arXiv:1408.3141 [hep-th].
  * (144) R. Shrock, Study of the six-loop beta function of the $\lambda\varphi_{4}^{4}$ theory, _Phys. Rev. D_ 94 (Dec., 2016) p. 125026, arXiv:1610.03733 [hep-th].
  * (145) A. V. Smirnov and M. Tentyukov, Four-loop massless propagators: A numerical evaluation of all master integrals, _Nucl. Phys. B_ 837 (Sept., 2010) pp. 40–49, arXiv:1004.1149 [hep-ph].
  * (146) A. I. Sokolov and M. A. Nikitina, Fisher exponent from pseudo-$\epsilon$ expansion, _Phys. Rev. E_ 90 (July, 2014) p. 012102, arXiv:1402.3894 [cond-mat.stat-mech].
  * (147) P. M. Stevenson, Resolution of the renormalisation-scheme ambiguity in perturbative QCD, _Physics Letters B_ 100 (Mar., 1981) pp. 61–64.
  * (148) I. M. Suslov, Structure of higher order corrections to the Lipatov asymptotic form, _J. Exp. Theor. Phys._ 90 (2000), no. 4 pp. 571–578, arXiv:hep-ph/0006074.
  * (149) G. ’t Hooft, Dimensional regularization and the renormalization group, _Nuclear Physics B_ 61 (1973) pp. 455–468.
  * (150) G. ’t Hooft and M. Veltman, Regularization and renormalization of gauge fields, _Nuclear Physics B_ 44 (July, 1972) pp. 189–213.
  * (151) A. N. Vasil’ev, _Quantum field renormalization group in critical behavior theory and stochastic dynamics_.  Chapman & Hall/CRC, Apr., 2004.  originally published in Russian in 1998 by St. Petersburg Institute of Nuclear Physics Press; translated by Patricia A. de Forcrand-Millard.
  * (152) A. N. Vasil’ev, Y. M. Pis’mak and Y. R. Honkonen, $1/n$ expansion: Calculation of the exponents $\eta$ and $\nu$ in the order $1/n^{2}$ for arbitrary number of dimensions, _Theoretical and Mathematical Physics_ 47 (1981), no. 3 pp. 465–475.
  * (153) A. N. Vasil’ev, Y. M. Pis’mak and Y. R. Honkonen, Simple method of calculating the critical indices in the $1/n$ expansion, _Theoretical and Mathematical Physics_ 46 (1981), no. 2 pp. 104–113.
  * (154) A. N. Vasil’ev, Y. M. Pis’mak and Y. R. Honkonen, $1/n$ expansion: Calculation of the exponent $\nu$ in the order $1/n^{3}$ by the conformal bootstrap method, _Theoretical and Mathematical Physics_ 50 (1982), no. 2 pp. 127–134.
  * (155) J. A. M. Vermaseren, Axodraw, _Computer Physics Communications_ 83 (Mar., 1994) pp. 45–58.
  * (156) J. A. M. Vermaseren, “New features of FORM.” Website: https://www.nikhef.nl/~form/, Oct., 2000, arXiv:math-ph/0010025.
  * (157) A. A. Vladimirov, Methods of calculating many-loop diagrams and renormalization-group analysis of the $\varphi^{4}$ theory, _Theoretical and Mathematical Physics_ 36 (Aug., 1978) pp. 732–737.
  * (158) A. A. Vladimirov, D. I. Kazakov and O. V. Tarasov, Calculation of critical exponents by quantum field theory methods, _Sov. Phys. JETP_ 50 (Sept., 1979) pp. 521–526. Russian original: Zh. Eksp. Teor. Fiz. 77, 1035–1045 (September 1979).
  * (159) K. G. Wilson and M. E. Fisher, Critical exponents in 3.99 dimensions, _Phys. Rev. Lett._ 28 (Jan., 1972) pp. 240–243.
  * (160) Wolfram Research, Inc., Mathematica 11.0, 2016\.
  * (161) V. I. Yukalov and E. P. Yukalova, Calculation of critical exponents by self-similar factor approximants, _The European Physical Journal B_ 55 (Feb., 2007) pp. 93–99, arXiv:0704.2125 [cond-mat.stat-mech].
  * (162) J. Zinn-Justin, _Quantum Field Theory and Critical Phenomena_ , vol. 113 of _International Series of Monographs on Physics_.  Clarendon Press, Oxford, 4 ed., 2002.
  * (163) J. Zinn-Justin, Summation of divergent series: Order-dependent mapping, _Applied Numerical Mathematics_ 60 (2010), no. 12 pp. 1454–1464, arXiv:1001.0675 [math-ph].