# Schema-Guided Paradigm for Zero-Shot Dialog

Shikib Mehri  Maxine Eskenazi
Language Technologies Institute, Carnegie Mellon University
<EMAIL_ADDRESS>
###### Abstract

Developing mechanisms that flexibly adapt dialog systems to unseen tasks and
domains is a major challenge in dialog research. Neural models implicitly
memorize task-specific dialog policies from the training data. We posit that
this implicit memorization has precluded zero-shot transfer learning. To this
end, we leverage the schema-guided paradigm, wherein the task-specific dialog
policy is explicitly provided to the model. We introduce the Schema Attention
Model (sam) and improved schema representations for the STAR corpus. sam
obtains significant improvement in zero-shot settings, with a +22
$\mathbf{F_{1}}$ score improvement over prior work. These results validate the
feasibility of zero-shot generalizability in dialog. Ablation experiments are
also presented to demonstrate the efficacy of sam.

##  1 Introduction

Task-oriented dialog systems aim to satisfy user goals pertaining to certain
tasks, such as booking flights (Hemphill et al., 1990), providing transit
information (Raux et al., 2005), or acting as a tour guide (Budzianowski et
al., 2018). Neural models for task-oriented dialog have become the dominant
paradigm (Williams and Zweig, 2016; Wen et al., 2016; Zhao et al., 2017).
These data-driven approaches can potentially learn complex patterns from large
dialog corpora without hand-crafted rules. However, the resulting models
struggle to generalize beyond the training data and under-perform on unseen
dialog tasks and domains (Zhao and Eskenazi, 2018; Rastogi et al., 2020b).

A long-standing challenge in dialog research is to flexibly adapt systems to
new dialog domains and tasks (Zhao and Eskenazi, 2018; Mosig et al., 2020).
Consider a system that has been trained to handle several different tasks
(e.g., restaurant reservations, ride booking, weather, etc.). How can this
dialog system be extended to handle a new task (e.g., hotel booking), without
collecting additional data? This paper tackles this challenge and aims to
address the problem of zero-shot generalization using the schema-guided
paradigm.

Figure 1: In the standard paradigm, data driven models implicitly learn the
task-specific dialog policies (i.e., schemas). This precludes generalization
to an unseen task at inference time. In contrast, in the schema-guided
paradigm, dialog policy is explicitly provided to the model through a schema
graph. At inference time, the model is given the schema for the new task and
can therefore generalize in a zero-shot setting.

The advent of large-scale pre-training (Devlin et al., 2019; Radford et al.,
2019) has led to significant progress in domain adaptation across areas in
NLP, including natural language understanding (Wang et al., 2018, 2019), open-
domain dialog (Zhang et al., 2020; Adiwardana et al., 2020) and language
understanding for task-oriented dialog (Wu et al., 2019; Mehri et al., 2020).
Generalization in end-to-end task-oriented dialog has proven to be
significantly more difficult, particularly in zero-shot settings where there
is no training data (Mosig et al., 2020). We posit that it is inherently
difficult to generalize to unseen dialog tasks because of the dialog policy.

Traditionally, an end-to-end dialog system must perform three distinct tasks.
First, it must understand the dialog history and identify any relevant user
intents or slots. Next, it must decide on the appropriate system action,
according to a task-specific dialog policy. Finally, it must generate a
natural language utterance corresponding to the system action. In a pipeline
dialog system, these three steps are performed by the NLU, DM and NLG
respectively (Jurafsky, 2000). Prior work has exhibited generalizability in
language understanding and, to a lesser extent, in language generation.
However for end-to-end dialog, the task-specific dialog policy inherently
precludes zero-shot generalization. An end-to-end dialog model trained on
several tasks, will implicitly learn the dialog policies from the data.
However, when generalizing to a new task in a zero-shot setting, the model has
no knowledge of the dialog policy for the new task.

To address the difficulty of generalizing to new task-specific dialog policies
and in order to facilitate zero-shot generalization, we present the schema-
guided paradigm. Generally, end-to-end neural models implicitly learn the
task-specific dialog policies from large corpora. In contrast, in the schema-
guided paradigm, we explicitly provide the task-specific dialog policies to
the model in the form of a schema graph. The schema graphs define the system’s
behavior for a specific task (e.g., when the user provides the reservation
time, ask them for the number of people). When transferring to an unseen task,
the corresponding schema graph is explicitly provided to the model. This
enables language understanding and the dialog policy to be decoupled. The
model no longer needs to implicitly memorize the task-specific policies from
the training data. Instead, the model learns to interpret the dialog history
and align it to the schema graph. As such, when transferring to a new task,
the schema graph serves as an inductive bias that provides the model with the
task-specific dialog policy.

To address the challenge of zero-shot transfer learning, Mosig et al. (2020)
presented the STAR corpus and several baseline experiments. We extend their
baselines for the task of next action prediction. We introduce the Schema
Attention Model (sam) and thorough schema representations for the 24 different
tasks in the STAR dataset. sam obtains a +22 $\mathbf{F_{1}}$ score
improvement over baseline approaches in the zero-shot setting, validating the
schema-guided paradigm and demonstrating the feasibility of zero-shot
generalization for task-oriented dialog. Our code and model checkpoints are
open-sourced and be found at https://github.com/shikib/schema_attention_model.

##  2 Related Work

###  2.1 Zero-Shot Dialog

Zero-shot transfer learning has been of interest to the dialog research
community. Many approaches have been proposed for zero-shot adaptation of
specific dialog components. Chen et al. (2016) present a zero-shot approach
for learning embeddings for unseen intents. Bapna et al. (2017) show that slot
names and descriptions can be leveraged to implicitly align slots across
domains and achieve better cross-domain generalization. Wu et al. (2019)
similarly use slot names, in combination with a generative model for state
tracking, to obtain strong zero-shot results. Shah et al. (2019) leverage
examples for zero-shot slot filling. Generally, approaches for zero-shot
generalizability leverage the similarity across domains (e.g., restaurant-area
and hotel-area are conceptually similar). The advent of large-scale pre-
training (Devlin et al., 2019; Radford et al., 2019) allows for language
understanding across dissimilar domains. Rastogi et al. (2020a) address zero-
shot domain adaptation in state tracking by leveraging BERT (Devlin et al.,
2019) with a domain-specific API specification.

Zhao and Eskenazi (2018) present an approach for zero-shot end-to-end dialog.
They leverage the Action Matching framework to learn a cross-domain latent
action space. Qian and Yu (2019) use model-agnostic meta learning to attain
stronger results in zero-shot dialog. Both these approaches rely on additional
annotations, which make them unsuitable for the STAR corpus. While there is a
significant amount of work in zero-shot generalizability for language
understanding, there is considerably less research in adaptation for end-to-
end dialog111While we focus on next action prediction, in the STAR dataset it
is trivial to go from a system action to a natural language response and as
such we consider our task to be end-to-end dialog.. This is in part because of
the difficulty of generalizing to unseen task-specific policies. To this end,
Mosig et al. (2020) presented STAR, a corpus consisting of 24 different dialog
tasks, and several baseline models for zero-shot adaptation on STAR. The
results in this paper significantly outperform the baselines introduced by
Mosig et al. (2020) as we leverage the schema-guided paradigm for zero-shot
generalizability in dialog.

###  2.2 Schema-Guided Paradigm

Plan-based dialog systems (Ferguson and Allen, 1998; Rich and Sidner, 1998;
Bohus and Rudnicky, 2009) reason about user intent, in the context of a dialog
plan. RavenClaw (Bohus and Rudnicky, 2009) consists of a task specification
that defines the behavior of a system depending on various user actions. Plan-
based dialog systems decouple the task-specific dialog policy from the task-
agnostic components of the system. This allows a system to be extended to a
new task by updating the task specification. The schema-guided paradigm shares
a similar motivation, and aims to disentangle the dialog policy in neural,
data-driven dialog systems.

Several approaches have been presented to discover dialog structure graphs
(similar to the schemas in this paper) from data in an unsupervised manner
(Shi et al., 2019; Qiu et al., 2020; Xu et al., 2020; Hu et al., 2019). These
approaches have been used to enhance generation for open-domain dialog (Qiu et
al., 2020; Hu et al., 2019). To the best of our knowledge, these dialog
structures have neither been used for generation in task-oriented dialog nor
in zero-shot settings. While our schemas are similar to these structure
graphs, they are hand-crafted similar to those in plan-based dialog systems.
Future work may extend our work by leveraging unsupervised structure graph
discovery as an alternative to hand-crafted schemas.

##  3 Task Definition

We address the problem of transferring dialog models to unseen tasks and
domains (Zhao and Eskenazi, 2018). This problem is especially important in
real world settings. It is impossible to preconceive every dialog task that
users may need (e.g., a COVID-19 information dialog system). Furthermore,
collecting new dialog data for each new task is inherently unscalable (Rastogi
et al., 2020b). While rule-based/pipeline dialog systems may be easier to
extend to new tasks (Bohus and Rudnicky, 2009), there is a tradeoff between
the adaptability of non-neural systems and the performance of neural models.

###  3.1 STAR Dataset

The STAR dataset (Mosig et al., 2020) was collected for the purpose of
studying transfer learning in dialog. The dataset spans 24 different tasks in
13 different domains (e.g., the restaurant domain has ‘restaurant-search’ and
‘restaurant-reservations’). The data collection procedure was designed to
reduce ambiguity in the system responses and make system actions
deterministic. As such, Amazon Mechanical Turk (AMT) workers were given a flow
chart diagram for each task. This flow chart defined the task, including the
order in which questions should be asked (e.g., ask date before city), how to
respond to various user responses and how to query a database. Additionally,
in order to minimize variance in the responses from the wizard, Mosig et al.
(2020) incorporate a suggestions module during data collection. This module
maps the wizard utterance to the closest pre-written response (e.g., ‘Give me
your name’ $\rightarrow$ ‘What is your name?’). In some cases, it is not
possible for the AMT worker to use the suggestions module. Nonetheless, the
module increases the consistency of the system actions.

Mosig et al. (2020) present baseline results on the tasks of next action
prediction and response generation. The present paper focuses on next action
prediction. The objective of next action prediction is to predict the correct
system action conditioned on the dialog history. Since there is a one-to-one
mapping between system actions and corresponding natural language responses,
the primary challenge in extending a next action prediction model to response
generation resides in learning to accurately fill in the response templates
(e.g., ‘Your reservation is confirmed for {date}’).

The STAR dataset consists of three different types of dialogs: (1) happy
single-task dialogs, (2) unhappy single-task dialogs and (3) multi-task
dialogs. Here, happy refers to dialogs where the users are cooperative and
complete the task. In contrast, unhappy dialogs consist of uncooperative users
that may change the subject, engage in irrelevant chit-chat and otherwise aim
to push the system beyond its capabilities. Since our primary objective is to
address zero-shot transfer, we only consider the happy single-task dialogs.
There are 1537 happy single-task dialogs and 10,364 turns.

###  3.2 Zero-Shot Setting

In the STAR dataset, there are 23 dialog tasks (13 domains) with happy single-
task dialogs. We perform two types of transfer learning experiments: task
transfer and domain transfer. In task transfer, a model is trained on $n-1$
tasks (i.e., 22) and evaluated on the last one. This is repeated for each of
the 23 tasks. For domain transfer, a model is trained on $n-1$ domains (i.e.,
12) and evaluated on the last one. In task transfer, there may be some overlap
between the training and testing, for example, the domain-specific
terminology. In contrast, in domain transfer there is very limited overlap.
When the model is tasked with generalizing to the restaurant domain, it has
seen nothing related to restaurants during training.

In both of these settings, the model is aware of which task it is being
evaluated on, meaning that it can leverage a task specification (e.g., schema)
for the new task. This experimental design resembles a real-world setting
where a system developer would be aware of the new task. For example, if a
developer wanted to extend a dialog system to handle a COVID-19 related
question, they would be able to create a new task specification. As such, our
goal is to develop a model that can generalize to an unseen task conditioned
on a task specification.

##  4 Methods

In order to enable zero-shot transfer to new dialog tasks and domains, the
Schema Attention Model (sam) is introduced. It leverages an external dialog
policy representation (i.e., the schema) to predict the next system action.
This section begins by describing the baseline model for the task of next
action prediction. Next, the schema-guided paradigm is introduced (Figure 1).
It includes a graph-based representation of the task-specific schema and sam,
a model that identifies the next system action by attending to a task-specific
schema representation.

###  4.1 Baseline

This section describes the baseline model proposed by Mosig et al. (2020).
Given an arbitrary language encoder, denoted as $\mathcal{F}$, the baseline
model obtains a vector representation of the dialog history, $c$. This
representation is then passed through a softmax layer to obtain a probability
distribution over the actions.

$\displaystyle{\bm{h}}=\mathcal{F}(c)$ (1) $\displaystyle
P_{\text{clf}}=\text{softmax}({\bm{W}}{\bm{h}}^{T}+{\bm{b}})$ (2)

Throughout this paper, BERT-base (Devlin et al., 2019) is used as the language
encoder.

###  4.2 Schema-Guided Paradigm

Our baseline model simultaneously needs to (1) interpret the dialog context
and identify the relevant intents and slots, and (2) learn the task-specific
dialog policies (i.e., if the user wants the weather, ask the city) for the
different tasks in the training data. This model is incapable of generalizing
to a new task in a zero-shot setting, as it would lack knowledge of the task-
specific policy for the new task. To mitigate this problem and to enable zero-
shot task transfer, we present the schema-guided paradigm which decouples the
task-specific dialog policy from the language understanding.

An example is shown in Figure 1: the schema-guided paradigm decouples the the
dialog policy from language understanding by explicitly providing task-
specific schema graphs as input to the model. These schema graphs serve as
complete representations of the dialog policy for a given task. Therefore,
while the baseline needs to implicitly learn the dialog policies, a schema-
guided model instead learns to leverage the explicit schema graphs. As such, a
schema-guided model can generalize to a new task as long as it is provided
with the corresponding schema graph.

In this paradigm, the role of the model is to interpret a dialog context and
align it to the explicit schema graph. The role of the schema graph is to
determine the next action according to the dialog policy. In this manner, the
language encoder is being trained for the task of sentence similarity. With
the help of pre-trained models, language understanding in a schema-guided
paradigm can be considered to be task-agnostic. By decoupling the task-
agnostic language understanding and the task-specific dialog policy, the
schema-guided paradigm better facilitates zero-shot transfer learning.

The schema-guided paradigm consists of the representation of the schema graph,
and a neural model which interprets the dialog context and aligns it to the
schema graph.

####  4.2.1 Schema Representation

In the schema-guided paradigm, the schema representation is the task-specific
dialog policy. To ensure the efficacy and robustness of the dialog system, it
is important that the schema representation be complete and informative. In
the case of ambiguity or incompleteness in the schema representation, the next
action will fail to be correctly predicted, regardless of the strength of the
model. The schema representations are manually constructed for every task. In
the schema-guided paradigm, to transfer to a new task, a system developer
would simply need to construct a new schema representation.

Mosig et al. (2020) propose a baseline schema representation wherein the nodes
of the graph correspond to system actions and database states. There are nodes
for user states only in situations where the system behavior differs depending
on the user’s actions (e.g., ‘Yes’ $\rightarrow$ ask-time, ‘No’ $\rightarrow$
ask-date). The consequence of this representation is that when the model
aligns the dialog history to the schema, it largely relies on the system
utterances. However, this representation fails to account for realistic user
behavior and therefore yields only marginal improvement over the baseline.

Specifically, users will often provide information out of turn (e.g., System:
‘Where would you like to go?’ $\rightarrow$ User: ‘Leaving from the airport
and going downtown’). In this example, it is difficult for the model to
realize that the question System: ‘Where are you leaving from?’ has also been
answered and therefore should not be the next system action. Users can also
ignore the system utterance (e.g., System: ‘Where would you like to go?’
$\rightarrow$ User: ‘Actually, what’s the weather?’). It is thus ineffective
to represent dialog policy only in terms of the system utterances. To this
end, we extend the schema representation by incorporating user utterances into
the schema graph.

As shown in Figure 2, our schema graph incorporates nodes corresponding to
user utterances. As such, if a user provides information out of turn or
changes the subject, our model will be able to effectively align the dialog to
the schema. To account for variance in the user utterances, future work could
extend this schema representation to include multiple variations of a given
user utterance. However, as the schema graphs are manually constructed for
every task, there is a trade-off between manual effort and
efficacy222Constructing the schema graphs is not particularly labor-intensive.
It took the first author between 15 and 45 minutes to create each schema
graph, depending on the complexity of the task..

Figure 2: A section of the task-specific schema graph for the bank-balance
task. The system must authenticate the user with their account number and PIN.
However, if the user has forgotten either of these, it must ask backup
security questions. The blue nodes correspond to system actions and the yellow
nodes denote user utterances.

The schema graph has several noteworthy properties. First, the system actions
are consistently deterministic. Nodes corresponding to a database response or
to a user utterance will always have a single outgoing edge to a system
response node. Furthermore, such nodes will also have a single incoming edge
from a system response node. For a given user/database node, $u$, we denote
the previous system response node as $\textbf{prev}(u)$ and the following
system response as $\textbf{next}(u)$. Each node has some text associated with
it, denoted as $\textbf{text}(u)$. This text is a template for either a system
utterance, database response or user utterance. System nodes will also have an
associated system action, $\textbf{act}(u)$. There is a one-to-one mapping
between the system actions and the system response templates.

####  4.2.2 Schema Attention Model

In the schema-guided paradigm, the role of the model is to understand the
dialog history and align it to the schema representation. We introduce the
Schema Attention Model, sam, which attends between the dialog history,
$c={c_{1},\dots,c_{N}}$ and the schema graph. sam extends the schema-guided
model presented by Mosig et al. (2020) by (1) leveraging a stronger attention
mechanism, (2) improving the training algorithm, and (3) removing the linear
classification layer which is detrimental to zero-shot performance.

The objective of sam is to predict the node in the schema graph that best
corresponds to the dialog context. sam will produce a probability distribution
over the nodes corresponding to user utterances and database responses. Given
an attention distribution over the nodes, we can obtain a probability
distribution over the set of actions by propagating the attention
probabilities over the graph. Concretely, if node $u$ has an attention weight
of $p$, we add $p$ to the probability of $\textbf{action}(\textbf{next}(u))$.

We consider every node $u$ that corresponds to either a database response or a
user utterance. We then represent each node $u$ as the concatenation of the
previous node and the current node, i.e.,
$\textbf{text}(\textbf{prev}(u))+\textbf{text}(u)$. For all nodes $u\in U$, we
obtain this textual representation denoted as $s\in S$.

We are given a language encoder, $\mathcal{F}$, the dialog context,
$c={c_{1},\dots,c_{N}}$, the nodes $U$, their corresponding textual
representations $S$, and the set of possible actions $A$. Note that unlike in
Equation 1, $\mathcal{F}$ is used to produce a vector representation of each
word in the input. sam produces a probability distribution over the actions as
follows:

$\displaystyle{\bm{h}}_{1,\dots,N}=\mathcal{F}(c\colon c_{1},\dots,c_{N})$ (3)
$\displaystyle{\bm{S}}_{i;1,\dots,M}=\mathcal{F}(S_{i}\colon
s_{1},\dots,s_{M})$ (4)
$\displaystyle{\bm{w}}^{i}_{j,k}={\bm{h}}_{j}^{T}{\bm{S}}_{i;k}$ (5)
$\displaystyle\mathbf{\alpha}=\text{softmax}({\bm{w}}^{1,\dots,|S|})$ (6)
$\displaystyle p_{i}=\sum_{j\leq N}\sum_{k\leq M}\alpha^{i}_{j,k}$ (7)

Here, ${\bm{w}}^{i}$ is an $N\times M$ dimensional matrix corresponding to the
dot product between the $N$ words of the dialog history and the $M$ words of
the $i$-th textual representation in $S$. To get the attention weights over
all of the words of the schema, we perform a softmax over all
${\bm{w}}^{i},1\leq i\leq|S|$. By summing over the attention weights in
$\alpha^{i}$, we get $p_{i}$, a scalar value which denotes the attention
between the dialog history and the $i$-th node (i.e., the corresponding
textual representation $S_{i}$). Given $p_{i}$ we produce a probability
distribution over the actions $A$ as follows:

$\displaystyle g(i,a)=\begin{cases}p_{i},&\text{if
}\textbf{action}(\textbf{next}(u_{i}))=a\\\ 0,&\text{otherwise}\end{cases}$
(8) $\displaystyle P(a)=\sum_{i\leq|S|}g(i,a)$ (9)

To align the dialog history to the schema graph, sam performs word-level
attention using a BERT-base model. In contrast, the schema-guided model of
Mosig et al. (2020) attends with the sentence level vector representation
produced by BERT. With the word-level attention, sam can better align
ambiguous dialog contexts, such as situations where the user provides multiple
pieces of information in a single utterance. Since this word-level attention
operates on the sub-word tokens used in BERT, it can also potentially handle
spelling errors in the user utterances.

Furthermore, in their schema-guided model, Mosig et al. (2020) combine the
probability distribution produced by attending to the schema graph with their
baseline model (i.e., Section 3.1). While this may result in better
performance on the tasks the model is trained with, the baseline model will
not generalize to unseen tasks. In contrast, sam computes the probability for
an action using only the attention over the schema graph.

Mosig et al. (2020) train their schema-guided model to predict the appropriate
node, $u_{i}$, from a set of nodes $U^{\prime}$ (s.t., $U^{\prime}\subset U$).
At training time, for efficiency reasons, the set of nodes $U^{\prime}$ is
obtained by using the corresponding node for every dialog context in the
training batch. Since the training batches are randomly sampled, this results
in $U^{\prime}$ including nodes from a variety of different schema graphs. At
inference time, the dialog task is known and therefore only the corresponding
schema graph needs to be attended to (i.e., $U^{\prime}$ will contain nodes
from a single schema graph). It is valuable to train the model to distinguish
between different nodes of the same schema graph. Specifically, the attention
mechanism (i.e., Equations 5 - 6) will learn stronger fine-grained
relationships when trained with negative samples from the same domain. As
such, we augment the training algorithm to sample batches from the same dialog
task, meaning that $U^{\prime}$ will only include nodes from a single schema.

sam improves on the baseline schema-guided model introduced by Mosig et al.
(2020) by (1) leveraging a stronger attention mechanism that better handles
realistic user behavior, (2) computing a probability distribution only by
attending to the schema graph and (3) modifying the training algorithm to have
in-domain negative samples which result in the model learning to identify
fine-grained relationships. In combination with the improved schema
representation, sam is better suited to handle realistic user behavior in
zero-shot settings.

Model |  $F_{1}$ score | Accuracy
---|---|---
Baseline $\diamond$ | 73.79 | 74.85
bert+s $\diamond$ | 71.59 | 72.27
sam $-$ [1] | 54.35 | 60.51
sam $-$ [2,3,4] | 70.22 | 71.01
sam $-$ [2] | 70.27 | 71.93
sam $-$ [3] | 70.18 | 71.64
sam $-$ [4] | 69.68 | 69.79
sam | 70.38 | 71.45
Table 1: Performance in the standard experimental setting. Models marked with $\diamond$ are attributed to Mosig et al. (2020). We denote their schema-guided model, ‘BERT + Schema’, as bert+s. sam consists of four improvements upon bert+s: (1) user-aware schema, (2) word-level attention, (3) using negative samples from the same task at training, (4) removing the linear classification layer. Results in boldface are statistically significant by t-test ($p<0.01$) Model | Task Transfer | Domain Transfer
---|---|---
|  $F_{1}$ score | Accuracy |  $F_{1}$ score | Accuracy
Baseline $\diamondsuit$ | 31.23 | 30.65 | 31.82 | 33.92
bert+s $\diamondsuit$ | 28.12 | 28.28 | 29.70 | 32.43
sam $-$ [1] | 33.81 | 37.84 | 41.77 | 45.64
sam $-$ [2,3,4] | 43.28 | 46.11 | 43.78 | 45.19
sam $-$ [2] | 50.72 | 53.69 | 52.20 | 54.68
sam $-$ [3] | 45.54 | 49.29 | 50.56 | 52.13
sam $-$ [4] | 47.26 | 47.99 | 47.67 | 48.92
sam | 53.31 | 55.51 | 55.74 | 57.75
Table 2: Performance in zero-shot transfer. We present results on both task
transfer and domain transfer. Models marked with $\diamondsuit$ are attributed
to Mosig et al. (2020). sam consists of four improvements upon bert+s: (1)
user-aware schema, (2) word-level attention, (3) using negative samples from
the same task at training, (4) removing the linear classification layer.
Results in bold-face are statistically significant by t-test ($p<0.01$).

##  5 Results

To validate the effectiveness of sam, a number of next action prediction
experiments are carried out on the STAR dataset (Mosig et al., 2020). First,
sam is evaluated in the standard experimental setting, i.e., training and
testing on the same tasks. Next, we carry out zero-shot transfer experiments,
as defined in Section 3. The evaluation uses accuracy and weighted $F_{1}$
score.

We rerun the experiments presented by Mosig et al. (2020) using code shared by
the authors. In our results, the model introduced by Mosig et al. (2020) is
denoted as bert+s. Their original results were obtained on an older version of
STAR, with annotation errors333Specifically, certain dialogs were
misattributed as being happy single-task dialogs. that have since been fixed.

###  5.1 Standard Experiments

In the standard experimental setting, models are trained and tested on the
same tasks. Following Mosig et al. (2020), 80% of the dialogs are used for
training and 20% for testing. All models are trained for 50 epochs.

The results shown in Table 1 show sam to be comparable to the baseline model
on the standard setting. Since the augmentations to sam are primarily intended
to improve zero-shot performance, it is unsurprising that there is no
performance improvement compared to the standard setting. When evaluating on
seen tasks, the linear classification layer is significantly more effective
than attending to the schema. This suggests that a large neural model (i.e.,
BERT) is able to implicitly learn meaningful dialog policies from dialog data.
It is possible that this performance difference may decrease with more
expressive schemas (e.g., having multiple examples for each user utterance,
automatically learning schemas from the dataset). The value of our schema
graphs is nonetheless shown when comparing sam to sam$-[1]$ (i.e., the old
schema graphs). These experiments provide an upper bound for the performance
in zero-shot transfer.

###  5.2 Zero-Shot Transfer

Table 2 shows the results of the zero-shot experiments. sam obtains strong
improvements over the baseline models for both zero-shot task transfer and
domain transfer. These experimental results validate the effectiveness of the
schema-guided paradigm, as well as the specific design of sam.

Compared to the baseline model (described in Section 3.1), sam obtains a
$\mathbf{+22~{}F_{1}}$ score improvement in task transfer and a
$\mathbf{+24~{}F_{1}}$ score improvement in domain transfer. Since the
baseline model is unable to predict classes it has not observed at training
time, its performance is limited to actions that are consistent across domains
(e.g., ‘hello’, ‘goodbye’, ‘anything-else’). This improvement highlights the
effectiveness of the schema-guided paradigm for zero-shot transfer learning.

bert+s also leverages schemas for transfer learning. Yet, it under-performs
relative to the baseline model. sam attains even larger improvements over this
baseline schema-guided model. As described in Section 4.2, the weak
performance of bert+s is largely a consequence of it being incapable of
handling realistic user behavior. The design of bert+s (i.e., the schema only
having system nodes) results in the model essentially predicting the
subsequent system actions. This is equivalent to sequentially predicting the
next system action, regardless of user behavior. With improved schema
representations and model architecture, sam achieves much stronger performance
in zero-shot transfer.

Our ablation experiments shed more light on the performance of sam relative to
bert+s. A significant performance drop is observed when removing the newly
constructed schema representations (i.e., sam$-[1]$). In contrast, adding the
schema graphs to bert+s (i.e., sam$-[2,3,4]$) results in a strong performance
improvement of $\mathbf{+15~{}F_{1}}$ score. This confirms the hypothesis that
the schema graphs of Mosig et al. (2020), which are largely comprised of
system action nodes are insufficient for modelling realistic user behavior.

Word-level attention is shown to give moderate, albeit statistically
significant, improvement. In contrast to sam$-[2]$, sam obtains a
$\mathbf{+3~{}F_{1}}$ score improvement. While word-level attention allows the
model to better align the dialog to the schema, it is an architectural
improvement that is not central to the schema-guided paradigm.

Modifying the training algorithm to sample batches from the same task results
in better negative samples during training. This allows the model to learn to
distinguish between nodes from the same schema graph when aligning the dialog
to the schema graph. When this modification is removed (i.e., sam$-[3])$, the
performance of sam drops by $\mathbf{8~{}F_{1}}$ score for zero-shot task
transfer.

The fourth and final component of sam is the removal of the linear
classification layer. Since this classification layer is unable to predict
classes it has not seen at training time, it is ineffective in zero-shot
settings. Unsurprisingly, removing it increases performance and sam obtains a
$\mathbf{+6~{}F_{1}}$ score improvement over sam$-[4]$.

The zero-shot experiments shown in Table 2 empirically validate several
hypotheses presented in this paper. First, the strong improvement over the
baseline demonstrates the efficacy of the schema-guided paradigm for zero-shot
generalizability in end-to-end dialog. Decoupling dialog policy and the
language understanding by explicitly representing the task-specific dialog
policies as schema graphs results in an improved ability to transfer to unseen
tasks. Next, we improve over the schema-guided model of Mosig et al. (2020)
through (1) an improved schema representation and (2) a collection of
modifications to the model. The improved schema representation better models
realistic user behaviors in dialog, and therefore results in better alignment
of the dialog and the schema. Our model modifications result in the model
being able to learn better fine-grained relationships during alignment (e.g.,
through better negative sampling and word-level attention) and better handle
zero-shot transfer (e.g., by removing the linear layer).

In contrast to prior work on zero-shot generalizability (Zhao and Eskenazi,
2018; Qian and Yu, 2019), our approach is shown to effectively transfer
between the vastly dissimilar domains of the STAR corpus (Mosig et al., 2020)
(e.g., trivia or spaceship maintenance). Rather than modelling a cross-domain
mapping and leveraging similar concepts across different domains, the schema-
guided paradigm decouples the domain-specific (i.e., the dialog policy) and
domain-agnostic (i.e., language understanding) aspects of dialog systems.
Through the schema-guided paradigm, we achieve strong performance in the zero-
shot setting and take an important step towards zero-shot dialog.

##  6 Conclusion

This paper shows strong results in zero-shot task transfer and domain transfer
using the schema-guided paradigm. We hypothesized that the difficulty of zero-
shot transfer in dialog stems from the dialog policy. When neural models
implicitly memorize dialog policies observed at training time, they struggle
to transfer to new tasks. To mitigate this, we explicitly provide the dialog
policy to the model, in the form of a schema graph. This paper introduces the
Schema Attention Model (sam) and shows improved schema graphs for the STAR
corpus. This approach attains significant improvement over prior work in the
zero-shot setting, with a $\mathbf{+22~{}F_{1}}$ score improvement.
Furthermore, the ablation experiments demonstrate the effectiveness of both
sam and the improved schema representations. Future work may explore (1)
improved schema representations to better capture dialog policy, (2) improved
model architectures to better align the dialog to the schema, and (3)
extensions to other problems (e.g., response generation).

## References

  * Adiwardana et al. (2020) Daniel Adiwardana, Minh-Thang Luong, David R So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, et al. 2020\.  Towards a human-like open-domain chatbot.  _arXiv preprint arXiv:2001.09977_.
  * Bapna et al. (2017) Ankur Bapna, Gokhan Tur, Dilek Hakkani-Tur, and Larry Heck. 2017.  Towards zero-shot frame semantic parsing for domain scaling.  _arXiv preprint arXiv:1707.02363_.
  * Bohus and Rudnicky (2009) Dan Bohus and Alexander I Rudnicky. 2009.  The ravenclaw dialog management framework: Architecture and systems.  _Computer Speech & Language_, 23(3):332–361.
  * Budzianowski et al. (2018) Paweł Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Iñigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gašić. 2018.  Multiwoz-a large-scale multi-domain wizard-of-oz dataset for task-oriented dialogue modelling.  _arXiv preprint arXiv:1810.00278_.
  * Chen et al. (2016) Yun-Nung Chen, Dilek Hakkani-Tür, and Xiaodong He. 2016.  Zero-shot learning of intent embeddings for expansion by convolutional deep structured semantic models.  In _2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 6045–6049. IEEE.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Ferguson and Allen (1998) George Ferguson and James Allen. 1998.  Trips: An intelligent integrated problem-solving assistant.  In _Proceedings of the Fifteenth National Conference on Artificial Intelligence (AAAI-98)_ , pages 567–573.
  * Hemphill et al. (1990) Charles T. Hemphill, John J. Godfrey, and George R. Doddington. 1990.  The ATIS spoken language systems pilot corpus.  In _Speech and Natural Language: Proceedings of a Workshop Held at Hidden Valley, Pennsylvania, June 24-27,1990_.
  * Hu et al. (2019) Wenpeng Hu, Zhangming Chan, Bing Liu, Dongyan Zhao, Jinwen Ma, and Rui Yan. 2019\.  Gsn: A graph-structured network for multi-party dialogues.  _arXiv preprint arXiv:1905.13637_.
  * Jurafsky (2000) Dan Jurafsky. 2000.  _Speech & language processing_.  Pearson Education India.
  * Mehri et al. (2020) Shikib Mehri, Mihail Eric, and Dilek Hakkani-Tur. 2020.  Dialoglue: A natural language understanding benchmark for task-oriented dialogue.  _arXiv preprint arXiv:2009.13570_.
  * Mosig et al. (2020) Johannes EM Mosig, Shikib Mehri, and Thomas Kober. 2020.  Star: A schema-guided dialog dataset for transfer learning.  _arXiv preprint arXiv:2010.11853_.
  * Qian and Yu (2019) Kun Qian and Zhou Yu. 2019.  Domain adaptive dialog generation via meta learning.  _arXiv preprint arXiv:1906.03520_.
  * Qiu et al. (2020) Liang Qiu, Yizhou Zhao, Weiyan Shi, Yuan Liang, Feng Shi, Tao Yuan, Zhou Yu, and Song-Chun Zhu. 2020.  Structured attention for unsupervised dialogue structure induction.  _arXiv preprint arXiv:2009.08552_.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9.
  * Rastogi et al. (2020a) Abhinav Rastogi, Xiaoxue Zang, Srinivas Sunkara, Raghav Gupta, and Pranav Khaitan. 2020a.  Schema-guided dialogue state tracking task at dstc8.  _arXiv preprint arXiv:2002.01359_.
  * Rastogi et al. (2020b) Abhinav Rastogi, Xiaoxue Zang, Srinivas Sunkara, Raghav Gupta, and Pranav Khaitan. 2020b.  Towards scalable multi-domain conversational agents: The schema-guided dialogue dataset.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 34, pages 8689–8696.
  * Raux et al. (2005) Antoine Raux, Brian Langner, Dan Bohus, Alan W Black, and Maxine Eskenazi. 2005\.  Let’s go public! taking a spoken dialog system to the real world.  In _Ninth European conference on speech communication and technology_.
  * Rich and Sidner (1998) Charles Rich and Candace L Sidner. 1998.  Collagen: A collaboration manager for software interface agents.  In _Computational models of mixed-initiative interaction_ , pages 149–184. Springer.
  * Shah et al. (2019) Darsh J Shah, Raghav Gupta, Amir A Fayazi, and Dilek Hakkani-Tur. 2019.  Robust zero-shot cross-domain slot filling with example values.  _arXiv preprint arXiv:1906.06870_.
  * Shi et al. (2019) Weiyan Shi, Tiancheng Zhao, and Zhou Yu. 2019.  Unsupervised dialog structure learning.  _arXiv preprint arXiv:1904.03736_.
  * Wang et al. (2019) Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2019.  Superglue: A stickier benchmark for general-purpose language understanding systems.  _arXiv preprint arXiv:1905.00537_.
  * Wang et al. (2018) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2018.  Glue: A multi-task benchmark and analysis platform for natural language understanding.  _arXiv preprint arXiv:1804.07461_.
  * Wen et al. (2016) Tsung-Hsien Wen, David Vandyke, Nikola Mrksic, Milica Gasic, Lina M Rojas-Barahona, Pei-Hao Su, Stefan Ultes, and Steve Young. 2016.  A network-based end-to-end trainable task-oriented dialogue system.  _arXiv preprint arXiv:1604.04562_.
  * Williams and Zweig (2016) Jason D Williams and Geoffrey Zweig. 2016.  End-to-end lstm-based dialog control optimized with supervised and reinforcement learning.  _arXiv preprint arXiv:1606.01269_.
  * Wu et al. (2019) Chien-Sheng Wu, Andrea Madotto, Ehsan Hosseini-Asl, Caiming Xiong, Richard Socher, and Pascale Fung. 2019.  Transferable multi-domain state generator for task-oriented dialogue systems.  _arXiv preprint arXiv:1905.08743_.
  * Xu et al. (2020) Jun Xu, Zeyang Lei, Haifeng Wang, Zheng-Yu Niu, Hua Wu, Wanxiang Che, and Ting Liu. 2020.  Discovering dialog structure graph for open-domain dialog generation.  _arXiv preprint arXiv:2012.15543_.
  * Zhang et al. (2020) Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and Bill Dolan. 2020.  DIALOGPT : Large-scale generative pre-training for conversational response generation.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations_ , pages 270–278, Online. Association for Computational Linguistics.
  * Zhao and Eskenazi (2018) Tiancheng Zhao and Maxine Eskenazi. 2018.  Zero-shot dialog generation with cross-domain latent actions.  _arXiv preprint arXiv:1805.04803_.
  * Zhao et al. (2017) Tiancheng Zhao, Allen Lu, Kyusong Lee, and Maxine Eskenazi. 2017.  Generative encoder-decoder models for task-oriented spoken dialog systems with chatting capability.  _arXiv preprint arXiv:1706.08476_.

##  Appendix A Example Outputs

We show a few example outputs in Table 3. These examples are model outputs
produced in the zero-shot domain transfer experiments. None of these models
have seen any training data pertaining to the specific domains. These examples
demonstrate the effectiveness of sam to handle ambiguous user intents and
effectively align them to the dialog schema.

Domain: |  Spaceship
---|---
Task: |  Spaceship Access Codes
Dialog History: |  USER: The ship is under attack! Need help ASAP SYSTEM: Please provide your name. USER: Rey. I’m the first officer SYSTEM: Please enter the code. USER: I think it’s 9315831
Baseline Output: |  Query
bert+s Output: |  Query
sam Output: |  Spaceship Ask Code Type (‘Please specify the code type.’)
sam Schema Node: |  SYSTEM: Please enter the code. USER: [NUMBER]
Domain: |  Bank
Task: |  Check Bank Balance
Dialog History: |  USER: Hi, can you help me check my balance? SYSTEM: Could I get your full name, please? USER: Jane Doe SYSTEM: Could you tell me your account number, please? USER: I don’t remember it or my PIN number unfortunately.
Baseline Output: |  Query
bert+s Output: |  Inform Cannot Authenticate (‘I am sorry, but I cannot authenticate you with the information you have provided.’)
sam Output: |  Ask Date of Birth (‘Could you provide your date of birth, please?’)
sam Schema Node: |  SYSTEM: Could you tell me your account number, please? USER: I don’t know my account number.
Table 3: Example outputs in the zero-shot domain transfer setting. The models
have not seen any data pertaining to the domains that are tested here. The
first example shows that sam can effectively align to the schema. The second
example is more complicated and requires sam to identify that the user
indicated that they don’t remember their account number (note that the most
typical system response following ‘Can you tell me your account number,
please?’) is asking for the user’s PIN.

##  Appendix B Example Schemas

All of the schema graphs produced for this paper are available in our GitHub
Repository444https://github.com/Shikib/schema_attention_model. These schemas
improve upon the schemas provided by Mosig et al. (2020). Nonetheless, this
schema representation is still far from perfect. Future work may explore
designing more expressive and thorough schemas. There are several approaches
for doing this: (1) having multiple examples for each user utterance, (2)
better representing loops in the schema graphs (e.g., query the database until
you find the cheapest flight), (3) representing more sophisticated
(potentially dynamic) scenarios (e.g., if the user’s answer matches the
correct answer to a trivia question, inform them that they are correct). The
proposed schema representations address several weaknesses, however there is
still significant room for improvement.


# Deep Learning-based Anomaly Detection on X-ray Images of Fuel Cell
Electrodes

Simon B. Jensen1, Thomas B. Moeslund1 and Søren J. Andreasen2
1Department of Architecture and Media Technology, Aalborg University, Aalborg,
Denmark
2Serenergy, Aalborg, Denmark
<EMAIL_ADDRESS><EMAIL_ADDRESS>
https://orcid.org/0000-0001-7584-5209

###### Abstract

Anomaly detection in X-ray images has been an active and lasting research area
in the last decades, especially in the domain of medical X-ray images. For
this work, we created a real-world labeled anomaly dataset, consisting of
16-bit X-ray image data of fuel cell electrodes coated with a platinum
catalyst solution and perform anomaly detection on the dataset using a deep
learning approach. The dataset contains a diverse set of anomalies with 11
identified common anomalies where the electrodes contain e.g. scratches,
bubbles, smudges etc. We experiment with 16-bit image to 8-bit image
conversion methods to utilize pre-trained Convolutional Neural Networks as
feature extractors (transfer learning) and find that we achieve the best
performance by maximizing the contrasts globally across the dataset during the
16-bit to 8-bit conversion, through histogram equalization. We group the fuel
cell electrodes with anomalies into a single class called abnormal and the
normal fuel cell electrodes into a class called normal, thereby abstracting
the anomaly detection problem into a binary classification problem. We achieve
a balanced accuracy of 85.18%. The anomaly detection is used by the company,
Serenergy, for optimizing the time spend on the quality control of the fuel
cell electrodes.

##  1 INTRODUCTION

Serenergy is a world-leading supplier of methanol-based fuel cell solutions
with more than a thousand active units deployed globally. The fuel cells
provide back-up power as well as temporary primary power or work in a hybrid
system with renewable sources such as solar and/or wind. The core component of
the fuel cell system, Serenergy, is a cell stack of 120 high temperature,
polymer electrolyte membranes. Each cell contains 2 fuel cell electrodes that
are coated with a platinum based catalyst. Meaning the fuel cell system is
made up of $120\times 2=240$ fuel cell electrodes in total. Examples of fuel
cell electrodes can be seen in figure 1. We will use the term fuel cell
electrode or electrode interchangeably.

The quality of the platinum based catalyst and the quality of how well it is
coated on to each electrode is paramount to the overall conductivity of the
fuel cell. The quality of an electrode is measured through a semi-
automatic/manual process where an X-ray image is manually captured of each
electrode and the X-ray image is analyzed by an image analysis tool which
outputs several quality parameters e.g., color histograms, box plots, standard
deviation of the colors of the platinum coating, the minimum- and maximum
colors of the platinum coating etc.

Figure 1: Fuel cell electrodes coated with a platinum catalyst

The process of analyzing the output of the image analysis tool is time
consuming and in most cases the electrodes have an acceptable quality.
Serenergy wishes to optimize the quality control process by using a deep
learning approach to perform automatic anomaly detection on the fuel cell
electrodes, by grouping them into two classes, normal or abnormal. Where a
normal electrode can be used in the final fuel cell system and an abnormal
electrode cannot.

In figure 2 examples of X-ray images of normal and abnormal electrodes are
shown. This paper describes our approach to this important problem and the
contributions are:

  * •

16-bit to 8-bit conversion methods for X-ray images of fuel cell electrodes
using histogram equalization.

  * •

A Deep Convolutional Neural Network classifier to perform anomaly detection of
X-ray images of fuel cell electrodes.

(a)

(b)

(c)

(d)

(e)

Figure 2: Examples of X-ray images of fuel cell electrodes. Figure 2(a), 2(b)
and 2(c) are examples of abnormal plates, while figure 2(d) and 2(e) are
examples of normal plates. 2(a) is abnormal due to issues with scratches near
its edges. 2(b) is abnormal due to issues with lines and bubbles and 2(b) is
abnormal due to issues with smudges.

###  1.1 Related Work

####  1.1.1 Deep Learning-based Classification on X-ray Image

Deep learning is a widely used technology for classification of images. More
specifically Deep Convolutional Neural Networks (CNNs). The technological
development of CNNs has been accelerated by image classification datasets and
challenges such as pascal VOC [Everingham et al., 2010] and ImageNet
[Russakovsky et al., 2014].

For X-ray images, there has been a lack of large publicly available dataset,
which means that few studies exists using CNNs for classification in X-ray
images. However, in the specific domain of pneumonia detection in chest X-ray
images, a number of dataset has recently been made publicly available for
example [Mooney, 2018], [Wang et al., 2017] and [Cohen et al., 2020]. This has
resulted in a large number of studies which use CNNs to classify X-ray images
in this specific field. [Çallı et al., 2021] reviews and compares a large
collection of these studies.

[Rahman et al., 2020] trains four well-recognized CNN models pre-trained on
the Imagenet dataset, AlexNet [Krizhevsky et al., 2012], ResNet-18 [He et al.,
2015], DenseNet201 [Wang and Zhang, 2020] and SqueezeNet [Iandola et al.,
2016] for detecting pneumonia in the Pneumonia Chest X-ray dataset and compare
their performance. They find that DenseNet201 achieves the best performance
for binary classification (normal/pneumonia) while only performing slightly
better than ResNet-18.

Similarly, [Jiang, 2020] trains a CNN on the Pneumonia Chest X-ray dataset to
detect pneumonia while utilizing the Dynamic Histogram Enhancement algorithm
[Abdullah-Al-Wadud et al., 2007] as pre-processing method to improve the
quality of X-ray images before training and evaluating the CNN model.

####  1.1.2 Transfer Learning for Image Classification

[Hussain et al., 2019] shows that transfer learning for image classification
using deep CNNs is a valid and efficient method for achieving high performance
in image classification tasks when dealing with datasets of limited size. They
use the Inception-v3 [Szegedy et al., 2015] CNN pre-trained on the ImageNet
dataset and re-train the model on the Caltech Face dataset consisting of only
450 images while achieving an accuracy of 65.7%.

[Rahman et al., 2020] validates transfer learning for image classification
when the base dataset (ImageNet) consist of 3-channel RGB images and the
target dataset consists of 1-channel grayscale X-Ray images. They do this by
fine-tuning CNN models pre-trained with ImageNet on chest X-ray datasets.

Several transfer learning methods have recently been publicized which further
optimizing the performance which can be achieved by the method. [Wang et al.,
2019] proposes a method called attentive feature distillation and selection
(AFDS), which adjusts the strength of transfer learning regularization and
also dynamically determines the important features to transfer. They impose
the method onto ResNet-101 and achieve state-of-the art computation reduction.

####  1.1.3 Anomaly Detection using Deep Learning

[Alloqmani et al., 2021] reviews twenty studies which utilizes deep learning
for anomaly detection and identify challenges and insights in the domain. They
identify the three main challenges for anomaly detection to be: (1) handling
the class imbalance of normal and abnormal data, (2) the availability of
labeled data and (3) the fact that there is often noise in the data that
appears to be close to the actual anomalies and thus it becomes difficult to
differentiate them.

[Ho and Wookey, 2020] proposes a solution to the class imbalance challenge, by
introducing a new loss function for binary- and multiclass classification
problems called Real-World-Weight Cross-Entropy loss function. Which allows
direct input of real world costs as weights. This could prove useful for
classification problems where there is a well-defined loss/cost for
misclassified samples.

[Elgendi et al., 2021] proposes a solution to the second challenge by
introducing and comparing four data augmentation methods for artificially
increasing the number of training samples of X-ray images, while performing
Covid-19 pneumonia detection using a CNN. The methods uses combinations of
random rotations, shear, translation, horizontal- and vertical flipping among
other data augmentation methods.

We discuss the three challenges and possible solutions further, in regards to
anomaly detection in X-ray images of fuel cell electrodes in section 4.

##  2 APPROACH

###  2.1 Overview

An overview of the anomaly detection approach proposed in this paper is seen
in figure 3. The approach is described by the following three steps:

  1. 1.

Convert the X-ray images from 16-bit to 8-bit, as described in section 2.3.

  2. 2.

Extract features using a pre-trained ResNet-34 CNN model as described in 2.4.

  3. 3.

Classify anomalies in the feature-maps generated by the CNN model, using fully
connected layers, as described in 2.4.1.

Figure 3: An overview of the anomaly detection approach proposed in this paper
made up by three main steps which are described in section 2. Some of the
layers of the ResNet-34 feature extractor are hidden for illustration
purposes.

###  2.2 The Fuel Cell Electrode X-Ray Dataset

A real-world labeled anomaly dataset consisting of 16-bit X-ray images of
platinum catalyst coated fuel cell electrodes is created for this work, as no
existing dataset in the domain of fuel cell electrodes for methanol fuel cells
exists to the best of our knowledge.

The fuel cell electrode X-ray dataset consists of 714 X-ray images. Each
electrode belongs to 1 of 12 batches, named batch 1, batch 2, …, batch 12.
Where a batch represents a collection of electrodes which are coated with a
platinum catalyst solution using the same coating method and mixture. The
batches in the dataset have varying number of X-ray images as can be seen from
figure 4. Batch 3 has most sample with 152 and batch 1 has fewest samples with
17. The class balance vary greatly for each batch as well. For batch 1 only 1
electrode is normal accounting for 5.88% of the images in the batch while 37
out of the 38 electrodes in batch 8 are labeled as normal accounting for
97.37% of the images in the batch. The dataset is generally imbalanced with a
over representation of normal samples. Across all 714 images 562 (78.71%) are
labeled as normal and 152 (21.29%) are labeled as abnormal.

A dataset size of 714 X-ray images is considered to be a small dataset when
utilizing a deep learning approach. We utilize transfer-learning to overcome
this problem, through the use of a pre-trained ResNet-34 model, as described
in section 2.4. Due to the limited size of the dataset, creating a
representative test set for the dataset proves difficult. We evaluated the
X-ray images of each batch through cross-validation. This is done by
evaluating each batch individually, while the remaining batches are used as
training samples and combining the results of each evaluation into a total
score. This is further described in section 3.

Figure 4: Overview of the number of X-ray images per batch as well as the
number of normal and abnormal samples. The number of X-ray images per batch
vary for each batch and so does the class balance. The dataset is generally
imbalanced with a over-representation of normal samples.

####  2.2.1 Fuel Cell Electrode X-Ray Images

The X-ray images of the dataset have varying dimensions with a minimum width
of 1119 pixels, maximum width of 1219 pixels and mean width of 1145 pixels.
The minimum height of an electrode is 2053 pixels, the maximum height is 2115
pixels and the mean 2072,1 pixels. During training and evaluation of the
anomaly detector, the X-ray images are transformed into a uniform size of
$2000\times 1000$. Examples of normal and abnormal electrode X-ray images can
be seen in figure 2.

####  2.2.2 Anomalies

Serenergy has identified 11 common anomaly types, which are grouped into a
single class called abnormal. The identified 11 common anomaly types are
named: scratches, lines, edge cuts, edge tensions, smudges, edge ink flow,
bubbles, missing ink, agglomerate, ink fluctuations, ink entry/exit.

The normal fuel cell electrodes can have minor representations of one or more
anomaly type, as long as the severity is not too great. Figure 2(d) is an
example of an electrode which have minor scratches, but the scratches are not
sever enough to be classified as abnormal, while figure 2(a) is an example of
a fuel cell electrode which is abnormal due to scratches near its edges.

###  2.3 X-ray Image Conversion Methods

To utilize pre-trained CNNs such as ResNet-34 as feature extractor in the
anomaly detector, the depth of the electrode X-ray images is extended from 1
channel to 3 channels and the pixel values of the images are converted from a
16-bit values into 8-bit values. Thous, increasing the similarity of the
electrode X-ray images to the images of ImageNet on which the CNN is pre-
trained on. This is further described in section 2.4.

The 16-bit color range consists of 2 to the power of 16 (65536) colors and the
8-bit color range consists of 2 to the power of 8 (256) colors. A loss of
information during the conversion is therefore inevitable.

We experiment with four methods for converting the X-ray images from 16- to
8-bits using histogram equalization and implemented using Python’s OpenCV
library [Itseez, 2015].

In section 2.3.1 a naive 16-bit to 8-bit conversion method is described which
is used as baseline. In section 2.3.2 we use histogram equalization with
global maximum and minimum bounds calculated across the entire dataset and in
section 2.3.3 we use histogram equalization with local maximum and minimum
bounds calculated for each individual X-ray image. Finally, we mix the methods
in section 2.3.4. Examples of the resulting 8-bit images for each conversion
are shown in figure 5.

(a)

(b)

(c)

(d)

Figure 5: Example of the 8-bit images obtained by applying conversion method
1, 2, 3 and 4 to a 16-bit X-ray image. Figure 5(a), 5(b), 5(c) and 5(d) shows
method 1, 2, 3 and 4 respectively.

####  2.3.1 Method 1: Naive Conversion

Method 1 is a naive conversion which is used as baseline. Method 1 simply
scales each 16-bit pixel value and convert it to an unsigned 8-bit type. The
conversion is given by equation 1.

$f_{1}(x_{i})=\text{usign}\left(\frac{x_{i}}{256}\right)$ (1)

Where $x_{i}$ corresponds to the $i$th pixel in a 16-bit X-ray image and usign
is a function which converts a number to an 8-bit unsigned integer. An example
of a resulting fuel cell electrode image after conversion can be seen in
figure 5(a).

Most pixels lie in the range 1700-2800 in the 16-bit X-ray images, as shown in
section 2.3.2, which means method 1 will appear very dark. Pixel values in the
16-bit color range of 1700-2800 corresponds to pixel values of 6-11 in the
8-bit range using naive conversion.

Finally, the resulting 1-channel 8-bit image is extended with 2 additional
channel, resulting in a 3. channel 8-bit image.

####  2.3.2 Method 2: Conversion by global min and max

For method 2 the global maximum- and minimum pixel value, $G_{max}$ and
$G_{min}$, of the 16-bit X-ray dataset is calculated and used as upper- and
lower-bounds for histogram equalization, during 16-bit to 8-bit conversion.

The global maximum pixel value is found by calculating the 99.99th percentile
of the pixel values in all X-ray images in the dataset and taking the maximum
pixel value found and round it to nearest hundred. Similarly, we calculate
$G_{min}$ by finding the 0.01th percentile. $G_{max}$ and $G_{min}$ are found
to be 28000 and 1700. Method 2 is given by equation 2 and illustrated in
figure 7.

$f_{2}(x_{i})=\text{usign}\left(\frac{\max(\min(x_{i},G_{max}),G_{min})}{G_{max}-G_{min}}\times
256\right)$ (2)

The reason for calculating the 99.99th and 0.01th percentiles of the pixel
values is to avoid that noise in the X-ray images will affect the values of
$G_{max}$ and $G_{min}$.

Figure 6: Method 2 uses the global minimum- and maximum pixel values,
$G_{max}$ and $G_{min}$, calculated across the entire fuel cell electrode
X-ray dataset as upper- and lower bounds to convert from 16-bit to 8-bit
through histogram equalization.

####  2.3.3 Method 3: Conversion by local min and max

Method 3 uses the local maximum- and minimum pixel value, $L_{max}$ and
$L_{min}$, found for each individual X-ray image, as upper- and lower-bound
for histogram equalization.

Thereby, maximizing the contrast in the 8-bit color range for each individual
X-ray image. Similarly to method 2, the 99.99th and 0.01th percentiles of the
pixel values for each X-ray image are used to avoid that noise will affect the
values of $L_{max}$ and $L_{min}$.

The danger however, can be that the converted pixel values in one X-ray image
lose there meaning relative to the converted pixel values of another X-ray
image. Consider an X-ray image, $\text{image}_{i}$ with a $L_{min}$ value of
2200 and another X-ray image, $\text{image}_{j}$ with an $L_{min}$ value of
1800. After conversion, a value of 0 in the 8-bit color range will have
corresponded to 2200 in $\text{image}_{i}$ and to 1800 in $\text{image}_{j}$.
Method 3 is given by equation 3.

$\begin{split}\begin{aligned}
f_{3}(k,x_{i})=\text{usign}\left(\frac{\max(\min(x_{i},L_{k,max}),L_{k,min})}{L_{k,max}-L_{k,min}}\times
256\right)\end{aligned}\end{split}$ (3)

Where $L_{k,max}$ and $L_{k,min}$ correspond to the local maximum and minimum
of the k’th image in the electrode dataset.

Figure 7: Method 3 uses the local minimum- and maximum pixel values, $L_{max}$
and $L_{min}$, calculated for each individual electrode X-ray image as upper-
and lower bounds to convert from 16-bit to 8-bit through histogram
equalization. Thereby, maximizing the contrast in the 8-bit color range for
each individual X-ray image.

####  2.3.4 Method 4: Mixing the methods

Finally, method 4 mixes the conversion methods from method 1, 2 and 3, such
that the resulting image will contain the 8-bit pixel values from method 1 in
its 1. channel, the pixel values from method 2 in its 2. channel and the pixel
values from method 3 in its 3. channel as seen in figure 8.

Figure 8: Method 4 mixes the conversion methods from method 1, 2, 3 into a 3
channel image, with 1 channel corresponding to the values achieved by each of
the methods.

###  2.4 Anomaly Detector Architecture

This paper proposes a fuel cell electrode anomaly detector which uses a
Convolutional Neural Network as feature extractor. For this purpose, a PyTorch
[Paszke et al., 2019] implementation of the ResNet-34 [He et al., 2015] CNN
model is utilized.

The ResNet-34 model is pre-trained on a base dataset (ImageNet [Russakovsky et
al., 2014]). The reason for using a pre-trained CNN, also referred to as
transfer learning, is that training CNNs from random initializations, usually
requires a large amount of data, to achieve a high performance. For many real-
world applications it can be both time consuming and expensive to collect the
required amount of data, as is also the case for this research.

The ResNet-34 model is extended with two fully connected layers of size 512
and 256 and finally with a softmax layer of size 2, to get an output for each
class, normal and abnormal. The model architecture is illustrated in figure 3.

####  2.4.1 Training

The anomaly detector model is trained on a Nvidia GeForce GTX 1080 GPU for
$12\times 50$ epochs for each of the 4 conversion methods. All layers in the
ResNet-34 CNN are frozen, accept for the fully connected layer, which is fine-
tuned together with the fully connected layers of the anomaly classifier. We
use the cross entropy loss as loss function and an initial learning rate of
0.001 which decays for each 10 epochs.

We augment the dataset with random horizontal and vertical flips (set to 50%
probability) and resize the electrode images to height of 2000 pixels and
width of 1000 pixels.

##  3 RESULTS

In this section we present the results achieved by the anomaly detector when
applying the 4 conversion methods described in section 2.3 to the fuel cell
electrode X-ray dataset. The anomaly detectors are evaluated using the
balanced accuracy metric described in section 3.1.

###  3.1 Balanced accuracy

We use the balanced accuracy metric to evaluate the anomaly detector, as it
proves useful for binary classification problems on datasets with class
imbalance, like in the fuel cell electrode dataset. Whereas, accuracy can be
misleading if the class imbalance is great. The balanced accuracy metric
overcomes this issue by weighting the positive and negative samples equally
significant despite one class being more numerous than the other. This is done
by adding the true positive rate (TPR) with the true negative rate (TNR) and
dividing them by 2, as can be seen in equation 4.

$\text{BALANCED ACC}=\frac{TPR+TNR}{2}$ (4)

$\text{TPR}=\frac{TP}{TP+FN}$ (5)

$\text{TNR}=\frac{TN}{TN+FN}$ (6)

For a dataset with 98 positive samples and 2 negative samples, a classifier
will achieve an accuracy score of 98% by simply classifying every sample as
positive. The balanced accuracy score will only be 50% in this case.

###  3.2 Cross-validation Evaluation

The dataset is evaluated through cross-validation, due to the limited size of
the dataset and the nature of the dataset where each batch of images are
coated with a platinum catalyst solution using the same coating method and
solution mixture. This means images from the same batches will have a high
similarity to one another. Utilizing samples from the same batch as both
training and testing samples will therefore inevitable occlude the performance
of the anomaly detectors.

We train the anomaly detector as described in section 2.4.1 using all but one
batch, which is used as test set. The same procedure is replicated until each
batch has been evaluated individually for each of the 4 conversion methods.
Meaning a total of $4\times 12$ training/evaluations are performed. For each
training/evaluation run, the balanced accuracy score is calculated. A combined
balanced accuracy score is then calculated for each conversion method by
adding the true positives (TP), false positives (FP), true negatives (TN) and
false negatives (FN) obtained when evaluating each of the 12 batches, for the
given conversion method. The results can be seen in table 1.

We find that the best anomaly detection performance is achieved by conversion
method 2, with a balanced accuracy of 85.18%. Surprisingly, we find that
method 4, which combines conversion method 1, 2 and 3 achieves the worst
overall anomaly detection performance. A possible explanation for this might
be that the dissimilarity between the combined 3-channel image created by
method 4 and the features of RGB images in ImageNet, which our feature
extractor CNN is pre-trained on, is too great.

Table 1: The balanced accuracy (%) achieved by the anomaly detector when
applying conversion method 1, 2, 3 and 4 to the fuel cell electrode X-ray
images. The performance achieved by each conversion method is measured through
cross-validation, where each batch is used as test set individually while the
remaining batches are used as training set. The green colors highlight the
method(s) which achieved the best balanced accuracy score for each batch. The
olive green color highlights the method which achieved the best overall
balanced accuracy score when combining the evaluations of each batch, which
was conversion method 2.

Test Batch | Method 1 | Method 2 | Method 3 | Method 4
---|---|---|---|---
Batch 1 | 50.00 | 96.88 | 50.00 | 40.63
Batch 2 | 75.00 | 50.00 | 75.00 | 75.00
Batch 3 | 50.00 | 50.00 | 50.00 | 50.00
Batch 4 | 90.54 | 58.65 | 67.30 | 67.30
Batch 5 | 75.00 | 62.50 | 75.00 | 62.50
Batch 6 | 66.15 | 87.05 | 77.56 | 85.13
Batch 7 | 82.39 | 89.53 | 73.89 | 72.17
Batch 8 | 50.00 | 50.00 | 100.00 | 50.00
Batch 9 | 68.53 | 50.00 | 90.00 | 70.00
Batch 10 | 50.00 | 50.00 | 54.17 | 54.17
Batch 11 | 75.00 | 99.40 | 75.00 | 75.00
Batch 12 | 53.33 | 80.70 | 75.52 | 67.76
Overall | 82.31 | 85.18 | 85.00 | 81.02

##  4 DISCUSSION

Labeling anomalies in X-ray images of fuel cell electrodes is a difficult and
time consuming task, which requires expertise in the specific domain and
knowledge about the severity and consequences different anomaly types impose
to the conductivity of the fuel cell systems in which the electrodes will be
used. To illustrate the difficulty, a few examples of false positives and
false negatives found during the evaluation, can be seen in figure 9, which
might be cause by faulty annotations.

Generally, when labeling samples for a binary classification problem, the
person labeling has to make a decision on which class the sample belong to,
for this work, whether an electrode is normal or abnormal.

We find that making this decision, for some samples, is a non-trivial task
prone to subjectivity. While one expert might label the sample as normal
another expert might label the same sample as abnormal. One possible solution,
which was chosen for this work, is to let the more experienced expert make the
final decision.

A second solution, could be to distribute the labeling task across a large
number of experts or non-experts and let the label with most votes represent
the sample. The method has been described and evaluated by [Younesian et al.,
2020] and it has the potential to remove subjectivity from the labels. The
drawback to this solution is that it can be expensive and time consuming and
in some cases a sample might end up having equally many votes for each class
in which case an additional solution for these cases needs to be found.

Other solutions could be to simply exclude such samples from the dataset or to
introduce a third class which represents samples which are undecidable, if one
or more experts disagree. Such a class would in our case be very small and
cause a highly imbalanced dataset.

(a)

(b)

(c)

(d)

Figure 9: Labeling fuel cell electrodes is a non-trivial task which can be
prone to subjectivity. Figure 9(a) and 9(b) are examples of false positives
and figure 9(c) and 9(d) are examples of false negatives found during
evaluation of conversion method 2. As can be seen the difference between e.g.
figure 9(a) and 9(c) is difficult to see and the mis-classification might as
well be due to faulty annotations.

##  5 CONCLUSIONS

This paper proposed an anomaly detector using a Deep Convolutional Neural
Network, an extended ResNet-34 model, for detecting anomalies in X-ray images
of fuel cell electrodes. For this purpose a dataset with normal and abnormal
fuel cell electrodes X-ray images was created. The dataset consists of 12
batches of images with a total of 714 X-ray images. The anomaly detector is
used by the company Serenergy for automatizing a time-consuming manual quality
control of the fuel cell electrode X-ray images. The anomaly detector was
trained and evaluated through cross-validation where a single batch of images
is used as test set and the remaining batches are used as training set. The
proposed anomaly detector was trained and evaluated using $12\times 50$ epochs
with a Nvidia GeForce 1080 GTX GPU and the PyTorch deep learning framework. We
compared 16-bit to 8-bit conversion methods for pre-processing the X-ray
images. We find that performing histogram equalization with upper- and lower
bounds set by the maximum and minimum pixel values calculated across the
entire dataset achieve a better performance than when using local maximum and
minimum as upper- and lower bounds calculated for each individual image. We
achieve a balanced accuracy of 85.18%.

In the future, we will continue to explorer approaches for performing more
accurate anomaly detection in X-ray images. Potential improvements could be
achieved by using variations of weighted cross-entropy loss and data
augmentation to cope with the imbalanced dataset and by utilizing different
histogram equalization methods e.g. the DHE algorithm. Further, we see a great
potential in using CNNs pre-trained on large-scale gray-scale image datasets
for classifying X-ray images. Whereas most CNNs today are pre-trained on RGB
image datasets e.g. ImageNet.

## ACKNOWLEDGEMENTS

We would like to thank Serenergy for their contributions, collaboration and
dataset for this paper. We would also like to thank Ambolt Aps for initiating
and facilitating the collaboration with Serenergy.

## REFERENCES

  * Abdullah-Al-Wadud et al., 2007 Abdullah-Al-Wadud, M., Kabir, M. H., Akber Dewan, M. A., and Chae, O. (2007).  A dynamic histogram equalization for image contrast enhancement.  IEEE Transactions on Consumer Electronics, 53(2):593–600.
  * Alloqmani et al., 2021 Alloqmani, A., B., Y., Khan, A., and Alsolami, F. (2021).  Deep learning based anomaly detection in images: Insights, challenges and recommendations.  International Journal of Advanced Computer Science and Applications, 12.
  * Cohen et al., 2020 Cohen, J. P., Morrison, P., Dao, L., Roth, K., Duong, T. Q., and Ghassemi, M. (2020).  Covid-19 image data collection: Prospective predictions are the future.  arXiv 2006.11988.
  * Elgendi et al., 2021 Elgendi, M., Nasir, M. U., Tang, Q., Smith, D., Grenier, J.-P., Batte, C., Spieler, B., Leslie, W. D., Menon, C., Fletcher, R. R., Howard, N., Ward, R., Parker, W., and Nicolaou, S. (2021).  The effectiveness of image augmentation in deep learning networks for detecting covid-19: A geometric transformation perspective.  Frontiers in Medicine, 8:153.
  * Everingham et al., 2010 Everingham, M., Gool, L. V., Williams, C. K. I., Winn, J., and Zisserman, A. (2010).  The pascal visual object classes (voc) challenge.
  * He et al., 2015 He, K., Zhang, X., Ren, S., and Sun, J. (2015).  Deep residual learning for image recognition.  CoRR, abs/1512.03385.
  * Ho and Wookey, 2020 Ho, Y. and Wookey, S. (2020).  The real-world-weight cross-entropy loss function: Modeling the costs of mislabeling.  CoRR, abs/2001.00570.
  * Hussain et al., 2019 Hussain, M., Bird, J. J., and Faria, D. R. (2019).  A study on cnn transfer learning for image classification.  In Lotfi, A., Bouchachia, H., Gegov, A., Langensiepen, C., and McGinnity, M., editors, Advances in Computational Intelligence Systems, pages 191–202, Cham. Springer International Publishing.
  * Iandola et al., 2016 Iandola, F. N., Moskewicz, M. W., Ashraf, K., Han, S., Dally, W. J., and Keutzer, K. (2016).  Squeezenet: Alexnet-level accuracy with 50x fewer parameters and <1mb model size.  CoRR, abs/1602.07360.
  * Itseez, 2015 Itseez (2015).  Open source computer vision library.
  * Jiang, 2020 Jiang, Z. (2020).  Chest x-ray pneumonia detection based on convolutional neural networks.  In 2020 International Conference on Big Data, Artificial Intelligence and Internet of Things Engineering (ICBAIE), pages 341–344.
  * Krizhevsky et al., 2012 Krizhevsky, A., Sutskever, I., and Hinton, G. E. (2012).  Imagenet classification with deep convolutional neural networks.  In Pereira, F., Burges, C. J. C., Bottou, L., and Weinberger, K. Q., editors, Advances in Neural Information Processing Systems 25, pages 1097–1105. Curran Associates, Inc.
  * Mooney, 2018 Mooney, P. (2018).  Chest x-ray images (pneumonia).
  * Paszke et al., 2019 Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Köpf, A., Yang, E., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., and Chintala, S. (2019).  Pytorch: An imperative style, high-performance deep learning library.  CoRR, abs/1912.01703.
  * Rahman et al., 2020 Rahman, T., Chowdhury, M. E. H., Khandakar, A., Islam, K. R., Islam, K. F., Mahbub, Z. B., Kadir, M. A., and Kashem, S. (2020).  Transfer learning with deep convolutional neural network (cnn) for pneumonia detection using chest x-ray.  Applied Sciences, 10(9).
  * Russakovsky et al., 2014 Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M. S., Berg, A. C., and Li, F. (2014).  Imagenet large scale visual recognition challenge.  CoRR, abs/1409.0575.
  * Szegedy et al., 2015 Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., and Wojna, Z. (2015).  Rethinking the inception architecture for computer vision.  CoRR, abs/1512.00567.
  * Wang et al., 2019 Wang, K., Gao, X., Zhao, Y., Li, X., Dou, D., and Xu, C.-Z. (2019).  Pay attention to features - transfer learn faster.
  * Wang and Zhang, 2020 Wang, S. and Zhang, Y.-D. (2020).  Densenet-201-based deep neural network with composite learning factor and precomputation for multiple sclerosis classification.  ACM Transactions on Multimedia Computing, Communications, and Applications, 16:1–19.
  * Wang et al., 2017 Wang, X., Peng, Y., Lu, L., Lu, Z., Bagheri, M., and Summers, R. M. (2017).  Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases.  CoRR, abs/1705.02315.
  * Younesian et al., 2020 Younesian, T., Hong, C., Ghiassi, A., Birke, R., and Chen, L. Y. (2020).  End-to-end learning from noisy crowd to supervised machine learning models.
  * Çallı et al., 2021 Çallı, E., Sogancioglu, E., van Ginneken, B., van Leeuwen, K. G., and Murphy, K. (2021).  Deep learning for chest x-ray analysis: A survey.  Medical Image Analysis, 72:102125.