# Knowledge Enhanced Neural Networks for relational domains

Alessandro Daniele
Data and Knowledge Managemen Research Unit
Fondazione Bruno Kessler
Trento, Italy
<EMAIL_ADDRESS>
&Luciano Serafini
Data and Knowledge Managemen Research Unit
Fondazione Bruno Kessler
Trento, Italy
<EMAIL_ADDRESS>

###### Abstract

In the recent past, there has been a growing interest in Neural-Symbolic
Integration frameworks, i.e., hybrid systems that integrate connectionist and
symbolic approaches to obtain the best of both worlds. In this work we focus
on a specific method, KENN (Knowledge Enhanced Neural Networks), a Neural-
Symbolic architecture that injects prior logical knowledge into a neural
network by adding on its top a residual layer that modifies the initial
predictions accordingly to the knowledge. Among the advantages of this
strategy, there is the inclusion of clause weights, learnable parameters that
represent the strength of the clauses, meaning that the model can learn the
impact of each rule on the final predictions. As a special case, if the
training data contradicts a constraint, KENN learns to ignore it, making the
system robust to the presence of wrong knowledge. In this paper, we propose an
extension of KENN for relational data. One of the main advantages of KENN
resides in its scalability, thanks to a flexible treatment of dependencies
between the rules obtained by stacking multiple logical layers. We show
experimentally the efficacy of this strategy. The results show that KENN is
capable of increasing the performances of the underlying neural network,
obtaining better or comparable accuracies in respect to other two related
methods that combine learning with logic, requiring significantly less time
for learning.

##  1 Introduction

In the last decade, deep learning approaches gained a lot of interest in the
AI community, becoming the state of the art on many fields, such as Computer
Vision [19], Machine Translation [2], Speech Recognition [15], etc. Indeed,
Neural networks (NNs) are suited for pattern recognition, even in the presence
of noisy data. They are particularly good at mapping low-level perceptions to
more abstract concepts (for instance, going from images to classes). However,
it is hard for a NN to reason with these high-level abstractions. Furthermore,
NNs are demanding in terms of training data. On the other hand, pure logical
approaches are not suited for learning from low-level features and they
struggle in the presence of noise. Nevertheless, they perform well in
reasoning with highly abstract concepts and learning from a small number of
samples. Given these opposite strengths and weaknesses, it is not a surprise
that a lot of interest has been drawn toward Neural-Symbolic (NeSy) systems.
Indeed, the goal is to combine these two paradigms to obtain the best of the
two worlds.

Among NeSy methods there is KENN (_Knowledge Enhanced Neural Network_) [6], a
model composed of a Neural Network enhanced with additional layers which
codify logical knowledge. KENN has multiple advantages over other NeSy
methods, such as its capacity to learn _clause weights_ and the ability to
impose the knowledge not only during training but even at inference time. In
particular, KENN showed remarkable results on the Predicate Detection task of
Visual Relationship Detection Dataset (VRD Dataset) [21] using a manually
curated prior knowledge proposed by [10], outperforming the previous state of
the art results, with really good performances on the _Zero Shot Learning_
sub-task [6]. Moreover, it outperformed Logic Tensor Networks [32], one of its
major competitors, using the same knowledge.

Despite its good empirical results, KENN has been applied only on multi-label
classification tasks with no relational data. Indeed, a limitation of KENN
resides in its inability to take into account binary predicates. This is
because KENN expects the NN’s predictions to be stored in a matrix format,
where the columns represent different unary predicates and the rows their
possible groundings (i.e., substitutions of the free variable for such
predicates). For this reason, it is not straightforward to apply KENN to
relational data, where binary predicates are available.

In this paper, we propose an updated version of KENN which can deal with
relational data. Particular attention was paid to defining a scalable strategy
to deal with binary predicates, obtaining good performances in terms of
execution time. Indeed, KENN assumes independence between the logical rules,
allowing for scalable inclusion of the underlying knowledge. However, the
assumption is often violated in real scenarios, in particular in the contexts
of relational domains. To deal with this problem, we propose a strategy that
consists of adding multiple logical layers inside the model. We provide proof
of the efficacy of this strategy in a simple scenario with two logical rules.
Additionally, we tested this idea on Citeseer, a dataset for Collective
Classification [31], showing that the additional layers improve the
performance of the model. Moreover, the experiments on this dataset provide a
comparison between KENN and two other approaches: Semantic Based
Regularization (SBR) [9] and Relational Neural Machines (RNM) [24].

##  2 Related works

Many previous works attempt to combine learning models with logical knowledge.
Among them there is Statistical Relational Learning (SRL), a subfield of
Machine Learning that aims at applying statistical methods in domains that
exhibit both uncertainty and relational structure [18]. Generally speaking,
SRL deals with the knowledge either by combining logic rules with
probabilistic graphical models (e.g., Markov Logic Networks [27], and
_Probabilistic Soft Logic_ (PSL) [1]) or by extending logic programming
languages to handle uncertainty (e.g., ProbLog [7]).

The recent achievements of deep learning methods lead to a renewed interest in
another line of research, called Neural-Symbolic Integration, which focuses on
combining neural network architectures with logical knowledge [3]. This can be
achieved in multiple ways depending on the role of the knowledge. For
instance, works like TensorLog [5], Neural Theorem Prover (NTP) [28, 29],
DeepProbLog [23], Neural Logic Machines [11], and NeuralLog [14] focus on the
development of differentiable approaches for reasoning, which can be used in
combination with neural networks. Another line of research comes from methods
like $\partial$ILP [12], [4], and Neural Logic Rule Layer (NLRL) [26]. In
these cases, the goal is to learn general knowledge from the data, either from
scratch or by refining an initial knowledge. Finally, more related to our
purposes, some methods focus on learning in the presence of prior knowledge,
which acts as additional supervision. In this section, we are going to focus
on these types of methods, since KENN falls in this category.

There are mainly two approaches for learning in the presence of prior
knowledge: the first consists of treating logical rules as constraints on the
predictions of the neural network. The problem is reduced to maximize the
satisfiability of the constraints and can be efficiently tackled by adding a
regularization term in the Loss function which penalizes the unsatisfaction of
the constraints. The second approach is to modify the neural network by
injecting the knowledge into its structure.

Two notable examples of regularization approaches are _Logic Tensor Network_
(LTN) [32] and _Semantic Based Regularization_ (SBR) [9]. Both methods
maximize the satisfaction of the constraints, expressed as FOL formulas, under
a fuzzy logic semantic. A similar strategy is employed also by _Semantic Loss
Function_ [34], but instead of relying on fuzzy logic, it optimizes the
probability of the rules being true. Nevertheless, this approach is restricted
to propositional logic. [13] introduces DL2. Nonetheless, it can be used only
in the context of regression tasks, where the predicates correspond to
comparison constraints (e.g. $=$, $\neq$, $\leq$). MultiplexNet [16] also uses
comparison constraints, but it is not restricted to them. In this case, the
interest is in safety-critical domains where complete satisfaction of the
constraints is desirable and the knowledge is expressed in disjunctive normal
form (DNF). [25] also proposes a method that regularizes the Loss, but they
focus on a specific task of Natural Language Processing. Their approach
differs from the others because it makes use of adversarial examples to
calculate the regularization term. Finally, in [33] the regularization term is
calculated from the unlabelled data.

In [17], a distillation mechanism is used to inject FOL rules. While this
approach seems different from the previous ones, we argue that it can be also
considered as a method based on regularization, since a teacher network (which
encodes the rules) is used to regularize the Loss applied to a student
network. Finally, in [8], a new technique is defined which adds the knowledge
at the level of the labels by modifying them after each training step.

Model-based approaches work pretty differently from the methods seen so far.
The main difference lies in the way logic formulas are used: the knowledge
becomes part of the classifier. Instead, approaches based on regularization
force the constraints satisfaction solely at training time. As a consequence,
there are no guarantees that they will be satisfied at inference time as well.
A way to reduce such a problem is to enforce the constraints not only during
training but also at inference time [30]. However, notice that in this case
the time complexity of inference is increased since the back-propagation needs
to be used even in this case. On the other hand, methods that inject knowledge
directly into the model structure are naturally capable of enforcing the
knowledge at inference time.

Another possible advantage of model based methods is the possibility to learn
a weight that codifies the importance of a logical rule directly from the
data. This is no possible at all with methods based on regularization, since
the logical formulas are directly codified inside the Loss function. As a
consequence, if some rule weight is available, it must be added as a hyper-
parameter.

Among the few methods that admit learnable weights associated with the logic
formulas there is KENN. This makes it suitable for scenarios where the given
knowledge contains errors or when rules are softly satisfied in the real world
but it is not known in advance the extent to which they are correct. KENN
injects knowledge on top of the NN model through an additional layer which
increases the satisfaction of the constraints in a fuzzy logic semantic.

Another approach is provided by Li and Srikumar who recently proposed a method
that codifies the logical constraints directly into the neural network model
[20]. However, they restrict the rules to implications with exactly one
consequent and they do not provide the possibility to learn clause weights,
which in their system are added as hyper-parameters.

Going in the same direction, [24] proposed Relational Neural Networks (RNM).
RNM can be also inserted in the set of approaches that add the logic directly
into the model and, as the best of our knowledge, it is the only method other
than KENN which is capable of integrating logical knowledge with a neural
network while learning the clause weights. RNM integrates a neural network
model with a FOL reasoner. This is done in two stages: in the first one, the
NN is used to calculate initial predictions for the atomic formulas; in the
second stage a graphical model is used to represent a probability distribution
over the set of atomic formulas. To obtain the final predictions a Maximum a
Posteriori (MAP) estimation is performed, finding the most probable assignment
to the grounded atoms given the output of the NN and the set of constraints.
At a high-level RNM approach is similar to KENN, since in both cases a NN
makes initial predictions and a post elaboration step is applied to such
predictions to provide the final classification. However, RNM requires to
solve an optimization problem at inference time and after each training step.
This has the advantage of considering all the logical rules together at the
same time at the expense of an increased computational effort. Contrary, in
KENN each rule is considered separately from the others, and the second stage
is directly integrated inside the model as a differentiable function that can
be trained end-to-end with the NN. However, with this strategy there could be
some contradictory changes when combining multiple clauses with the same
predicates. We will further analyze this aspect in Section 3.4, proposing a
strategy to handle this limitation. Moreover, in Section 4, we analyze this
strategy empirically.

##  3 Knowledge Enhanced Neural Networks

We define the prior knowledge in terms of formulas of a function-free first
order language $\mathcal{L}$. Its signature is defined with a set of domain
constants $\mathcal{C}\triangleq\\{a_{1},a_{2},...a_{m}\\}$ and a set of
predicates $\mathcal{P}\triangleq\\{P_{1},P_{2}...P_{q}\\}$. In our setting,
predicates can be unary or binary/ Binary predicates can express relations
among pairs of objects in the domain, e.g. $Friends(a,b)$ states that person
$a$ is a friend of $b$. The prior knowledge is defined as a set of clauses:
$\mathcal{K}\triangleq\\{c_{1},c_{2},...c_{r}\\}$. A clause is a disjunction
of literals, each of which is a possibly negated atom:
$c\triangleq\bigvee\limits_{i=1}^{k}l_{i}$, where $k$ is the number of
literals in $c$ and $l_{i}$ is the $i^{th}$ literal. We assume that there are
no repeated literals. Since we are interested in representing only general
knowledge, the literals do not contain any constant, only variables that are
assumed to be universally quantified. If the predicate is binary, the two
variables are $x$ and $y$, otherwise only $x$. When an entire clause contains
only variable $x$ (i.e., only unary predicates), we call it _unary_.
Similarly, if it contains both $x$ and $y$ we call it _binary_ 111We restrict
to the case where clauses contain at most two variables.

As an example, the clause $\lnot Smoker(x)\lor Cancer(x)$ is unary and states
that all smokers have also cancer (notice that the clauses are not assumed to
be hard constraints). Instead, the clause

$\lnot Smoker(x)\lor\lnot Friends(x,y)\lor Smoker(y)$ (1)

is binary. It states that if a person $x$ is a smoker and he is a friend of
another person $y$, then $y$ is also a smoker. We will use extensively this
clause in the remaining of the paper, referring to it as $c_{SF}$.

We define the grounding of a unary clause $c$, denoted by $c[a]$, as the
clause obtained by substituting the $x$ variable with constant $a$. Similarly,
if $c$ is binary, its grounding $c[a,b]$ is obtained by substituting $x$ and
$y$ with $a$ and $b$ respectively. For instance, the grounding $c_{SF}[a,b]$
of the clause defined in Eq.1 correspond to $\lnot Smoker(a)\lor\lnot
Friends(a,b)\lor Smoker(b)$.

###  3.1 KENN architecture

Suppose we have a NN for a classification task which takes as input a matrix
$\mathbf{x}\in\mathbb{R}^{d\times n}$ containing $n$ features for $d$ samples,
and returns an output $\mathbf{y}\in[0,1]^{d\times q}$ which contains the
predictions for $q$ classes corresponding to the $q$ predicates. A prior
knowledge $\mathcal{K}$ is also provided. It can be used by KENN to improve
the predictions of the NN.

Figure 1(left) shows a high-level overview of KENN where a residual layer,
called _Knowledge Enhancer_ (KE), is inserted between the NN and the final
activation function. The role of KE is to revise the final predictions
returned by the NN in order to increase the truth value of each clause
$c\in\mathcal{K}$. It does so by calculating a residue $\bm{\delta}$, a matrix
that is added to the predictions of the NN.

Figure 1: Model architecture. Left: KENN model overview. Right: Knowledge
Enhancer.

The KE works in the pre-activations space, i.e. on $\mathbf{z}$, and the
activation function ($\sigma$) is called later. In order for KENN to work, the
activation function must be monotonic and return values in the range $[0,1]$
222For more details on why the KE is applied on pre-activations, please refer
to [6]. Since both NN and KE are differentiable, the entire architecture is
differentiable end-to-end, making it possible to apply back-propagation
algorithm on the whole model. Figure 1(right) shows the architecture of KE,
which calculates the residual matrix $\bm{\delta}$.

More in details, for each clause $c\in\mathcal{K}$, the KE contains a
submodule, the _Clause Enhancer_ (CE), which proposes the changes
$\bm{\delta}_{c}$ to be applied on the NN’s preactivations in order to
increase the satisfaction of $c$. Indeed, the CE computes a soft
differentiable approximation of a function called _t-conorm boost function_
(TBF). Intuitively, a TBF is a function
$\phi:\mathbb{R}^{k}\to\mathbb{R}_{+}^{k}$ that proposes the changes to be
applied on the pre-activations $\mathbf{z}$ of $k$ truth values, such that
$\bot(\sigma(\mathbf{z}+\phi(\mathbf{z})))\geq\bot(\sigma(\mathbf{z}))$, where
$\bot:[0,1]^{k}\to[0,1]$ is a t-conorm function, used in fuzzy logic to
represent the semantics of the disjunction operator 333In [6], function $\phi$
is called $\delta$. Here we changed the name to avoid confusions with its
output which is also referred as $\delta$. In [6] it has been defined the
function

$\phi(\mathbf{z})_{i}=\left\\{\begin{array}[]{ll}1&\mbox{if
$i=\operatorname*{\arg\\!max}_{j=1}^{n}z_{j}$}\\\
0&\mbox{otherwise}\end{array}\right.$ (2)

and proved that such a function is the optimal TBF for the Gödel t-conorm.
KENN employs the softmax function as a continuous and differentiable
approximation of $\phi$.

The $\bm{\delta}_{c}$ matrices are combined linearly inside the KE to obtain
the final change $\bm{\delta}$ to be applied on the NN’s predictions, and
finally the $\bm{\delta}$ is summed to the initial pre-activations
$\mathbf{z}$ and passed to the activation function:

$\begin{split}y_{P(a)}&=\sigma\bigl{(}z_{P(a)}+\sum_{\begin{subarray}{c}c\in\mathcal{K}\\\
P(x)\in c\end{subarray}}w_{c}\cdot\delta_{c[a],P(a)}\bigr{)}\end{split}$ (3)

where $w_{c}$ is the _clause weight_ , $P(a)$ a grounded atom, $y_{P(a)}$ its
final prediction, and $z_{P(a)}$ the NN’s pre-activations. Finally,
$\delta_{c[a],P(a)}$ is the change applied to $P(a)$ based on the grounded
clause $c[a]$:

$\delta_{c[a],P(a)}=\begin{cases}\phi(\mathbf{z}_{c})_{P(a)}&\mbox{if $P(a)\in
c[a]$}\\\ -\phi(\mathbf{z}_{c})_{\lnot P(a)}&\mbox{if $\neg{P(a)}\in
c[a]$}\end{cases}$ (4)

where $\mathbf{z}_{c}$ are the pre-activations of literals of $c$.

Note that applying a linear combination of the $\bm{\delta}_{c}$ matrices can
be done under the assumption of independence between the clauses. When
multiple clauses share common predicates the changes proposed by KENN could
only partially improve the satisfaction of the knowledge. We will further
analyze this problem in Section 3.4.

###  3.2 Extending KENN for relational domains

In the architecture defined so far, the groundings involve a single object and
$\mathbf{z}$ is defined as a matrix, where columns represent predicates and
rows constants. Fig. 2(left) introduces the representation of $\mathbf{z}$: it
is defined as a matrix such that the element $z_{ij}$ contains the pre-
activation of $P_{j}(a_{i})$, with $P_{j}$ the $j^{th}$ predicate and $a_{i}$
the $i^{th}$ constant.

Figure 2: The representation of NN’s final pre-activations. Left: unary case.
Right: representation of relational data. Pre-activations are represented as
integers instead of reals to simplify the figure.

Note that this kind of representation is common when working with neural
networks since the columns (predicates) correspond to the labels and the rows
(groundings) to the samples. An important aspect of this representation lies
in the fact that each grounded atom can be found in the matrix exactly one
time. This allows to parallelize computations of Equation 3 since a grounded
clause involves only atoms in the same row, and each row can be managed in
parallel inside a GPU. This can be done only if the same atom does not appear
in multiple rows, since the changes are applied independently to each row and
are not aggregated together. This property always holds with unary clauses.

To represent relational data, we extend KENN with an extra matrix
$\mathbf{z}_{B}$, which contains the binary predicates’ pre-activations. For
uniformity of notation we use $\mathbf{z}_{U}$ to denote the unary matrix
$\mathbf{z}$ of the not relational KENN. Matrix $\mathbf{z}_{B}$ contains one
row for every pair of objects we are interested in and a column for each
binary predicate. Fig. 2(right) shows this representation using the classical
Smoker-Friends-Cancer example, where the domain is composed of three constants
(persons) $\mathcal{C}=\\{a,b,c\\}$, the unary predicates are $S$ and $C$ (for
$Smoker$ and $Cancer$), and a binary predicate $F$ (for $Friends$). The blue
box shows the graph representation with nodes and edges labelled with pre-
activation of unary and binary predicates respectively. The grey box shows the
corresponding matrix representation used by KENN. Notice that it is not
required that the entire graph is computed by the NN. For instance, in the
experiments on Citeseer, the $Cite$ predicate is provided directly as a
feature (see Section 4).

The architecture of KENN for relational domains is very similar to the
architecture of traditional KENN of Fig. 1, with KE substituted by a
_Relational KE_ (RKE). From a high level perspective, the RKE differs from the
traditional KE on the amount of inputs and outputs. As seen before, in the
relational case the pre-activations are divided in two different matrices
($\mathbf{z}_{U}$ and $\mathbf{z}_{B}$) and, as a consequence, also the
$\bm{\delta}$ matrix and predictions $\mathbf{y}$ are now splitted in unary
and binary matrices ($\bm{\delta}_{U}$ and $\bm{\delta}_{B}$ for the residues,
$\mathbf{y}_{U}$ and $\mathbf{y}_{B}$ for the final predictions).

The RKE has the same role as the KE in the unary case. However, it is capable
to consider also binary predicates. When binary knowledge is available,
additional steps are required since the independence between object can not be
assumed anymore.

Let $\mathcal{K}_{U}$ be the set of unary clauses and $\mathcal{K}_{B}$ the
set of binary clauses. The prior knowledge is now defined as
$\mathcal{K}=\mathcal{K}_{U}\cup\mathcal{K}_{B}$. The idea is to apply the KE
to these two sets separately. Equation 3 can be decomposed using the new
defined partition of the knowledge:
$y_{A}=\sigma\left(z_{A}+\sum_{c\in\mathcal{K}_{U}[\mathcal{C}]}w_{c}\cdot\delta_{c,A}+\sum_{c\in\mathcal{K}_{B}[\mathcal{C}]}w_{c}\cdot\delta_{c,A}\right)$,
where $A$ is a grounded atom (i.e. $P(a)$ or $P(a,b)$, depending on the arity
of $P$). We define $\bm{\delta}_{\mathcal{K}_{U}}$ as the changes deriving
from unary clauses:

$\delta_{\mathcal{K}_{U},P(a)}=\sum_{\begin{subarray}{c}c\in\mathcal{K}_{U}\\\
P(x)\in c\end{subarray}}w_{c}\cdot\delta_{c[a],P(a)}$ (5)

Similarly, $\bm{\delta}_{\mathcal{K}_{B}}$ are the changes calculated from
$\mathcal{K}_{B}$.

Notice that the approach defined so far can be directly applied to the unary
knowledge $\mathcal{K}_{U}$ to calculate $\bm{\delta}_{U}$ since traditional
KE can manage unary knowledge. Indeed, internally the RKE contains a standard
KE which manages the unary clauses.

We need to define a strategy to deal with binary clauses. Indeed, when a
clause $c$ contain two variables, a grounding of a unary predicate may occur
in multiple groundings of $c$. For instance, consider the clause of Eq. 1. The
two groundings $c_{SF}[a,b]$ and $c_{SF}[b,c]$ share a common grounded atom:
$Smoker(b)$. For this reason, when dealing with the predictions of a unary
predicate in a relational domain, we need to account for such repetitions:

$\delta_{\mathcal{K}_{B},P(a)}=\sum_{b\neq
a}\Big{(}\sum_{\begin{subarray}{c}c\in\mathcal{K}_{B}\\\ P(x)\in
c\end{subarray}}w_{c}\cdot\delta_{c[a,b],P(a)}+\sum_{\begin{subarray}{c}c\in\mathcal{K}_{B}\\\
P(y)\in c\end{subarray}}w_{c}\cdot\delta_{c[b,a],P(a)}\Big{)}$ (6)

Putting all together, the predictions $y_{P(a)}$ for a grounded unary
predicate $P$ are defined as:

$y_{P(a)}=\sigma(z_{P(a)}+\delta_{\mathcal{K}_{U},P(a)}+\delta_{\mathcal{K}_{B},P(a)})$
(7)

The predictions for a binary predicate $R$ are easier to be computed since
they can be found only in binary clauses and any possible grounding of $R$ can
be found in only one corresponding grounding of each clause
$y_{R(a,b)}=\sigma(z_{R(a,b)}+\delta_{\mathcal{K}_{B},R(a,b)})$, with

$\delta_{\mathcal{K}_{B},R(a,b)}=\sum_{\begin{subarray}{c}c\in\mathcal{K}_{B}\\\
R(x,y)\in c\end{subarray}}w_{c}\cdot\delta_{c[a,b],R(a,b)}$ (8)

###  3.3 Time complexity

Here we analyze the time complexity of an RKE layer with respect to domain
size $m$, number of predicates $|\mathcal{P}|$, and number of rules
$|\mathcal{K}|$. We also assume the maximum number $L$ of literals in a clause
to be a small constant.

Let us first analyze the time complexity for calculating the $\delta_{c[a]}$
used in Equations 5, and 6. Each $\delta_{c[a],P(a)}$ can be calculated in
time $O(1)$ (see Equation 4). Computing $\delta_{c[a]}$ also requires constant
time. The sum of Equation 5 require time $O(|\mathcal{K}|)$, which is the time
necessary to compute $\delta_{\mathcal{K}_{U},P(a)}$. Note that neural
networks are usually run on GPUs, where the computations can be parallelized.
Assuming enough parallel processes ($|\mathcal{K}|$ in this case), a sum can
be performed in a time logarithmic with respect to the number of addends, and
complexity for $\delta_{\mathcal{K}_{U},P(a)}$ becomes
$O(log(|\mathcal{K}|))$. Finally, Equation 5 needs to be calculated for all
the grounded unary predicates $P(a)$, for a total time of
$O(m\cdot|\mathcal{P}|\cdot|\mathcal{K}|)$ in a single process, and
$O(log(|\mathcal{K}|))$ with multiple parallel processes (each of the grounded
atom can be considered independently from the others). With a similar
reasoning, we found the time complexity of Equations 6 and 8 to be
$O(m^{2}\cdot|\mathcal{P}|\cdot|\mathcal{K}|)$. Note that with enough parallel
processes we can compute all the deltas in $O(log(m)+log(|\mathcal{K}|))$.

###  3.4 Treatment of dependencies among the rules

In the previous section we showed the efficacy of the method in terms of
execution time, which can be achieved thanks to the assumption of
independence. However, when this assumption is violated, KENN does not provide
any guarantees on the satisfaction of the knowledge. As an example, suppose
that we have two grounded clauses $c_{1}:\lnot A\lor B$ and $c_{2}:\lnot B\lor
C$ with their respective clause enhancers CE1 and CE2, where $A$, $B$ and $C$
are grounded unary or binary predicates. The atom $B$ appears in both clauses
with opposite signs. Since the CE increase the highest literal value (see Eq.
2), if $A<B$ 444With an abuse of notation, we use the atoms symbols to refer
also to their truth value and $C<\lnot B$, then CE1 increases $B$ and CE2
decreases it. As a consequence, the satisfaction of only one between $c_{1}$
and $c_{2}$ is increased. Furthermore, the satisfaction of the entailed clause
$\lnot A\lor C$ is also not improved.

For any grounded atom $G$, lets define $G^{(0)}$ as its initial prediction and
$G^{(i)}$ as the prediction of the $i^{th}$ KE layer. Moreover, suppose that
all KEs share the same clause weights $w_{1}$ and $w_{2}$ (for $c_{1}$ and
$c_{2}$ respectively). From Equations 3 and 4 we can derive
$B^{(1)}=B^{(0)}+w_{1}-w_{2}$, and $\lnot B^{(1)}=\lnot B^{(0)}+w_{2}-w_{1}$.
If $w_{1}\geq w_{2}$, then $A^{(1)}=A^{(0)}<B^{(0)}\leq B^{(1)}$. As a
consequence, the first rule will increase again $B$ even at the next KE layer.
On the other hand, the value of $\lnot B$ is reduced, which means that there
is an increased chance for $C^{(1)}>\lnot B^{(1)}$, which would solve the
problem since CE2 would increase $C$ instead of $\lnot B$. Notice that, since
the weights are the same at each level, it is always true that $\lnot
B^{(i+1)}\leq\lnot B^{(i)}$, meaning that with enough KE layers both clauses’
satisfaction will be increased (and as a consequence, also their entailments).

The problem analyzed in this section becomes even more relevant in relational
domains since in these contexts an atom can be shared not only by multiple
clauses but also by different groundings of the same clause (for instance, in
$c_{SF}[a,b]$ and $c_{SF}[b,c]$). For this reason, in these contexts stacking
multiple RKEs is recommended (more details in Section 4.2).

##  4 Evaluation of the model

In this section, the relational extension of KENN is tested on the task of
Collective Classification: given a graph, we are interested in finding a
classification for its nodes using both features of the nodes (the objects)
and the information coming from the edges of the graph (relations between
objects) [31].

In Collective Classification, there are two different learning tasks:
inductive and transductive learning. In inductive learning, there are two
separate graphs, one for training and the other for testing. On the contrary,
in transductive learning, there is only one graph that contains nodes both for
training and testing. In other words, in inductive learning, there are no
edges between nodes for training and testing, while in transductive learning
there are. The tests have been performed on both tasks to analyze the behavior
of KENN in the contexts of relational domains. In particular, we tested KENN
with a varying number of KEs layers to validate the proposal of Section
3.4.555Source code of the experiments are available on
https://github.com/rmazzier/KENN-Citeseer-Experiments .

###  4.1 Experimental setup

We followed the evaluation methodology of [24], where the experiments have
been carried out on Citeseer dataset [22] using SBR and RNM. The Citeseer
dataset used in the evaluation is a citation network: the graph’s nodes
represent documents and the edges represent citations. The nodes’ features are
bag-of-words vectors, where an entry is zero if the corresponding word of the
dictionary is absent in the document, and one if it is present. The classes to
be predicted represent possible topics for a document. The dataset contains
3312 nodes that must be classified in 6 different classes: AG, AI, DB, IR, ML,
and HCI. The classification is obtained from the 3703 features of the nodes,
with the addition of the information coming from the citations (4732 edges).

We use the same NN and knowledge as in [24], allowing for the comparison with
SBR and RNM. The NN is a dense network with 3 hidden layers, each with 50
hidden nodes and ReLU activation function. The knowledge consists of six rules
obtained by substituting the topic $T$ in $\lnot T(x)\lor\lnot Cite(x,y)\lor
T(y)$ with all the classes, and they codify the idea that papers cite works
that are related to them.

The training set dimension is changed multiple times to evaluate the efficacy
of the three methods on the varying of training data. More precisely, tests
have been conducted by selecting 10%, 25%, 50%, 75%, and 90% of nodes for
training. For each of these values, the training and evaluation were performed
100 times, each with a different split of the dataset. At each run the
training set is created by selecting random nodes of the graph, with the
constraints that the dataset must be balanced.

###  4.2 Results

Figure 3: Accuracies of KENN at the varying of KEs layers.

Figure 3 shows the test accuracies obtained by KENN while increasing the
number of KEs layers, starting from $0$ (corresponding to the NN accuracy) up
to 6. Note that, for each line in the figure, there is a surrounding border
corresponding to a 99% confidence interval. To calculate the intervals, we
assumed the distribution of improvements obtained by the injection of the
logical rules to be a normal distribution (see figures in Appendix B and C).
We also computed the p-values for each setting, assuming as null Hypothesis
that the distribution of accuracies of the NN is the same as KENN. Since the
number of runs, as well as the obtained improvements, are quite high, the
resulting p-values are very small. For this reason, we can safely reject the
null Hypothesis, and we are very confident that the improvements given by KENN
do not depend on the random initialization of the models’ parameters or the
specific choices of the splits of the dataset.

More in detail, we found p-values in the range from 8.2e-42 to 1.6e-09 in the
inductive case, and from 53e-72 to 2.1e-23 for the transductive one. The only
exception is with 90% of the samples in the inductive scenario where the
p-value is 0.35. This is because the improvements over the NN are very small.
Indeed, in both learning paradigms, the effect of the knowledge is reduced
when the amount of available data is larger. This behavior is consistent with
the simple intuition that, when the training data is scarce, the usage of
knowledge should bring higher benefits.

A more important result coming from these experiments is the fact that in all
cases adding a new KE layer does not reduce the test accuracy. On the
contrary, most of the time the metric is increased until a certain number of
layers is reached, and after that, the accuracy stabilizes. This behavior is
in line with the discussion of Section 3.4 and confirms the efficacy of the
proposed strategy to deal with the violation of independece assumption.

Finally, Figure 3 provide also a measure of the amount of information carried
out by the knowledge. For instance, consider blue and yellow lines,
corresponding to a training set with 25% and 50% of the samples, respectively.
In the inductive scenario, the accuracy obtained with 25% with the addition of
the knowledge is almost the same as the standard NN with 50% of the data (even
higher in the transductive scenario). In this case, adding the knowledge has
the same effect of doubling up the training data! Indeed, one of the main
motivations behind Neural-Symbolic Integration consists in reducing the
required amount of training samples since collecting labeled data is costly in
practice.

###  4.3 Comparison with other NeSy frameworks

Table 1 shows a comparison of KENN with SBR and RNM. We used the results of
KENN with 3 KEs since more layers do not provide a significant advantage (see
Section 4.2).

Table 1: Improvements in terms of accuracy on inductive and transductive learning.  Inductive | Transductive
---|---
% Tr | SBR | RNM | KENN | SBR | RNM | KENN
10 | 0.005 | 0.040 | 0.052 | 0.063 | 0.068 | 0.110
25 | 0.008 | 0.035 | 0.044 | 0.062 | 0.068 | 0.074
50 | 0.005 | 0.019 | 0.036 | 0.052 | 0.058 | 0.064
75 | 0.002 | 0.009 | 0.021 | 0.056 | 0.058 | 0.057
90 | 0.003 | 0.009 | 0.001 | 0.054 | 0.054 | 0.043

As we can see from the table, in the inductive case SBR produces much lower
improvements compared to the other two methods. Note that these results are in
line with previous results obtained on VRD dataset, where another
regularization approach (LTN) was compared with KENN [6]. Indeed, the results
obtained in both VRD and Citeseer suggest better performances of model-based
approaches as compared to the ones based on regularization. Note that methods
based on regularization of the loss do not impose the knowledge at inference
time. In the transductive scenario, the situation is different and SBR behaves
similarly to the other two. Indeed, in this case, citations between training
and test nodes are available and there is no distinction between training and
inference.

Finally, the results suggest that KENN is particularly useful when the
training data available is scarce. On the contrary, when data is abundant, our
results tend to degrade faster than RNM and SBR. However, the greatest
advantage of KENN over other architectures is its scalability. This is
confirmed by the comparison of the execution times of the three methods: we
found KENN to be very fast as compared to the other two methods with an
average of 7.96s required for a single run, as compared to the NN which
requires 2.46s (on average of 1.83 seconds for each KE layer). A run of SBR
cost 87.36s (almost 11 times slower than KENN), while RNM required 215.69s per
run (27 times slower) 666All the experiments have been run on the same
architecture, an NVIDIA Tesla v100.

##  5 Conclusions

KENN is a NeSy architecture that injects prior logical knowledge inside a
neural network by stacking a residual layer on its top. In [6], it proved to
be able to effectively inject knowledge in the context of multi-label
classification tasks. In this work, we extended KENN for relational domains,
where the presence of both unary and binary predicates doesn’t allow for the
usage of the simple tabular representation of the data used in the previous
version of the framework. Moreover, we propose a strategy to deal with the
violation of the independence assumption made by KENN. The experiments on
Citeseer show the effectiveness of this strategy, obtaining statistically
relevant improvements over the NN performances, meaning that KENN can
successfully inject knowledge even in the presence of relational data.
Finally, KENN provided quality results also in comparison with other two NeSy
frameworks. In particular, the large difference in performances between
KENN/RNM and SBR provides additional evidence in support of model-based
approaches in comparison to regularization ones, with KENN the best option in
terms of scalability.

## References

  * Bach et al. [2017] S. H. Bach, M. Broecheler, B. Huang, and L. Getoor.  Hinge-loss Markov random fields and probabilistic soft logic.  _Journal of Machine Learning Research_ , 18(109):1–67, 2017.
  * Bahdanau et al. [2014] D. Bahdanau, K. Cho, and Y. Bengio.  Neural machine translation by jointly learning to align and translate.  _arXiv preprint arXiv:1409.0473_ , 2014.
  * Besold et al. [2017] T. R. Besold, A. S. d’Avila Garcez, S. Bader, H. Bowman, P. M. Domingos, P. Hitzler, K. Kühnberger, L. C. Lamb, D. Lowd, P. M. V. Lima, L. de Penning, G. Pinkas, H. Poon, and G. Zaverucha.  Neural-symbolic learning and reasoning: A survey and interpretation.  _CoRR_ , abs/1711.03902, 2017.  URL http://arxiv.org/abs/1711.03902.
  * Campero et al. [2018] A. Campero, A. Pareja, T. Klinger, J. Tenenbaum, and S. Riedel.  Logical rule induction and theory learning using neural theorem proving.  _arXiv preprint arXiv:1809.02193_ , 2018.
  * Cohen [2016] W. W. Cohen.  Tensorlog: A differentiable deductive database.  _arXiv preprint arXiv:1605.06523_ , 2016.
  * Daniele and Serafini [2019] A. Daniele and L. Serafini.  Knowledge enhanced neural networks.  In A. C. Nayak and A. Sharma, editors, _PRICAI 2019: Trends in Artificial Intelligence_ , pages 542–554, Cham, 2019. Springer International Publishing.  ISBN 978-3-030-29908-8.
  * De Raedt et al. [2007] L. De Raedt, A. Kimmig, and H. Toivonen.  Problog: A probabilistic prolog and its application in link discovery.  In _IJCAI_ , volume 7, pages 2462–2467. Hyderabad, 2007.
  * Detassis et al. [2021] F. Detassis, M. Lombardi, and M. Milano.  Teaching the old dog new tricks: Supervised learning with constraints.  In _Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021_ , pages 3742–3749. AAAI Press, 2021.  URL https://ojs.aaai.org/index.php/AAAI/article/view/16491.
  * Diligenti et al. [2017] M. Diligenti, M. Gori, and C. Saccà.  Semantic-based regularization for learning and inference.  _Artif. Intell._ , 244:143–165, 2017.
  * Donadello [2018] I. Donadello.  _Semantic Image Interpretation - Integration of Numerical Data and Logical Knowledge for Cognitive Vision_.  PhD thesis, Trento Univ., Italy, 2018.
  * Dong et al. [2019] H. Dong, J. Mao, T. Lin, C. Wang, L. Li, and D. Zhou.  Neural logic machines.  _arXiv preprint arXiv:1904.11694_ , 2019.
  * Evans and Grefenstette [2018] R. Evans and E. Grefenstette.  Learning explanatory rules from noisy data.  _Journal of Artificial Intelligence Research_ , 61:1–64, 2018.
  * Fischer et al. [2019] M. Fischer, M. Balunovic, D. Drachsler-Cohen, T. Gehr, C. Zhang, and M. Vechev.  Dl2: Training and querying neural networks with logic.  In _International Conference on Machine Learning_ , pages 1931–1941, 2019.
  * Guimarães and Costa [2021] V. Guimarães and V. S. Costa.  Neurallog: a neural logic language.  _CoRR_ , abs/2105.01442, 2021.  URL https://arxiv.org/abs/2105.01442.
  * Hinton et al. [2012] G. Hinton, L. Deng, D. Yu, G. Dahl, A.-r. Mohamed, N. Jaitly, A. Senior, V. Vanhoucke, P. Nguyen, B. Kingsbury, et al.  Deep neural networks for acoustic modeling in speech recognition.  _IEEE Signal processing magazine_ , 29, 2012.
  * Hoernle et al. [2021] N. Hoernle, R. Karampatsis, V. Belle, and K. Gal.  Multiplexnet: Towards fully satisfied logical constraints in neural networks.  _CoRR_ , abs/2111.01564, 2021.  URL https://arxiv.org/abs/2111.01564.
  * Hu et al. [2016] Z. Hu, X. Ma, Z. Liu, E. H. Hovy, and E. P. Xing.  Harnessing deep neural networks with logic rules.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany, Volume 1: Long Papers_. The Association for Computer Linguistics, 2016\.  ISBN 978-1-945626-00-5.  URL http://aclweb.org/anthology/P/P16/P16-1228.pdf.
  * Koller et al. [2007] D. Koller, N. Friedman, S. Džeroski, C. Sutton, A. McCallum, A. Pfeffer, P. Abbeel, M.-F. Wong, D. Heckerman, C. Meek, et al.  _Introduction to statistical relational learning_.  MIT press, 2007.
  * Krizhevsky et al. [2012] A. Krizhevsky, I. Sutskever, and G. E. Hinton.  Imagenet classification with deep convolutional neural networks.  In _Proceedings of the 25th International Conference on Neural Information Processing Systems - Volume 1_ , NIPS’12, pages 1097–1105, USA, 2012\. Curran Associates Inc.  URL http://dl.acm.org/citation.cfm?id=2999134.2999257.
  * Li and Srikumar [2019] T. Li and V. Srikumar.  Augmenting neural networks with first-order logic.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 292–302, Florence, Italy, July 2019. Association for Computational Linguistics.  doi: 10.18653/v1/P19-1028.  URL https://www.aclweb.org/anthology/P19-1028.
  * Lu et al. [2016] C. Lu, R. Krishna, M. S. Bernstein, and F. Li.  Visual relationship detection with language priors.  In _ECCV (1)_ , volume 9905 of _Lecture Notes in Computer Science_ , pages 852–869. Springer, 2016.
  * Lu and Getoor [2003] Q. Lu and L. Getoor.  Link-based classification.  In _Proceedings of the Twentieth International Conference on International Conference on Machine Learning_ , ICML’03, page 496–503. AAAI Press, 2003.  ISBN 1577351894.
  * Manhaeve et al. [2018] R. Manhaeve, S. Dumancic, A. Kimmig, T. Demeester, and L. De Raedt.  Deepproblog: Neural probabilistic logic programming.  In _Advances in Neural Information Processing Systems_ , pages 3749–3759, 2018.
  * Marra et al. [2020] G. Marra, M. Diligenti, F. Giannini, M. Gori, and M. Maggini.  Relational neural machines.  _arXiv preprint arXiv:2002.02193_ , 2020.
  * Minervini and Riedel [2018] P. Minervini and S. Riedel.  Adversarially regularising neural nli models to integrate logical background knowledge.  _arXiv preprint arXiv:1808.08609_ , 2018.
  * Reimann and Schwung [2019] J. N. Reimann and A. Schwung.  Neural logic rule layers.  _arXiv preprint arXiv:1907.00878_ , 2019.
  * Richardson and Domingos [2006] M. Richardson and P. Domingos.  Markov logic networks.  _Mach. Learn._ , 62(1-2):107–136, Feb. 2006.  ISSN 0885-6125.
  * Rocktäschel and Riedel [2016] T. Rocktäschel and S. Riedel.  Learning knowledge base inference with neural theorem provers.  In _Proceedings of the 5th Workshop on Automated Knowledge Base Construction_ , pages 45–50, 2016.
  * Rocktäschel and Riedel [2017] T. Rocktäschel and S. Riedel.  End-to-end differentiable proving.  In _Advances in Neural Information Processing Systems_ , pages 3788–3800, 2017.
  * Saccá et al. [2013] C. Saccá, M. Diligenti, and M. Gori.  Collective classification using semantic based regularization.  In _2013 12th International Conference on Machine Learning and Applications_ , volume 1, pages 283–286. IEEE, 2013.
  * Sen et al. [2008] P. Sen, G. M. Namata, M. Bilgic, L. Getoor, B. Gallagher, and T. Eliassi-Rad.  Collective classification in network data.  _AI Magazine_ , 29(3):93–106, 2008.
  * Serafini and d’Avila Garcez [2016] L. Serafini and A. S. d’Avila Garcez.  Logic tensor networks: Deep learning and logical reasoning from data and knowledge.  _CoRR_ , abs/1606.04422, 2016.
  * Van Krieken et al. [2019] E. Van Krieken, E. Acar, and F. Van Harmelen.  Semi-supervised learning using differentiable reasoning.  _arXiv preprint arXiv:1908.04700_ , 2019.
  * Xu et al. [2018] J. Xu, Z. Zhang, T. Friedman, Y. Liang, and G. Van den Broeck.  A semantic loss function for deep learning with symbolic knowledge.  In J. Dy and A. Krause, editors, _Proceedings of the 35th International Conference on Machine Learning_ , volume 80 of _Proceedings of Machine Learning Research_ , pages 5502–5511, Stockholmsmässan, Stockholm Sweden, 10–15 Jul 2018. PMLR.  URL http://proceedings.mlr.press/v80/xu18h.html.


# Understanding and Improving Encoder Layer Fusion in Sequence-to-Sequence
Learning

Xuebo Liu1 , Longyue Wang2, Derek F. Wong1, Liang Ding3, Lidia S. Chao1 &
Zhaopeng Tu2
1NLP2CT Lab, Department of Computer and Information Science, University of
Macau
2Tencent AI Lab 3The University of Sydney
<EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>Work was done when Xuebo
Liu and Liang Ding were interning at Tencent AI Lab.

###### Abstract

Encoder layer fusion (EncoderFusion) is a technique to fuse all the encoder
layers (instead of the uppermost layer) for sequence-to-sequence (Seq2Seq)
models, which has proven effective on various NLP tasks. However, it is still
not entirely clear why and when EncoderFusion should work. In this paper, our
main contribution is to take a step further in understanding EncoderFusion.
Many of previous studies believe that the success of EncoderFusion comes from
exploiting surface and syntactic information embedded in lower encoder layers.
Unlike them, we find that the encoder embedding layer is more important than
other intermediate encoder layers. In addition, the uppermost decoder layer
consistently pays more attention to the encoder embedding layer across NLP
tasks. Based on this observation, we propose a simple fusion method,
SurfaceFusion, by fusing only the encoder embedding layer for the softmax
layer. Experimental results show that SurfaceFusion outperforms EncoderFusion
on several NLP benchmarks, including machine translation, text summarization,
and grammatical error correction. It obtains the state-of-the-art performance
on WMT16 Romanian-English and WMT14 English-French translation tasks.
Extensive analyses reveal that SurfaceFusion learns more expressive bilingual
word embeddings by building a closer relationship between relevant source and
target embeddings. Source code is freely available at
https://github.com/SunbowLiu/SurfaceFusion.

##  1 Introduction

Sequence-to-Sequence (Seq2Seq) learning (Sutskever et al., 2014) has advanced
the state of the art in various natural language processing (NLP) tasks, such
as machine translation (Bahdanau et al., 2015; Vaswani et al., 2017; Wu et
al., 2019), text summarization (Wang et al., 2019b; Zhang et al., 2020), and
grammatical error correction (Kiyono et al., 2019; Kaneko et al., 2020).
Seq2Seq models are generally implemented with an encoder-decoder framework, in
which a multi-layer encoder summarizes a source sequence into a sequence of
representation and another multi-layer decoder produces the target sequence
conditioned on the encoded representation.

Recent studies reveal that fusing the intermediate encoder layers
(EncoderFusion) is beneficial for Seq2Seq models, such as layer attention
(Bapna et al., 2018), layer aggregation (Dou et al., 2018; Wang et al.,
2019c), and layer-wise coordination (He et al., 2018). Despite its
effectiveness, not much is known about how fusing encoder layer
representations work. The intuitive explanation is that fusing encoder layers
exploits surface and syntactic information embedded in the lower encoder
layers (Belinkov et al., 2017; Peters et al., 2018). However, other studies
show that attending to lower encoder layers (excluding the encoder embedding
layer) does not improve model performance (Domhan, 2018), which is conflicted
with existing conclusions. It is still unclear why and when fusing encoder
layers should work in Seq2Seq models.

This paper tries to shed light upon behavior of Seq2Seq models augmented with
EncoderFusion method. To this end, we propose a novel fine-grained layer
attention to evaluate the contribution of individual encoder layers. We
conduct experiments on several representative Seq2Seq NLP tasks, including
machine translation, text summarization, and grammatical error correction.
Through a series of analyses, we find that the uppermost decoder layer pays
more attention to the encoder embedding layer. Masking the encoder embedding
layer significantly drops model performance by generating hallucinatory (i.e.
fluent but unfaithful to the source) predictions. The encoded representation
of the standard Seq2Seq models (i.e. w/o fusing encoder layers) may not have
enough capacity to model both semantic and surface features (especially at the
encoder embedding layer). We call the problem described above the source
representation bottleneck.

Based on this observation, we simplify the EncoderFusion approaches by only
connecting the encoder embedding layer to softmax layer (SurfaceFusion). The
SurfaceFusion approach shortens the path distance between source and target
embeddings, which can help to learn better bilingual embeddings with direct
interactions. Experimental results on several Seq2Seq NLP tasks show that our
method consistently outperforms both the vanilla Seq2Seq model and the layer
attention model. Extensive analyses reveal that our approach produces more
aligned bilingual word embeddings by shortening the path distance between
them, which confirm our claim.

Our main contributions are as follows:

  * •

We introduce a fine-grained layer attention method to qualitatively and
quantitatively evaluate the contribution of individual encoder layers.

  * •

We demonstrate that the encoder embedding layer is essential for fusing
encoder layers, which consolidates conflicted findings reported by previous
studies.

  * •

We propose a simple yet effective SurfaceFusion approach to directly exploit
the encoder embedding layer for the decoder, which produces more expressive
bilingual embeddings.

##  2 Preliminaries

###  2.1 Sequence-to-sequence learning

Seq2Seq learning aims to maximize the log-likelihood of a target sequence
$\mathbf{y}=\\{y_{1},\dots,y_{J}\\}$ conditioned on a source sequence
$\mathbf{x}=\\{x_{1},\dots,x_{I}\\}$, which is formulated as:
$\mathbf{\hat{y}}=\arg\max\log P(\mathbf{y}|\mathbf{x})$. Typically, Seq2Seq
learning can be implemented as various architectures (Bahdanau et al., 2015;
Gehring et al., 2017; Vaswani et al., 2017; Wu et al., 2019), among which the
Transformer (Vaswani et al., 2017) has advanced the state of the art. Without
loss of generality, we introduce Transformer as the testbed in this paper.
Transformer consists of an encoder $\mathcal{E}$ equipped with $N$ identical
layers to map the source sequence $\mathbf{x}$ into distributed
representations, based on which a decoder $\mathcal{D}$ equipped with $M$
identical layers generates the target sequence $\mathbf{y}$:

$\displaystyle\mathbf{X}^{N}=\mathcal{E}(\mathbf{X}^{0})$
$\displaystyle\stackrel{{\scriptstyle N}}{{\underset{n=1}{:=}}}$
$\displaystyle\textsc{Ffn}\left(\textsc{Att}(\mathbf{X}^{n-1},\mathbf{X}^{n-1},\mathbf{X}^{n-1})\right)$
(1) $\displaystyle\mathbf{Y}^{M}=\mathcal{D}(\mathbf{Y}^{0},\mathbf{X}^{N})$
$\displaystyle\stackrel{{\scriptstyle M}}{{\underset{m=1}{:=}}}$
$\displaystyle\textsc{Ffn}\left(\textsc{Att}\left(\textsc{Att}(\mathbf{Y}^{m-1},\mathbf{Y}^{m-1},\mathbf{Y}^{m-1}),\mathbf{X}^{N},\mathbf{X}^{N}\right)\right)$
(2)

where $\mathbf{X}^{0}$ denotes the sum of the word embeddings
$\mathbf{X}_{\mathrm{emb}}$ and position embeddings
$\mathbf{X}_{\mathrm{pos}}$ of $\mathbf{x}$, $\mathbf{Y}^{0}$ denotes that of
the shifted right $\mathbf{y}$, $\textsc{Ffn}(\cdot)$ denotes a position-wise
feed-forward network, and $\textsc{Att}(\cdot)$ denotes a multi-head dot-
product attention network with three arguments–query, key and value. Residual
connection (He et al., 2016) and layer normalization (Ba et al., 2016) are
used in each sub-layer, which are suppressed in Equation 1 and 2 for clarity.
Finally, the output representation $\mathbf{Y}^{M}$ of the decoder is
projected into the probability $P(\mathbf{y}|\mathbf{x})$, which is optimized
during model training.

###  2.2 Experimental setup

To validate the universality of source representation bottleneck in Seq2Seq
models, we conducted experiments on three representative tasks, which vary
from the distance between input and output domains and the scale of training
data:

Machine translation takes a sentence in one language as input, and outputs a
semantically-equivalent sentence in another language. We conducted experiments
on three benchmarking datasets: small-scale WMT16 Romanian-English (Ro-En;
0.6M instances), medium-scale WMT14 English-German (En-De; 4.5M instances),
and large-scale WMT14 English-French (En-Fr; 36.0M instances). The tokenized
BLEU score (Papineni et al., 2002) was used for all the translation tasks.

Text summarization takes a long-text document as input, and outputs a short
and adequate summary in the same language. We used the CNN/Daily Mail corpus
(0.3M instances). We evaluated with the standard ROUGE metric (Lin, 2004),
i.e. Rouge-1, Rouge-2, and Rouge-L.

Grammatical error correction takes a sentence with grammatical errors as
input, and outputs a corrected sentence. We used CONLL14 datasets as the
testbed (1.4M instances). The MaxMatch (M2) scores (Dahlmeier & Ng, 2012) were
used for evaluation with precision, recall, and F0.5 values.

The machine translation task has distant input/output domains (i.e. in
different languages), while the other tasks have similar input/output domains
(i.e. in the same language). We used Transformer (Vaswani et al., 2017) as the
Seq2Seq model. Details of the datasets and model training are listed in
Appendix A.1.

##  3 Behavior of EncoderFusion

In this section, we first formulate our research hypothesis of source
representation bottleneck (§3.1) that EncoderFusion expects to solve. In the
following subsections, we propose a fine-grained layer attention model (§3.2)
to validate our hypothesis on well-designed experiments (§3.3).

###  3.1 Source representation bottleneck

Seq2Seq models learn more abstract features with the increase of layer level
(i.e. $\mathbf{X}^{0}\rightarrow\mathbf{X}^{N}$ and
$\mathbf{Y}^{0}\rightarrow\mathbf{Y}^{M}$) (Belinkov et al., 2017). It has
been extensively validated that a reasonable use of both the abstract
representations (at higher-level layers) and the surface representations (at
lower-level layers) is beneficial for various NLP (Lu & Li, 2013; Hu et al.,
2014; Dou et al., 2018; Peters et al., 2018) and CV (Long et al., 2014;
Pinheiro et al., 2016; Lin et al., 2017; Chen et al., 2018a) tasks.

However, the Seq2Seq decoder only takes the abstract representations at
uppermost layer $\mathbf{X}^{N}$ as input (Equation 2), while ignores other
usefully surface representations at other layers $\mathbf{X}^{n}$ ($n<N$).
Although $\mathbf{X}^{N}$ has encoded surface features from low-level
representations through layer-by-layer abstraction and residual connections,
we hypothesize that its limited representation capacity may not sufficiently
model those surface features from lower encoder layers, especially the
embedding layer. We call such an issue as source representation bottleneck.

###  3.2 Fine-grained layer attention

For each decoder layer, layer attention (Bapna et al., 2018; Peters et al.,
2018) assigns normalized scalar weights to all encoder layers, providing a
direct way for evaluating the contributions made by each encoder layer.
However, the capacity of a simple scalar weight is limited, leading to
insufficient evaluation of the contributions.

Motivated by fine-grained attention (Choi et al., 2018) that each element of a
context vector receives an individual attention weight, we propose a fine-
grained layer attention model to combine the advantages of both techniques.
This allows us to more convincingly evaluate the contribution of individual
encoder layer to the model performance. Besides, the nature of fine-grained
attention enables us to give in-depth analyses of the representation power in
§3.3.

Specifically, we replace the layer-agnostic source representation
$\mathbf{X}^{N}$ with the layer-aware representation $\mathbf{S}^{m}$ for each
decoder layer ${\bf Y}^{m}$, which is calculated as:

$\mathbf{S}^{m}=\sum_{n=0}^{N}\hat{\mathbf{w}}^{m,n}\odot\mathbf{X}^{n},\quad\hat{\mathbf{w}}^{m,n}=\left[\hat{w}^{m,n,1},\dots,\hat{w}^{m,n,D}\right],\quad\hat{w}^{m,n,d}=\frac{\exp(w^{m,n,d})}{\sum_{n^{\prime}=0}^{N}\exp(w^{m,n^{\prime},d})}$

where $\odot$ denotes an element-wise multiplication, and $w^{m,n,d}$ denotes
an element in the learnable attention weight
$\mathbf{W}\in\mathds{R}^{M\times(N+1)\times D}$, where $D$ is the
dimensionality of the source representation. When $n=0$, we use the word
embeddings $\mathbf{X}_{\mathrm{emb}}$ without position embeddings as
$\mathbf{X}^{0}$, which has been empirically proved effective. We applied a
regularization technique – DropConnect (Wan et al., 2013) to the attention
weight $\mathbf{W}$ for a stable training, which randomly drops each
$w^{m,n,d}$ with a probability $p$ and divides $\mathbf{W}$ by $1-p$. We set
it to 0.3 for all the experiments.

Table 2 lists the results. The proposed fine-grained layer attention model
consistently outperforms the vanilla Transformer across Seq2Seq tasks,
demonstrating the benefit of fusing surface features at lower-level layers.

Table 1: Results of existing encoder layer fusion methods on the WMT16 Ro-En translation task. Model | BLEU
---|---
Vanilla Transformer | 33.80
Layer aggregation | 34.05
Layer-wise coordination | 34.19
Coarse-grained layer attention | 34.32
Fine-grained layer attention | 34.45

We evaluated several EncoderFusion methods in Table 1, including layer
aggregation (Dou et al., 2018), layer-wise coordination (He et al., 2018), and
coarse-grained layer attention (Bapna et al., 2018). Their results are
respectively 34.05, 34.19, and 34.32, which are all lower than that of fine-
grained layer attention (34.45). Based on these experimental results, we thus
choose fine-grained layer attention as a representative of EncoderFusion in
the following analyses.

###  3.3 Behavior changes across encoder layers

In this section, we investigate whether the surface features at lower encoder
layers (especially the encoder embedding layer) contribute to the model
performance via carefully designed experiments.

(a) Translation: Ro-En

(b) Summarization

(c) Correction

Figure 1: Attention distribution that each decoder layer ($x$-axis) attending
to encoder layers ($y$-axis).

#### Visualization of layer attention

We first visualize the learned layer attention distribution in Figure 1, in
which each weight is the averaged attention weights over all dimensions.
Generally, a higher weight denotes more contribution of an encoder layer to
the corresponding decoder layer.

Clearly, in all tasks higher decoder layers especially the uppermost ones pay
more attention to the encoder embedding layer, which indicates that the
surface representations potentially bring some additional useful features to
the model performance. Voita et al. (2019); Wang & Tu (2020) reveal that the
upper layers of decoder are responsible for the translation part while the
lower layers for the language modeling part. Similarly, our results show that
surface representations might play an important role in learning to translate
source tokens.

Among the Seq2Seq models, there are still considerable differences in the
attention heatmaps. In the summarization model, almost all decoder layers
focus more on the encoder embedding layer, while in the other two models the
intermediate decoder layers pay more attention to the higher-level encoder
layers. This is consistent with the findings of Rothe et al. (2019), in which
they reveal that the summarization task, as a typical extractive generation
task, tends to use more surface features to generate extractive summaries. In
contrast, both machine translation and error correction tasks require a large
amount of syntactic and semantic information, which are generally embedded in
higher-level encoder layers (Peters et al., 2018).

However, we still cannot conclude that source representation bottleneck does
exist in Seq2Seq models, since the surface features might act as a noise
regularizer to improve the robustness of encoder output representations. To
dispel the doubt, we further design two experiments to directly evaluate the
effectiveness of surface features at the encoder embedding layer.

#### Contribution of individual encoder layer

In this experiment, we quantitatively analyze the behaviors change of a
trained Seq2Seq model when masking a specific encoder layer (i.e. turning its
attention weight to zero and redistribute the other attention weights). Note
that the masking operation does not affect the information flow of encoding
calculation, i.e. keeping Equation 1 unchanged.

(a) Model performance

(b) Length of output

Figure 2: Relative changes of (a) model performance and (b) length of output
when masking individual encoder layer in the trained Seq2Seq models. As seen,
masking the embedding layer leads to a significant drop of model performance
and increase of output length.

Figure 2(a) shows the contribution of individual encoder layer to model
performance. As seen, masking the encoder embedding layer seriously harms the
model performance in all tasks, which confirms our claim that the surface
features in the embedding layer are essential to Seq2Seq models.

Figure 2(b) shows the results on the output length. Masking the encoder
embedding layer consistently increases the length of generated output, which
is especially significant for the summarization model. One possible reason is
that the instances in translation and correction tasks have similar
input/output lengths, while the summarization instances have distant
input/output lengths.

By analyzing the model outputs, we found that the Seq2Seq models tend to
generate some hallucinatory (i.e. fluent but unfaithful to the source)
predictions (Lee et al., 2019; Wang & Sennrich, 2020) when masking the
embedding layer. Taking the correction task for an example, a right prediction
“anyone” was replaced by the hallucinatory prediction “friends of anyone” in
the masked model, in which the corresponding source contains no information
related to “friends”. This issue becomes worse in the summarization task,
since the hallucinatory prediction is more likely to be a sentence.

The additional hallucinations will increase the output length and reduce the
model performance. In addition, Lee et al. (2019) point out that even if
hallucinations occur only occasionally, the Seq2Seq model may evidently lose
user trust than other prediction problems, indicating the importance to fuse
surface features at the embedding layer. More cases are studied in Appendix
A.2.

(a) Translation: Ro-En

(b) Summarization

(c) Correction

Figure 3: Log scale singular values of the three sub-embedding matrices in the
fine-grained layer attention models. Higher log eigenvalues denote more
expressivity of the dimensions.

#### Expressivity of attended dimensions in the encoder embedding layer

As shown in Figure 1, the uppermost decoder layer pays most attention to the
encoder embedding layer (i.e. the lower right corner). If the embedding layer
acts as a noise regularizer, the layer dimensions would be randomly attended
by the fine-grained model; otherwise, the dimensions of higher attention
weights should be distinguished from the other dimensions.

Starting from this intuition, we reordered the dimensions of the encoder
embedding layer according to the attention weights $\hat{\mathbf{w}}^{M,0}$,
and split it into two equal sub-embedding matrices, i.e. more attended
dimensions and less attended dimensions. We compared the expressivity of the
two sub-embedding matrices by the commonly-used singular value decomposition
(Gao et al., 2019; Wang et al., 2019a; Shen et al., 2020), in which higher
normalized singular values denote that the embedding is more uniformly
distributed, thus are more expressive. The singular values are normalized by
dividing them by the largest value and their log scale values are reported for
better clarity.

Figure 3 depicts the singular value results. For comparison, we also report
the values of the randomly selected dimensions. Clearly, the more attended
dimensions are most expressive, while the less attended dimensions are least
expressive. These results demonstrate that the fine-grained attention model
indeed extracts useful surface information from the encoder embedding layer,
which does not play the role of a noise regularizer.

From the above experiments, we prove that the encoder embedding layer indeed
provides useful surface information, which is not fully exploited by the
standard Seq2Seq models.

##  4 Our method

In Section 3, we show that the uppermost decoder layer requires more surface
features for better representation learning. One possible reason is that the
uppermost decoder layer is used for predicting individual target token, which
naturally benefits from more token-level surface features than sequence-level
abstract features. To validate this assumption, we simplify fine-grained layer
attention that only the uppermost decoder layer can attend to the embedding
layer and output layer of the encoder. Empirical results show that the
simplified variant works on par with the original one, revealing that the
surface features embed at the source embedding layer is expressive.

Although layer attention model partially alleviates source representation
bottleneck, it potentially introduces unnecessary intermediate encoder
representations. To address this gap, we propose to directly connect the
decoder softmax layer and the encoder embedding layer with a simple
SurfaceFusion method.

###  4.1 SurfaceFusion

Seq2Seq learning aims to maximize the log-likelihood of a target sequence
$\mathbf{y}$ given a source sequence $\mathbf{x}$. In practice, it factorizes
the likelihood of the target sequence into individually token likelihoods:

$\mathbf{\hat{y}}=\arg\max\prod\limits_{j=1}^{J}\log
P(y_{j})=\arg\max\prod\limits_{j=1}^{J}\log P(y_{j}|y_{<j},\mathbf{x})$ (3)

We rewrite $P(y_{j})$ as a fused probability with the second condition term
$\bf x$:

$\log P(y_{j})=\Phi\big{(}P(y_{j}|y_{<j},\mathbf{x}),\leavevmode\nobreak\
P(y_{j}|\mathbf{x})\big{)}$ (4)

where $\Phi(\cdot)$ is a fusion method that will be described later, and
$P(y_{j}|\mathbf{x})$ is a probability conditioned on the source surface
features. Specifically, we employ a multi-head dot-product attention network
(Vaswani et al., 2017) with a decoder output representation
$\mathbf{y}_{j}^{M}$ as a query, encoder output representations
$\mathbf{X}^{N}$ as keys , and encoder surface representations
$\mathbf{X}_{\mathrm{emb}}$ as values, to calculate a surface representation
$\mathbf{r}(y_{j},\mathbf{x})$.

Then we use the pre-softmax weight
$\mathbf{V}\in\mathds{R}^{d\times|\mathcal{V}_{y}|}$ of the vanilla model to
transform the surface representation
$\mathbf{r}(y_{j},\mathbf{x})\in\mathds{R}^{d}$ into a pre-softmax logit
$\widetilde{\mathbf{r}}(y_{j},\mathbf{x})\in\mathds{R}^{|\mathcal{V}_{y}|}$.
The final surface constraint probability is calculated as:

$\displaystyle
P(y_{j}|\mathbf{x})=\frac{\exp({\mathds{1}_{y_{j}}(\widetilde{\mathbf{r}}(y_{j},\mathbf{x}))/\tau)}}{\sum_{w\in\mathcal{V}_{y}}\exp({\mathds{1}_{w}\widetilde{\mathbf{r}}(y_{j},\mathbf{x})/\tau)}}$
(5)

where $\mathds{1}_{w}(\cdot)$ denotes an index function to take the logit of a
target token $y$, and $\tau$ denotes a softmax temperature parameter to
control the smoothness of the probability distribution $P(y_{j}|\mathbf{x})$.
As $\tau$ approaches to 0, the distribution tends to be an one-hot
distribution representing the token of the maximum probability. The
distribution becomes uniform at a higher $\tau$.

#### Choices of fusion function $\Phi$

There are many variants of fusion methods (Gulcehre et al., 2015; Sriram et
al., 2017; Stahlberg et al., 2018). The aim of this paper is not to explore
this whole space but simply to show that two fairly straightforward
implementations works well and that SurfaceFusion helps for sequence-to-
sequence models:

Hard fusion: $\displaystyle\Phi_{\mathrm{hard}}$ $\displaystyle=\lambda\log
P(y_{j}|y_{<j},{\bf x})+(1-\lambda)\log P(y_{j}|{\bf x})$ (6) Soft fusion:
$\displaystyle\Phi_{\mathrm{soft}}$
$\displaystyle=\log(\mathrm{softmax}(E(y_{j}|y_{<j},{\bf x})+\log P(y_{j}|{\bf
x}))$ (7)

where $\lambda$ is a pre-defined interpolation weight, and
$E(y_{j}|y_{<j},{\bf x})$ is the pre-softmax logit of the probability
$P(y_{j}|y_{<j},{\bf x})$. Compared to hard fusion, soft fusion removes the
need for manually setting the hyperparameter $\lambda$.

The proposed SurfaceFusion method is easy to use. There are only two
additional hyperparameters, i.e. $\lambda$ (Equation 6) and $\tau$ (Equation
5). We find that $\lambda$ is sensitive to the corpus scale but insensitive to
the relationship of input/output domain, which was set to 0.9 for the En-De,
En-Fr and correction tasks, and 0.8 for the Ro-En and summarization tasks. For
$\tau$, it was set to 5 for soft fusion and 1 for hard fusion across different
benchmarks. We kept other settings all the same with the vanilla models. In
practice, we observed an additional 10% inference latency with the
introduction of SurfaceFusion.

###  4.2 Experimental results

#### Model performance

Table 2: Results of the proposed SurfaceFusion methods on the Seq2Seq tasks. “FGLA” denotes fine-grained layer attention. The existing results are Ghazvininejad et al. (2019) for Ro-En, Ott et al. (2018) for En-De and En-Fr, Ott et al. (2019) for summarization, and Chollampatt & Ng (2018) for correction. All reported scores are the higher the better. | Translation | Summarization | Correction
---|---|---|---
Ro-En | En-De | En-Fr | RG-1 | RG-2 | RG-L | Prec. | Recall | F0.5
Existing | 34.0 | 29.3 | 43.2 | 40.1 | 17.6 | 36.8 | 65.5 | 33.1 | 54.8
Vanilla | 33.8 | 28.9 | 43.4 | 40.4 | 17.7 | 37.2 | 64.7 | 33.2 | 54.3
FGLA | 34.5 | 29.1 | 43.5 | 40.8 | 18.0 | 37.5 | 67.7 | 31.9 | 55.3
Hard fusion | 35.1 | 29.5 | 43.9 | 40.9 | 18.2 | 37.7 | 67.0 | 34.4 | 56.3
Soft fusion | 34.0 | 29.0 | 43.6 | 41.0 | 18.3 | 37.9 | 66.8 | 35.0 | 56.6

Table 2 lists the results of the proposed approach on different tasks. In
addition to the vanilla Seq2Seq model (“Vanilla”), we also report the results
of existing studies on the same datasets (“Existing”) for better comparison.
Our re-implementation of the vanilla models matches the results reported in
previous works, which we believe make the evaluation convincing.

Clearly, the proposed fusion approaches outperform the baselines (i.e.
“Vanilla” and “FGLA”) in all cases, while there are still considerable
differences among model variations. Hard fusion performs better on the
translation tasks, while soft fusion is superior on the summarization and
correction tasks. Unlike hard fusion that performs at the probability level,
soft fusion performs at the logit level to provide an earlier and direct way
for fusing surface features, which might be a better solution for the tasks
with a similar input/output domain.

Table 3:  Cosine similarities between aligned source and target word embeddings. “All” and “Non-Shared” denotes keeping or removing the aligned pair when the source and target words are the same, which are easier to be aligned. | All | Non-Shared
---|---|---
Vanilla | 0.602 | 0.338
SurfaceFusion | 0.650 | 0.417

#### Closeness of word embeddings

SurfaceFusion shortens the path distance between source and target embeddings,
which can help to learn better bilingual embeddings with direct interactions.
Table 3 shows the cosine similarities between the tied source and target
embeddings on the Ro-En translation task.

In the experiment, we first train an additional aligner (i.e. fast-align (Dyer
et al., 2013)) on the training corpus and use the alignment links to construct
a word dictionary. The results calculated over the dictionary show that the
relationship between the source and target embedding becomes much closer (i.e.
high cosine similarities). This can help each other to learn better
representations, and has been validated to be beneficial for Seq2Seq models
(Press & Wolf, 2017; Liu et al., 2019).

#### Expressivity of word embeddings

In this experiment, we quantitatively evaluate the expressivity of the word
embeddings learned by different models using the singular value decomposition.
The related experimental details and executions are similar to that of Figure
3.

Figure 4:  Log scale singular values of the embeddings.

Figures 4 shows the results of the tied source and target embeddings on the
Ro-En translation task. The word embeddings of the vanilla model have fast
decaying singular values, which limits the representational power of
embeddings to a small sub-space. The SurfaceFusion model slows down the
decaying and the singular values become more uniformly distributed, which
demonstrates that the fused surface features remarkably enhance the
representation learning of embeddings. This provides a better starting point
for the model to effectively extract surface and abstract features, which
leads to an improvement of model performance.

##  5 Related work

#### EncoderFusion in Seq2Seq

Lower encoder layers that embed useful surface features are far away from the
training signals, which poses difficulty for deep Seq2Seq models to exploit
such useful features. Although residual connections (He et al., 2016) have
been incorporated to combine layers, these connections have been “shallow”
themselves, and only fuse by simple, one-step operations (Yu et al., 2018). In
response to this problem, several approaches have been proposed to fuse the
encoder layers with advanced methods, such as layer attention (Bapna et al.,
2018; Shen et al., 2018; Wang et al., 2019c), layer aggregation (Dou et al.,
2018; Wang et al., 2018a; Dou et al., 2019; Li et al., 2020), and layer-wise
coordination (He et al., 2018; Liu et al., 2020). Although these methods show
promising results on different NLP tasks, not much is known about how the
EncoderFusion works. In addition, some other studies show that exploiting low-
layer encoder representations fail to improve model performance (Domhan,
2018).

In this paper, we consolidate the conflicting conclusions of existing studies
by pointing out that the encoder embedding layer is the key, which can help
Seq2Seq models to precisely predict target words. Based on this finding, we
propose a novel SurfaceFusion to directly connecting the encoder embedding
layer and the softmax layer, which consistently outperform current
EncoderFusion approaches across different NLP tasks.

#### Variants of Feature Fusion

Feature fusion aims to merge two sets of features into one, which is
frequently employed in CV tasks, such as semantic segmentation (Long et al.,
2014; Chen et al., 2018a; Zhang et al., 2018) and object detection (Pinheiro
et al., 2016; Lin et al., 2017). Zhang et al. (2018) shows that simply fusing
surface and abstract features tends to be less effective due to the gap in
semantic levels.

For NLP tasks, researchers investigated fusion models for language
understanding (Lu & Li, 2013; Hu et al., 2014; Peters et al., 2018) and
language generation (Gulcehre et al., 2015; Sriram et al., 2017; Stahlberg et
al., 2018). Nguyen & Chiang (2019) propose to fuse features at representation-
level, but we empirically find this kind of fusion method is not orthogonal to
multi-layer models due to the large semantic gap. Gulcehre et al. (2015)
combine the predictions produced by the Seq2Seq model and external LM
predictions in a later fusion manner, which pose little impact to the original
information flow. Stahlberg et al. (2018) improve upon it by removing the
dependence on the manually defined hyper-parameter. In this work, we
demonstrate the effectiveness of the two typical probability-level fusion
methods on sequence-to-sequence learning tasks. Unlike them that rely on an
external model, our approach only requires a surface attention module that can
be jointly trained with the vanilla Seq2Seq model.

##  6 Conclusion and future work

In this paper, we investigate how encoder layer fusion works on solving the
source representation bottleneck. Based on a series of experiments on
different Seq2Seq tasks, we find that the encoder embedding layer is important
to the success of EncoderFusion by exploiting the useful surface information.
Based on this observation, we propose a novel SurfaceFusion to directly
connect the encoder embedding layer and softmax layer. Experiments show that
SurfaceFusion consistently outperforms the conventional EncoderFusion in
several datasets. Extensive analyses reveal that SurfaceFusion enhances the
learning of expressive bilingual word embeddings for Seq2Seq models, which
confirm our claim.

Future directions include validating our findings on more Seq2Seq tasks (e.g.
dialogue and speech recognition) and model architectures (RNMT+ (Chen et al.,
2018b) and DynamicConv (Wu et al., 2019)). It is also worthwhile to explore
more alternatives to EncoderFusion from the perspective of exploiting the
embedding layer.

##  7 Acknowledgments

This work was supported in part by the Science and Technology Development
Fund, Macau SAR (Grant No. 0101/2019/A2), and the Multi-year Research Grant
from the University of Macau (Grant No. MYRG2020-00054-FST). We thank the
anonymous reviewers for their insightful comments.

## References

  * Ba et al. (2016) Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton.  Layer normalization.  _In arXiv_ , 2016.
  * Bahdanau et al. (2015) Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio.  Neural machine translation by jointly learning to align and translate.  In _ICLR_ , 2015.
  * Bapna et al. (2018) Ankur Bapna, Mia Xu Chen, Orhan Firat, Yuan Cao, and Yonghui Wu.  Training deeper neural machine translation models with transparent attention.  In _EMNLP_ , 2018.
  * Belinkov et al. (2017) Yonatan Belinkov, Nadir Durrani, Fahim Dalvi, Hassan Sajjad, and James Glass.  What do neural machine translation models learn about morphology?  In _ACL_ , 2017.
  * Chen et al. (2018a) Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam.  Encoder-decoder with atrous separable convolution for semantic image segmentation.  In _ECCV_ , 2018a.
  * Chen et al. (2018b) Mia Xu Chen, Orhan Firat, Ankur Bapna, Melvin Johnson, Wolfgang Macherey, George Foster, Llion Jones, Niki Parmar, Mike Schuster, Zhifeng Chen, Yonghui Wu, and Macduff Hughes.  The best of both worlds: Combining recent advances in neural machine translation.  In _ACL_ , 2018b.
  * Choi et al. (2018) Heeyoul Choi, Kyunghyun Cho, and Yoshua Bengio.  Fine-grained attention mechanism for neural machine translation.  _Neurocomputing_ , 2018.
  * Chollampatt & Ng (2018) Shamil Chollampatt and Hwee Tou Ng.  A multilayer convolutional encoder-decoder neural network for grammatical error correction.  In _AAAI_ , 2018.
  * Dahlmeier & Ng (2012) Daniel Dahlmeier and Hwee Tou Ng.  Better evaluation for grammatical error correction.  In _NAACL_ , 2012.
  * Domhan (2018) Tobias Domhan.  How much attention do you need? a granular analysis of neural machine translation architectures.  In _ACL_ , 2018.
  * Dou et al. (2018) Zi-Yi Dou, Zhaopeng Tu, Xing Wang, Shuming Shi, and Tong Zhang.  Exploiting deep representations for neural machine translation.  In _EMNLP_ , 2018.
  * Dou et al. (2019) Zi-Yi Dou, Zhaopeng Tu, Xing Wang, Longyue Wang, Shuming Shi, and Tong Zhang.  Dynamic layer aggregation for neural machine translation with routing-by-agreement.  In _AAAI_ , 2019.
  * Dyer et al. (2013) Chris Dyer, Victor Chahuneau, and Noah A. Smith.  A simple, fast, and effective reparameterization of IBM model 2.  In _NAACL_ , 2013.
  * Gao et al. (2019) Jun Gao, Di He, Xu Tan, Tao Qin, Liwei Wang, and Tieyan Liu.  Representation degeneration problem in training natural language generation models.  In _ICLR_ , 2019.
  * Gehring et al. (2017) Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N. Dauphin.  Convolutional sequence to sequence learning.  In _ICML_ , 2017.
  * Ghazvininejad et al. (2019) Marjan Ghazvininejad, Omer Levy, Yinhan Liu, and Luke Zettlemoyer.  Mask-predict: Parallel decoding of conditional masked language models.  In _EMNLP_ , 2019.
  * Gulcehre et al. (2015) Caglar Gulcehre, Orhan Firat, Kelvin Xu, Kyunghyun Cho, Loic Barrault, Huei-Chi Lin, Fethi Bougares, Holger Schwenk, and Yoshua Bengio.  On using monolingual corpora in neural machine translation.  _In arXiv_ , 2015.
  * He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _CVPR_ , 2016.
  * He et al. (2018) Tianyu He, Xu Tan, Yingce Xia, Di He, Tao Qin, Zhibo Chen, and Tie-Yan Liu.  Layer-wise coordination between encoder and decoder for neural machine translation.  In _NIPS_ , 2018.
  * Hu et al. (2014) Baotian Hu, Zhengdong Lu, Hang Li, and Qingcai Chen.  Convolutional neural network architectures for matching natural language sentences.  In _NIPS_ , 2014.
  * Kaneko et al. (2020) Masahiro Kaneko, Masato Mita, Shun Kiyono, Jun Suzuki, and Kentaro Inui.  Encoder-decoder models can benefit from pre-trained masked language models in grammatical error correction.  In _ACL_ , 2020.
  * Kiyono et al. (2019) Shun Kiyono, Jun Suzuki, Masato Mita, Tomoya Mizumoto, and Kentaro Inui.  An empirical study of incorporating pseudo data into grammatical error correction.  In _EMNLP_ , 2019.
  * Lee et al. (2019) Katherine Lee, Orhan Firat, Ashish Agarwal, Clara Fannjiang, and David Sussillo.  Hallucinations in neural machine translation.  _In OpenReview_ , 2019.
  * Li et al. (2020) Jian Li, Xing Wang, Baosong Yang, Shuming Shi, Michael R Lyu, and Zhaopeng Tu.  Neuron interaction based representation composition for neural machine translation.  In _AAAI_ , 2020.
  * Lin (2004) Chin-Yew Lin.  ROUGE: A package for automatic evaluation of summaries.  In _Text Summarization Branches Out_ , 2004.
  * Lin et al. (2017) Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie.  Feature pyramid networks for object detection.  In _CVPR_ , 2017.
  * Liu et al. (2020) Fenglin Liu, Xuancheng Ren, Guangxiang Zhao, and Xu Sun.  Layer-wise cross-view decoding for sequence-to-sequence learning.  _arXiv_ , 2020.
  * Liu et al. (2019) Xuebo Liu, Derek F. Wong, Yang Liu, Lidia S. Chao, Tong Xiao, and Jingbo Zhu.  Shared-private bilingual word embeddings for neural machine translation.  In _ACL_ , 2019.
  * Long et al. (2014) Jonathan Long, Evan Shelhamer, and Trevor Darrell.  Fully convolutional networks for semantic segmentation.  In _CVPR_ , 2014.
  * Lu & Li (2013) Zhengdong Lu and Hang Li.  A deep architecture for matching short texts.  In _NIPS_ , 2013.
  * Nguyen & Chiang (2019) Toan Nguyen and David Chiang.  Improving lexical choice in neural machine translation.  In _NAACL_ , 2019.
  * Ott et al. (2018) Myle Ott, Sergey Edunov, David Grangier, and Michael Auli.  Scaling neural machine translation.  In _WMT@EMNLP_ , 2018.
  * Ott et al. (2019) Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli.  fairseq: A fast, extensible toolkit for sequence modeling.  In _NAACL_ , 2019.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu.  Bleu: A method for automatic evaluation of machine translation.  In _ACL_ , 2002.
  * Paulus et al. (2018) Romain Paulus, Caiming Xiong, and Richard Socher.  A deep reinforced model for abstractive summarization.  In _ICLR_ , 2018.
  * Peters et al. (2018) Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer.  Deep contextualized word representations.  In _NAACL_ , 2018.
  * Pinheiro et al. (2016) Pedro H. O. Pinheiro, Tsung-Yi Lin, Ronan Collobert, and Piotr Dollár.  Learning to refine object segments.  In _ECCV_ , 2016.
  * Press & Wolf (2017) Ofir Press and Lior Wolf.  Using the Output Embedding to Improve Language Models.  In _EACL_ , 2017.
  * Rothe et al. (2019) Sascha Rothe, Shashi Narayan, and Aliaksei Severyn.  Leveraging pre-trained checkpoints for sequence generation tasks.  _In arXiv_ , 2019.
  * Sennrich et al. (2016) Rico Sennrich, Barry Haddow, and Alexandra Birch.  Neural machine translation of rare words with subword units.  In _ACL_ , 2016.
  * Shen et al. (2020) Sheng Shen, Zhewei Yao, Amir Gholami, Michael Mahoney, and Kurt Keutzer.  Rethinking batch normalization in transformers.  In _ICML_ , 2020.
  * Shen et al. (2018) Yanyao Shen, Xu Tan, Di He, Tao Qin, and Tie-Yan Liu.  Dense information flow for neural machine translation.  In _NAACL_ , 2018.
  * Sriram et al. (2017) Anuroop Sriram, Heewoo Jun, Sanjeev Satheesh, and Adam Coates.  Cold fusion: Training seq2seq models together with language models.  _In arXiv_ , 2017.
  * Srivastava et al. (2014) Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov.  Dropout: A simple way to prevent neural networks from overfitting.  _JMLR_ , 2014.
  * Stahlberg et al. (2018) Felix Stahlberg, James Cross, and Veselin Stoyanov.  Simple fusion: Return of the language model.  In _WMT_ , 2018.
  * Sutskever et al. (2014) Ilya Sutskever, Oriol Vinyals, and Quoc V Le.  Sequence to sequence learning with neural networks.  In _NIPS_ , 2014.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In _NIPS_ , 2017.
  * Voita et al. (2019) Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov.  Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned.  In _ACL_ , 2019.
  * Wan et al. (2013) Li Wan, Matthew Zeiler, Sixin Zhang, Yann Le Cun, and Rob Fergus.  Regularization of neural networks using dropconnect.  In _ICML_ , 2013.
  * Wang & Sennrich (2020) Chaojun Wang and Rico Sennrich.  On exposure bias, hallucination and domain shift in neural machine translation.  _In arXiv_ , 2020.
  * Wang et al. (2019a) Dilin Wang, Chengyue Gong, and Qiang Liu.  Improving neural language modeling via adversarial training.  In _ICML_ , 2019a.
  * Wang et al. (2019b) Liang Wang, Wei Zhao, Ruoyu Jia, Sujian Li, and Jingming Liu.  Denoising based sequence-to-sequence pre-training for text generation.  In _EMNLP_ , 2019b.
  * Wang et al. (2018a) Qiang Wang, Fuxue Li, Tong Xiao, Yanyang Li, Yinqiao Li, and Jingbo Zhu.  Multi-layer representation fusion for neural machine translation.  In _COLING_ , 2018a.
  * Wang et al. (2019c) Qiang Wang, Bei Li, Tong Xiao, Jingbo Zhu, Changliang Li, Derek F Wong, and Lidia S Chao.  Learning deep transformer models for machine translation.  In _ACL_ , 2019c.
  * Wang & Tu (2020) Wenxuan Wang and Zhaopeng Tu.  Rethinking the value of transformer components.  In _COLING_ , 2020.
  * Wang et al. (2018b) Xinyi Wang, Hieu Pham, Zihang Dai, and Graham Neubig.  Switchout: An efficient data augmentation algorithm for neural machine translation.  In _EMNLP_ , 2018b.
  * Wu et al. (2019) Felix Wu, Angela Fan, Alexei Baevski, Yann Dauphin, and Michael Auli.  Pay less attention with lightweight and dynamic convolutions.  In _ICLR_ , 2019.
  * Wu et al. (2016) Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al.  Google’s neural machine translation system: Bridging the gap between human and machine translation.  _In arXiv_ , 2016.
  * Yu et al. (2018) Fisher Yu, Dequan Wang, Evan Shelhamer, and Trevor Darrell.  Deep layer aggregation.  In _CVPR_ , 2018.
  * Zhang et al. (2020) Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter J Liu.  Pegasus: Pre-training with extracted gap-sentences for abstractive summarization.  In _ICML_ , 2020.
  * Zhang et al. (2018) Zhenli Zhang, Xiangyu Zhang, Chao Peng, Dazhi Cheng, and Jian Sun.  Exfuse: Enhancing feature fusion for semantic segmentation.  In _ECCV_ , 2018.

##  Appendix A Appendix

###  A.1 Experimental setup

Table 4: Statistics of the datasets and hyperparameters for the experiments. All the data have been tokenized and split into joint sub-word units (Sennrich et al., 2016). “Batch” denotes the number of source tokens and target tokens used in each training step. “DP” denotes the dropout value (Srivastava et al., 2014). “LP” denotes the length penalty (Wu et al., 2016). “Base” and “Big” denote the two kinds of model variants of Transformer. We chose the checkpoint with best validation ppl for testing. | Vocab | #Sents | Training | Testing
---|---|---|---|---
| Src/Tgt | Train | Dev | Test | Model | Batch | Step | DP | Beam | LP
Ro-En | 34,976 | 0.6M | 2K | 2K | Base | 16K | 60K | 0.3 | 4 | 1.0
En-De | 32,768 | 4.5M | 3K | 3K | Big | 460K | 30K | 0.3 | 5 | 0.6
En-Fr | 36,736 | 35.5M | 6K | 3K | Big | 460K | 80K | 0.1 | 5 | 0.9
CNN/DM | 50,264 | 0.3M | 13K | 11K | Base | 64K | 70K | 0.1 | 4 | 2.0
CONLL | 33,352 | 1.3M | 5K | 1K | Base | 64K | 80K | 0.2 | 6 | 0.6

Machine translation For WMT16 Romanian-English, we used the prepossessed
data111https://drive.google.com/uc?id=1YrAwCEuktG-iDVxtEW-FE72uFTLc5QMl and
existing result from Ghazvininejad et al. (2019). The validation set is
newsdev2016 and the test set is newtest2016. For WMT14 English-German, the
prepossessed data222https://drive.google.com/uc?id=0B_bZck-
ksdkpM25jRUN2X2UxMm8 and existing result are derived from Ott et al. (2018).
The validation set is newstest2013 and the test set is newstest2014. For WMT14
English-French, we reported the existing result from Ott et al. (2018) and
followed them to preprocess the data sets. The validation set is
newstest2012+2013 and the test set is newstest2014.

Text summarization For CNN/Daily Mail dataset, we used the existing result and
preprocessing method of Ott et al. (2019). During testing, the minimum length
was set to 55 and the maximum length was set to 140, which were tuned on the
development data. We also followed Paulus et al. (2018) to disallow repeating
the same trigram.

Grammatical error correction For CONLL14 benchmark, the preprocessing
script333https://github.com/nusnlp/mlconvgec2018/blob/master/data/prepare_data.sh
and existing result are given by Chollampatt & Ng (2018). We applied the
regularization technique SwitchOut (Wang et al., 2018b) in this task to
prevent overfitting, which was set to 0.8 for the source and 0.9 for the
target.

Table 4 gives more details of the benchmarks. It is noted that other
unmentioned hyperparameters keep the same with the original paper of
Transformer (Vaswani et al., 2017). All the models are implemented by the
open-source toolkit fairseq (Ott et al.,
2019).444https://github.com/pytorch/fairseq

###  A.2 Case study

Tables 5, 6 and 7 give the cases from the three tasks. We can see that the
hallucination issue related to surface features consistently appear over the
different Seq2Seq tasks. The most representative cases are those from the
correction task, in which very similar input/output sequences still make such
mistakes.

Another observation is the prediction omission problem when masking the
encoder output layer. The lack of abstract features leads to incomplete
semantics of source representations, thus making Seq2Seq models omit
generating a part of source, hurting the model performance. By looking at the
cases over the three tasks, we find that the prediction omission is widespread
in the prediction of modifiers, e.g. adjectives and adverbs.

Table 5: Examples from the Ro-En translation task. Red words are good predictions, while blue words are bad predictions. Masking the embedding layer (“Mask Emb”) of the fine-grained layer attention model leads to hallucinatory predictions, prolonging the prediction length. While masking the output layer (“Mask Out”) leads to prediction omissions, shortening the length. _Hallucination_ | Source |  diseara voi merge acasa si voi dormi linistit .
---|---|---
Reference |  i will go home tonight and sleep well .
Vanilla |  i will go home and sleep quietly .
Mask Emb |  the device will go home and i will sleep peacefully .
Mask Out |  i will go home and sleep quietly .
_Omission_ | Source |  radem adesea mult atunci cand vorbim .
Reference |  we often laugh a lot when we talk .
Vanilla |  we often laugh a lot when we talk .
Mask Emb |  we often laugh a lot when we talk .
Mask Out |  we often laugh when we talk .
Table 6: Examples from the CNN/DM summarization task. _Hallucination_ | Source |  … But it is able to carry just as much power - 400,000 volts . It is designed to be less obtrusive and will be used for clean energy purposes …
---|---|---
Reference |  … But it is able to carry just as much power - 400,000 volts . It is designed to be less obtrusive and will be used for clean energy .
Vanilla |  … But it is able to carry just as much power - 400,000 volts . It is designed to be less obtrusive and will be used for clean energy .
Mask Emb |  … It is able to carry just as much power - 400,000 volts . The design is a T-shape , with two ‘ hanging baskets ’ either side …
Mask Out |  … But it is able to carry just as much power - 400,000 volts . It is designed to be less obtrusive and will be used for clean energy .
_Omission_ | Source |  … Opening statements in his trial are scheduled to begin Monday …
Reference |  … Opening statements are scheduled Monday in the trial of James Holmes …
Vanilla |  … Prosecutors are not swayed, will seek the death penalty . Opening statements in his trial are scheduled to begin Monday . Holmes says he was suffering ‘ a psychotic episode ’ at the time …
Mask Emb |  … Prosecutors are not swayed, will seek the death penalty . Opening statements in his trial are scheduled to begin Monday . Holmes says he was suffering ‘ a psychotic episode ’ at the time …
Mask Out |  … Prosecutors are not swayed and will seek the death penalty . Holmes says he was suffering ‘ a psychotic episode ’ at the time …
Table 7: Examples from the CONLL correction task. _Hallucination_ | Source |  They can become anyone .
---|---|---
Reference |  They can become anyone .
Vanilla |  They can become anyone .
Mask Emb |  They can become friends with anyone .
Mask Out |  They can become anyone .
_Omission_ | Source |  In conclude , people should think carefully of what is the consequences of telling the relatives his or her generic disorder issue .
Reference |  In conclusion , people should think carefully about what is the consequences of telling the relatives his or her generic disorder issue .
Vanilla |  In conclusion , people should think carefully about what is the consequences of telling the relatives his or her generic disorder issue .
Mask Emb |  In conclusion , people should think carefully about what is the consequences of telling the relatives his or her generic disorder issue .
Mask Out |  In conclusion , people should think carefully about what is the consequences of telling the relatives his or her generic issue .