# Semantic Role Labeling for Knowledge Graph Extraction from Text

Mehwish Alam<EMAIL_ADDRESS>Aldo Gangemi<EMAIL_ADDRESS>Valentina Presutti<EMAIL_ADDRESS>Diego Reforgiato Recupero
<EMAIL_ADDRESS>ISTC, CNR, Rome, Italy.  University of Bologna,
Bologna, Italy.  Università degli Studi di Cagliari, Cagliari, Italy.

###### Abstract

This paper introduces TakeFive, a new semantic role labeling method that
transforms a text into a frame-oriented knowledge graph. It performs
dependency parsing, identifies the words that evoke lexical frames, locates
the roles and fillers for each frame, runs coercion techniques, and formalises
the results as a knowledge graph. This formal representation complies with the
frame semantics used in Framester, a factual-linguistic linked data resource.
The obtained precision, recall and F1 values indicate that TakeFive is
competitive with other existing methods such as SEMAFOR, Pikes, PathLSTM and
FRED. We finally discuss how to combine TakeFive and FRED, obtaining higher
values of precision, recall and F1.

###### keywords:

Semantic Role Labeling , Frame Semantics , Framester , Dependency Parsing ,
Role Oriented Knowledge Graphs

##  1 Introduction

Most knowledge in linked data and knowledge graphs is of a relational nature:
people participating in events, products having prices, artifacts with parts,
works of art produced by artists, beers sold at a bar, etc. For that reason, a
good part of integration and interoperability ends up consisting in aligning
relations among heterogeneous schemas and data.

Less known is the fact that the relations holding between entities are usually
part of a larger context or situation: beers can be found at a bar because
there is a selling/purchase situation; artists produce works because there is
a creative process involved; artifacts are assembled through craftmanship or
industrial procedures, products are assigned prices in the market, people are
assigned roles in events, etc.

Regardless of the representation language used and its serialization, existing
knowledge graphs share a common limit: they express facts that typically lack
contextual and situational information. This limit makes interoperability
difficult. When two different datasets need to be integrated, implicit
situations need to be reconstructed: This happens quite smoothly in humans,
but not in knowledge-based systems.

A method to contextualise knowledge graphs is to express the facts that they
capture as _projections of frames_. _Frames_ are cognitive structures that are
used by humans for organising their knowledge, as well as for interpreting,
processing or anticipating information (cf. [1] for a discussion encompassing
both linguistic and knowledge-based approaches to frames). In linguistics, a
reference model for frames is Fillmore’s Frame Semantics [2], where a frame is
introduced intuitively as “a kind of outline figure with not necessarily all
of the details filled in”. More precisely, a frame is a structure that reifies
an n-ary relation with multi-varied arguments, denotes a situation, event,
state, or configuration, and is supposed to bear representational similarity
to the knowledge encoded in cognitive systems. Any binary projection of a
frame is called a semantic role. For example, in the sentence I bought a pair
of shoes, the word “bought” identifies an occurrence of a commercial event,
where “I” and “pair of shoes” are objects that play the roles of “buyer” and
“‘goods” respectively in the Commerce_buy frame. Fillmore’s Frame Semantics
has been substantiated by FrameNet [3]: a long-standing, manually developed
resource of (English) frames represented in a structured format by a group of
linguists in Berkeley.

Recently, two resources have been introduced which support semantic
interoperability by using frames: FrameBase [4] and Framester [5]. The idea is
simple in principle: since situations are frame occurrences, let’s align any
schema to a set of frames from a stable ontology, and make data interoperate
along that path (if a schema fragment 1 and a schema fragment 2 align to a
same frame, the respective data can be jointly queried modulo ontology-based
data access, where the ontology of frames is uniform across resources). This
is apparently good news, but while an initial ontology of frames can be found
in FrameNet, the methods by which we can actually align any existing schema to
frames is much less obvious. And the main reason is that the relations defined
in schemas, ontologies, and knowledge graphs cannot be trivially aligned to
semantic roles.

For example, we need to assign the relation foaf:knows to a semantic role
within frames such as framenet:Personal_relationship or framenet:Familiarity.
However, there is no semantic role corresponding to foaf:knows. The alignment
would work only as a result of a path internal to the frame, e.g. an OWL
property chain on roles, such as: $isPartnerIn~{}o~{}hasPartner$.

Current approaches are still struggling with this problem: FrameBase manually
aligns relations to semantic roles, leading to scalability issues, while
Framester provides an extensive amount of linguistic mappings that help a
semi-automated alignment, but a previous linguistic parsing of the relations
and their context is required, which is still non-standard, specially
considering that only a few ontologies explicitly encode the competency
questions that led to the form of their relations.

In practice, the integration of a knowledge extraction approach from
competency questions or other textual material, its alignment to Framester,
and the usage of dereferencing methods as proposed by FrameBase, collectively
seem a viable automated integration solution in the future. In order to foster
the research, in this paper we propose a knowledge-graph-based algorithm for
labeling semantic roles from an arbitrary text, thus accommodating for the
linguistic parsing needed to perform frame alignment prior to
interoperability. Our algorithm, called TakeFive, is evaluated, and compared
to alternative approaches, with respect to metrics widely applied in two NLP
tasks: frame detection and semantic role labeling. The first refers to the
ability to automatically detect occurrences of frames in natural language
text. The second refers to identifying the fragments of text denoting the
entities that play specific roles in a frame occurrence. In this paper, we
extend our preliminary work [6], and use Framester [5], a frame-based
knowledge graph, to address frame detection and SRL with
TakeFive111https://github.com/TakeFiveSRL/TakeFiveSRL. TakeFive uses NLP
resources and software components, but integrates them in a semantic web
pipeline that produces knowledge graphs ready to be used for interoperability
across data and schemas. We intend to verify if a knowledge-based hybrid
method is comparable to purely statistical methods, while retaining the
ability to extract a properly linked knowledge graph from a sentence.

As an example of the output of SRL, let us consider the following sentence

Despite recent declines in yields, investors continue to pour cash into money
funds. (1)

By performing frame detection, we recognize that to pour evokes the frame
Pour.v from Framester, subsumed by the frame Cause_motion from FrameNet,
meaning that the sentence expresses an occurrence of this frame. By performing
SRL, TakeFive then labels the investors and cash respectively with the
Agent.cause_motion role, and the Theme.cause_motion role, as both involved in
the Cause_motion situation occurrence (cf. Figure 1). The annotations use
entities from reference ontologies for frames and semantic roles.

Despite & recent & declines & in & yields & , & investors & continue & to &
pour & cash & into & money & funds

8root 87nsubj [edge below,red,densely dotted]87agent 810xcomp [edge
below,red,densely dotted]810during$\\_$event [arc edge]83nmod [edge below,,arc
edge,red,densely dotted]83oblique 32amod 31case 35nmod 54case 109mark 1011dobj
[edge below,red,densely dotted]1011product 1014nmod [edge below,arc
edge,red,densely dotted]1014oblique 1413compound 1412case

Figure 1: Dependency tree obtained from CoreNLP. Dotted lines represent the
role labeling by TakeFive.

The remainder of the paper is organized as follows: Section 2 describes
related work for SRL and SRL-based knowledge extraction, Section 3 briefly
introduces the data, resources and components used by TakeFive. Section 4
presents the TakeFive algorithm. Section 5 describes the evaluation setting,
the performance measures, and the results, with a comparison to other tools
and a discussio.

##  2 Related Methods

Semantic technologies usually leverage syntactic resources to improve their
accuracy. Stanford CoreNLP222https://stanfordnlp.github.io/CoreNLP/ [7], which
we use in TakeFive, is one of the most used full-fledged NLP tools, and has
been used in the Semantic Web context. However, it has not been extensively
employed for SRL (an exception is [8]).

After the development of PropBank [9], where semantic information has been
added to the Penn English Treebank data set, and the CoNLL shared tasks on
semantic role labeling [10, 11], there has been a lot of research in this
domain, typically using PropBank as the reference ontology for roles. PropBank
is a data set consisting of the phrase-structure syntactic trees from the WSJ
section of the Penn Treebank. Its annotations include predicate-argument
structures for verbs and defines a small number of roles: core roles are ARG0
through ARG5, which can be interpreted differently for different predicates.
Further modifier roles ARGM* include e.g. ARGM-TMP (temporal) and ARGM-DIR
(directional).

The semantics of the core roles ARG0-ARG5 is not straightforwardly clear. The
study described in [12] shows that the roles ARG2-ARG5 serve many different
purposes for different verbs, and points out that they are inconsistent and
highly overloaded. In order to improve the performance for the SRL task, the
arguments were mapped to VerbNet thematic roles. Others, e.g. [13] revised the
syntactic subcategorization patterns for FrameNet lexical units, using
VerbNet. While PropBank labels the roles of verbs with a limited number of
tags, frame-semantic parsing labels frame arguments with frame-specific roles,
making it clearer what those arguments may mean. Therefore, for frame-semantic
parsing, sentences may contain multiple frames that need to be detected along
with their arguments. SemEval 2007 task 19 [14] addressed this problem. The
task leveraged FrameNet 1.3 and released a small corpus containing more than
2000 sentences with full text annotations.

The work described in [15] projects predicate-argument structures from seed
examples to unlabeled sentences using linear program formulation to find the
best alignment related to the projection. The projected information and the
seeds are both used to train statistical models for SRL. In addition, the
authors introduce a method for finding examples for unseen verbs using a graph
alignment tool, which was used to project annotations from seed examples to
unlabeled sentences.

In [16, 17] the authors use an unsupervised approach for SRL that aims at
inducing semantic roles automatically from unannotated data. Although this can
be useful to discover new semantic frames and roles, in this paper we focus on
the concrete representation provided in FrameNet and VerbNet, without
expanding their inventory of semantic types.

Authors in [18] introduce a semantic parser that uses a broad knowledge base
created by interconnecting FrameNet, VerbNet and PropBank. SEMAFOR
[19]333http://www.ark.cs.cmu.edu/SEMAFOR is a well known system for frame-
semantic parsing, based on the combination of knowledge from FrameNet, two
probabilistic models trained on full text annotations released along the
FrameNet lexicon, and expedient heuristics. At SemEval 2007 it outperformed
existing approaches.

FRED444http://wit.istc.cnr.it/stlab-tools/fred [20, 21] is the state-of-the-
art tool for producing framed knowledge graphs for the Semantic Web. It
consists of a complex pipeline of NLP and Semantic Web components for parsing
text, representing it to a neo-Davidsonian logical form, extracting entities,
disambiguating predicates, linking them to public resources, and creating a
well-connected, formal and queryable knowledge graph out of that. FRED uses a
“greedy” approach for SRL, i.e. it labels roles with reference labels (from
either VerbNet or FrameNet) when the confidence of its categorial parser is
high, otherwise it uses other heuristics to provide meaningful local labels
that make sense in that textual context.

PIKES [22]555http://pikes.fbk.eu is a tool that automatically extracts things
of interest and facts about them from text. PIKES applies a number of NLP
tools to annotate a text, and applies a linked-data-oriented approach to
generate RDF graphs.

PathLSTM is a SRL system introduced in [23], which builds on top of the mate-
tools semantic role labeler666http://code.google.com/p/mate-tools/. It
leverages neural sequence modeling techniques: the authors model semantic
relationships between a predicate and its arguments by analyzing the
dependency path between a predicate word, and each argument head word.
Lexicalised paths are considered, which are decomposed into sequences of
individual items, namely the words and dependency relations on a path. Long-
short term memory networks are then applied to find a recurrent composition
function that can reconstruct an appropriate representation of the full path
from its individual parts.

##  3 Lexical resources used by TakeFive

In this section we describe the lexical resources that we leverage in order to
design TakeFive.

###  3.1 VerbNet

VerbNet [24] is a broad coverage verb lexicon in English, with links to other
data sources such as WordNet [25] and FrameNet [3]. It contains semantic roles
and verb classes corresponding to Levin’s classes [26], and including multiple
verb senses. Verb classes can therefore be considered akin to word synsets.
They generalise the verbs based on their shared syntactic behavior. These verb
classes feature a simple two-layer hierarchy. For example, the verb conquer is
a member of the class subjugate-42.3, and hence a sense Conquer_42030000 is
created (the sense of conquer in that class).

VerbNet further contains semantic roles, which correspond to the relations
between a verb sense and its arguments. Each class has multiple _frames_
(either syntactic- or semantic-oriented), which define a list of _predicates_
associated with their _arguments_. There is a (partial) morphism between
syntactic and semantic frames, so that semantic roles (“arguments”) are also
associated with patterns that characterize the syntactic behavior of a verb in
that class. For example, the roles defined for the class subjugate-42.3 are
_Agent_ , _Patient_ and _Instrument_ meaning that an agent subjugates a
patient with some instrument. Here _Agent_ and _Patient_ are necessary roles,
and _Instrument_ is an optional role. Verb senses help in determining if a
particular verb instance conforms to the underlying semantics of the class.
For the case of the verb conquer its only sense is included in the class
subjugate-42.3. VerbNet maps verbs to FrameNet frames, e.g. the verb sense
Conquer_42030000 is mapped to the frame Conquering. The version of VerbNet
used in TakeFive evaluation is 3.1, and the data come from the RDF porting of
VerbNet 3.1 that is included in
Framester777http://github.com/framester/Framester [5].

###  3.2 FrameNet

FrameNet [3] contains _frames_ , which describe a situation, state or action.
Each frame has semantic roles (“frame elements”) that are much more
semantically detailed than VerbNet ones. FrameNet also defines a subsumption
relation between either frames or roles. The subsumption relation can be used
to create a hierarchy of classes, as shown in [27]. Each frame can be evoked
by _lexical Units (LUs)_ belonging to different parts of speech. In version
1.5, FrameNet covers about 10,000 lexical units and 1024 frames. Let us
consider the following sentence:

[The Spaniards]Conqueror [conquered]${}_{Lexical~{}Unit}$ [the Incas]Theme.
(2)

In the above example, The Spaniards is the argument (we will also refer to it
as filler) of the role Conqueror and the Incas is the argument (or filler) of
the role Theme and conquered is the lexical unit evoking the frame.

###  3.3 Framester

Framester [5] is a large RDF888https://www.w3.org/TR/rdf11-primer/ knowledge
graph (currently including about 50 million triples), acting as a hub between
several predicate oriented linguistic resources such as FrameNet, WordNet,
VerbNet [24], BabelNet [28], Predicate Matrix [29], as well as many other
linguistic, factual, and foundational knowledge graphs. It leverages this
wealth of links to create an interoperable and homogeneous _predicate space_
represented in a formal rendering of frame semantics [2] and semiotics [30].
Framester uses a novel mapping between WordNet, BabelNet, VerbNet and FrameNet
at its core, expands it to other linguistic resources transitively, and
represents all of this formally. It further links these resources to other
important ontological and linked data resources

Framester is accessible through its SPARQL
endpoint999http://etna.istc.cnr.it/framester2/sparql. Framester also features
a subsumption hierarchy of semantic roles (i.e. frame elements) and adds
generic roles on top of frame-specific roles.

Framester also offers a Word Frame Disambiguation (WFD) service based on the
mappings defined within the resource. It is available as a frame detection API
which is implemented using UKB and Babelfy as the Word Sense Disambiguation
algorithms as a first step and then the second step uses the mappings between
WordNet/BabelNet synsets and FrameNet frames. The associated REST API is
available online101010https://lipn.univ-paris13.fr/framester/en/wfd_html.

##  4 Semantic Role Labeling Algorithm

TakeFive generates role oriented knowledge graphs given an input sentence. The
algorithm begins by detecting the verb (lemma and VerbNet verb class), along
with its arguments, and then it relates it to their corresponding VerbNet
roles. According to our running example 2, TakeFive detects the verb conquered
and then extracts the VerbNet roles of this verb i.e., Conqueror and Theme.
Finally, it assigns the role fillers i.e., The Spaniards as a filler of
Conqueror and the Incas as the filler of the VerbNet role Theme. The backbone
of TakeFive follows a step-wise approach: (i) preprocessing step for
extracting dependencies and frame annotations using existing tools (i.e.,
CoreNLP and Word Frame Disambiguation respectively), (ii) detecting (CoreNLP-
derived, mainly syntactic) interface roles, (iii) VerbNet specific roles
(mainly semantic) for a certain frame, and then finally, (iv) checking the
compatibility between interface and semantically specific roles. In other
words, we aim at using background knowledge and formal reasoning in order to
associate semantic roles with syntactic dependencies. In the rest of the
paper, we use the following terminologies:

  * –

CoreNLP interface roles for the roles generalising CoreNLP dependencies as
well as resource-specific semantic roles;

  * –

VerbNet specific roles for the VerbNet roles related to a certain verb sense;

  * –

VerbNet interface roles for the roles that subsume VerbNet specific roles in
Framester, and are subsumed by an interface role.

###  4.1 Pre-processing step

##### Framester and CoreNLP

For a given input sentence, frame detection using Word Frame Disambiguation
(WFD) is performed. It uses Babelfy as a WSD algorithm and then uses the
mappings between BabelNet Synsets and FrameNet frames as given in Framester.
The dependency tree is extracted associated to a given input sentence using
CoreNLP. Listing 1 shows a dependency tree returned by CoreNLP for the running
example.

Listing 1: Dependency tree triples representation of CoreNLP

⬇

det, Spaniards-2, The-1,

nsubj, conquered-3, Spaniards-2,

root, ROOT-0, conquered-3,

det, Incas-5, the-4,

dobj, conquered-3, Incas-5

##### Assigning Interface roles to CoreNLP dependencies

TakeFive is based on 23 simple heuristics for mapping CoreNLP dependency
triples to interface roles. For the running example, we have a dependency
nsubj, conquered-3, Spaniards-2 related to the verb conquered, and its
argument Spaniards. Dependency types such as nsubj, dobj,... are generalized
to _CoreNLP interface roles_ through a set of heuristics e.g., by applying the
rule $nsubj\rightarrow Agent$ i.e., the role Agent is assigned to the argument
Spaniards. The set of interface roles include {Agent, Undergoer, Recipient,
Eventuality, Oblique}.

###  4.2 TakeFive: A Semantic Role Labeling Algorithm

This section discusses the two algorithms proposed for labeling a given
sentence with the VerbNet specific and VerbNet interface as well as a way to
check the the compatibility between the CoreNLP interface roles (as assigned
previously) and VerbNet interface roles. Algorithm 1 computes VerbNet
interface and specific roles of extracted verbs from an input sentence.

####  4.2.1 Computing VN Interface and Specific Roles

Algorithm 1 takes the pre-processed information as an input i.e., (a) the
sentence, (b) dependency tree obtained by CoreNLP and (c) the output of frame
annotations obtained using WFD. It then returns the input sentence labeled
with VerbNet specific as well as VerbNet Interface roles. If the verb is
polysemic then it uses frame detection for extracting VerbNet roles (line 2-4,
see Algorithm 2), otherwise it gets the verb sense using the SPARQL query in
listing 3. If it returns more than one verb sense it selects the one which is
most frequent (see query in listing 6) and extracts the VerbNet specific roles
along with VN interface roles, if any (line 6-9, see listing 5). If the result
is empty, it uses frame detection for obtaining the VerbNet roles described in
Algorithm 2.

Algorithm 2 and 3 are used in case of polysemous verbs. The algorithm takes as
an input a sentence and annotates it with frames (line 1). If there are no
frame annotations it takes the most frequent verb sense using SPARQL query in
listing 6 and then the VN specific and interface roles associated to the this
verb sense through SPARQL query in listing 5 (line 2-5). If the Word Frame
Disambiguation API returns multiple frames and there is a relation between
these frames, the most specific frame is chosen (line 7-16) (see Algorithm 3).
Then, given the $verb$ and the chosen frame, VerbNet senses are extracted
using listing 4 (line 17). If no verb senses are returned then the most
frequent verb senses are extracted for getting the VerbNet specific roles
(line 18-20). However, if there are more than one verb senses then the most
frequent verb sense is chosen. An intersection of both the queried and
returned sets of verb senses is taken and the verb sense with highest ranking
based on frequency of verb senses in WordNet is selected and the corresponding
VN role is returned (line 22-30). If both the above cases are false the VN
role associated to the verb sense is selected (line 32).

Data: CoreNLP and Framester pre-processed information of an input sentence; an
input sentence

Result: Pairs (VerbNet interface roles, VerbNet specific roles) for each verb
senses of the extracted verbs in the input sentence

1 foreach  _verb_ do

2

3 if  _verb is polysemic_ then

4 see Algorithm 2

5 end if

6 else

7 $verbSenses\leftarrow$ get verb sense using query in listing 3;

8 if  _ $verbSenses!=\emptyset$_ then

9 $verbSense\leftarrow$ obtain most frequent verb senses using SPARQL query in
listing 6 ;

10 $vnrole\leftarrow$ given verbSense extract VN roles using SPARQL query in
listing 5 ;

11

12 end if

13 else

14 see Algorithm 2

15 end if

16

17 end if

18

19 end foreach

Algorithm 1 Computation of pairs (VerbNet interface roles, VerbNet specific
roles) for each verb in a given sentence.

Input: sentence, verb

Output: 3-tuple containing verb, its VerbNet specific role, VerbNet Interface
role

1 $frames\leftarrow getFrames(sentence)$;

2 if  _frames is $\emptyset$_ then

3 $verbSense\leftarrow$ obtain most frequent verb senses using query 6 ;

4 $vnrole\leftarrow$ given verbSense extract VN roles using query 5 ;

5

6 end if

7

Algorithm 2 Obtaining VerbNet Roles using Frame Detection if no frames are
obtained.

6 for  _$f_{1}\in frames$_ do

7 for  _$f_{2}\in frames$_ do

8 if  _ $f_{1}\sqsubseteq f_{2}$ _ then

9 $frameSpecific\leftarrow f_{1}$

10 end if

11 else if  _ $f_{2}\sqsubseteq f_{1}$ _ then

12 $frameSpecific\leftarrow f_{2}$

13 end if

14

15 end for

16

17 end for

18$verbSenses\leftarrow$ given $verb$ and $frameSpecific$ using query 4;

19 if  _ $verbSenses==\emptyset$ _ then

20 $verbSense\leftarrow$ obtain most frequent verb senses using query 6 ;

21 $vnrole\leftarrow$ given verbSense extract VN roles using query 5

22 end if

23else if  _ $len(verbSenses)>1$ _ then

24 $verbSenseQueried\leftarrow$ obtain most frequent verb senses using query 6
;

25 if  _ $verbSenseQueried\in verbSenses$ _ then

26 $vnrole\leftarrow$ given verbSense extract VN roles using query 5

27 end if

28 else

29 $verbSense\leftarrow$ get the next most frequent verb sense obtained by
query 6 and so on until the verbSense matches.

30 end if

31

32 end if

33

34else

35 $vnrole\leftarrow$ given verbSense extract VN roles using query 5

36 end if

37

Algorithm 3 Obtaining VerbNet Roles using Frame Detection if frames are
detected.

####  4.2.2 Checking Compatibility of CoreNLP interface roles

The objective here is to return, all roles and fillers for each argument of
verbs from the input sentence if the interface roles assigned using the two
methods (i.e., heuristics and algorithm 1) are compatible. Let
$O=\\{Agent,~{}Undergoer,$ $Recipient,~{}Eventuality\\}$, $C$ be the CoreNLP
interface roles (assigned using heuristics such as $nsubj\rightarrow Agent$),
$V$ be the VerbNet interface roles and $R$ be the VerbNet specific roles;
where $V$ and $R$ are returned by algorithm 1 for a given sentence. For
$v_{1}\in V$, $c_{1}\in C$ and $r_{1}\in R$, if $v_{1}=\emptyset$ and
$r_{1}=\emptyset$, then $c_{1}$ is assigned. However, if $v_{1}\neq\emptyset$
and $r_{1}\neq\emptyset$, then the following rules are defined:

  * –

The algorithm starts by choosing $verb$ having at least one VN sense and takes
$c_{1}$ and $v_{1}$. If $c_{1}\in O$ and $v_{1}\in O$ then the pair
$(c_{1},v_{1})$ is marked compatible and $r_{1}$ is returned such that
$r_{1}\in R$ and $v_{1}$ associated_to $r_{1}$ (returned by Algorithm 1).

  * –

If $c_{1}$ and $v_{1}$ associated with $verb$ are oblique check for a
preposition along with CoreNLP dependencies triples having the modifier _nmod_
and then return $r_{1}$ such that $r_{1}$ is compatible with the preposition
according to VerbNet verb arguments. The association between the VerbNet
arguments and the prepositions are already defined in VerbNet and now
standardized in RDF in Framester linguistic linked data hub as shown in
Listing 2 (we also make use of the Preposition Project dataset, which is
another linked dataset in Framester).

  * –

If $c_{1}=Agent$ or $c_{1}=Undergoer$ and $c_{1}\neq v_{1}$ then select the
top role of the subsumption hierarchy associated to VerbNet interface role
(defined by the predicate
fschema111111https://w3id.org/framester/schema/:subsumedUnder). If the top
role is _Theme_ then select $v_{1}$.

  * –

If all the previously defined rules are false or if there is no mapping
between $c_{1}$ and $v_{1}$ then return $c_{1}$.

Listing 2: Triples for obtaining VerbNet arguments.

⬇

vsprepositionselection:Conquer_42030000-with-Instrument

a vnschema:SensePrepSelection ;

vnschema:hasGenericArgument vndata:Instrument ;

vnschema:hasPreposition vndataprep:with ;

vnschema:hasVerbSense vndata:Conquer_42030000 .

Listing 3: Query for extracting Verb Senses given a verb lemma.

⬇

SELECT DISTINCT ?verbsense

WHERE {

?verbsense rdfs:label ?verblemma ;

a fschema:VerbSense .

filter regex(?verblemma, ”conquer”, ”i”) }

Listing 4: Query for mapping frames to verb senses when the verb is polysemous
in VerbNet.

⬇

SELECT DISTINCT ?verbsense

WHERE {

?verbsense rdfs:label ?verblemma ;

a vn31schema:VerbSense;

skos:closeMatch ?frame }

Listing 5: Query to retrieve the VerbNet role for a certain (optional)
interface role and a certain verb sense.

⬇

SELECT DISTINCT ?interfacerole ?verbnetrole

WHERE {

?verbnetrole a vn31schema:Argument;

vn31schema:inVerbSense <VERBNET SENSE>

OPTIONAL

{

?verbnetrole fschema:subsumedUnder+ ?interfacerole .

?interfacerole a fschema:InterfaceRole } }

Listing 6: Query to retrieve most frequent verb senses given a verb lemma.

⬇

SELECT DISTINCT ?verbsense

WHERE {

?verbsense rdfs:label <verblemma> ;

a vn31schema:VerbSense .

?verbsense skos:closeMatch ?wnsense .

?wnsense schema:tagCount ?freq .

FILTER ((datatype(?freq)) = xsd:int) }

ORDER BY DESC(?freq) LIMIT 1

##  5 Evaluation

This section details the experimental setting, and the two evaluation
procedures for measuring the performance of the TakeFive SRL algorithm. It
also describes a comparison between TakeFive and other SRL tools.

###  5.1 Implementation details

The algorithm is developed in Python and uses REST-APIs for Framester and
Stanford CoreNLP. It also employs Py4J121212https://www.py4j.org/ as a bridge
between Python and Java. A Java class was developed which can directly be
called from the main Python code through Py4J. It can be faster if a cache
mechanism is used to store Framester results, and SPARQL queries to the
Framester endpoint. The TakeFive SRL tool is available on-
line131313https://github.com/TakeFiveSRL/TakeFiveSRL.

###  5.2 Evaluation setting

The performance evaluation was conducted for verifying if the chosen VerbNet
roles associated with fillers are correct or not. We used the WSJ section of
the Penn Treebank annotated with VerbNet and PropBank
labels141414https://github.com/ibeltagy/pl-
semantics/blob/master/resources/semlink-1.2.2c/1.2.2c.okay. These annotations
include the VerbNet and PropBank roles associated with each verb of each
sentence of the dataset, and related to each filler. As an example, consider
the following sentence contained in the WSJ annotated dataset:

The Canadian pig herd totaled 10,674,500 at Oct. 1, down 3 from a year
earlier, said Statistics Canada, a federal agency. (3)

The two verbs totaled and said are indicated in the annotations together with
their VerbNet verb classes, as well as their VerbNet and PropBank roles and
fillers. In particular, Table 1 shows the annotations for sentence 3.

Verb | Verb class | VerbNet Role | PropBank Role | Filler
---|---|---|---|---
Say | 37.7-1 | Topic | ARG1 | The Canadian pig herd totaled
|  |  |  | 10,674,500 at Oct. 1, down 3
|  |  |  | from a year earlier
Say | 37.7-1 | Agent | ARG0 | Statistics Canada, a federal agency
Total | 54.1-1 | Theme | ARG1 | The Canadian pig herd
Total | 54.1-1 | Value | ARG2 | 10,674,500

Table 1: Annotations for example sentence 3

Performance evaluation was conducted by computing precision, recall and F1
score using the official CoNLL-2009
scorer151515https://ufal.mff.cuni.cz/conll2009-st/scorer.html [31]. The
CoNLL-2009 scorer evaluates the semantic frames by reducing them to semantic
dependencies. A semantic dependency from every predicate to all its arguments
is created. These dependencies are labeled according to their corresponding
arguments. Additionally, a semantic dependency from each predicate to a
virtual ROOT node is added. The latter dependencies are labeled with the
predicate senses. This approach guarantees that the semantic dependency
structure forms a single-rooted, connected (but not necessarily acyclic)
graph. It can be seen that the scoring strategy gains some points even though
a system assigns the incorrect predicate sense. For further details refer to
[32, 31].

In order to use the CoNLL scorer when comparing to other methods, we formatted
the output of TakeFive as well as that of the other tools as required by the
CoNLL-2009 scorer. Finally, for compliance purposes, we employed
SemLink161616https://verbs.colorado.edu/semlink/ to map VerbNet roles to
PropBank roles.

###  5.3 Results

The results obtained by TakeFive have been compared to other state-of-the-art
methods, including SEMAFOR, Pikes, FRED, and PathLSTM (for details (see
Section 2)). As FRED and TakeFive are two resources maintained by overlapping
teams, we have also combined their results by including all the VerbNet roles
extracted by FRED into the results of TakeFive; we named this new algorithm as
TakeFive+FRED. First of all, the onf files (2454) of the WSJ corpus171717The
dataset was made available by the Linguistic Data Consortium:
https://www.ldc.upenn.edu. were processed. Each file contains input sentences
and their parse trees. The gold standard has 74977 rows. Each row corresponds
to a verb in a given sentence and includes the VerbNet verb class, fillers and
VerbNet and PropBank roles associated to that verb. Different rows might refer
to the same sentence (as there might be several verbs within a given
sentence). As already mentioned, when labeling each sentence using TakeFive,
we extracted Framester and CoreNLP information (frames, dependency triples,
POS tags, etc.). To speed up the experiments, we used a cache mechanism so
that the information is downloaded only once (the cache mechanism is not
currently available in the on-line version of TakeFive software).

Table 2 shows the labeled and unlabeled precision, recall and F1-measure
values of TakeFive, TakeFive+FRED, and the competitors, SEMAFOR, Pikes,
PathLSTM and FRED. _Labeled scores_ are related to the correct identification
of labeled dependency whereas _Unlabeled scores_ do not take into account
labels. For example, for the correct proposition: verb.01: ARG0, ARG1, ARGM-
TMP, the system that generates the following output for the same argument
tokens verb.02: ARG0, ARG1, ARGM-LOC receives a labeled precision score of 2/4
because two out of four semantic dependencies are incorrect: the ROOT
dependency is labeled ”02” instead of ”01” and the dependency to the ”ARGM-
TMP” is incorrectly labeled ”ARGM-LOC”. On the other hand, the same example
would receive an unlabeled precision score of 4/4. SEMAFOR is the method with
the highest accuracy. Our proposed approach is the second best, however the
numbers are very close to that of SEMAFOR. We noticed that FRED has some
internal issues with the offset extraction of words of the sentence and this
affects the output representation. As far as Pikes is concerned we have
observed that it misses some important roles for tokens. Its SRL engine is
based on mate-tools181818http://code.google.com/p/mate-tools/, further
developed in [33] in 2009.Similar conclusions, and similar performances with
Pikes, can be drafted for PathLSTM which is also based on mate-tools.

The combined approach TakeFive+FRED is able to slightly outperform SEMAFOR.
FRED captures complementary roles which TakeFive is unable to detect. Based on
this intuition, we believe that future experiments on optimal combination of
multiple SRL approaches might yield the best results.

Method | Lab. | Lab. | Lab. | Unlab. | Unlab. | Unlab.
---|---|---|---|---|---|---
| Prec. | Recall | F1 | Prec. | Recall | F1
TakeFive | 80.12% | 76.04% | 78.02% | 85.09% | 80.44% | 82.70%
SEMAFOR | 81.05% | 77.01% | 78.97% | 87.32% | 82.97% | 85,09%
TakeFive+FRED | 82.55% | 78.48% | 80.46% | 87.60% | 83.18% | 85.33%
FRED | 74.02% | 72.36% | 73.18% | 83.11% | 81.65% | 82.37%
Pikes | 72.11% | 70.62% | 71.35% | 79.27% | 78.15% | 78.70%
PathLSTM | 73.66% | 71.65% | 72.64% | 82.64% | 80.63% | 81.62%

Table 2: Labeled and unlabeled precision, recall and F1 values of TakeFive,
TakeFive+FRED, SEMAFOR, Pikes, FRED and PathLSTM.

##### A different strategy?

While the results with CoNLL indicate that TakeFive performs as good as state-
of-the-art methods, and better in ensemble, in a semantic web context the
evaluation strategy may be too lightweight. In order to test this, a different
evaluation strategy has been conducted (not detailed here for space reasons,
see [6]), which follows more closely the kind of SRL extraction that is
supposed to be represented in a knowledge graph. The results with this second
evaluation strategy show a lower accuracy (more than half than the one
obtained with the CoNLL scorer) because i) in strategy 2 we defined the score
so that a matching is verified only when the role filler contains all the
exact words of the gold standard, ii) in strategy 2 we took into account
VerbNet roles. The first evaluation method is based on CoNLL2009 score which
takes into account the head words only, and PropBank roles. The latter are
much lower in number with respect to their corresponding VerbNet elements.
Moreover, matching the head word only for a certain filler probably
oversimplifies the matches.

The main lesson learned is that NLP evaluation settings may be inadequate when
measuring the absolute performance of a semantic task as complex as SRL. Since
the contrastive results show that the differences in method performance are
consistent, even if at different accuracy levels, the accuracy seems to
entirely depend on the “resolution” or sensibility of the setting. We
recommend to define knowledge-graph-oriented benchmarks and scorers, and, in
the particular case of SRL, to revisit the way role ontologies are designed.

##  6 Conclusions and Future Work

In this paper we have addressed the problem of frame parsing jointly using
Framester and Stanford CoreNLP in a novel implemented algorithm, TakeFive. In
particular, we aimed at detecting verb frames and their labeled arguments
(semantic roles and fillers). In order to assess the quality of our approach,
we have carried out a comparative performance evaluation between TakeFive, and
other SRL tools including SEMAFOR, FRED, Pikes, PathLSTM. TakeFive, the only
one using a hybrid knowledge-based approach, is close to the best with the
Wall Street Journal corpus from the Penn TreeBank. We have also observed that
a simple ensemble of TakeFive, and the FRED machine reader, produces the best
overall results.

We have noticed that natural texts (even in the reasonably controlled
production of the Wall Street Journal) contain many more linguistic phenomena
than expected in existing manually developed resources such as VerbNet (for
this reason, FRED uses a greedy algorithm for SRL instead of one that is
closed under one specific resource).

As ongoing and future work, we aim at designing new role ontologies that
respond to best practices in knowledge graph design. We also want to use
ensemble learning approaches by combining multiple methods, and
feeding/controlling the ensemble pipeline by using existing linguistic
resources for SRL, and heuristical methods.

## References

## References

  * [1] A. Gangemi, V. Presutti, A pattern science for the semantic web, Semantic Web Volume 1 (2010) 61–68.
  * [2] C. J. Fillmore, Frame semantics and the nature of language, Annals of the New York Academy of Sciences 280 (1) (1976) 20–32.
  * [3] C. F. Baker, C. J. Fillmore, J. B. Lowe, The berkeley framenet project, in: Proceedings of the 17th International Conference on Computational Linguistics \- Volume 1, 1998.
  * [4] J. Rouces, G. de Melo, K. Hose, Framebase: Representing n-ary relations using semantic frames, in: European Semantic Web Conference, Springer, 2015, pp. 505–521.
  * [5] A. Gangemi, M. Alam, L. Asprino, V. Presutti, D. R. Recupero, Framester: A Wide Coverage Linguistic Linked Data Hub, in: EKAW, 2016.
  * [6] D. R. Recupero, M. Alam, A. Gangemi, V. Presutti, Frame semantic parsing using framester knowledge graphs, in: Posters & Demonstrations and Industry Tracks co-located with ISWC, 2017.
  * [7] C. D. Manning, M. Surdeanu, J. Bauer, J. Finkel, S. J. Bethard, D. McClosky, The Stanford CoreNLP natural language processing toolkit, in: ACL, System Demonstrations, 2014.
  * [8] A. Kilgarriff, The sketch engine: ten years on, Lexicography (2014) 1–30.
  * [9] P. Kingsbury, M. Palmer, From treebank to propbank, in: In Language Resources and Evaluation, 2002.
  * [10] X. Carreras, L. Màrquez, Introduction to the conll-2004 shared task: Semantic role labeling (2004).
  * [11] X. Carreras, L. Màrquez, Introduction to the conll-2005 shared task: Semantic role labeling, in: Proceedings of the Ninth Conference on Computational Natural Language Learning, CONLL ’05, 2005.
  * [12] S. T. Yi, E. Loper, M. Palmer, Can semantic roles generalize across genres, in: In Proceedings of the Human Language Technology Conference/North American chapter of the Association for Computational Linguistics annual meeting (HLTNAACL2007, 2007.
  * [13] D. Bauer, O. Rambow, Increasing coverage of syntactic subcategorization patterns in framenet using verbnet, in: IEEE Fifth International Conference on Semantic Computing, 2011.
  * [14] C. Baker, M. Ellsworth, K. Erk, Semeval task 19: Frame semantic structure extraction, in: 4th International Workshop on Semantic Evaluations, 2007.
  * [15] H. Fürstenau, M. Lapata, Semi-supervised semantic role labeling via structural alignment, Comput. Linguist. 38 (1) (2012) 135–171.
  * [16] I. Titov, A. Klementiev, A bayesian approach to unsupervised semantic role induction, in: 13th Conference of the European Chapter of the Association for Computational Linguistics, 2012.
  * [17] J. Lang, M. Lapata, Unsupervised semantic role induction with graph partitioning, in: International Conference on Empirical Methods in Natural Language Processing, 2011.
  * [18] A.-M. Giuglea, A. Moschitti, Semantic role labeling via FrameNet, VerbNet and PropBank, in: International Conference on Computational Linguistics and Annual Meeting of the Association for Computational Linguistics, ACL-44, 2006.
  * [19] D. Das, N. Schneider, D. Chen, N. A. Smith, SEMAFOR 1.0: A probabilistic frame-semantic parser, Tech. rep., Carnegie Mellon University (2010).
  * [20] V. Presutti, F. Draicchio, A. Gangemi, Knowledge extraction based on discourse representation theory and linguistic frames, in: International Conference on Knowledge Engineering and Knowledge Management, 2012.
  * [21] A. Gangemi, V. Presutti, D. R. Recupero, A. G. Nuzzolese, F. Draicchio, M. Mongiovi, Semantic Web Machine Reading with FRED, Semantic Web.
  * [22] F. Corcoglioniti, M. Rospocher, A. P. Aprosio, Extracting knowledge from text with PIKES, in: ISWC 2015 Posters & Demonstrations Track, 2015.
  * [23] M. Roth, M. Lapata, Neural semantic role labeling with dependency path embeddings, CoRR abs/1605.07515.
  * [24] K. Kipper Schuler, Verbnet: A broad-coverage, comprehensive verb lexicon, Ph.D. thesis, Philadelphia, PA, USA, aAI3179808 (2005).
  * [25] C. Fellbaum (Ed.), WordNet: an electronic lexical database, MIT Press, 1998.
  * [26] B. Levin, English verb classes and alternations: A preliminary investigation, University of Chicago press, 1993.
  * [27] A. G. Nuzzolese, A. Gangemi, V. Presutti, Gathering lexical linked data and knowledge patterns from FrameNet, in: International Conference on Knowledge Capture (K-CAP), 2011.
  * [28] R. Navigli, S. P. Ponzetto, Babelnet: The automatic construction, evaluation and application of a wide-coverage multilingual semantic network, Artif. Intell. 193 (2012) 217–250.
  * [29] M. L. de Lacalle, E. Laparra, G. Rigau, Predicate Matrix: extending SemLink through WordNet mappings, in: International Conference on Language Resources and Evaluation, 2014.
  * [30] A. Gangemi, What’s in a Schema?, Cambridge University Press, Cambridge, UK, 2010, pp. 144–182.
  * [31] J. Hajič, M. Ciaramita, R. Johansson, D. Kawahara, M. A. Martí, L. Màrquez, A. Meyers, J. Nivre, S. Padó, J. Štěpánek, P. Straňák, M. Surdeanu, N. Xue, Y. Zhang, The CoNLL-2009 Shared Task: Syntactic and Semantic Dependencies in Multiple Languages, in: Conference on Computational Natural Language Learning, 2009.
  * [32] M. Surdeanu, R. Johansson, A. Meyers, L. Màrquez, J. Nivre, The conll-2008 shared task on joint parsing of syntactic and semantic dependencies, in: Conference on Computational Natural Language Learning, 2008.
  * [33] A. Björkelund, L. Hafdell, P. Nugues, Multilingual semantic role labeling, in: 13th Conference on Computational Natural Language Learning: Shared Task, CoNLL 2009.


# TDAF: Top-Down Attention Framework for Vision Tasks

Bo Pang1, Yizhuo Li1, Jiefeng Li1, Muchen Li2,
Hanwen Cao1, Cewu Lu 1
Cewu Lu is the corresponding author.

###### Abstract

Human attention mechanisms often work in a top-down manner, yet it is not well
explored in vision research. Here, we propose the Top-Down Attention Framework
(TDAF) to capture top-down attentions, which can be easily adopted in most
existing models. The designed Recursive Dual-Directional Nested Structure in
it forms two sets of orthogonal paths, recursive and structural ones, where
bottom-up spatial features and top-down attention features are extracted
respectively. Such spatial and attention features are nested deeply,
therefore, the proposed framework works in a mixed top-down and bottom-up
manner. Empirical evidence shows that our TDAF can capture effective
stratified attention information and boost performance. ResNet with TDAF
achieves $2.0\%$ improvements on ImageNet. For object detection, the
performance is improved by $2.7\%$ AP over FCOS. For pose estimation, TDAF
improves the baseline by $1.6\%$. And for action recognition, the 3D-ResNet
adopting TDAF achieves improvements of $1.7\%$ accuracy.

##  1 Introduction

When observing an unfamiliar scene, it’s natural for humans to first take a
glimpse to grasp the part of interest and then look closer to its details.
Such a top-down attention mechanism allows global context information to be
considered by humans at an early stage so that they can quickly and precisely
understand the surroundings (Corbetta and Shulman 2002; Buschman and Miller
2007). In contrast to human’s vision system, deep neural networks(DNN), the
most powerful vision tools, work in a bottom-up manner. They extract local
features first then integrate them into global features. Thus, in DNN, local
features are aggregated step by step to help networks interpret global
context. Observing from above we intend to build a model that works in a mixed
top-down and bottom-up manner, incorporating the strong traits of deep model
and mimicking the human attention.

Despite that the design of attention mechanism has been intensively studied in
previous literature, little research has been conducted to introduce the
human-like top-down attention into mainstream deep models. Some works
formulate attention mask as a residual module (Li, Zhu, and Gong 2018; Zhang
et al. 2018; Wang et al. 2017) or built-in block (Hu, Shen, and Sun 2018; Woo
et al. 2018), where hierarchical attention features are extracted along with
basic bottom-up spatial features. Thus, this kind of attention has to focus on
local details first then on the global information. Since these bottom-up
mechanisms only access to local features at early stages, they will first
generate trivial attention features, failing to focus models on the part of
interest.

Figure 1:  Top-Down Attentions. The model introduces global information at
“top” attention stage to focus on coarse areas and then gradually adjusts the
attention to elaborate areas that contain the most important features. Such a
manner can generate fine-grained attention maps with high-level semantics and
reduce the loss of information.  Figure 2: Structure of R2DNS. a) The R2DNS
consists of multiple stages and at each stage, there is a spatial feature
extractor $h_{l}[\cdot]$ and an attention module $g_{l}[\cdot]$ (ANAR). The
dashed line denotes that it is a path across the timestamps. b) R2DNS first
samples the original input down to different scales as inputs of each
recursive stamp (vertical flow). $h_{l}[\cdot]$ and $g_{l}[\cdot]$ share the
same parameters among different flows. The attention module outputs “top”
attention maps and multiplies them on “down” features to form the top-down
flows in horizontal paths as the red lines show, while in vertical flows, the
spatial blocks extract basic features in a bottom-up manner as the blue lines
show. Note that the front vertical flows are shorter than the hind ones

In this paper, to extract the top-down attention features (see Fig. 1), we
cannot simply extract the attention along with the basic features, instead, we
utilize a recursive structure. By adding the recursive dimension, the proposed
model can establish the top-down features step by step. It consists of two
principal techniques, namely Recursive Dual-Directional Nested Structure
(R2DNS) and Attention Network across Recurrence (ANAR). As its name, R2DNS is
a recursive structure and information flows in two nested directions:
recursive and structural directions. At each recursive step, it takes as input
the images with different scales, from small to large. The small scale inputs
contain global but coarse information while the large ones contain local but
detailed information. Thus, in the recursive direction, information flows from
“global” to “local”, forming the top-down path. While in the structural
direction, the multi-layer deep model still works in the bottom-up scheme.
Therefore, R2DNS works in a mixed top-down and bottom-up manner.

Attention Network across Recurrence (ANAR) serves as a recursive bridge and
takes charge of propagating attention features along the recursive paths of
R2DNS. It is an hourglass-like network which has been successfully applied to
human pose estimation (Newell, Yang, and Deng 2016; Fang et al. 2017; Xiu et
al. 2018) and image segmentation (Long, Shelhamer, and Darrell 2015; Fang et
al. 2019). We adopt such a structure to compute attention maps based on
features of previous recursive step and then add them as soft weights on next
step’s spatial features. Thus, it is ANAR that acts as the connection bridges
of the recursive steps, instead of the hidden states and it deep nests top-
down attention to bottom-up features to work cooperatively.

We conduct comprehensive experiments on several tasks to evaluate our proposed
Top-Down Attention Framework. Results reveal that: 1) The ANAR can generate
effective attention maps with top-down characteristics. 2) The R2DNS is easy
to train in the end-to-end setting. 3) The Top-Down Attention Framework can
enjoy accuracy gained from the mixed top-down and bottom-up features, greatly
surpassing the corresponding baselines and other attention methods.

We evaluate our framework on several visual tasks: image classification on
CIFAR-10 (Krizhevsky, Hinton et al. 2009) and ImageNet (Russakovsky et al.
2015), action recognition on Kinetics (Kay et al. 2017), objection detection,
and human pose estimation on COCO (Lin et al. 2014). For image classification
and action recognition, TDAF modified on ResNet50 (He et al. 2016) and
3D-ResNet50 (Hara, Kataoka, and Satoh 2018) achieves performances better than
ResNet152 and 3D-ResNet101. For object detection, TDAF improves FCOS’s (Tian
et al. 2019) performance by 2.7% AP. For pose estimation, the performances
improve 1.6% AP on the SimplePose (Xiao, Wu, and Wei 2018).

##  2 Related Work

Attention Mechanisms Attention mechanisms widely exist in the information
processing system of human brains (Buschman and Miller 2007; Corbetta and
Shulman 2002). Many recent works have incorporated them into artificial neural
networks. In sequential problems of NLP (Bahdanau, Cho, and Bengio 2014;
Vaswani et al. 2017; Lin et al. 2017b; Xu et al. 2015), attention mechanisms
are widely adopted in recurrent neural networks (RNN) (Pang et al. 2019), Long
Short Term Memory (LSTM) (Hochreiter and Schmidhuber 1997), SCS (Pang et al.
2020b), and Transformer (Vaswani et al. 2017) to capture the relationships
between words or sentences. In computer vision, many tasks like fine-grained
recognition (Fu, Zheng, and Mei 2017; Wang et al. 2015; Fang et al. 2018; Pang
et al. 2020c), image captioning (Anderson et al. 2018; Anne Hendricks et al.
2016; Xu et al. 2015), classification (Mnih et al. 2014; Hu, Shen, and Sun
2018; Woo et al. 2018; Wang et al. 2017; Tang et al. 2020), and segmentation
(Ren and Zemel 2017; Chen et al. 2016; Cao et al. 2020) also utilize attention
mechanisms based on soft attention maps or bounding-boxes to search salient
areas. Moreover, self-attention structures (Wang et al. 2018; Zhu et al. 2019;
Huang et al. 2018; Dai et al. 2019) focusing on the combination weight of
elements (pixels in vision) are another attention method that adopts adjacent
matrix to present attentions.

The above attention mechanisms work in the bottom-up pattern. They capture
attentions first from local information, leading to neglect of context
information (Oliva et al. 2003). To this end, we propose the new recurrent
top-down attention which is closer to human’s attention (Corbetta and Shulman
2002; Buschman and Miller 2007), and inchoate attempts (Oliva et al. 2003)
have proved its effectiveness.

Multi-Scale Structures Multi-scale structures which have great scale
invariance are widely adopted in vision models. In (Chen et al. 2016; Piao et
al. 2019; Tian et al. 2019; Lin et al. 2017a; Pang et al. 2020a), different
scales take charge of detecting the object with different sizes. DLA (Yu et
al. 2018) aggregates features from different layers to form more efficient
ones. SlowFast network (Feichtenhofer et al. 2018) adopts different scales
both in the spatial dimension and temporal dimension to split the works of
temporal and spatial feature extracting and reduce the resource consumption.
HRNet (Sun et al. 2019) consists of several parallel flows with inputs of
different scales, which aims at maintaining the high-resolution feature map
across the whole network, instead of regaining them from the low-resolution
map.

In our model, multi-scale inputs are not treated equally like previous works.
The attention information flows only from small scale inputs to large ones.
Because the attention features from small scale inputs are used as soft masks
added onto large ones, these attention features can also be treated as control
gates working on the spatial features.

##  3 Approach

To imitate human brains that observe environment with a top-down attention
mechanism, we propose the Top-Down Attention Framework (TDAF) with a recursive
structure. In this section, we will introduce its two main parts in turn: 1)
the Recursive Dual-Directional Nested Structure (R2DNS) designed to form the
mixed top-down and bottom-up information paths in dual directions; 2) the
Attention Network across Recurrence (ANAR) designed to generate and propagate
top-down attention along top-down (recursive) paths.

###  3.1 Recursive Dual-Directional Nested Structure

Inspired from the working manner of human attention, we need to build top-down
information flow paths in the model, which gradually form attentions from
coarse to detailed.

Challenge It is not easy to endow the existing deep vision models a top-down
information flow, as the stacked CNN layers need to combine low-level local
features into high-level global ones and this is a bottom-up process. Adopting
residual or parallel modules like (He et al. 2016; Wang et al. 2017; Zhang et
al. 2018) can add more flows but they still behave like traditional CNN,
working in a bottom-up manner. Adding a reverse flow from high layers like
(Lin et al. 2016, 2017a; Tian et al. 2019) cannot combine the bottom-up
spatial features and top-down attention maps.

Overall Structure Therefore, we go down to consider how to form the top-down
path and keep the high-level spatial features simultaneously. For this, we
design the Recursive Dual-Directional Nested Structure (R2DNS). It adds
another dimension, the recursive dimension, into the traditional bottom-up
networks, with which the model can form orthogonal information paths in
recursive and structural directions marked as horizontal paths (red lines) and
vertical flows (blue lines) in Fig. 2 b. The same as other CNN networks,
vertical flows are bottom-up ones which are responsible for capturing deep
spatial representations. And in horizontal direction, the recursive paths
extract top-down attention features with a tailored multi-scale mechanism.

As shown in Fig. 2 a, in vertical flows, the computing units are
$\\{h_{l}\\}$, responsible for capturing deep spatial representations as other
CNN networks, where $l\in\\{1,2,...,L\\}$ and $L$ is the number of stages in
R2DNS. Besides $\\{h_{l}\\}$, we design recursive modules $\\{g_{l}\\}$. They
take spatial features of the last vertical flow as input to generate attention
maps and map them onto the next flow. These recursive operations between
adjacent flows form the horizontal information paths.

Vertical Flows The unrolled and detailed calculation procedure is shown in
Fig. 2 b. In each vertical flow, the network is a typical deep CNN model with
multiple stages: $\\{h_{l}|l\in\\{0,1,...,L\\}\\}$. It can be implemented
based on popular image or video backbones. Note that we apply the asymmetric
recursive structure: the front vertical flows are shorter (have fewer stages)
than the hind ones. With an original backbone having $L$ stages, the number of
stages $S$ in vertical flow $n$ can be formalized as:

$S^{(n)}=L-(N-n)~{},\vspace{-0.05in}$ (1)

where $N\leq L$ is the total number of flows. And weights are shared among the
vertical flows.

In order to form top-down information along the horizontal paths, we adopt a
down-sample function (DSF) to generate the input $\mathbf{x}_{1}$ of each
vertical flow $n$ as:

$\displaystyle\mathbf{x}_{1}^{(N)}$ $\displaystyle=\mathbf{X}$ (2)
$\displaystyle\mathbf{x}_{1}^{(n)}$ $\displaystyle={\rm
downsample}[\mathbf{x}_{1}^{(n+1)}]~{},$

where $\mathbf{X}$ is the original input of the whole framework. DSF removes
details, therefore, the same receptive field will contain more global
information after DSF. This setting makes front vertical flows access to
global but coarse information while hind flows get local and detailed one,
which form top-down flows in horizontal paths.

Figure 3: a) Structure of ANAR. As an hourglass module, ANAR first reduces the
resolution to enlarge the receptive field. Then DeCNN layers are adopted to
map the features to the same size of basic features in the next flow. “SC”, a
single layer CNN, denotes the skip connects which assist to capture the multi-
scale information. b) Attention feature maps with different degrees of
fineness generated in different recursive flows. As expected, the front flows
capture coarse attention heat-maps to form the “top” attention and the
following flows refine them to more accurate ones and gradually form the
“down” attention. This can reduce the information loss caused by the
inaccurate attention generated by a single-step. Here we only exhibit the maps
of the first stage. c) Besides the top-down attention in the recursive
direction, the deep model also forms the bottom-up feature maps along the
structural direction

Horizontal Paths In the horizontal direction, $g_{l}$ takes global coarse
spatial features from the last vertical flow as input, generates attention
maps based on them, and maps the attention onto local detailed spatial
features of next flow. These recursive operations link up the vertical flows
to form new horizontal information paths, and at each stage of R2DNS, due to
the global-to-local setting, a series of attention maps from “top” to “down”
are formed in these paths. Note that weights of $g_{l}$ are shared among the
recursive steps. The detailed structure of $g_{l}$ is introduced in Sec. 3.2.

Formal Representation The calculation procedure can be written as:

$\displaystyle\mathbf{x}_{l+1}^{(n)}=h_{l}[\mathbf{x}_{l}^{(n)}]*g_{l}[\mathbf{x}_{l+1}^{(n-1)}]~{},$
(3)

where $\mathbf{x}_{l}^{(n)}$ is the input of the $l$th stage in the $n$th flow
and $*$ is the element-wise multiplication. Note that at the start of each
horizontal path, the attention feature $g_{l}[\mathbf{x}_{l+1}^{(n-1)}]$ is
initialized to $\mathbf{1}$ as Fig. 2 b shows. From Eq. 3, we can see that
$\mathbf{x}_{l+1}^{(n)}$ is generated by merging the output of the spatial
block $h_{l}$ in the last stage and the output of attention module $g_{l}$ in
the front flow. As $h_{l}[\cdot]$ extracts bottom-up features, and
$g_{l}[\cdot]$ works in a top-down manner, the R2DNS is a mixed top-down and
bottom-up model. $g_{l}[\cdot]$ outputs attention masks and they are
multiplied onto the basic CNN features generated by $h_{l}[\cdot]$. These two
different information flows transfer in different directions and they are
nested deeply at each element-wise multiplication junction.

Let’s take a fresh look at the asymmetric recursive structure. From the
viewpoint of vertical flows, inputs of front flow with small scales and fewer
details do not need deep models. While for horizontal paths, high-level
spatial features that are more abstract and have smaller scales do not need
long attention chains to form top-down attention features gradually. Thus,
this asymmetric structure can reduce unnecessary calculations.

Multi-Flow Batch Normalization Batch normalization (BN) (Ioffe and Szegedy
2015) is a powerful module designed to facilitate the training process of deep
model. As a recursive structure, multiple vertical flows share the same batch
normalization module at each stage. However, the activation of different flows
does not share the same distribution. Because the hind flows are masked by the
front ones’ attention maps, they usually have smaller mean. To this end, we
propose the Multi-Flow Batch Normalization (MFBN), which shares the affine
parameters among the vertical flows, while for each flow, it has independent
mean and variance:

$\displaystyle\hat{x}^{(n)}_{i}$
$\displaystyle=\frac{x^{(n)}_{i}-\mu_{\beta^{(n)}}}{\sqrt{\sigma_{\beta^{(n)}}^{2}+\epsilon}}$
(4) $\displaystyle y^{(n)}_{i}$
$\displaystyle=\gamma\hat{x}^{(n)}_{i}+\alpha\equiv{\rm
MFBN}_{\gamma,\alpha}(x_{i}^{(n)})~{},$

where $\beta^{(n)}={x_{1...m}^{(n)}}$ is a mini-batch of $x^{(n)}$, $\gamma$
and $\alpha$ are the affine parameters shared among the flows. The mean
$\mu_{\beta^{(n)}}$ and variance $\sigma_{\beta^{(n)}}$ are independent among
the flows.

###  3.2 Attention Network across Recurrence

Attention Network across Recurrence (ANAR) is the computing module responsible
for transferring information among the recursive flows of R2DNS. It is also
the attention module ($g_{l}$) that generates soft attention maps and masks
them on basic spatial features outputted by $h_{l}[\cdot]$. This soft
attention mechanism can be seen as a control gate of spatial features like
(Srivastava, Greff, and Schmidhuber 2015; Wang et al. 2017; Pang et al. 2019),
aiming at strengthening foreground features and weakening background ones.

Structure Extracting attention feature maps is a pixel-level task, thus we
design ANAR following the hourglass network (Newell, Yang, and Deng 2016)
which is usually used in pose estimation. The detailed structure is shown in
Fig. 3 a. In the first half, we apply CNN with stride 2 to reduce the feature
space and extract the attention related information. Transposed CNN (DeCNN) is
adopted in the second half to map attention maps back to the same size of
basic features in the next flow. Like (Newell, Yang, and Deng 2016; Wang et
al. 2017), we also add skip connections between the corresponding down-sample
and up-sample parts to maintain the multi-scale spatial information. The final
activation of ANAR is the sigmoid function which converts feature maps to soft
attention masks. To fit different backbones and balance performances with
computational requirements, we propose three ANAR structures with different
depths. Details are shown in Tab. 1.

Due to the sigmoid activation function which generates values less than 1, the
soft attention mask can only reduce the strength of basic spatial activation.
In a deep nested and recursive structure, this property will largely degrade
the value of final features. To deal with these two problems, we add a scaled
shortcut connection based on Eq. 3 as the following equation shows.

$\displaystyle\mathbf{x}_{l+1}^{(n)}$
$\displaystyle=h_{l}[\mathbf{x}_{l}^{(n)}]*g_{l}[\mathbf{x}_{l+1}^{(n-1)}]+\eta*h_{l}[\mathbf{x}_{l}^{(n)}]$
(5)
$\displaystyle=h_{l}[\mathbf{x}_{l}^{(n)}]*(g_{l}[\mathbf{x}_{l+1}^{(n-1)}]+\eta)~{},$

where $\eta\in[0,1]$ is the shortcut parameter. When $\eta$ is set as 1, this
shortcut connection forms a residual learning structure (He et al. 2016),
which will solve the problem of activation value drop. However, if $\eta=1$,
$(g_{l}[\mathbf{x}_{l+1}^{(n-1)}]+\eta)\in(1,2)$ holds true, therefore, the
soft mask instead can only enhance the spatial activation. In order to both
enhance useful features and suppress noise features, we set $\eta$ as 0.5, and
we call this “half-scale” residual learning.

In summary, ANAR forms a series of top-down attention features by a group of
recursive operations: firstly generate small coarse attention maps based on
the global information, then map this attention onto the more detailed basic
spatial feature in the next flow and again, utilize this mapped spatial
feature to generate larger and more detailed attention maps. Different from
existing attention works (Wang et al. 2017; Xu et al. 2015), besides the
multi-level attention formed alone the structural direction, our model also
outputs multiple attention maps with different degrees of fineness (bottom-up
and top-down attentions in Fig. 3 b, c). This process can significantly reduce
the information loss caused by the one-step generated inaccurate attention.

Table 1: Detailed settings of ANAR architecture. The structure is shown in Fig. 3 a. $c$ denotes the number of input channels which depends on $h_{l}[\cdot]$, “Down-SP” denotes down-sample and “/2” means the stride is 2. We adopt MFBN after convolutional layers. Except for the last layer which adopts sigmoid function, ReLU is the activation function of each CNN and DeCNN layer Stage | ANAR-7 | ANAR-5 | ANAR-3
---|---|---|---
Trans (CNN) | 1$\times$1, $c/4$ | 1$\times$1, $c/4$ | 1$\times$1, $c/8$
Down-SP (CNN) |  | 3$\times$3, $c/8$, /2
---
3$\times$3, $c/8$, /2
3$\times$3, $c/8$, /2 | -
Up-SP (DeCNN) |  | 4$\times$4, $c/16$, /2
---
4$\times$4, $c/16$, /2
4$\times$4, $c/32$, /2
| 4$\times$4, $c/16$, /2
---
4$\times$4, $c/32$, /2
4$\times$4, $c/32$, /2
Trans (CNN) | 1$\times$1, $1$ | 1$\times$1, $1$ | 1$\times$1, $1$

###  3.3 Comparsion with Other Attention Models

Main stream attention mechanisms can be divided into soft attention (Hu, Shen,
and Sun 2018; Woo et al. 2018; Chen et al. 2016; Wang et al. 2017; Fukui and
et al. 2019) and self-attention (Wang et al. 2018; Huang et al. 2018; Zhou et
al. 2018; Chen et al. 2018; Yue and et al. 2018). Of course, there are some
attention achieved by detecting bounding-box (Fu, Zheng, and Mei 2017;
Anderson et al. 2018; Mnih et al. 2014), but this is not the focus of this
paper. The main purpose of conventional bottom-up soft attention is to build a
group of new weights that is relevant to the input, instead of the fixed
convolutional kernel. One problem of this mechanism is that in low layers the
receptive field of attention module is too small to form effective attentions.
For self-attention, it replaces convolutional operation by attention mechanism
in order to enlarge the receptive field, where the essence is to construct
high-level features by calculating the “relevance” between pixels. But its
computing complexity is too large to applied on low layers. While our TDAF
aims at forming top-down attention chains, building attention maps from coarse
to fine, solving the above problems. In the term of design purpose, the TDAF
is a new attention mechanism.

##  4 Experiments

In this section, we evaluate our TDAF on four vision tasks: image
classification, object detection, pose estimation, and action recognition. The
results reveal TDAF’s great performance and universality.

Table 2: Performance (Acc) on CIFAR-10 test set. The 2-layer ANAR replaces the up-sample layer in ANAR-3 with a parameter-free interpolation. All the models with attention recur three times, i.e. 3 recursive flows Backbone | Attention | Params | Acc
---|---|---|---
VGG16 | No Att | 138.4M | 92.64
ANAR-2 | 138.4M | 93.02
ANAR-3 | 138.5M | 93.43
VGG19 | No Att | 143.7M | 93.13
ANAR-3 | 143.8M | 93.78
Res50 | No Att | 25.6M | 93.62
ANAR-3 | 29.9M | 94.13
Res101 | No Att | 44.5M | 93.75
ANAR-3 | 48.9M | 94.21

###  4.1 Image Classification

We first evaluate TDAF on image classification with CIFAR-10 (Krizhevsky,
Nair, and Hinton 2014) and ImageNet-2012 (Russakovsky et al. 2015) datasets
and compare it with the following baselines: VGG (Simonyan 2014), ResNet (He
et al. 2016), ResNeXt (Xie et al. 2017), MobileNet (Howard et al. 2017), and
SENet (Hu, Shen, and Sun 2018).

Implementation For VGG, we insert 5 ANAR modules after each max-pooling layer
to form the recursive attention structure. While for the ResNe(X)t, we insert
ANARs after each stage respectively. All the BN layers are replaced by our
MFBN. Baseline network and its TDAF counterpart are trained with identical
optimization scheme following (He et al. 2016). The parameters are initialized
by “Kaiming initialization” proposed in (He et al. 2015). We use SGD optimizer
with 256 mini-batch on 8 GPUs to train.

Table 3: a) Performance on ImageNet validation set (10-crop test). “# L” and
“# F” denote the number of ANAR’s layers and flows (recursive steps). b)
Comparison of performance improvement (PI) on Top-1 accuracy (1-crop test) and
computational overheads on ImageNet between different attention mechanisms

(a)  Backbone | ANAR | Params | GFlops | Accuracy
---|---|---|---|---
# L | # F | Top-1 | Top-5
VGG16 (Simonyan 2014) | - | - | 138.4M | 15.7 | 74.7 | 92.0
5 | 3 | 138.9M | 21.6 | 75.9 | 92.7
5 | 2 | 138.9M | 20.6 | 75.5 | 92.4
Res50 (He et al. 2016) | - | - | 25.6M | 4.1 | 77.1 | 93.3
3 | 3 | 29.9M | 5.4 | 78.4 | 94.1
5 | 3 | 30.3M | 5.8 | 78.6 | 94.3
7 | 3 | 30.5M | 5.8 | 78.8 | 94.5
3 | 4 | 29.9M | 5.4 | 78.7 | 94.4
Res101 (He et al. 2016) | - | - | 44.5M | 7.9 | 78.2 | 93.9
3 | 3 | 48.9M | 10.0 | 79.9 | 94.8
3 | 4 | 48.9M | 10.0 | 80.2 | 95.0
Res152 (He et al. 2016) | - | - | 60.2M | 11.6 | 78.6 | 94.3
3 | 3 | 64.5M | 14.8 | 80.4 | 95.1
ResX50 (Xie et al. 2017) | - | - | 25.0M | 4.3 | 79.4 | 94.7
3 | 3 | 29.4M | 5.6 | 80.3 | 95.0
ResX101 (Xie et al. 2017) | - | - | 88.8M | 16.5 | 81.0 | 95.5
3 | 3 | 93.1M | 20.8 | 81.8 | 95.8
MobNets (Howard et al. 2017) | - | - | 4.2M | 0.57 | 68.6 | 88.49
3 | 2 | 5.1M | 0.71 | 71.0 | 90.01

(b)  Baseline | Model | GFLOPs | PI
---|---|---|---
VGG16 | TDAF | +37.5% | +1.3
ABN (Fukui and et al. 2019) | - | +0.7
Res50 | TDAF | +31.7% | +1.7
$\rm A^{2}$-Net (Chen et al. 2018) | +58.5% | +1.7
NL (Wang et al. 2018) | +12.5% | +0.8
ResAtt (Wang et al. 2017) | +53.6% | +1.1
GCNL (Yue and et al. 2018) | - | +1.2
| SAST (Parmar et al. 2019) | +34.5% | +1.2
| AAAtt (Bello et al. 2019) | +1.3% (29%) | +1.3
Res152 | TDAF | +27.5% | +2.0
ABN (Fukui and et al. 2019) | - | +0.8
GCNL (Yue and et al. 2018) | - | +1.1
ResX101 | TDAF | +24.5% | +0.9
NL (Wang et al. 2018) | +9.9% | +0.4
CBAM (Woo et al. 2018) | +0.1% | +0.4
SENet | TDAF | +26.0% | +1.3
ABN (Fukui and et al. 2019) | - | +0.8
CBAM (Woo et al. 2018) | +0.1% | +0.5
CBAM | TDAF | +25.9% | +0.7
ABN (Fukui and et al. 2019) | - | +0.3
NL (Wang et al. 2018) | +12.5% | +0.3

CIFAR-10 and Analysis We first evaluate TDAF on the small dataset CIFAR-10 to
reveal its effectiveness. As the resolution is only 32 $\times$ 32, we adopt
the 3-layer ANAR. We also replace the DeCNN layer in ANAR-3 with a parameter-
free interpolation function to build a much smaller ANAR-2 specifically for
this small-scale task.

The experiment results are shown in Tab. 2. We observe that TDAF consistently
improves the performance across different backbones and model depths with a
small increase in parameters. VGG19 with attention exceeds baseline by 0.65%,
and ResNet-50 by 0.51%, even better than the much deeper baseline ResNet-101
with much fewer parameters (29.9M vs. 45.5M). In order to test whether we can
get great attention with nearly no parameters in ANAR, we replace ANAR-3’s
DeCNN layer with a parameter-free interpolation to build a much smaller
ANAR-2. Unfortunately, this trial leads to a great drop in performance,
revealing the importance of the learnable parameters in ANAR.

ImageNet and Analysis We further analyze different attention settings on
ImageNet-2012 (Russakovsky et al. 2015). Training images are resized randomly
to [256, 480] with its shorter side and a 224 $\times$ 224 crop is sampled
from it or its horizontal flip. With larger input size, we apply deeper ANAR
and more recursive steps to maximize its potential.

The results are shown in Tab. 3(a). The networks with TDAF outperform all the
baselines significantly. Note that even applying the smallest attention
setting (ANAR-3 with 3 recursive flows), our model adopting ResNet50 as the
backbone exceeds deeper ResNet-101 (78.4% vs. 78.2%) with much fewer
parameters and FLOPs. And this pattern holds at ResNet101 which outperforms
ResNet152 by 1.3% in Top-1 Accuracy. On ResNeXt50 and ResNeXt101 (Xie et al.
2017), there are still considerable improvements (0.9% and 0.8%). For
MobileNet (Howard et al. 2017), our ANAR adopts depthwise convolutional
filters proposed in (Howard et al. 2017) and the performance is improved by
2.4%.

As for different attention settings, the results reveal that a deeper ANAR
module or more recursive steps will lead to better results due to better
abilities to extract attention features and longer top-down attention paths
which endows the model larger initial receptive fields.

Performance Comparison of Attention Models Here, we compare our TDAF with
other attention models on ImageNet. As shown in table 3(b), on different
baselines, TDAF is competitive among all the mechanisms, achieving great
performance boost with relatively small Flops overheads. As for parameter
overhead, TDAF adopts 10% more parameter on ResNet101 and 7% on ResNet152.
This overhead less than 10% leads to 2% Top-1 accuracy improvements. Other
attention models like Non-local (20% overhead, 0.8% improvements), A2 Net
(10%, 1.7%), ResAtt (24%, 1.1%), ABN (>20%, 0.8%), and GCNL (>20%, 1.2%) adopt
more parameters and gains less improvements. For each unit of parameters
increase, our model delivers 3$\sim$5 times performance increases of Non-
Local, ResAtt, ABN, and GCNL. Note that compared with CBAM which directly
generates maps based on the original convolutional features, which cannot be
adjusted to top-down attention, TDAF introduces more overhead like other
attention methods. To further demonstrate the performance of TDAF, we treat
CBAM as an advanced baseline. After adopting TDAF, it achieves a considerable
performance boost, revealing that the top-down and bottom-up soft attentions
can work cooperatively and propel each other to better performances.

###  4.2 Object Detection

We also evaluate TDAF on detection task with the COCO-2017 dataset (Lin et al.
2014). We choose the one-stage anchor-free model FCOS (Tian et al. 2019) as
baseline.

Implementation Still, the $\rm downsample$ stride is 2. We modify the backbone
of FCOS with our TDAF. The data pre-processing and training methods follow the
original FCOS (“2$\times$” training schedule with multi-scale input). The
weights of all the backbones are initialized by the parameters pre-trained on
the ImageNet dataset.

Table 4: Performance and training curve on COCO-2017. “4F” denotes the number
of recursive flows is 4

Backbone | Top-Down Att | AP | $\text{AP}_{S}$ | $\text{AP}_{M}$ | $\text{AP}_{L}$ | Params
---|---|---|---|---|---|---
ResNet50 (He et al. 2016) | No Att | 37.1 | 21.3 | 41.0 | 47.8 | 32.244M
ANAR-3, 3F | 39.6 | 23.9 | 43.4 | 51.1 | 32.462M
ANAR-5, 3F | 39.8 | 24.0 | 43.3 | 51.2 | 32.885M
ResNet101 (He et al. 2016) | No Att | 41.5 | 24.4 | 44.8 | 51.6 | 51.184M
ANAR-3, 3F | 42.0 | 25.5 | 46.4 | 54.4 | 51.454M
ANAR-3, 4F | 42.3 | 25.3 | 46.7 | 55.1 | 51.454M
ANAR-5, 3F | 42.2 | 25.3 | 46.6 | 55.3 | 51.877M

Tab. 4 reports the performances. TDAF-Res50-FCOS outperforms Res50-FOCS by
2.7% on the standard AP metric (7.3% relatively). Top-down attention also
benefits the deeper Res101-FCOS and achieves 0.9% improvements.

To provide some insights into the influence of Top-Down Attention Framework on
the optimization process, the training curves are depicted in Tab. 4. The TDAF
yields lower training errors throughout the optimization procedure and more
recursive steps or deeper ANAR lead to lower training losses throughout the
optimization procedure.

###  4.3 Pose Estimation

We further evaluate TDAF on pose estimation task using the COCO-2017 dataset.
The mainstream methods of this task utilize heat-maps to localize key point
positions. To some extent, this is a kind of attention-based solution. We want
to evaluate whether our top-down attention mechanism can benefit this already
attention-driven task.

Implementation We adopt SimplePose (Xiao, Wu, and Wei 2018) as our baseline
and modify its deconvolutional layers into our TDAF version with 3 recursive
flows. The details of training processes maintain the same with the original
baseline. We adopt Faster-RCNN (Girshick 2015) and YOLO-v3 (Redmon and Farhadi
2018) detectors.

The results are shown in Tab. 5. With ResNet50 as the backbone, our TDAF
improves AP by 1.6%, while for ResNet101, the improvements are 0.8%.
Considering pose estimation is already based on attention, these promotions
are considerable, demonstrating the potential of the top-down manner in
attention mechanism. Also, a deeper ANAR structure leads to better
performances.

Table 5: Performance (AP in %) of the SimplePose (Xiao, Wu, and Wei 2018) on COCO-2017 validation set. “3F” denotes 3 recursive flows. “FRCNN” and “YOLO” denote Faster-RCNN and YOLO-V3 detectors Backbone | Top-Down Att | Detector
---|---|---
GT | FRCNN | YOLO
ResNet50 | No Att | 72.7 | 70.1 | 70.4
ANAR-3, 3F | 73.8 | 71.3 | 71.4
ANAR-5, 3F | 74.1 | 71.5 | 71.5
ANAR-7, 3F | 74.3 | 71.6 | 71.7
ResNet101 | No Att | 73.7 | 70.9 | 71.3
ANAR-3, 3F | 74.4 | 71.9 | 72.4
ANAR-5, 3F | 74.5 | 72.0 | 72.3

###  4.4 Action Recognition

Next, we apply the proposed framework onto the video action recognition task.
We adopt 3D-ResNet (Hara, Kataoka, and Satoh 2018) as our baseline and the
experiments are conducted on Kinetics-400 dataset (Kay et al. 2017).

Implementation In these experiments, the $\rm downsample$ function is
conducted on 3 dimensions, where we treat the temporal and spatial dimensions
equally, just as the 3D-CNN does. In 3D-ResNet, all the 2D kernels are
extended to 3D, therefore, we also adopt a 3D version ANAR. Here, the ANAR-5
with 3 recursive steps is utilized. Similar to (Carreira and Zisserman 2017),
we inflate the 2D weights pre-trained on ImageNet to initialize the model.

We split the video into clips of 16 frames as input and the frames are resized
to 112 $\times$ 112\. The other training and inference settings all follow
(Hara, Kataoka, and Satoh 2018).

Results on the Kinetics dataset shown in Tab. 6 demonstrate consistent
performances of TDAF with the above tasks. It outperforms 3D-ResNet50 by 1.8%
without ImageNet pre-training and 1.2% with pre-training. The same, TDAF with
ResNet50 outperforms the ResNet101 baseline (67.5% vs. 67.2%), revealing that
the TDAF also has great performances in capturing the spatial and temporal
attention simultaneously and can be applied to video tasks.

Table 6: Performance on the Kinetics validation set. We adopt ANAR-5 and 3 recursive flows in our TDAF models Model | Params | Acc (%)
---|---|---
3D-ResNet50 | 47.019M | 65.7
3D-ResNet50, Pre-trained | 47.019M | 68.9
3D-ResNet101 | 86.065M | 67.2
TDAF-3D-ResNet50 | 48.417M | 67.5
TDAF-3D-ResNet50, Pre-trained | 48.417M | 70.1

###  4.5 Analysis

On Multi-Scale and Attention Until now, there is a question that whether the
great improvements are achieved by our top-down attention mechanism or simply
by the multi-scale information. Note that we only feed the features from the
last flow into the task heads as shown in Fig. 2, thus from the view of the
task head, TDAF does not access to the multi-scale spatial features, instead,
it only utilizes the attention features from other flows. Even so, we conduct
controlled trials on ImageNet to show that it is the top-down attention that
works in the whole framework. On ResNet-50, we adopt the same R2DNS setting (3
flows) and remove the attention modules among the recursive steps. In this way
the multiple flows are standalone without attention connections. Then the
features of all the flows are concatenated to conduct the classification in a
multi-scale way. Compared with the corresponding TDAF with ANAR-3, the
performance of this multi-scale setting drops 1.1% Top-1 accuracy on ImageNet,
achieving 77.5%, only 0.4% improvements over the baseline without multi-scale
setting (more results in supplementary files). This reveals that multi-scale
information can lead to better performances, but the great improvements are
actually achieved by our top-down attention mechanism.

On Time Overhead One may be concerned that the recurrent structure cannot be
parallelled and lead to a long time delay. But actually, only the ANAR parts
cannot be parallelled, while the multiple backbone flows can. As Fig. 2b
shows, in first stage, we could calculate $h_{1}^{(1)}$ to $h_{1}^{(N)}$
parallelly, then calculate $g_{1}^{(1)}$ to $g_{1}^{(N-1)}$ recursively, so
can other stages. Thus, the total time of our model is just the sum of the
time of the original backbone and the attention layers. Because the attention
layers and the additional backbone flows are much lighter than the original
backbone, the latency overhead is less than 25% for Res50 and 15% for Res101.
On TitanXP with 256 batch size, the forward time of Res50 and its attention
version is 0.015s and 0.018s, and 0.031s vs 0.035s for Res101. Compared with
GALA (Linsley et al. 2018) (86%), CBAM (56%), and AAAtt (29%) (these results
are published in (Bello et al. 2019)), this latency is in a reasonable range.
For detection, because the head and NMS consume much time, the total time
overhead on Res50 is less than 10%.

##  5 Conclusion

In this paper, we proposed TDAF to mix top-down attention features with
bottom-up spatial features. The first part of our framework is the Recursive
Dual-Directional Nested Structure (R2DNS) designed to form the orthogonal top-
down and bottom-up information paths and the second part is the Attention
Network across Recurrence (ANAR) aiming at extracting attention maps to
enhance useful features and suppress noises. Compared with baselines and other
attention mechanisms, our TDAF achieves remarkable improvements on image
classification, object detection, pose estimation, and action recognition.
Comprehensive analyses were presented to further validate different settings
of TDAF.

##  6 Acknowledgement

This work is supported in part by the National Key R&D Program of China, No.
2017YFA0700800, National Natural Science Foundation of China under Grants
61772332 and Shanghai Qi Zhi Institute, SHEITC (018-RGZN-02046).

## References

  * Anderson et al. (2018) Anderson, P.; He, X.; Buehler, C.; Teney, D.; Johnson, M.; Gould, S.; and Zhang, L. 2018.  Bottom-up and top-down attention for image captioning and visual question answering.  In _CVPR_ , 6077–6086.
  * Anne Hendricks et al. (2016) Anne Hendricks, L.; Venugopalan, S.; Rohrbach, M.; Mooney, R.; Saenko, K.; and Darrell, T. 2016.  Deep compositional captioning: Describing novel object categories without paired training data.  In _CVPR_ , 1–10.
  * Bahdanau, Cho, and Bengio (2014) Bahdanau, D.; Cho, K.; and Bengio, Y. 2014.  Neural machine translation by jointly learning to align and translate.  _arXiv preprint arXiv:1409.0473_ .
  * Bello et al. (2019) Bello, I.; Zoph, B.; Vaswani, A.; Shlens, J.; and Le, Q. V. 2019.  Attention augmented convolutional networks.  In _ICCV_ , 3286–3295.
  * Buschman and Miller (2007) Buschman, T. J.; and Miller, E. K. 2007.  Top-down versus bottom-up control of attention in the prefrontal and posterior parietal cortices.  _science_ 315(5820): 1860–1862.
  * Cao et al. (2020) Cao, H.; Lu, Y.; Lu, C.; Pang, B.; Liu, G.; and Yuille, A. 2020.  ASAP-Net: Attention and Structure Aware Point Cloud Sequence Segmentation.  _arXiv preprint arXiv:2008.05149_ .
  * Carreira and Zisserman (2017) Carreira, J.; and Zisserman, A. 2017.  Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset .
  * Chen et al. (2016) Chen, L.-C.; Yang, Y.; Wang, J.; Xu, W.; and Yuille, A. L. 2016.  Attention to scale: Scale-aware semantic image segmentation.  In _CVPR_ , 3640–3649.
  * Chen et al. (2018) Chen, Y.; Kalantidis, Y.; Li, J.; Yan, S.; and Feng, J. 2018.  A^ 2-nets: Double attention networks.  In _NeurIPS_ , 352–361.
  * Corbetta and Shulman (2002) Corbetta, M.; and Shulman, G. L. 2002.  Control of goal-directed and stimulus-driven attention in the brain.  _Nature reviews neuroscience_ 3(3): 201.
  * Dai et al. (2019) Dai, Z.; Yang, Z.; Yang, Y.; Cohen, W. W.; Carbonell, J.; Le, Q. V.; and Salakhutdinov, R. 2019.  Transformer-xl: Attentive language models beyond a fixed-length context.  _arXiv preprint arXiv:1901.02860_ .
  * Fang et al. (2018) Fang, H.-S.; Cao, J.; Tai, Y.-W.; and Lu, C. 2018.  Pairwise body-part attention for recognizing human-object interactions.  In _ECCV_ , 51–67.
  * Fang et al. (2019) Fang, H.-S.; Sun, J.; Wang, R.; Gou, M.; Li, Y.-L.; and Lu, C. 2019.  Instaboost: Boosting instance segmentation via probability map guided copy-pasting.  In _Proceedings of the IEEE International Conference on Computer Vision_ , 682–691.
  * Fang et al. (2017) Fang, H.-S.; Xie, S.; Tai, Y.-W.; and Lu, C. 2017.  Rmpe: Regional multi-person pose estimation.  In _ICCV_ , 2334–2343.
  * Feichtenhofer et al. (2018) Feichtenhofer, C.; Fan, H.; Malik, J.; and He, K. 2018.  Slowfast networks for video recognition.  _arXiv preprint arXiv:1812.03982_ .
  * Fu, Zheng, and Mei (2017) Fu, J.; Zheng, H.; and Mei, T. 2017.  Look closer to see better: Recurrent attention convolutional neural network for fine-grained image recognition.  In _CVPR_ , 4438–4446.
  * Fukui and et al. (2019) Fukui, H.; and et al. 2019.  Attention Branch Network: Learning of Attention Mechanism for Visual Explanation arXiv:1812.10025v2.  In _CVPR_.
  * Girshick (2015) Girshick, R. 2015.  Fast r-cnn.  In _ICCV_ , 1440–1448.
  * Hara, Kataoka, and Satoh (2018) Hara, K.; Kataoka, H.; and Satoh, Y. 2018.  Can Spatiotemporal 3D CNNs Retrace the History of 2D CNNs and ImageNet?  In _CVPR_ , 6546–6555.
  * He et al. (2015) He, K.; Zhang, X.; Ren, S.; and Sun, J. 2015.  Delving deep into rectifiers: Surpassing human-level performance on imagenet classification.  In _ICCV_ , 1026–1034.
  * He et al. (2016) He, K.; Zhang, X.; Ren, S.; and Sun, J. 2016.  Deep residual learning for image recognition.  In _CVPR_ , 770–778.
  * Hochreiter and Schmidhuber (1997) Hochreiter, S.; and Schmidhuber, J. 1997.  Long Short-Term Memory.  _Neural Computation_ 9: 1735–1780.
  * Howard et al. (2017) Howard, A. G.; Zhu, M.; Chen, B.; Kalenichenko, D.; Wang, W.; Weyand, T.; Andreetto, M.; and Adam, H. 2017.  Mobilenets: Efficient convolutional neural networks for mobile vision applications.  _arXiv preprint_ .
  * Hu, Shen, and Sun (2018) Hu, J.; Shen, L.; and Sun, G. 2018.  Squeeze-and-excitation networks.  In _CVPR_ , 7132–7141.
  * Huang et al. (2018) Huang, Z.; Wang, X.; Huang, L.; Huang, C.; Wei, Y.; and Liu, W. 2018.  Ccnet: Criss-cross attention for semantic segmentation.  _arXiv preprint arXiv:1811.11721_ .
  * Ioffe and Szegedy (2015) Ioffe, S.; and Szegedy, C. 2015.  Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift .
  * Kay et al. (2017) Kay, W.; Carreira, J.; Simonyan, K.; Zhang, B.; Hillier, C.; Vijayanarasimhan, S.; Viola, F.; Green, T.; Back, T.; Natsev, P.; et al. 2017.  The kinetics human action video dataset.  _arXiv preprint_ .
  * Krizhevsky, Hinton et al. (2009) Krizhevsky, A.; Hinton, G.; et al. 2009.  Learning multiple layers of features from tiny images.  Technical report, Citeseer.
  * Krizhevsky, Nair, and Hinton (2014) Krizhevsky, A.; Nair, V.; and Hinton, G. 2014.  The CIFAR-10 dataset.  _online: http://www. cs. toronto. edu/kriz/cifar. html_ 55\.
  * Li, Zhu, and Gong (2018) Li, W.; Zhu, X.; and Gong, S. 2018.  Harmonious attention network for person re-identification.  In _CVPR_ , 2285–2294.
  * Lin et al. (2016) Lin, T. Y.; Dollár, P.; Girshick, R.; He, K.; and Belongie, S. 2016.  Feature Pyramid Networks for Object Detection .
  * Lin et al. (2017a) Lin, T. Y.; Goyal, P.; Girshick, R.; He, K.; and Dollar, P. 2017a.  Focal Loss for Dense Object Detection.  _TPAMI_ PP(99): 2999–3007.
  * Lin et al. (2014) Lin, T.-Y.; Maire, M.; Belongie, S.; Hays, J.; Perona, P.; Ramanan, D.; Dollár, P.; and Zitnick, C. L. 2014.  Microsoft coco: Common objects in context.  In _ECCV_ , 740–755. Springer.
  * Lin et al. (2017b) Lin, Z.; Feng, M.; Santos, C. N. d.; Yu, M.; Xiang, B.; Zhou, B.; and Bengio, Y. 2017b.  A structured self-attentive sentence embedding.  _arXiv preprint arXiv:1703.03130_ .
  * Linsley et al. (2018) Linsley, D.; Scheibler, D.; Eberhardt, S.; and Serre, T. 2018.  Global-and-local attention networks for visual recognition.  _arXiv preprint arXiv:1805.08819_ .
  * Long, Shelhamer, and Darrell (2015) Long, J.; Shelhamer, E.; and Darrell, T. 2015.  Fully convolutional networks for semantic segmentation.  In _CVPR_ , 3431–3440.
  * Mnih et al. (2014) Mnih, V.; Heess, N.; Graves, A.; et al. 2014.  Recurrent models of visual attention.  In _NeurIPS_ , 2204–2212.
  * Newell, Yang, and Deng (2016) Newell, A.; Yang, K.; and Deng, J. 2016.  Stacked hourglass networks for human pose estimation.  In _ECCV_ , 483–499. Springer.
  * Oliva et al. (2003) Oliva, A.; Torralba, A.; Castelhano, M. S.; and Henderson, J. M. 2003.  et al. Top-down control of visual attention in object detection.  _ICIP_ 5(8): 706–710.
  * Pang et al. (2020a) Pang, B.; Li, Y.; Zhang, Y.; Li, M.; and Lu, C. 2020a.  TubeTK: Adopting Tubes to Track Multi-Object in a One-Step Training Model.  In _CVPR_ , 6308–6318.
  * Pang et al. (2019) Pang, B.; Zha, K.; Cao, H.; Shi, C.; and Lu, C. 2019.  Deep RNN Framework for Visual Sequential Applications.  In _CVPR_ , 423–432.
  * Pang et al. (2020b) Pang, B.; Zha, K.; Cao, H.; Tang, J.; Yu, M.; and Lu, C. 2020b.  Complex sequential understanding through the awareness of spatial and temporal concepts.  _Nature Machine Intelligence_ 2(5): 245–253.
  * Pang et al. (2020c) Pang, B.; Zha, K.; Zhang, Y.; and Lu, C. 2020c.  Further Understanding Videos through Adverbs: A New Video Task.  In _AAAI_ , 11823–11830.
  * Parmar et al. (2019) Parmar, N.; Ramachandran, P.; Vaswani, A.; Bello, I.; Levskaya, A.; and Shlens, J. 2019.  Stand-alone self-attention in vision models.  In _NeurIPS_ , 68–80.
  * Piao et al. (2019) Piao, Y.; Ji, W.; Li, J.; Zhang, M.; and Lu, H. 2019.  Depth-induced Multi-scale Recurrent Attention Network for Saliency Detection.  In _ICCV_.
  * Redmon and Farhadi (2018) Redmon, J.; and Farhadi, A. 2018.  Yolov3: An incremental improvement.  _arXiv preprint arXiv:1804.02767_ .
  * Ren and Zemel (2017) Ren, M.; and Zemel, R. S. 2017.  End-to-end instance segmentation with recurrent attention.  In _CVPR_ , 6656–6664.
  * Russakovsky et al. (2015) Russakovsky, O.; Deng, J.; Su, H.; Krause, J.; Satheesh, S.; Ma, S.; Huang, Z.; Karpathy, A.; Khosla, A.; Bernstein, M.; et al. 2015.  Imagenet large scale visual recognition challenge.  _International journal of computer vision_ 115(3): 211–252.
  * Simonyan (2014) Simonyan, K. 2014.  Very deep convolutional networks for large-scale image recognition.  _arXiv preprint arXiv:1409.1556_ .
  * Srivastava, Greff, and Schmidhuber (2015) Srivastava, R. K.; Greff, K.; and Schmidhuber, J. 2015.  Highway networks.  _arXiv preprint arXiv:1505.00387_ .
  * Sun et al. (2019) Sun, K.; Xiao, B.; Liu, D.; and Wang, J. 2019.  Deep high-resolution representation learning for human pose estimation.  _arXiv preprint arXiv:1902.09212_ .
  * Tang et al. (2020) Tang, J.; Xia, J.; Mu, X.; Pang, B.; and Lu, C. 2020.  Asynchronous Interaction Aggregation for Action Detection.  _arXiv preprint arXiv:2004.07485_ .
  * Tian et al. (2019) Tian, Z.; Shen, C.; Chen, H.; and He, T. 2019.  FCOS: Fully Convolutional One-Stage Object Detection.  _arXiv preprint arXiv:1904.01355_ .
  * Vaswani et al. (2017) Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, Ł.; and Polosukhin, I. 2017.  Attention is all you need.  In _NeurIPS_ , 5998–6008.
  * Wang et al. (2015) Wang, D.; Shen, Z.; Shao, J.; Zhang, W.; Xue, X.; and Zhang, Z. 2015.  Multiple granularity descriptors for fine-grained categorization.  In _ICCV_ , 2399–2406.
  * Wang et al. (2017) Wang, F.; Jiang, M.; Qian, C.; Yang, S.; Li, C.; Zhang, H.; Wang, X.; and Tang, X. 2017.  Residual attention network for image classification.  In _CVPR_ , 3156–3164.
  * Wang et al. (2018) Wang, X.; Girshick, R.; Gupta, A.; and He, K. 2018.  Non-local neural networks.  In _CVPR_ , 7794–7803.
  * Woo et al. (2018) Woo, S.; Park, J.; Lee, J.-Y.; and So Kweon, I. 2018.  Cbam: Convolutional block attention module.  In _ECCV_ , 3–19.
  * Xiao, Wu, and Wei (2018) Xiao, B.; Wu, H.; and Wei, Y. 2018.  Simple baselines for human pose estimation and tracking.  In _ECCV_ , 466–481.
  * Xie et al. (2017) Xie, S.; Girshick, R.; Dollár, P.; Tu, Z.; and He, K. 2017.  Aggregated residual transformations for deep neural networks.  In _CVPR_ , 1492–1500.
  * Xiu et al. (2018) Xiu, Y.; Li, J.; Wang, H.; Fang, Y.; and Lu, C. 2018.  Pose flow: Efficient online pose tracking.  _arXiv preprint arXiv:1802.00977_ .
  * Xu et al. (2015) Xu, K.; Ba, J.; Kiros, R.; Cho, K.; Courville, A.; Salakhudinov, R.; Zemel, R.; and Bengio, Y. 2015.  Show, attend and tell: Neural image caption generation with visual attention.  In _ICML_ , 2048–2057.
  * Yu et al. (2018) Yu, F.; Wang, D.; Shelhamer, E.; and Darrell, T. 2018.  Deep Layer Aggregation.  In _CVPR_.
  * Yue and et al. (2018) Yue, K.; and et al. 2018.  Compact Generalized Non-local Network.  In _NeurIPS_.
  * Zhang et al. (2018) Zhang, Y.; Li, K.; Li, K.; Wang, L.; Zhong, B.; and Fu, Y. 2018.  Image super-resolution using very deep residual channel attention networks.  In _ECCV_ , 286–301.
  * Zhou et al. (2018) Zhou, B.; Andonian, A.; Oliva, A.; and Torralba, A. 2018.  Temporal Relational Reasoning in Videos.  In _ECCV_.
  * Zhu et al. (2019) Zhu, X.; Cheng, D.; Zhang, Z.; Lin, S.; and Dai, J. 2019.  An empirical study of spatial attention mechanisms in deep networks.  _arXiv preprint arXiv:1904.05873_ .