# Intent-based Network Management and Orchestration for Smart Distribution
Grids

Kashif Mehmood, H. V. Kalpanie Mendis, Katina Kralevska, and Poul E. Heegaard
Department of Information Security and Communication Technology,
NTNU - Norwegian University of Science and Technology, N-7491 Trondheim,
Norway
Email: {kashif.mehmood; kalpanie.mendis; katinak<EMAIL_ADDRESS>
###### Abstract

5G technology complements the enabling of communication services for different
vertical industries such as smart distribution grids. Automation is an
integral and necessary part of the power distribution grid operation and
management. This paper postulates a framework by which the smart distribution
grid can obtain service-oriented communication services using 5G network
slicing and intent-based networking (IBN). IBN provides an interface to
service users and network stakeholders to cooperate through a high level
abstraction model of service provisioning in a network agnostic manner. The
automation and adaptability of the distribution grid are facilitated by using
the dynamic and closed-loop mechanism of IBN together with network slicing and
network function virtualization for network management and orchestration. We
identify the automation parts of the power distribution grid and illustrate
the intent processing and its inclusion in the definition of network slice
instances, service and network configuration models.

##  I Introduction

Network virtualization has transformed the design, deployment, control,
management and orchestration of network components. Next-generation
communication networks such as 5G, with their programmable nature, enable
creating and managing customized networks (slices) triggered by on-demand
service requests [1].

Intent-based networking (IBN) is a novel technology for automating and
orchestrating networks that can quickly adapt to ever-changing applications
and services [2]. Intent is a declaration of operational goals that a network
should meet and outcomes that the network is supposed to deliver, without
specifying how to achieve them. IBN management provides the ability to couple
services and network functions to offer on-demand and swift service to various
verticals with requirements (functionality, quality-of-service (QoS))
expressed by the stakeholders in a vertical [3]. User intents help in
determining the scope of the requested services, and an intent controller
extracts the required service requirements from the intent representation and
established service level agreements (SLAs). An intent-based network is an
intelligent network that can automatically convert, deploy, and configure
network resources according to the users’ intents. A closed-loop feedback
mechanism provides the ability to continuously monitor and adapt the
established services to network conditions and stakeholders’ intents during
orchestration life-cycle.

5G and beyond networks have enabled the application of information and
communication technologies (ICT) in a diverse set of verticals and industries,
and one of them is power distribution. Network providers in smart grids can
utilize evolving 5G, network slicing, and IBN frameworks to provide
differentiated services, efficient distribution and power consumption, reduced
transmission loss, improved reliability of the energy grid operations to the
customers, reduction of human intervention, and reduced operational cost [4].

The service requirements for the smart grids are based upon several possible
application scenarios stated as but not restricted to 1) distribution system
monitoring and control, 2) distribution system protection, 3) management of
distributed energy sources (DERs), and 4) demand response. These application
use cases require the utilization of diverse services offered by the 5G
networks due to their heterogeneous performance metrics thus, belong to the
three main use case categories: enhanced mobile broadband (eMBB), ultra-
reliable and low latency communications (URLLC), and massive machine-type
communications (mMTC). For example, feeder automation requires latency in the
order of milliseconds with very high reliability of 99.999% (URLLC), and the
distributed nature of power generation and other grid units requires massive
access support for a large number of devices simultaneously (mMTC) [5]. Hence,
the provisioning of differentiated services to the smart distribution grid
requires accurate and timely orchestration and automation of the necessary
network functions and resources. The inclusion of user intents in the
management of the smart distribution grid communication provides vast benefits
of cost reduction, enhanced power reliability and quality, less power outages,
improved grid resiliency, safety and security to the different stakeholders.

The application of IBN for different industrial use cases, such as protection
paths in ring networks and data dissemination in the vehicular edge computing
ecosystem, has been investigated in [6] and [7], respectively. A recent study
[8] focuses on the utilization of higher-level instructions in the form of
intents to perform configuration and management of 5G networks in a cross-
platform environment. The work in [9] reinforces the premise of IBN management
as a promising direction in solving the connectivity and configuration issues
in smart grids as well as making the network resilient.

In this paper, we apply the concept of intent-based network management in the
domain of smart distribution grids. Our contributions are summarized as
follows:

  * •

An intent and service orchestration mechanism is devised for intent
translation and creation of slice instances, service delivery and network
configuration models.

  * •

Identification of means of automation for the smart grid use case with a
discussion of different stakeholders and how IBN can help in automating
dynamic communication service life-cycle.

  * •

We propose an intent-based service management and orchestration framework that
incorporates the smart grid customers’ intents and the SLAs with the service
providers and network operators, reflected in the obtained service models.

  * •

Smart grid vertical example is provided to identify the service design and
orchestration process using network slicing and YANG service model.

The remainder of this paper is organized as follows. Sec. II provides an
overview of IBN management and orchestration. Sec. III provides a description
of the smart grid use case. Sec. IV presents the proposed intent-based network
management architecture for smart grids. Sec. V concludes the paper.

Figure 1: Intent processing and translation into slicing profiles, service
delivery models, and network configuration models.

##  II Intent-based Network Management and Orchestration

The service-based architecture of 5G networks supports the provisioning of
communication services and network infrastructures based on differentiated
service models. The ultimate goal of IBN management is to provide a
generalized input from the stakeholders to the management plane in order to
orchestrate efficiently and provision resources for different services.

###  II-A Intents: Definition, Life-cycle and Network Integration

An intent can be defined as an abstract, high-level policy used to operate a
network with different types of services based on the feedback received from
the intent and network management layers [10]. Intents are different from
other policy-based user inputs in the sense that they provide a general
direction about the objectives of the intents of the users like the service
customers. The network must understand these objectives and implement the
necessary configurations and technologies to satisfy the intents. The
orchestration of the required resources can be done by the network
orchestrators in order to deliver the desired service and user key performance
indicators (KPIs) such as latency, availability and security. Intents are
integrated into the service design process and augment an automated intent
processing mechanism to convert abstracted user requirements into service and
resource provisioning configurations for the underlying network orchestrators.
The network creates a service model comprising of a customer service model and
a service delivery model from the translated user requirements acquired
through the customer intents [11]. Network function virtualization (NFV) has
enabled the automation in the deployment of required network functions for
different service models in a service-centric network environment.

###  II-B From Intents to Service and Network Configuration Models

Intents are converted into a form which the network and service orchestrators
can use to meet end-to-end (E2E) customer performance requirements. In an IBN
design, the service model is obtained after the processing of user intents
along with the consideration of relevant SLAs from the network operators, as
presented in Fig.1.

A service model defines the service provided to the user in a network
environment. The intents are translated into a set of service requirements for
each user, step (1) in Fig.1, and converted into different available service
profiles supported by the network, step (2), [2]. A smart grid user can
require different services according to the application use cases explained in
Section III. The service profiles are optimized and rigorously evaluated for
potential trade-offs and conflicts of interest in a closed-loop system
consisting of service, relevant configuration, and performance evaluation as
per required resources, step (3). The service delivery models are forwarded to
network orchestration, step (4), consisting of optimized configuration
guidelines in order to meet the relevant service requirements. The network
orchestrator generates the network configurations for different requested
services and assists the network controllers in the deployment of relevant
network functions, step (5).

###  II-C From Intents and Service Requests to Slice Instances

Various organizations have provided frameworks for orchestration and
implementation of communication service requirements in 5G networks. GSMA
utilizes the concept of generic network slice templates (GST) to specify the
communication service requirements [12]. Network slicing is utilized to
orchestrate resources by the slicing orchestrator through the management and
network orchestrator (MANO). Intent orchestrator provides the required set of
service requirements for the user intent after validating with the agreed SLA.
GST is network deployment agnostic representation of a slicing module with
specified slice type acting as one of the several network slice type (NEST).
Several network slice instances (NSIs) can be generated to fulfill service
requirements from a single customer depending upon the propagated service
requests [13]. A network slice provider accepts the NEST and triggers the
network slice instantiation process in order to perform necessary resource
allocations from the network management layer as depicted in Fig. 1.

The slice orchestrator is responsible for the instantiation of different NSIs
in order to fulfill the requested service. This is accomplished through the
definition of network function (NF) service chains by the MANO framework. The
process is completed in accordance with the service requirements and mapped
onto user intents through the intent and service orchestrators. A key
consideration here is the ability to modify the slice life-cycle through the
slicing orchestrator in order to address the dynamic nature of the grid
environment and changing user intents accordingly.

###  II-D Automation in Intent Processing and Service Design

Automation in service and network orchestration requires a dynamic framework
to include service users in the service design process. A closed optimization
loop exists between an intent monitor module and the intent orchestrator in
order to achieve a dynamic and user intent-based service life-cycle
management. The intent processing utilizes language processing mechanisms to
convert the high-level network agnostic intents into different service
profiles and service-specific configurations to be provided to the network
elements in the network management layer. The definition of service objectives
and profiles requires exposure of network capabilities and SLAs agreed upon
between the customer and the service providers and between the other
stakeholders [10]. The automation required during the intent processing and
service design process itself can be achieved through a policy based
optimization framework involving machine learning algorithms. The intent and
service orchestrators utilize local databases to store different predefined
service and intent templates in order to map and adapt different network
agnostic intent translation and service mapping policies. The required data
for the learning based automation consists of the consumer groups, service
provider and operator SLAs, service slicing templates for eMBB, mMTC and uRLLC
and the monitoring data from the smart grid network.

##  III Power Distribution Grid Automation

The well-functioning of modern society is highly dependent on the reliability
and resiliency of the power distribution grid, which plays a key role as a
critical infrastructure. As more distributed energy resources (DERs) are
integrated at the low and medium voltage levels, the flexibility of the power
supply decreases while hindering the reliability of the power grid. The next-
generation power distribution grid or the smart distribution grid is
forecasted to be more flexible and resilient with increased power consumption,
local production, and storage options with added communication capabilities,
control, and intelligence.

TABLE I: Service requirements of distribution grid operations for communication networks [14]. DG operation | 5G use case | Reliability | Latency | Bandwidth | #Devices
---|---|---|---|---|---
WAMS | URLLC | high | low | low | low
Protection, FLISR | URLLC | high | low | low | low
AMI | mMTC | low | high | low | high
Remote inspection | eMBB | low | low | high | low

###  III-A Main power distribution grid operations

Smart grid provides monitoring, protection, and optimization functions
automatically to the operation of the interconnected elements covering
generation, transmission, distribution, and to the consumers [15]. It can
intelligently integrate the needs and capabilities of all grid stakeholders
connected to it [16]. Power distribution automation plays a crucial role in
smart grids. Distribution automation is the path towards a self healing grid.
A self healing grid is a system that uses information, sensing, control and
communication technologies to allow it to deal with unforeseen events and
minimize their adverse impact. As the grid dynamics increase and more closed-
loop controls are implemented, communication networks become more integral to
the smart distribution grid operations. With the enhanced real-time monitoring
and controlling capability provided by the deployment of new sensors, digital
measuring equipment, and automated protection relays, there is a need for a
low latency, high reliable, real-time, and interactive communications system.

####  III-A1 Distribution system monitoring and control

Smart monitoring and control are essential elements of the smart distribution
grid. Smart operation of distribution grid requires several measurements
including voltage, current, frequency, power flows and equipment status
information.

Several ICT-based applications are used to monitor the status of the power
distribution grid. The distribution management system (DMS) is capable of
collecting, and analyzing real-time information on the distribution grid.
Existing operation and (remote) control of the power grid heavily rely on the
supervisory control and data acquisition (SCADA) system. It mainly handles the
local and remote operation of the high voltage equipment, information and
measurement acquisition, and emergency events (alarms). Phasor measurement
units (PMUs) obtain synchronized measurements across a wide area of the power
grid and constitute the wide area measuring system (WAMS). WAMS aims to detect
power grid disturbances in real-time using state estimation algorithms with
extreme reliability and ultra-low latency. Intelligent electronic devices
(IEDs) are used for protection purposes, and use IEC 61850 protocol to
communicate with other IEDs. We assume that the technology and cost
development allow the deployment of PMUs in the future distribution grid,
which are mostly deployed in the transmission grid today.

####  III-A2 Distribution system protection

A failure in the power distribution grid is stressful. Power outages have to
be handled in milliseconds for critical power consumers. Thus, self-healing
solutions empowered by automated switching, fault location, isolation and
service restoration (FLISR) are vital. Several automated protection functions
in the distribution grid require a time-critical exchange of information
between IEDs. Typically, these functions are not using human interaction and
are performed via IEC 61850 publisher-subscriber communication. These are
time-critical and the maximum accepted communication delay is in the range of
several milliseconds.

####  III-A3 Management of DERs

Increasing penetration of DERs at the consumer end adds a high degree of
unpredictability to the power distribution grid’s operation. Local distributed
power generation from volatile DERs such as roof-top solar panels and small-
scale wind turbines by prosumers (i.e. users who are able to both consume and
produce electricity), challenges the flexibility of the power supply. The
resulting bi-directional power flows complicate the protection mechanisms.
Recently, microgrids have become an alternative to supplying power to isolated
communities. A microgrid is a self-sufficient entity comprising of distributed
generation, loads and local storage. For a flexible and reliable operation,
proper control and automated protection system must prevail to overcome the
challenges that emerge with DER integration in the islanded microgrids.

####  III-A4 Demand response

The balance between the supply and demand of power is vital for the reliable
operation of the power distribution grid, since storing energy in large scale
is infeasible. This is largely dependent on the advanced metering
infrastructure (AMI) based on smart meters which facilitates bi-directional
communication between the power grid and the consumers. By careful observation
of the smart meter parameters, suppliers and consumers can make wiser
decisions such as peak demand saving.

####  III-A5 Managing big data

The addition of intelligent sensors, IEDs, PMUs and AMI enables the power
distribution grid to acquire a massive amount of new information about
different aspects of the grid. Data management and advanced analytics can
support data-driven decision making and planning.

###  III-B Stakeholders

The primary stakeholders involved in the power grid fall under the different
areas of power generation, transmission and distribution. However, in this
paper, we focus only on the intents and SLAs between the key actors in the
power distribution grid.

  * •

Distribution system operators (DSO): DSO is responsible for the operation of
local electrical grid between power transmission system and the consumers.

  * •

Prosumers or customers: In smart grid context, the end users play a more
active role by contributing towards the power generation and actively
participating in management of electric power supply and demand.

  * •

Demand response (DR) aggregator: DR aggregators create customized, automated
controls for consumer loads and appliances that enable remote access while
considering their preferences and behavioral patterns. DR aggregators
facilitate the integration of demand-side technologies with ICT and AMI to
encourage end-consumers and prosumers to participate in electricity markets.

  * •

Communication service provider (CSP): The DSOs may demand different types of
communication services from the CSP based on the characteristics of the
different distribution grid operations.

The distribution grid operations impose stringent service requirements upon
the ICT infrastructure. Table I categorizes them into the three major 5G use
case categories: URLLC, mMTC and eMBB.

Figure 2: Proposed Intent-based network management and service orchestration
architecture.

##  IV Proposed Intent- and Network Slicing-Based Framework

In this section, we propose an intent-based architecture for communication
services in smart distribution grid given in Fig. 2 and explain it with one
example use case in Fig. 3. The consumer/prosumer intents couple with the SLAs
between the consumer/prosumer and CSP and the SLAs between the DSO and CSP,
and the service provider follows them in order to provide the requested
service creation and to orchestrate the framework. The proposed architecture
is composed of three management layers consisting of intent, network, and
distribution grid components.

Note that this architecture can be interchangeably applied to any other
vertical industry; thus, we also refer to it as a framework.

Figure 3: An example of IBN WAMS application use case.

###  IV-A Intent Management Layer

The intent management layer is the critical part and provides the required
functionality to implement the intent and the service life-cycle. The intent
processing starts with the information regarding SLAs from DSO and the grid
consumer/prosumer relayed through the CSP. Grid users specify a high-level
expectation for the network like connect residential consumers to the closest
substation. The intent orchestrator considers the SLA information and the
users’ intents during the intent translation and feasibility analysis of the
required services. The intent translation consists of language processing by
an intelligent module in order to iteratively produce the required service
expectations for different grid users (see Table I). A closed-loop
optimization is performed with the intent monitor and optimizer module in the
grid layer to ensure compliance of user intents with the configured service
and network configuration. The intent monitor module also maintains an updated
status of the resource utilization and availability in the grid management
layer, which is utilized in the intent and service orchestration for the
generation of feasible service profiles. The intent orchestrator cross
validates the optimized service requirements with the assured SLAs for the
users with the CSP and the DSO (Fig. 1) before forwarding the service
requirements to the service orchestrator. The service orchestration consists
of generating a customer service model as per the provided service
requirements from the intent orchestrator. Several feasibility checks are
performed in order to ensure the required services are made available to the
users, including the possible tradeoffs between different services and the
dynamic nature of user requirements. A service model can be represented in the
YANG modeling language [11], and following service orchestration, it instructs
the network orchestrators to provision the required functionality for ensuring
the generated service profiles for different user applications. The
information exchange in the form of NEST profiles (Fig. 1) happens between the
service and slice orchestrators, followed by the network function provisioning
through the management and network orchestrator.

###  IV-B Network Management Layer

The middle layer consists of two functional blocks, namely, the MANO entity
and the slice orchestrator. The slice orchestrator maintains the life-cycle
(creation, update, deletion) of the network slices. The MANO entity manages
and orchestrates the resources needed to realize network slice instances over
the virtual network functions (VNFs) [17]. The NFV orchestrator (NFVO) manages
NSIs; thus, it is responsible for the life-cycle management, fault, and
performance management of NSIs. The VNF manager (VNFM) is responsible for the
life-cycle (setting up, maintaining, and tearing down) of VNF instances. The
virtualized infrastructure manager (VIM) connects to NFV infrastructures and
sets up virtual links.

###  IV-C Distribution Grid Management Layer

The bottom layer comprises the underlying physical infrastructure of the power
distribution grid and 5G radio access network (RAN), and core network. The
physical and logical components of the power distribution grid such as
substations, PMUs, IEDs, circuit breakers, smart meters connect to the RAN in
order to send the necessary information to the data aggregation points located
in the proximity. These are then used by SCADA and DMS to execute the
distribution grid automation processes. The intent monitor and optimizer
entity continuously observes the performance of the communication services
delivered via the network slices. The observations are dispensed to the intent
orchestrator, which makes a comparison between the desired and the observed
KPIs and makes the adjustments accordingly.

###  IV-D Example

We consider an example of the WAMS use case in Fig. 3 requiring the
establishment of network slices that deliver URLLC service. PMUs are
responsible for the collection of phasor measurement data from the grid
towards phasor data concentrators (PDC) and form the basis of the WAMS. An
automated service provisioning is usually completed in the following steps:

  1. 1.

The interested service user (DSO) generates an intent ”Need to communicate
with central PDC”, and enters in to an SLA.

  2. 2.

The intent orchestrator translates the intent to a service request with
specific requirements and cross validates it with the SLAs.

  3. 3.

The service request is forwarded to the service orchestrator that creates a
service profile including the specifications for the GST and then a specific
slice type is selected as NEST. The slice type selected for the WAMS use case
is the URLLC and the slicing profile is generated by the service orchestrator
for the slice orchestrator.

  4. 4.

The generated slicing profile is implemented via the network orchestrator
(MANO) and the service provisioning life-cycle is completed for the requested
service for the PMUs.

The intent monitor and optimizer, together with the intent orchestrator,
dynamically adapt the KPIs to the network conditions and send updates to the
service orchestrator. A new service slicing profile is generated likewise to
accommodate the changing resource and network state in order to ensure the
SLAs are satisfied for the users. The service slicing profiles consist of
various parameters that are adjusted in order to enable the generated profiles
to continuously meet the service requirements for the intent provided by the
WAMS users.

##  V Concluding Remarks

Automating the management and control of the power distribution grids provides
benefits such as real-time operation of time-critical tasks, reduction of
human intervention, thus, leading to reduced associated cost, high level of
security of supply, increased productivity, resiliency, and dependability. In
this paper, we proposed an architecture to facilitate the distribution grid
automation using the dynamic and closed-loop mechanism of IBN and
communication service provisioning using 5G network slicing. The architecture
is service based consisting of distribution, network management and intent
management layers. By providing one example, we illustrated the workflow from
intent definition to slice realization in the distribution grid.

## Acknowledgment

This paper has been funded by CINELDI - Centre for intelligent electricity
distribution, an 8 year Research Centre under the FME-scheme (Centre for
Environment-friendly Energy Research, 257626/E20). The authors gratefully
acknowledge the financial support from the Research Council of Norway and the
CINELDI partners.

## References

  * [1] A. Esmaeily, K. Kralevska, and D. Gligoroski, “A cloud-based sdn/nfv testbed for end-to-end network slicing in 4g/5g,” in 2020 6th IEEE Conference on Network Softwarization (NetSoft), pp. 29–35, 2020.
  * [2] A. Clemm, L. Ciavaglia, L. Granville, and J. Tantsura, “Intent-based networking-concepts and definitions,” tech. rep., IETF, 2020.
  * [3] A. Kaloxylos, R. Peppe, and A. Gavras, “Empowering vertical industries through 5G networks - current status and future trends.” Available at https://5g-ppp.eu/wp-content/uploads/2020/09/5GPPP-VerticalsWhitePaper-2020-Final.pdf.  (Accessed: 2021/02/15).
  * [4] X. Fang, S. Misra, G. Xue, and D. Yang, “Smart grid — the new and improved power grid: A survey,” IEEE Communications Surveys Tutorials, vol. 14, no. 4, pp. 944–980, 2012.
  * [5] H. C. Leligou, T. Zahariadis, L. Sarakis, E. Tsampasis, A. Voulkidis, and T. E. Velivassaki, “Smart grid: a demanding use case for 5G technologies,” in IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 215–220, 2018.
  * [6] B. K. Saha, D. Tandur, L. Haab, and L. Podleski, “Intent-based networks: An industrial perspective,” in Proc. of the 1st Int. Workshop on Future Industrial Communication Networks, FICN ’18, p. 35–40, ACM, 2018.
  * [7] A. Singh, G. S. Aujla, and R. S. Bali, “Intent-based network for data dissemination in software-defined vehicular edge computing,” IEEE Transactions on Intelligent Transportation Systems, pp. 1–9, 2020.
  * [8] T. A. Khan, A. Mehmood, J. J. Diaz Rivera, and W. Song, “Machine learning approach for automatic configuration and management of 5g platforms,” in 20th Asia-Pacific Network Operations and Management Symposium (APNOMS), pp. 1–6, 2019.
  * [9] S. Dobson, D. Hutchison, A. Mauthe, A. Schaeffer-Filho, P. Smith, and J. P. G. Sterbenz, “Self-organization and resilience for networked systems: Design principles and open research issues,” Proceedings of the IEEE, vol. 107, no. 4, pp. 819–834, 2019.
  * [10] M. Behringer, M. Pritikin, S. Bjarnason, A. Clemm, B. Carpenter, S. Jiang, and L. Ciavaglia, “Autonomic networking: Definitions and design goals.” Available at https://tools.ietf.org/html/rfc7575, 2015.  Last accessed 25 December 2020.
  * [11] Q. Wu, W. Liu, and A. Farrel, “Service models explained,” IETF RFC8309,[Online]. Available: https://tools. ietf. org/html/rfc8309, 2017.
  * [12] GSMA, “NG.116 generic network slice template v2.0.” Available at https://www.gsma.com/newsroom/wp-content/uploads/NG.116-v2.0.pdf (Accessed: 2021/02/20).
  * [13] ETSI, “5G; System architecture for the 5G System (5GS).” Available at https://www.etsi.org/deliver/etsi_ts/123500_123599/123501/16.06.00_60/ts_123501v160600p.pdf (Accessed: 2021/02/20).  (3GPP TS 23.501 version 16.6.0 Release 16).
  * [14] H. V. K. Mendis, P. E. Heegaard, and K. Kralevska, “5G network slicing as an enabler for smart distribution grid operations,” Proc. Int. Conf. and Exhibition on Electricity Distribution (CIRED), 2019.
  * [15] Y. Yan, Y. Qian, H. Sharif, and D. Tipper, “A survey on smart grid communication infrastructures: Motivations, requirements and challenges,” IEEE comm. surveys & tutorials, vol. 15, no. 1, pp. 5–20, 2012.
  * [16] B. Panajotovic, M. Jankovic, and B. Odadzic, “ICT and smart grid,” in 10th Int. Conference on Telecommunication in Modern Satellite Cable and Broadcasting Services (TELSIKS), vol. 1, pp. 118–121, 2011.
  * [17] etsi, “Network functions virtualisation: Architectural framework,” ETsI GS NFV, vol. 2, no. 2, p. V1, 2013.


#  SpanDrop: Simple and Effective Counterfactual Learning for Long Sequences

Peng Qi∗†1 Guangtao Wang∗†2 Jing Huang†3
1 AWS AI, Seattle, WA 2 TikTok, Mountain View, CA 3 Alexa AI, Sunnyvale, CA
<EMAIL_ADDRESS>
###### Abstract

Distilling supervision signal from a long sequence to make predictions is a
challenging task in machine learning, especially when not all elements in the
input sequence contribute equally to the desired output. In this paper, we
propose SpanDrop, a simple and effective data augmentation technique that
helps models identify the true supervision signal in a long sequence with very
few examples. By directly manipulating the input sequence, SpanDrop randomly
ablates parts of the sequence at a time and ask the model to perform the same
task to emulate counterfactual learning and achieve input attribution. Based
on theoretical analysis of its properties, we also propose a variant of
SpanDrop based on the beta-Bernoulli distribution, which yields diverse
augmented sequences while providing a learning objective that is more
consistent with the original dataset. We demonstrate the effectiveness of
SpanDrop on a set of carefully designed toy tasks, as well as various natural
language processing tasks that require reasoning over long sequences to arrive
at the correct answer, and show that it helps models improve performance both
when data is scarce and abundant.

**footnotetext: Equal contribution.††footnotetext: Work done at JD AI
Research.

##  1 Introduction

Building effective machine learning systems for long sequences is a
challenging and important task, which helps us better understand underlying
patterns in naturally occurring sequential data like long texts (Radford et
al., 2019), protein sequences (Jumper et al., 2021), financial time series
(Bao et al., 2017), etc. Recently, there is growing interest in studying
neural network models that can capture long-range correlations in sequential
data with high computational, memory, and statistical efficiency, especially
widely adopted Transformer models (Vaswani et al., 2017).

Previous work approach long-sequence learning in Transformers largely by
introducing computational approaches to replace the attention mechanism with
more efficient counterparts. These approaches include limiting the scope of
the attention mechanism (Kitaev et al., 2019), limiting sequence-level
attention to only a handful of positions (Beltagy et al., 2020; Zaheer et al.,
2020), or borrowing ideas from the kernel trick to eliminate the need to
compute or instantiate the costly attention matrix (Peng et al., 2020;
Katharopoulos et al., 2020; Choromanski et al., 2020). Essentially, these
approaches aim to approximate the original pair-wise interaction with lower
cost, and are often interested in capturing the effect of every input element
on the outcome (e.g., arithmetic operations over a long list of numbers and
operators, as proposed by Tay et al., 2020). While these tasks present great
challenges for designing effective models to handle long sequences, many real-
world problems involving long sequences share properties that make them
amenable to more effective approaches than modeling the raw input-output
relationship directly (e.g., via decomposition).

Figure 1: SpanDrop compared to masking- and substitution-based augmentation
techniques (e.g., word dropout) commonly used in sequence learning. SpanDrop
can not only work with arbitrary granularities of input units, but also
introduce augmented sequences without artificial tokens and alleviates
potential positional bias in the data.

In this paper, we focus on a learning problem for long sequences motivated by
real-world tasks, where not all input elements might contribute to the desired
output. Natural examples that take this form include sentiment classification
for long customer review documents (where a few salient sentiment words and
conjunctions contribute the most), question answering from a long document
(where each question typically requires a small number of supporting sentences
to answer), key phrase detection in audio processing (where a small number of
recorded frames actually determine the prediction), as well as detecting a
specific object from a complex scene (where, similarly, a small amount of
pixels determine the outcome), to name a few. In these problems, it is usually
counterproductive to try and make direct use of the entire input if the
contributing portion is small or sparse, which results in a problem of
_underspecification_ (i.e., the training data does not sufficiently define the
goal for statistical models).

One approach to address this problem is to annotate the parts of input that
directly contribute to the outcome. This could take the form of a subset of
sentences that answer a question or describe the relation between entities in
a paragraph (Yang et al., 2018; Yao et al., 2019). However, such annotation is
not always technically or financially feasible, so researchers and
practitioners often need to resort to either collecting more input-output
pairs or designing problem-specific data augmentation techniques to make up
for the data gap. For real-valued data, augmentation often translates to
random transformations (e.g., shifting or flipping an image); for symbolic
data like natural language, techniques like masking or substitution are more
commonly used (e.g., randomly swapping words with a special mask token or
other words). While these approaches have proven effective in some tasks, each
has limitations that prevents it from being well-suited for the
underspecification scenario. For instance, while global feature
transformations enhance group-invariance in learned representations, they do
not directly help with better locating the underlying true stimulus. On the
other hand, while replacement techniques like masking and substitution help
ablate parts of the input, they are susceptible to the position bias of where
the true stimulus might occur in the input. Furthermore, while substitution
techniques can help create challenging contrastive examples, they are often
significantly more difficult to implement (e.g., replacing a phrase in a
sentence without losing fluency).

To address these challenges, we propose SpanDrop, a simple and effective
technique that helps models distill sparse supervision signal from long
sequences when the problem is underspecified. SpanDrop randomly ablates parts
of the input to construct _counterfactual_ examples that preserves the
original output and supervision signal with high probability. Unlike
replacement-based techniques, however, SpanDrop removes ablated elements from
the input and concatenate the remainder. This avoids introducing artificial
representations that are not used at test time, and mitigates potential
spurious correlation to absolute positions of outcome-determining parts of the
input (see Figure 1). We further propose a theoretically motivated, more
effective variant of SpanDrop based on the beta-Bernoulli distribution that
enhances the consistency of the augmented objective function with the original
one. We demonstrate via carefully designed toy experiments that SpanDrop not
only helps models achieve up to $20\times$ sample-efficiency in low-data
settings, but also further reduces overfitting even when training data is
abundant. We find that it is very effective at mitigating position bias
compared to replacement-based counterfactual approaches, and enhances out-of-
distribution generalization effectively. We further experiment on four natural
language processing tasks that require models to answer question or extract
entity relations from long texts, where, motivated by our theoretical
analysis, we further propose an adaptive span segmentation. Our experiments
demonstrate that SpanDrop can improve the performance of competitive neural
models without any architectural change.

To summarize, our contributions in this paper are: 1) we propose SpanDrop and
Beta-SpanDrop, two effective approaches to generate counterfactual examples
for tasks of long sequence learning; 2) we study the theoretical properties of
these approaches and verify them with carefully designed experiments on
synthetic data; 3) we demonstrate that both approaches improve upon strong
neural models on four NLP datasets, which can be furthered by a theoretically
motivated span segmentation approach we propose.

##  2 Method

In this section, we first formulate the problem of sequence inference, where
the model takes sequential data as input to make predictions. Then, we
introduce SpanDrop, a simple and effective data augmentation technique for
long sequence inference, and analyze its theoretical properties.

###  2.1 Problem Definition

Sequence Inference. We consider a task where a model takes a sequence $\bm{S}$
as input and predicts the output $\bm{y}$. We assume that
$\bm{S}=(\bm{s}_{1},\ldots,\bm{s}_{n})$ consists of $n$ disjoint but
contiguous _spans_ , and each span represents a part of the sequence in order.
One example of sequence inference is sentiment classification from a paragraph
of text, where $\bm{S}$ is the paragraph and $\bm{y}$ the sentiment label.
Spans could be words, phrases, sentences, or a mixture of these in the
paragraph. Another example is time series prediction, where $\bm{S}$ is
historical data, $\bm{y}$ is the value at the next time step.

Supporting Facts. Given an input-output pair $(\bm{S},\bm{y})$ for sequence
prediction, we assume that $\bm{y}$ is truly determined by only a subset of
spans in $\bm{S}$. More formally, we assume that there is a subset of spans
$S_{\mathrm{sup}}\subset\\{\bm{s}_{1},\bm{s}_{2},\ldots,\bm{s}_{n}\\}$ such
that $\bm{y}$ is independent of $\bm{s}_{i}$, if $\bm{s_{i}}\notin
S_{\mathrm{sup}}$. In sentiment classification, $S_{\mathrm{sup}}$ could
consist of important sentiment words or conjunctions (like “good”, “bad”,
“but”); in time series prediction, it could reflect the most recent time steps
as well as those a few cycles away if the series is periodic. For simplicity,
we will denote the size of this set $m=|S_{\mathrm{sup}}|$, and restrict our
attention to tasks where $m\ll n$, such as those described in the previous
section.

###  2.2 SpanDrop

In a long sequence inference task with sparse support facts ($m\ll n$), most
of the spans in the input sequence will not contribute to the prediction of
$\bm{y}$, but they will introduce spurious correlation in a low-data scenario.
SpanDrop generates new data instances $(\tilde{\bm{S}},\bm{y})$ by ablating
these spans at random, while preserving the supporting facts with high
probability so that the model is still trained to make the correct prediction
$\bm{y}$. This is akin to counterfactually determining whether each span truly
determines the outcome $\bm{y}$ by asking what the prediction would have been
without it.

######  Definition 1 (SpanDrop).

Formally, given a sequence $\bm{S}$ that consists of spans
$(\bm{s}_{1},\bm{s}_{2},\cdots\bm{s}_{n})$, SpanDrop generates a new sequence
$\tilde{\bm{S}}$ as follows:

$\displaystyle\delta_{i}\overset{i.i.d.}{\sim}$
$\displaystyle\mathrm{Bernoulli}(1-p),$ $\displaystyle\tilde{\bm{S}}=$
$\displaystyle(\bm{s}_{i})_{i=1,\delta_{i}=1}^{n},$

where $p$ is the hyperparameter that determines the probability to drop a
span.

Note that SpanDrop does not require introducing substitute spans or artificial
symbols when ablating spans from the input sequence. It makes the most of the
natural sequence as it occurs in the original training data, and preserves the
relative order between spans that are not dropped, which is often helpful in
understanding sequential data (e.g., time series or text). It is also not
difficult to establish that the resulting sequence $\tilde{\bm{S}}$ can
preserve all of the $m$ supporting facts with high probability regardless of
how large $n$ is.

######  Remark 1.

The new sequence length $n^{\prime}=|\tilde{\bm{S}}|$ and the number of
preserved supporting facts $m^{\prime}=|\tilde{\bm{S}}\cap S_{\mathrm{sup}}|$
follow binomial distributions $\mathrm{Bin}(n,p)$ and $\mathrm{Bin}(m,p)$,
respectively, where $P(x=k|N,p)={N\choose k}(1-p)^{k}p^{N-k}$ for
$X\sim\mathrm{Bin}(N,p)$.

Therefore, the proportion of sequences where all supporting facts are retained
(i.e., $m^{\prime}=m$) is $(1-p)^{m}$, which is independent of $n$. This means
that as long as the total number of supporting facts in the sequence is
bounded, then regardless of the sequence length, we can always choose $p$
carefully such that we end up with many valid new examples with bounded noise
introduced to supporting facts. Note that our analysis so far relies only on
the assumption that $m$ is known or can be estimated, and thus it can be
applied to tasks where the precise set of supporting facts $S_{\mathrm{sup}}$
is unknown. More formally, the amount of new examples can be characterized by
the size of the _typical set_ of $\tilde{\bm{S}}$, i.e., the set of sequences
that the randomly ablated sequence will fall into with high probability. The
size of the typical set for SpanDrop is approximately $2^{nH(p)}$, where
$H(p)$ is the binary entropy of a Bernoulli random variable with probability
$p$. Intuitively, these results indicate that the amount of total
counterfactual examples generated by SpanDrop scales exponentially in $n$, but
the level of supporting fact noise can be bounded as long as $m$ is small.

However, this formulation of SpanDrop does have a notable drawback that could
potentially hinder its efficacy. The new sequence length $n^{\prime}$ follows
a binomial distribution, thus for sufficiently large $n$, most
$\tilde{\bm{S}}$ lengths will concentrate around the mean $n(1-p)$ with a
width of $O(\sqrt{n})$. This creates an artificial and permanent distribution
drift from the original length (see Figure 2(a)). Furthermore, even if we know
the identity of $S_{\mathrm{sup}}$ and keep these spans during training, this
length reduction will bias the training set towards easier examples to locate
spans in $S_{\mathrm{sup}}$, potentially hurting generalization performance.
In the next subsection, we will introduce a variant of SpanDrop based on the
beta-Bernoulli distribution that alleviates this issue.

Relation to word dropout. A commonly used data augmentation/regularization
technique in NLP, word dropout Dai & Le (2015); Gal & Ghahramani (2016), is
closely related to SpanDrop. Two crucial difference, however, set SpanDrop
apart from it and similar techniques: 1) word dropout masks or replaces
symbols in the input, while SpanDrop directly removes them. Thus SpanDrop can
avoid introducing artificial representations not used at test time, and affect
sequence length and the absolute position of remaining elements in the
sequence, which we will demonstrate alleviates the effect of spurious
correlations between the outcome and absolute element positions in the
training data; 2) SpanDrop can operate on input spans segmented at arbitrary
granularity (not just words, or even necessarily contiguous), and our
theoretical analysis holds as long as these spans are _disjoint_. We will show
that task-informed span segmentation leads to further performance gains.

$0$$20$$40$$60$$80$$100$$0$$5$$10$Sequence length ($n^{\prime}$)Proportion
(%)SpanDropBeta-SpanDrop (a) Length of $\tilde{\bm{S}}$ ($n=100,p=0.2$)

$0$$10$$20$$-4$$-2$$0$$2$Supporting facts ($m$)Log noise-free
prob.$\gamma=\infty$$\gamma=100$$\gamma=10$$\gamma=1$$\gamma=0.1$$\gamma=0.01$
(b) Supporting fact noise ($p=0.2$)

$10^{0}$$10^{2}$$10^{4}$$0$$0.2$$0.4$$0.6$Sequence length ($n$)Avg.
nats/span$\gamma=\infty$$\gamma=100$$\gamma=10$$\gamma=1$$\gamma=0.1$$\gamma=0.01$
(c) Typical set size ($p=0.1$)

Figure 2: Theoretical comparison between SpanDrop and Beta-SpanDrop.

###  2.3 Beta-SpanDrop

To address the problem of distribution drift with SpanDrop, we introduce a
variant that is based on the beta-Bernoulli distribution. The main idea is
that instead of dropping each span in a sequence independently with a fixed
probability $p$, we first sample a sequence-level probability $\pi$ at which
spans are dropped from a Beta distribution, then use this probability to
perform SpanDrop.

######  Definition 2 (Beta-SpanDrop).

Let $\alpha=\gamma,\beta=\gamma\cdot\frac{1-p}{p}$, where $\gamma>0$ is a
scaling hyperparameter. Beta-SpanDrop generates $\tilde{\bm{S}}$ over $\bm{S}$
as:

$\displaystyle\pi\sim$ $\displaystyle\mathrm{B}(\alpha,\beta),$
$\displaystyle\delta_{i}\overset{i.i.d.}{\sim}$
$\displaystyle\mathrm{Bernoulli}(1-\pi),$ $\displaystyle\tilde{\bm{S}}=$
$\displaystyle(\bm{s}_{i})_{i=1,\delta_{i}=1}^{n},$

where $\mathrm{B}(\alpha,\beta)$ is the beta-distribution with parameters
$\alpha>0$ and $\beta>0$.

It can be easily demonstrated that in Beta-SpanDrop, the probability that each
span is dropped is still $p$ on average:
$\mathbb{E}[\delta_{i}|p]=\mathbb{E}[\mathbb{E}[\delta_{i}|\pi]|p]=\mathbb{E}[1-\pi|p]=1-\frac{\alpha}{\alpha+\beta}=1-p$.
In fact, we can show that as $\gamma\to\infty$, Beta-SpanDrop degenerates into
SpanDrop since the beta-distribution would assign all probability mass on
$\pi=p$. Despite the simplicity in its implementation, Beta-SpanDrop is
significantly less likely to introduce unwanted data distribution drift, while
is capable of generating diverse counterfactual examples to regularize the
training of sequence inference models. This is due to the following
properties:

######  Remark 2.

The new sequence length $n^{\prime}=|\tilde{\bm{S}}|$ and the number of
preserved supporting facts $m^{\prime}=|\tilde{\bm{S}}\cap S_{\mathrm{sup}}|$
follow beta-binomial distributions $\mathrm{B\mbox{-}Bin}(n,\beta,\alpha)$ and
$\mathrm{B\mbox{-}Bin}(m,\beta,\alpha)$, respectively, where
$P(x=k|N,\alpha,\beta)=\frac{\Gamma(N+1)}{\Gamma(k+1)\Gamma(N-k+1)}\frac{\Gamma(k+\beta)\Gamma(N-k+\alpha)}{\Gamma(N+\alpha+\beta)}\frac{\Gamma(\alpha+\beta)}{\Gamma(\alpha)\Gamma(\beta)}$
for $X\sim\mathrm{B\mbox{-}Bin}(N,\beta,\alpha)$, and
$\Gamma(z)=\int_{0}^{\infty}x^{z-1}e^{-x}dx$ the gamma function.

As a result, we can show that the probability that Beta-SpanDrop preserves the
entire original sequence with the following probability

$\displaystyle P(n^{\prime}=n|n,\alpha,\beta)=$
$\displaystyle\frac{\Gamma(n+\beta)\Gamma(\alpha+\beta)}{\Gamma(n+\alpha+\beta)\Gamma(\beta)}.$

When $\gamma=1$, this expression simply reduces to $\frac{\beta}{n+\beta}$;
when $\gamma\neq 1$, this quantity tends to $O(n^{-\gamma})$ as $n$ grows
sufficiently large. Comparing this to the $O((1-p)^{n})$ rate from SpanDrop,
we can see that when $n$ is large, Beta-SpanDrop recovers more of the original
distribution represented by $(\tilde{\bm{S}},\bm{y})$ compared to SpanDrop. In
fact, as evidenced by Figure 2(a), the counterfactual sequences generated by
Beta-SpanDrop are also more spread-out in their length distribution besides
covering the original length $n$ with significantly higher probability. A
similar analysis can be performed by substituting $n$ and $n^{\prime}$ with
$m$ and $m^{\prime}$, where we can conclude that as $m$ grows, Beta-SpanDrop
is much better at generating counterfactual sequences that preserve the entire
supporting fact set $S_{\mathrm{sup}}$. This is shown in Figure 2(b), where
the proportion of “noise-free” examples (i.e., $m^{\prime}=m$) decays
exponentially with SpanDrop ($\gamma=\infty$) while remaining much higher when
$\gamma$ is sufficiently small. For instance, when $p=0.1$, $\gamma=1$ and
$m=10$, the proportion of noise-free examples for SpanDrop is just 34.9%,
while that for Beta-SpanDrop is 47.4%.

As we have seen, Beta-SpanDrop is significantly better than its Bernoulli
counterpart at assigning probability mass to the original data as well as
generated sequences that contain the entire set of supporting facts. A natural
question is, does this come at the cost of diverse counterfactual examples? To
answer this question we study the entropy of the distribution that
$\tilde{\bm{S}}$ follows by varying $\gamma$ and $n$, and normalize it by $n$
to study the size of typical set of this distribution. As can be seen in
Figure 2(c), as long as $\gamma$ is large enough, the average entropy per span
$\bar{H}$ degrades very little from the theoretical maximum, which is $H(p)$,
attained when $\gamma=\infty$. Therefore, to balance between introducing noise
in the supporting facts and generating diverse examples, we set $\gamma=1$ in
our experiments.

Beta-Bernoulli distribution in dropout. The beta-Bernoulli distribution has
been studied in prior work in seeking replacements for the (Bernoulli) dropout
mechanism (Srivastava et al., 2014). Liu et al. (2019a) set $\alpha=\beta$ for
the beta distribution in their formulation, which limits the dropout rate to
always be 0.5. Lee et al. (2018) fix $\beta=1$ and vary $\alpha$ to control
the sparsity of the result of dropout, which is similar to Beta-SpanDrop when
$\gamma=1$. However, we note that these approaches (as with dropout) are
focused more on adding noise to internal representations of neural networks to
introduce regularization, while SpanDrop operates directly on the input to
ablate different components therein, and thus orthogonal (and potentially
complementary) to these approaches. Further, SpanDrop has the benefit of not
having to make any assumption about the model or any change to it during
training, which makes it much more widely applicable.

##  3 FindAnimals: Distilling Supervision from Long-Sequences

In this section, we design a synthetic task of finding a specific subsequence
in a character sequence to: a) demonstrate the effectiveness of SpanDrop and
Beta-SpanDrop in promoting the performance over a series of problems with
different settings, b) analyze the various factors that may affect the
efficacy of these approaches, and c) compare it to other counterfactual
augmentation techniques like masking on mitigating position bias.

###  3.1 Experimental Setup

FindAnimals. To understand the effectiveness of SpanDrop and Beta-SpanDrop
empirically, we designed a synthetic task called FindAnimals where the model
is trained to discern that given an animal name $\bm{a}$, e.g., “cat”, whether
a character string contains it as a subsequence (i.e., contains characters in
“cat” in order, for instance, “abcdafgbijktma”) or not (e.g.,
“abcdefhtijkamn”). This allows us to easily control the total sequence length
$n$, the supporting facts size $m$, as well as easily estimate the supporting
fact noise that each SpanDrop variant might introduce.111To generate the
synthetic training data of FindAnimals, we first generate sequences consisting
of lowercase letters (a to z) that each sequence $\bm{S}$ does not contain the
animal name $\bm{a}$; then randomly choose half of these sequences, and
replace letters with those in $\bm{a}$ from arbitrary (but not necessarily
contiguous) positions in $\bm{S}$ to generate positive examples, and label the
rest negative.

In all of our experiments, we evaluate model performance on a held-out set of
10,000 examples to observe classification error. We set sequence length to
$n=300$ where each letter is a separate span, and chose positions for the
letters in the animal name $\bm{a}$ ($|\bm{a}|=m=3$) uniformly at random in
the sequence unless otherwise mentioned.

Model. We employ a three-layer Transformer model (Vaswani et al., 2017) with
position embeddings (Devlin et al., 2019) as the sequence encoder implemented
with HuggingFace Transformers (Wolf et al., 2019). For each example
$(\bm{a},\bm{S},y)$, we feed “[CLS] $\bm{a}$ [SEP] $\bm{S}$ [SEP]” to the
model and then perform binary classifier over the “[CLS]” representation to
predict $y\in\\{0,1\\}$, where $y=1$ means $\bm{a}$ appears in $\bm{S}$. To
investigate the effectiveness of SpanDrop, we apply SpanDrop to $\bm{S}$ first
before feeding the resulting sequence into the Transformer classifier.

###  3.2 Results and Analysis

$10^{1}$$10^{3}$$10^{5}$131030# Training examplesError/Noise (%)(a) Data
efficiency$0$$0.1$$0.2$$0.3$$0.4$$0.5$131030Drop rate $p$(b) Noise in
supporting factsNoise, SpanDropNoise, Beta-SpanDrop10301003000.1110Train/test
sequence length(c) Varying sequence lengthNoise, SpanDropNoise, Beta-
SpanDrop246810131030Supporting fact size(d) Varying supporting fact sizeNoise,
SpanDropNoise, Beta-SpanDropBaselineSpanDropSpanDrop (noise-free)Beta-SpanDrop
Beta-SpanDrop (noise-free)

$200$$225$$250$$275$$300$$325$$0$$10$$20$$30$Test sequence lengthError(%)(e)
OOD length generalizationBernoulliBeta-Bernoulli$0$$20$$40$Drop/mask
distribution(f) SpanDrop vs SpanMaskSpanMaskFixedFirst
100$0$$20$$40$$60$Setting(g) Position&0-shot generalizationSpanMask02.55
$\cdot$10431030Training steps(h) Dev error during training

Figure 3: Experimental results of SpanDrop variants and SpanMask on the
FindAnimals synthetic tasks.

In each experiment, we compare SpanDrop and Beta-SpanDrop at the same drop
ratio $p$. And we further use rejection sampling to remove examples that do
not preserve the desired supporting facts to understand the effect of
supporting fact noise.

Data efficiency. We begin by analyzing the contribution of SpanDrop and Beta-
SpanDrop to improving the sample efficiency of the baseline model. To achieve
this goal, we vary the training set size from 10 to 50,000 and observe the
prediction error on the held-out set. We observe from the results in Figure
3(a) that: 1) Both SpanDrop and Beta-SpanDrop significantly improve data
efficiency in low-data settings. For instance, when trained on only 200
training examples, SpanDrop variants can achieve the generalization
performance of the baseline model trained on 5x to even 20x data. 2) Removing
supporting fact noise typically improves data efficiency further by about 2x.
This indicates it is helpful not to drop spans in $S_{\mathrm{sup}}$ during
training when possible, so that the model is always trained with true
counterfactual examples rather than sometimes noisy ones. 3) Beta-SpanDrop
consistently improves upon the baseline model even when data is abundant. This
is likely due to the difficulty of the task when $n=300$ and $m=3$. Similar to
many real-world tasks, the task remains underspecified even when the
generalization error is already very low, thanks to the large amount of
training data available. 4) SpanDrop introduces inconsistent training
objective with the original training set, which leads to performance
deterioration when there is sufficient training data, which is consistent with
our theoretical observation.

Effect of supporting fact noise. Since SpanDrop introduces noise in the
supporting facts (albeit with a low probability), it is natural to ask if such
noise is negatively correlated with model performance. We study this by
varying the drop ratio $p$ from $\\{0.05,0.1,0.15,0.2,0.3,0.4,0.5\\}$ on fixed
training sets of size 1,000, and observe the resulting model performance and
supporting fact error. As can be seen in Figure 3(b), supporting fact noise
increases rapidly as $p$ grows.222Note that the noise in our experiments are
lower than what would be predicted by theory, because in practice the initial
sequence $\bm{S}$ might already contain parts of $\bm{a}$ before it is
inserted. This creates redundant sets of supporting facts for this task and
reduces supporting fact noise especially when $n$ is large. However, we note
that although the performance of SpanDrop deteriorates as $p$ increases, that
of Beta-SpanDrop stays relatively stable. Inspecting these results more
closely, we find that even the performance of the noise-free variants follow a
similar trend, which should not be affected by supporting fact noise.

Effect of sequence and supporting fact lengths. In our theoretical analysis,
sequence length $n$ determines regularization strength and the drift in
sequence length distribution, and supporting fact size $m$ determines the
probability that the counterfactual example retains the correct supervision
signal. To study their effects empirically, we conduct three separate sets of
experiments: 1) training and testing the model on varying sequence lengths
{10, 20, 30, 50, 100, 200, 300, 500}; 2) varying supporting fact size between
2 and 10; and 3) testing the model trained on $n=300$ on test sets of
different lengths.

As can be seen from Figures 3(c) and 3(e), our experimental results seem to
well supporting this hypothesis that the gap between the two SpanDrop variants
has a lot to do with the discrepancy in length distributions. Specifically, in
Figure 3(c), while the performance of both SpanDrop variants deteriorates as
$n$ grows and the task becomes more challenging and underspecified, SpanDrop
deteriorates at a faster speed even when we remove the effect of supporting
fact noise. On the other hand, we see in Figure 3(e) that SpanDrop performance
peak around sequences of length 270 ($=n(1-p)$=300$\times$(1-0.1)) before
rapidly deteriorating, while Beta-SpanDrop is unaffected until test sequence
length exceeds that of all examples seen during training.

In Figure 3(d), we can also see that as predicted by Remarks 1 and 2, Beta-
SpanDrop is better at preserving all supporting facts than SpanDrop agnostic
of their positions, which translates to superior performance.

Mitigating position bias. Besides SpanDrop, replacement-based techniques like
masking can also be applied to construct counterfactual examples, where
elements in the sequence are replaced by a special symbol that is not used at
test time. We implement SpanMask in the same way as SpanDrop except spans are
replaced rather than removed when the sampled “drop mask” $\delta_{i}$ is 0.
We first inspect whether SpanMask benefits from the same beta-Bernoulli
distribution we use in SpanDrop. As can be seen in Figure 3(f), the gain from
switching to a beta-Bernoulli distribution provides negligible benefit to
SpanMask, which does not alter the sequence length of the input to begin with.
We also see that SpanMask results in significantly higher error than both
SpanDrop and Beta-SpanDrop in this setting.

We further experiment with introducing position bias into the training data
(but not the test data) to test whether these method help the model generalize
to an unseen setting. Specifically, instead of selecting the position for the
characters in $\bm{a}$ uniformly at random, we train the model with a “fixed
position” dataset where they always occur at indices (10, 110, 210), and a
“first 100” dataset where they are uniformly distributed among the first 100
letters. As can be seen in Figure 3(g), both the baseline and SpanMask models
overfit to the position bias in the “fixed” setting, while SpanDrop techniques
significantly reduce zero-shot generalization error. In the “first 100”
setting, Beta-SpanDrop consistently outperforms its Bernoulli counterpart and
SpanMask at improving the performance of the baseline model as well,
indicating that SpanDrop variants are effective at reducing the position bias
of the model.

Impact on convergence speed. Regularization is commonly shown to slow down
model convergence, resulting in the need for much longer training time for
marginal improvements in performance. We show in Figure 3(h), however, that
SpanDrop approaches do not seem to suffer from this issue on the synthetic
task. In the contrary, both approaches help the model generalize significantly
better within the same amount of training time.

##  4 Experiments on Natural Language Data

To examine the efficacy of the proposed SpanDrop techniques on realistic data,
we conduct experiments on four NLP datasets that represent a variety of tasks.
We focus on showing the effect of SpanDrop instead of pursuing the state-of-
the-art in these experiments.

###  4.1 Setup and Main Results

Model |  Ans F1 |  Sup F1 |  Joint F1
---|---|---|---
RoBERTa-base | 73.5 | 83.4 | 63.5
Longformer-base | 74.3 | 84.4 | 64.4
SAE BERT-base | 73.6 | 84.6 | 65.0
Our implementation |  |  |
ELECTRA-base | 74.2 | 86.3 | 66.2
\+ SpanDrop | 74.7 | 86.7 | 66.8
\+ Beta-SpanDrop | 74.7 | 86.9 | 67.1
(a) HotpotQA dev

Model | EM |  F1
---|---|---
BERT-base | 26.6/24.1 | 71.8/70.1
RoBERTa-base |  38.7/0—0 |  77.1/0—0
REPT RoBERTa-base |  40.4/0—0 |  80.0/0—0
Our implementation |  |
ELECTRA-base | 40.1/39.1 | 80.4/78.2
\+ SpanDrop | 42.3/39.9 | 81.7/78.5
\+ Beta-SpanDrop | 44.8/41.1 | 81.6/79.8
(b) MultiRC dev/test

Model |  Ign F1 |  RE F1 |  Evi F1
---|---|---|---
E2GRE BERT-base | 55.2 | 58.7 | 47.1
ATLOP BERT-base | 59.2 | 61.1 | —
SSAN BERT-base | 57.0 | 59.2 | —
Our implementation |  |  |
ELECTRA-base | 59.6 | 61.6 | 50.8
\+ SpanDrop | 59.9 | 61.9 | 51.2
\+ Beta-SpanDrop | 60.1 | 62.1 | 51.2
(c) DocRED dev

Model | EM |  F1
---|---|---
RoBERTa-base | — | 90.6
ELECTRA-base | 84.5 | 90.8
XLNet-large | 89.7 | 95.1
Our implementation |  |
ELECTRA-base | 86.6 | 92.4
\+ SpanDrop _w/ adaptive spans_ | 87.4 | 92.9
\+ Beta-SpanDrop _w/ adaptive spans_ | 87.3 | 92.8
(d) SQuAD dev

Table 1: Main results on four natural language processing datasets.

Datasets. We use four natural language processing datasets: SQuAD 1.1
(Rajpurkar et al., 2016), where models answer questions on a paragraph of text
from Wikipedia; MultiRC (Khashabi et al., 2018), which is a multi-choice
reading comprehension task in which questions can only be answered by taking
into account information from multiple sentences; HotpotQA (Yang et al.,
2018), which requires models to perform multi-hop reasoning over multiple
Wikipedia pages to answer questions; and DocRED (Yao et al., 2019), which is a
document-level data set for relation extraction.

For the SQuAD dataset, we define spans as collections of one or more
consecutive tokens to show that SpanDrop can be applied to different
granularities. Guided by our theoretical analysis, we also experiment with an
_adaptive span segmentation_ approach to reduce the number of supporting facts
($m$) by combining all $N$-gram overlaps between the question and context into
a single span ($N\geq 2$). For the rest three datasets, we define spans to be
sentences since supporting facts are provided at sentence level. For all of
these tasks, we report standard exact match (EM) and F1 metrics where
applicable, for which higher scores are better. We refer the reader to the
appendix for details about the statistics and metrics of these datasets.

Model. We build our models based on ELECTRA (Clark et al., 2019), since it is
shown to perform well across a range of NLP tasks recently. We introduce
randomly initialized task-specific parameters designed for each task following
prior work on each dataset, and finetune these models on each dataset to
report results. We refer the reader to the appendix for training details and
hyperparameter settings.

Main Results. We first present the performance of our implemented models and
their combination with SpanDrop variants on the four natural language
processing tasks. We also include results from representative prior work on
each dataset for reference (detailed in the appendix), and summarize the
results in Table 1. We observe that: 1) our implemented models achieve
competitive and sometimes significantly better performance (in the cases of
HotpotQA, SQuAD, and DocRED) compared to published results, especially
considering that we do not tailor our models to each task too much; 2)
SpanDrop improves the performance over these models even when the training set
is large and that the model is already performing well; 3) Models trained with
Beta-SpanDrop consistently perform better or equally well with their SpanDrop
counterparts across all datasets, demonstrating that our observations on the
synthetic datasets generalize well to real-world ones. We note that the
performance gains on real-world data is less significant, which likely results
from the fact that non-supporting spans in the synthetic task are independent
from each other, which is not the case in natural language data.

###  4.2 Analysis

To understand whether the properties of SpanDrop and Beta-SpanDrop we observe
on FindAnimals generalize to real-world data, we further perform a set of
analysis experiments on SQuAD. Specifically, we are interested in studying the
effect of the amount of training data, the span drop ratio $p$, and the choice
of span size on performance.

Effect of low data. To understand SpanDrop’s regularizing effect when training
data is scarce, we study the model’s generalization performance when training
on only 0.1% of the training data (around 100 examples) to using the entire
training set (around 88k examples). As shown in Figure 4 (left), both SpanDrop
and Beta-SpanDrop significantly improve model performance when the amount of
training data is extremely low. As the amount of training data increases, this
gap slowly closes but remains consistently positive. When all training data is
used, the gap is still sufficient to separate top-2 performing systems on this
dataset.

0.1110100$40$$60$$80$% of training set usedDev F1 (%)BaselineSpanDropBeta-
SpanDrop$0$$0.2$$0.4$$91$$92$$93$Drop ratio $p$ad141664$91$$92$$93$Span size
Figure 4: Effect analysis of training data size, drop ratio and span size on
performance of models trained with SpanDrop and Beta-SpanDrop over SQuAD

Impact of drop ratio. We compare SpanDrop and Beta-SpanDrop by controlling how
likely each span is dropped on average (drop ratio $p$). Recall from our
experiments on FindCats that larger $p$ will result in distribution drift from
the original training set for SpanDrop but not Beta-SpanDrop, thus the
performance of the former deteriorates as $p$ increases while the latter is
virtually not affected. As can be seen in Figure 4 (middle), our observation
on real-world data is consistent with this theoretical prediction, and
indicate that Beta-SpanDrop is a better technique for data augmentation should
one want to increase sequence diversity by setting $p$ to a larger value.

Impact of span selection. We train the model with SpanDrop on SQuAD with
varying fixed span sizes of {1, 2, 4, 8, 16, 32, 64} tokens per span, as well
as task-motivated _adaptive_ span sizes, to understand the effect of this
hyperparameter. We observe in Figure 4 (right) that as span size grows, the
generalization performance of the model first holds roughly constant, then
slowly deteriorates as span size grows too large. Adaptive spans (“ad”)
notably outperforms all fixed span sizes, suggesting that while larger fixed
span sizes reduce $m$ and thus noise in supporting facts, the resulting
reduction in $n$ hinders regularization strength and thus hurting
generalization. This observation is consistent with that on our synthetic data
as well as what would be predicted by our theoretical analysis. This also
suggests that while SpanDrop works with arbitrary span sizes, the optimal
choice of spans for different tasks warrants further investigation, which we
leave to future work.

##  5 Related Work

Long Sequence Inference. Many applications require the prediction/inference
over long sequences, such as multi-hop reading comprehension (Yang et al.,
2018; Welbl et al., 2018), long document summarization (Huang et al., 2021),
document-level information extraction (Yao et al., 2019) in natural language
processing, long sequence time-series prediction (Zhou et al., 2021a),
promoter region and chromatin-profile prediction in DNA sequence (Oubounyt et
al., 2019; Zhou & Troyanskaya, 2015) in Genomics etc, where not all elements
in the long sequence contribute equally to the desired output. Aside from
approaches that attempt to approximate all pair-wise interactions between
elements in a sequence, more recent work has also investigated compressing
long sequences into shorter ones to distill the information therein for
prediction or representation learning (Rae et al., 2020; Goyal et al., 2020;
Kim & Cho, 2021).

Sequence Data Augmentation. Data augmentation is an effective common technique
for underspecified tasks like long sequence inference. Feng et al. (2021)
propose to group common data augmentation techniques in natural language
processing into three categories: 1) rule-based methods (Zhang et al., 2015;
Wei & Zou, 2019; Şahin & Steedman, 2018), which apply a set of predefined
operations over the raw input, such as removing, adding, shuffling and
replacement; 2) example mixup-based methods (Guo et al., 2019; Guo, 2020; Chen
et al., 2020; Jindal et al., 2020), which, inspired from Mixup in computer
vision (Zhang et al., 2018), perform interpolation between continuous features
like word embeddings and sentence embeddings; 3) model-based methods (Xie et
al., 2020; Sennrich et al., 2016), which use trained models to generate new
examples (e.g., back translation Xie et al., 2020).

Most existing rule-based data augmentation methods operate at the token/word
level (Feng et al., 2021), such as word shuffle/replacement/addition (Wei &
Zou, 2019). Shuffle-based techniques are less applicable when order
information is crucial in the raw data (Lan et al., 2019, e.g., in natural
language). Clark et al. (2019) and Lewis et al. (2020) have recently shown
that well designed word-level augmentation can also lead to improved
pretrained models, but generalizing this idea to phrases or sentences is less
straightforward. In contrast, our proposed SpanDrop supports data augmentation
in multiple granularity, and is able to reserve sequence order since drop
operation does not change the relative order of the original input, which is
important in many kinds of sequence data such as natural language.

##  6 Conclusion

In this paper, we presented SpanDrop, a simple and effective method for
learning from long sequences, which ablates parts of the sequence at random to
generate counterfactual data to distill the sparse supervision signal that is
predictive of the desired output. We show via theoretical analysis and
carefully designed synthetic datasets that SpanDrop and its variant based on
the beta-Bernoulli distribution, Beta-SpanDrop, help models achieve
competitive performance with a fraction of the data by introducing diverse
augmented training examples, and generalize better to previously unseen data.
Our experiments on four real-world NLP datasets confirm these theoretical
findings, and demonstrate SpanDrop’s efficacy on strong neural models even
when data is abundant.

## References

  * Bao et al. (2017) Wei Bao, Jun Yue, and Yulei Rao.  A deep learning framework for financial time series using stacked autoencoders and long-short term memory.  _PloS one_ , 12(7):e0180944, 2017.
  * Beltagy et al. (2020) Iz Beltagy, Matthew E Peters, and Arman Cohan.  Longformer: The long-document transformer.  _arXiv preprint arXiv:2004.05150_ , 2020.
  * Chen et al. (2020) Jiaao Chen, Zichao Yang, and Diyi Yang.  MixText: Linguistically-informed interpolation of hidden space for semi-supervised text classification.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pp. 2147–2157, 2020.
  * Choromanski et al. (2020) Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, et al.  Rethinking attention with Performers.  In _International Conference on Learning Representations_ , 2020.
  * Clark et al. (2019) Kevin Clark, Minh-Thang Luong, Quoc V Le, and Christopher D Manning.  ELECTRA: Pre-training text encoders as discriminators rather than generators.  In _International Conference on Learning Representations_ , 2019.
  * Dai & Le (2015) Andrew M Dai and Quoc V Le.  Semi-supervised sequence learning.  _Advances in neural information processing systems_ , 28:3079–3087, 2015.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _NAACL-HLT_ , 2019.
  * Feng et al. (2021) Steven Y Feng, Varun Gangal, Jason Wei, Sarath Chandar, Soroush Vosoughi, Teruko Mitamura, and Eduard Hovy.  A survey of data augmentation approaches for nlp.  _Findings of ACL_ , 2021.
  * Gal & Ghahramani (2016) Yarin Gal and Zoubin Ghahramani.  A theoretically grounded application of dropout in recurrent neural networks.  In _Proceedings of the 30th International Conference on Neural Information Processing Systems_ , 2016.
  * Goyal et al. (2020) Saurabh Goyal, Anamitra Roy Choudhury, Saurabh Raje, Venkatesan Chakaravarthy, Yogish Sabharwal, and Ashish Verma.  Power-bert: Accelerating bert inference via progressive word-vector elimination.  In _International Conference on Machine Learning_ , pp. 3690–3699. PMLR, 2020.
  * Guo (2020) Hongyu Guo.  Nonlinear mixup: Out-of-manifold data augmentation for text classification.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 34, pp. 4044–4051, 2020.
  * Guo et al. (2019) Hongyu Guo, Yongyi Mao, and Richong Zhang.  Augmenting data with mixup for sentence classification: An empirical study.  _arXiv preprint arXiv:1905.08941_ , 2019.
  * Huang et al. (2020) Kevin Huang, Qi Peng, Guangtao Wang, Tengyu Ma, and Jing Huang.  Entity and evidence guided relation extraction for docred.  _arXiv preprint arXiv:2008.12283_ , 2020.
  * Huang et al. (2021) Luyang Huang, Shuyang Cao, Nikolaus Parulian, Heng Ji, and Lu Wang.  Efficient attentions for long document summarization.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pp. 1419–1436, 2021.
  * Jiao et al. (2021) Fangkai Jiao, Yangyang Guo, Yilin Niu, Feng Ji, Feng-Lin Li, and Liqiang Nie.  REPT: Bridging language models and machine reading comprehensionvia retrieval-based pre-training.  _arXiv preprint arXiv:2105.04201_ , 2021.
  * Jindal et al. (2020) Amit Jindal, Arijit Ghosh Chowdhury, Aniket Didolkar, Di Jin, Ramit Sawhney, and Rajiv Shah.  Augmenting nlp models using latent feature interpolations.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pp. 6931–6936, 2020.
  * Jumper et al. (2021) John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Žídek, Anna Potapenko, et al.  Highly accurate protein structure prediction with alphafold.  _Nature_ , 596(7873):583–589, 2021.
  * Katharopoulos et al. (2020) Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret.  Transformers are RNNs: Fast autoregressive transformers with linear attention.  In _International Conference on Machine Learning_. PMLR, 2020.
  * Khashabi et al. (2018) Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth.  Looking beyond the surface: A challenge set for reading comprehension over multiple sentences.  In _Proceedings of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , 2018.
  * Kim & Cho (2021) Gyuwan Kim and Kyunghyun Cho.  Length-adaptive transformer: Train once with length drop, use anytime with search.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing_ , pp. 6501–6511, 2021.
  * Kitaev et al. (2019) Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya.  Reformer: The efficient transformer.  In _International Conference on Learning Representations_ , 2019.
  * Lan et al. (2019) Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut.  ALBERT: A lite bert for self-supervised learning of language representations.  In _International Conference on Learning Representations_ , 2019.
  * Lee et al. (2018) Juho Lee, Saehoon Kim, Jaehong Yoon, Hae Beom Lee, Eunho Yang, and Sung Ju Hwang.  Adaptive network sparsification with dependent variational beta-bernoulli dropout.  _arXiv preprint arXiv:1805.10896_ , 2018.
  * Lewis et al. (2020) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer.  BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , 2020.
  * Liu et al. (2019a) Lei Liu, Yuhao Luo, Xu Shen, Mingzhai Sun, and Bin Li.  $\beta$-dropout: A unified dropout.  _IEEE Access_ , 7:36140–36153, 2019a.
  * Liu et al. (2019b) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov.  RoBERTa: A robustly optimized BERT pretraining approach.  _arXiv preprint arXiv:1907.11692_ , 2019b.
  * Oubounyt et al. (2019) Mhaned Oubounyt, Zakaria Louadi, Hilal Tayara, and Kil To Chong.  Deepromoter: robust promoter predictor using deep learning.  _Frontiers in genetics_ , 10:286, 2019.
  * Peng et al. (2020) Hao Peng, Nikolaos Pappas, Dani Yogatama, Roy Schwartz, Noah Smith, and Lingpeng Kong.  Random feature attention.  In _International Conference on Learning Representations_ , 2020.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9, 2019.
  * Rae et al. (2020) Jack W. Rae, Anna Potapenko, Siddhant M. Jayakumar, Chloe Hillier, and Timothy P. Lillicrap.  Compressive transformers for long-range sequence modelling.  In _International Conference on Learning Representations_ , 2020.
  * Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang.  SQuAD: 100, 000+ questions for machine comprehension of text.  In _EMNLP_ , 2016.
  * Şahin & Steedman (2018) Gözde Gül Şahin and Mark Steedman.  Data augmentation via dependency tree morphing for low-resource languages.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pp. 5004–5009, 2018.
  * Sennrich et al. (2016) Rico Sennrich, Barry Haddow, and Alexandra Birch.  Improving neural machine translation models with monolingual data.  In _54th Annual Meeting of the Association for Computational Linguistics_ , pp. 86–96, 2016.
  * Srivastava et al. (2014) Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov.  Dropout: a simple way to prevent neural networks from overfitting.  _Journal of Machine Learning Research_ , 15(1):1929–1958, 2014.
  * Tay et al. (2020) Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler.  Long range arena: A benchmark for efficient transformers.  In _International Conference on Learning Representations_ , 2020.
  * Tu et al. (2020) Ming Tu, Kevin Huang, Guangtao Wang, Jing Huang, Xiaodong He, and Bowen Zhou.  Select, answer and explain: Interpretable multi-hop reading comprehension over multiple documents.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 34, pp. 9073–9080, 2020.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In _Advances in Neural Information Processing Systems_ , pp. 5998–6008, 2017.
  * Wei & Zou (2019) Jason Wei and Kai Zou.  EDA: Easy data augmentation techniques for boosting performance on text classification tasks.  In _Proceedings of the Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing_ , pp. 6382–6388, 2019.
  * Welbl et al. (2018) Johannes Welbl, Pontus Stenetorp, and Sebastian Riedel.  Constructing datasets for multi-hop reading comprehension across documents.  _Transactions of the Association for Computational Linguistics_ , 6:287–302, 2018.
  * Wolf et al. (2019) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al.  Huggingface’s transformers: State-of-the-art natural language processing.  _arXiv preprint arXiv:1910.03771_ , 2019.
  * Xie et al. (2020) Qizhe Xie, Zihang Dai, Eduard Hovy, Thang Luong, and Quoc Le.  Unsupervised data augmentation for consistency training.  In _Advances in Neural Information Processing Systems_ , 2020.
  * Xu et al. (2021) Benfeng Xu, Quan Wang, Yajuan Lyu, Yong Zhu, and Zhendong Mao.  Entity structure within and throughout: Modeling mention dependencies for document-level relation extraction.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 35, pp. 14149–14157, 2021.
  * Yang et al. (2018) Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D Manning.  HotpotQA: A dataset for diverse, explainable multi-hop question answering.  In _Proceedings of the Conference on Empirical Methods in Natural Language Processing_ , pp. 2369–2380, 2018.
  * Yang et al. (2019) Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le.  XLNet: Generalized autoregressive pretraining for language understanding.  In _Advances in Neural Information Processing Systems_ , 2019.
  * Yao et al. (2019) Yuan Yao, Deming Ye, Peng Li, Xu Han, Yankai Lin, Zhenghao Liu, Zhiyuan Liu, Lixin Huang, Jie Zhou, and Maosong Sun.  DocRED: A large-scale document-level relation extraction dataset.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pp. 764–777, 2019.
  * Zaheer et al. (2020) Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al.  Big Bird: Transformers for longer sequences.  In _Advances in Neural Information Processing Systems_ , 2020.
  * Zhang et al. (2018) Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz.  mixup: Beyond empirical risk minimization.  In _International Conference on Learning Representations_ , 2018.
  * Zhang et al. (2015) Xiang Zhang, Junbo Zhao, and Yann LeCun.  Character-level convolutional networks for text classification.  In _Advances in Neural Information Processing Systems_ , pp. 649–657, 2015.
  * Zhou et al. (2021a) Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang.  Informer: Beyond efficient transformer for long sequence time-series forecasting.  In _Proceedings of AAAI_ , 2021a.
  * Zhou & Troyanskaya (2015) Jian Zhou and Olga G Troyanskaya.  Predicting effects of noncoding variants with deep learning–based sequence model.  _Nature methods_ , 12(10):931–934, 2015.
  * Zhou et al. (2021b) Wenxuan Zhou, Kevin Huang, Tengyu Ma, and Jing Huang.  Document-level relation extraction with adaptive thresholding and localized context pooling.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , 2021b.

##  Appendix A Appendix

###  A.1 Statistics of Benchmark Data Sets

In this section, we summarize the statistics of the four natural language
processing datasets used in our experiments in Table 2. For SQuAD, since the
dataset does not annotate supporting facts, we approximately estimate
supporting facts by counting tokens that are part of a bigram that appears in
the question.

Data | Train | Dev | # of Spans | # of Supporting facts
---|---|---|---|---
HotpotQA(Yang et al., 2018)$\dagger$ | 90,564 |  07,405 |  14.45/11/17/2/96$\diamond$ | 2.38/2/3/2/12
MultiRC(Khashabi et al., 2018)$\dagger$ |  05,131 |  00,953 | 14.72/12/18/6/41 | 2.32/2/2/2/6
DocRED(Yao et al., 2019)$\dagger$ | 38,180 | 12,323 | 8.24/6/10/3/25 | 1.67/1/2/1/11
SQuAD(Rajpurkar et al., 2016)$\star$ | 87,599 | 10,570 | 156.26/114/186/25/853 |  7.74/3/11/0/82$\ddagger$

  * $\dagger$

Sentence-level span and supporting facts, and MultiRC is the second version
and available at https://cogcomp.seas.upenn.edu/multirc/, where the column
“train” and “dev” w.r.t. MultiRC reports the number of questions in training
data and dev data, respectively.

  * $\star$

Token-level span, and the statistics of # of spans/supporting facts are
collected by setting span = 1 token;

  * $\diamond$

This is collected based on the top 4 documents selected from 10 candidate
documents by document retriever. The five numbers correspond to Average, 25%
Percentile, 75% Percentile, Min and Max, respectively.

  * $\ddagger$

We define the supporting facts as the spans of the context which have bigrams
appearing in the question.

Table 2: Statistics of Benchmark Data sets

###  A.2 Experimental Setup for Different Tasks

This section introduces the detailed implementations of our methods on four
benchmark data sets, as well as the hyper-parameter setting for model
optimization and baselines to compare with our implementations.

####  A.2.1 HotpotQA

Implementation details. The objective of HotpotQA is answering questions from
a set of 10 paragraphs where two paragraphs are relevant to the question and
the rest are distractors. HotpotQA presents two tasks: answer span prediction
and evidence sentence (i.e., supporting fact) prediction. Our HotpotQA model
consists of two stages: the first stage selects top 4 paragraphs from 10
candidates by a retrieval model. The second stage finds the final answer span
and evidence over the selected 4 paragraphs. We particularly feed the
following input format to encoder: “[CLS] question [SEP] sent1,1 [SEP] sent1,2
$\cdots$ [SEP] sent4,1 [SEP] sent4,2 $\cdots$ [SEP]”. And we apply the
proposed span drop methods over all the sentences except for supporting facts.

For answer span prediction, we use the answer span prediction model in (Devlin
et al., 2019) with an additional task of question type (yes/no/span)
classification head over the first special token ([CLS]). For evidence
extraction, we apply two-layer MLPs on top of the representations
corresponding to sentence and paragraph to get the corresponding evidence
prediction scores and use binary cross entropy loss to train the model.
Finally, we combine answer span, question type, sentence evidence, and
relevant paragraph losses and train the model in a multitask way using linear
combination of losses. The hyper-parameter search space for our models on
HotpotQA is given in Table 3.

Baselines. We compare our implementation of HotpotQA model over Electra with
the following strong baselines: 1) RoBERTa (Liu et al., 2019b) based model, 2)
long sequence encoder Longformer (Beltagy et al., 2020) based model and 3) SAE
(Tu et al., 2020) which combines graph neural network and pretrained language
models for multi-hop question answer and is the current SOTA model on
HotpotQA. The numbers reported in our paper for the first two models come from
(Zaheer et al., 2020).

Parameter name | HotpotQA | MultiRC | DocRED | SQuAD
---|---|---|---|---
Batch size | {4, 8} | {8, 16} | {4, 8} | {16, 32}
Learning rate | {3e-5, 2e-5, 1e-5, 1e-4} | {3e-5, 2e-5, 1e-5, 1e-4} | {1e-4, 5e-5} | {1e-4, 5e-5}
Span Drop ratio | {0.1, 0.15, 0.2, 0.25} | {0.1, 0.15, 0.2, 0.25} | {0.05, 0.1, 0.15, 0.2} | {0.03, 0.05, 0.1, 0.15, 0.2, 0.25}
Optimizer | AdamW | AdamW | AdamW | AdamW
Epochs | 10 | 15 | {10,20,30} | {2,4,6,8}
Table 3: Hyper-parameter search space for models on different benchmarks

####  A.2.2 MultiRC

Implementation details. MultiRC is a multi-choice question answer task, which
supplies a set of alternatives or possible answers to the question and
requires to select the best answer(s) based on multiple sentences while a few
of these sentences (i.e., supporting facts) are relevant to the questions. For
given example of $k$ candidate answer and $n$ sentences, we first feed the
following input to the encoder: “[CLS] question [SEP] answer1 [SEP] answer2
[SEP] $\cdots$ answerk [SEP] sentence1 [SEP] sentence2 [SEP] $\cdots$
sentencen [SEP]”. For answer prediction, we apply two-layer MLPs on top of the
representations corresponding to candidate answers and sentences to get the
corresponding answer and sentences scores, and use a combination of two binary
cross entropy losses to train the model in a multi-task way. We apply our span
drop methods over all input sentences except for the evidence sentences. The
hyper-parameter search space for our MultiRC models is given in Table 3.

Baselines. We compare our implementation of MultiRC model over Electra with
the following baselines: 1) BERT based model, 2) RoBERTa (Liu et al., 2019b)
based model and 3) REPT (RoBERTa-base) trained with new additional training
tasks (Jiao et al., 2021).

####  A.2.3 Relation extraction task: DocRED

Implementation details. DocRED is document-level relation extraction which
consists of two tasks: relation prediction of a given pair of entities and the
evidence prediction. We construct the entity-guided inputs to the encoder
following prior work Huang et al. (2020). Each training example is organized
by concatenating the head entity, together with the $n$ sentences in the
document as “[CLS] head entity [SEP] sentence1 [SEP] sentence2 [SEP] $\cdots$
sentencen [SEP]”.

For both relation extraction and evidence prediction, we apply a biaffine
transformation that combines entity representations and entity/sentence
representations, respectively, and score them using the adaptive threshold
loss proposed by Zhou et al. (2021b). We train the model in a multi-task
setting by using a linear combination of relation extraction and evidence
prediction losses. And we apply the proposed span drop methods over all input
sentences except for those that serve as evidence for entity relations with
the head entity. Please refer to Table 3 for the hyper-parameter search space
of our DocRED models.

Baselines. We compare against a set of strong baselines w.r.t. document-level
relation including: 1) E2GRE (Huang et al., 2020), 2) ATLOP (Zhou et al.,
2021b) and 3) SSAN (Xu et al., 2021).

####  A.2.4 SQuAD

Implementation details. SQuAD aims at extracting a segment of text from a
given paragraph as answer to the question. We format the input example as
“[CLS] question [SEP] paragraph [SEP]” and feed to the encoder. Following
prior work, we employ a span prediction mechanism where the end of the span is
conditioned on the representation of the start of the span, originally
presented in the XLNet paper (Yang et al., 2019). And we apply our proposed
span drop methods over the paragraph before feeding it into the model for
training, where spans that contain question bigrams are preserved. The hyper-
parameter space to optimize the SQuAD model is shown in Table 3.

Baselines. We implemented models for SQuAD based on ELECTRA (Clark et al.,
2019) and compare them with existing model implementations for SQuAD based on
1) ELECTRA (Clark et al., 2019), 2) RoBERTa (Liu et al., 2019b) and 3) XLNet
(Yang et al., 2019), respectively.