# Bridge Data: Boosting Generalization of Robotic Skills with Cross-Domain
Datasets

Frederik Ebert*1, Yanlai Yang*1, Karl Schmeckpeper3, Bernadette Bucher3,
Georgios Georgakis3,
Kostas Daniilidis3, Chelsea Finn2, Sergey Levine1 * denotes equal
contribution1 University of California Berkeley, 2 Stanford University, 3
University of Pennsylvania

###### Abstract

Robot learning holds the promise of learning policies that generalize broadly.
However, such generalization requires sufficiently diverse datasets of the
task of interest, which can be prohibitively expensive to collect. In other
fields, such as computer vision, it is common to utilize shared, reusable
datasets, such as ImageNet, to overcome this challenge, but this has proven
difficult in robotics. In this paper, we ask: what would it take to enable
practical data reuse in robotics for end-to-end skill learning? We hypothesize
that the key is to use datasets with multiple tasks and multiple domains, such
that a new user that wants to train their robot to perform a new task in a new
domain can include this dataset in their training process and benefit from
cross-task and cross-domain generalization. To evaluate this hypothesis, we
collect a large multi-domain and multi-task dataset, with 7,200 demonstrations
constituting 71 tasks across 10 environments, and empirically study how this
data can improve the learning of new tasks in new environments. We find that
jointly training with the proposed dataset and 50 demonstrations of a never-
before-seen task in a new domain on average leads to a 2x improvement in
success rate compared to using target domain data alone. We also find that
data for only a few tasks in a new domain can bridge the domain gap and make
it possible for a robot to perform a variety of prior tasks that were only
seen in other domains. These results suggest that reusing diverse multi-task
and multi-domain datasets, including our open-source dataset, may pave the way
for broader robot generalization, eliminating the need to re-collect data for
each new robot learning project.

##  I Introduction

Humans and animals can generalize a learned skill to a wide variety of
contexts without needing to relearn the skill every time. Endowing robots with
the same capability would be a significant advance toward making robots more
applicable to a range of real-world settings. However, the prevailing paradigm
of robot learning is to repeat data collection and policy training from
scratch for every new task and environment. Learning policies in isolation not
only increases the costs of data collection, but also limits the policy’s
scope of generalization.

In other fields, such as computer vision [1] and natural language processing
(NLP) [2], utilizing large, diverse datasets has shown considerable success in
enabling generalization to new problems or domains with a small amount of data
(e.g., via pretraining and finetuning). However, in robotics, datasets are
usually collected with a specific robotic platform and domain in mind,
typically by the same researcher who intends to use that dataset. What would
it take to make datasets reusable in robotics in the same way as large
supervised datasets are reused (e.g., ImageNet [3])? Each end-user of such a
dataset might want their robot to learn a different task, which would be
situated in a different domain (e.g., a different laboratory, home, etc.). It
is currently an open question whether such reuse is feasible in robotics, and
we posit that any such dataset would need to cover both multiple different
tasks and multiple different domains. To this end, the aim of our paper is to
investigate the degree to which such a multi-task and multi-domain dataset,
which we refer to as a _bridge_ dataset, can enable a new robot in a new
domain (which was not seen in the bridge data) to more effectively generalize
when learning a new task (which was also not seen in the bridge data), as well
as to transfer tasks from the bridge data to the target domain. We also
propose a new dataset that enables this goal in the context of kitchen-themed
tasks with a low-cost robotic arm and is intended to be reused by other
researchers.

Figure 1: Illustration of our bridge dataset. The dataset includes
demonstrations in 10 environments (4 toy kitchens and 5 toy sinks and 1 real
kitchen), collected using a WidowX250 robot controlled via an Oculus Quest2 VR
device, and consists of 7200 demonstrations. The red arrows indicate the
desired movement of the target object.

The notion that multi-task data can speed up learning or improve
generalization has been studied in many prior works [4, 5]. However, unlike
this paper, the focus in these prior works, as we discuss in Section II, is
not on enabling new users to quickly train generalizable skills in a new
setting or domain, but rather to utilize multi-task learning to lower the data
requirements of acquiring a pre-defined set of tasks. More closely related to
our work, RoboNet [6] contains data from multiple robots and domains, but this
data is collected using random motions, and does not provide examples of
multiple different tasks that can be used for more complex task-directed
manipulation. We discuss other datasets in Section II; but in summary, no
existing dataset covers _both_ multiple tasks _and_ multiple domains in a way
that is suitable to study our central hypothesis: can prior data be used to
improve the generalization of _new_ tasks in _new_ domains? We will call this
the _bridge data hypothesis_. We believe this is a critical requirement for
effective data reuse in robotics, where different labs and researchers can all
bootstrap from the same shared datasets. To study this, we collected a new
multi-domain manipulation dataset with 7,200 demonstrations of 71 distinct and
semantically meaningful tasks, themed around household tasks in kitchen
environments. The data was collected across 10 distinct “toy” kitchens, as
shown in Figure 1. This data is suitable for imitation learning, which is the
focus of our work, though it could also be repurposed for offline RL and other
algorithms in the future. We present our new dataset, and then use it to
evaluate the bridge data hypothesis that is stated above, using three types of
transfer scenarios: (1) When the user needs to train an existing task in a new
domain, does the inclusion of bridge data boost performance? This roughly
corresponds to a standard domain adaptation setting. (2) After the user has
collected some data for a few tasks in a new domain, can their robot then
perform other tasks that were _not_ seen in the new domain, but are only
present in the bridge data (i.e., can it “import” tasks from the bridge data)?
(3) When the user collects some data in a new domain for a task that was _not_
seen in the bridge data, can the performance and generalization of this task
be boosted by including the bridge data in training? Scenario (3) directly
evaluates our central hypothesis, while the other scenarios illustrate other
potential uses for bridge data.

The main contributions of our work consist of an empirical evaluation of the
bridge data hypothesis and a practical example of a bridge dataset with 7,200
demonstrations for 71 tasks in 10 environments, which we have released
publicly on the project website.111https://sites.google.com/view/bridgedata To
the best of our knowledge, our work is also the first to demonstrate transfer
scenarios (2) and (3) above. This is significant, because (2) provides users
with a low-cost way to “import” all of the skills in the bridge dataset into
their own domain with just a small number of demonstrations in their domain,
while (3) provides for a way to boost the performance of an entirely new skill
with previously collected reusable bridge data. Our results suggest that
accumulating and reusing diverse multi-task and multi-domain datasets, at
least when all data is collected with the same type of robot, may make it
possible for researchers to endow robots with generalizable skills using only
a modest amount of in-domain data for their desired task.

##  II Related Work

While most prior work on deep visuomotor learning trains a single task in a
single domain [7, 8, 9, 10, 11, 12, 13, 14, 15], our goal is not to develop
better learning methods, but rather to illustrate how generic multi-domain,
multi-task datasets can be used with existing algorithms to boost the
generalization of _new_ tasks in _new_ domains. Prior work on multi-task
reinforcement learning [5] has shown that data from other tasks can boost
generalization of new tasks, however this study is carried out in a _single_
domain.

Existing robot learning datasets do not exhibit the right properties for
boosting the generalization of new tasks in _new_ domains or zero-shot
transferring skills from the prior dataset to a target domain. We provide an
overview of the most related datasets in Figure 2. Most existing robot
datasets, such as MIME [16], DAML [10], RoboTurk [17, 18], and many others
[19, 20, 21, 22, 23, 24, 5] only feature a single domain, making them
difficult to use for boosting the generalization in _other_ domains. Merging
multiple existing datasets into one multi-domain dataset is difficult due to
inconsistencies in data collection protocols, time discretization, robot
morphologies, and sensors. Learning from multiple robots has been studied with
RoboNet [6], which provides a dataset with 7 different robots in different
domains. Here the data is generated with random motions which do not produce
semantically meaningful tasks. This limits task complexity to pushing and
basic grasping, and makes the data poorly suited for imitation learning.

Dataset | # Tasks | # Trajec. | # Domains |  suitable for BC/IL
---|---|---|---|---
DAML [10] | 3 | 2.9k | 1 | ✓
MIME [16] | 22 | 8.2k | 1 | ✓
RoboNet [6] | N/A | 162k | 7 | ✗
RoboTurk [17, 18] | 3 | 2.1k | 1 | ✓
Vis. Imit. Made [25] | 2 | 2k | 50 | ✓
Ours | 71 | 7.2k | 10 | ✓
Figure 2: Comparison of our dataset and prior works. Our dataset has by far
the most tasks, and is the only dataset with more than 2 tasks that has many
domains. This is critical for evaluating the bridge data hypothesis.

Some prior works have also used datasets collected by humans without a robot,
across multiple domains. For example, Young et al. [25] presents results on
data across many more domains than our bridge data, collected via a hand-held
gripper, but only presents two grasping tasks.

##  III Bridge Datasets

In this section, we describe the basic principles behind bridge datasets and
how they can be used to boost generalization. Then, we present a description
of the specific bridge dataset that we collected using teleoperation of a low-
cost robotic arm for a range of kitchen-themed manipulation tasks. We use the
term _bridge dataset_ to refer to a large and diverse dataset of robotic
behaviors collected in a range of settings (e.g., different viewpoints,
lighting conditions, objects, and scenes), for a range of _different_ tasks,
so as to make it possible to “bridge” gaps in the generalization that arise
when the user provides a small to medium amount of data in their specific
target domain. We define the term “target domain” to refer to the environment
where the robot must perform the desired task. This target domain is distinct
from _any_ of the settings seen in the bridge dataset: the intent is for the
same large bridge dataset to be used by all users for whichever target domain
they require.

Figure 3: Demonstration data collection setup using VR Headset. The scene is
captured by 5 cameras simultaneously. While one of the cameras is fixed, the
others are mounted on flexible rods. Figure 4: Scenario (1): transfer with
matching behaviors. In this setting, bridge data is used to improve the
performance and generalization of tasks in the target domain for which the
user has collected some amount of data. These tasks must also be present in
the bridge data. In this example, the user demonstrates the “turn lever,”
“squash into pot,” and “flip cup” tasks in the target domain, and these tasks
are also present in several domains in the bridge data. After including the
bridge data in training, the performance and generalization of these tasks in
the target is significantly higher.

###  III-A Boosting Generalization via Bridge Datasets

We consider three types of generalization in our experiments, though other
modes may also be feasible:

(1) Transfer with matching behaviors, where the user collects some small
amount of data in their target domain for tasks that are also present in the
bridge data (e.g., around 50 demos per task), and uses the bridge data to
boost the performance and generalization of these tasks. We illustrate this
scenario in Figure 4. This scenario is the most conventional, and resembles
domain adaptation in computer vision, but it is also the most limiting, since
it requires the user’s desired tasks to be present in the bridge data.
However, as we will show, bridge data can enable very significant performance
and generalization boosts in this setting.

(2) Zero-shot transfer with target support, where the user utilizes data from
a few tasks in their target domain to “import” other tasks that are present in
the bridge data _without_ additionally collecting new demonstrations for them
in the target domain. For example, the bridge data contains the tasks of
putting a sweet potato into a pot or a pan, the user provides data in their
domain for putting brushes in pans, and the robot is then able to _both_ put
brushes as well as put sweet potatoes in pans. We illustrate this scenario in
Figure 5. This scenario increases the repertoires of skills that are available
in the user’s target environment, simply by including the bridge data, thus
eliminating the need to recollect data for every task in every target
environment.

Figure 5: Scenario (2): zero-shot transfer with target support. In this
setting, the goal is to “import” a task from the bridge data that was _not_
seen in the target domain. The user provides a few tasks in the target domain
that are used to connect to the bridge data, and then asks the robot to
perform a task that they did not provide, but which was seen in the bridge
data. In this case, the “put sweet potato in pot” task is present in the toy
kitchen 1 domain in the bridge data, but is _not_ demonstrated by the user in
the target domain. After training with user-provided data for other tasks, the
robot is able to perform “put sweet potato in pot” in the target domain.

(3) Boosting generalization of new tasks, where the user provides a small
amount of data (50 demonstrations in practice) for a new task that is not
present in the bridge data, and then utilizes the bridge data to boost
generalization and performance of this task. This scenario, illustrated in
Figure 6, most directly reflects our primary goals, since it uses the bridge
data without requiring _either_ the domains or tasks to match, leveraging the
diversity of the data and structural similarity to boost performance and
generalization of entirely new tasks.

Figure 6: Scenario (3): boosting generalization of new tasks. The user
provides some data for a _new_ task that was not seen in the bridge data, and
the bridge data is included in training to boost performance and
generalization for this new task.

To enable this kind of generalization boosting, we conjecture that the key
features that bridge datasets must have are: (i) a sufficient variety of
settings, so as to provide for good generalization; (ii) shared structure
between bridge data domains and target domains (i.e., it is unreasonable to
expect generalization for a construction robot using bridge data of kitchen
tasks); (iii) a sufficient range of tasks that breaks unwanted correlations
between tasks and domains. Analogously to how the ImageNet dataset [3]
provides broad coverage that makes it possible to boost generalization for a
range of computer vision tasks, the broader a bridge dataset is, the more
likely target tasks receive a generalization boost in a particular target
domain.

###  III-B A Bridge Dataset of Large-Scale Kitchen Tasks

We instantiate a bridge dataset based on the principles above as follows:

Robotic system overview. Since our dataset is likely the most useful for users
with the same or similar type of robot, we chose to use a low-cost and widely
available robot, a 6-dof WidowX250s (US$2900), which many other users of our
dataset are likely to be able to obtain. The total cost of the setup is less
than US$3600 (excluding the computer). To collect demonstrations, we use an
Oculus Quest headset, where we put the headset on a table as illustrated in
Figure 3 next to the robot and track the user’s handset while applying the
user’s motions to the robot end-effector via inverse kinematics. We capture
images from 3 to 5 cameras concurrently, using standard webcams as well as
Intel RealSense depth cameras.

Data collection protocol. Our proposed bridge dataset, illustrated in Figure 1
consists of a total of 7200 demonstrations for 71 different tasks, collected
in 10 different environments, focusing on the theme of household kitchen
tasks. Each task has between 50 and 300 demonstrations. We opted to use
kitchen and sink ”play sets” for children, since they are smaller than real-
world kitchens and therefore ideal for small-scale robots, and they are
comparatively robust and low-cost, while still providing settings that
resemble typical household scenes. During data collection we randomize the
kitchen position (translations of 0-20cm) and the camera positions
(translations of 0-10cm and rotations of 0-30 degrees) for all cameras on
flexible rods every 25 trajectories. The positions of distractor objects (i.e.
objects not needed for a task) are randomized at least every 5 trajectories.
All environments except toy sink 4, toy sink 5, and kitchen 3 were collected
at UC Berkeley and use Logitech C920 webcams, the three remaining environments
were collected at the University of Pennsylvania and use Intel RealSense RGB-D
cameras. The trajectories collected at the University of Pennsylvania
randomize all camera positions once every 50 trajectories. Instructions for
how users can reproduce our setup and collect data in new environments can be
found on the project website.222https://sites.google.com/view/bridgedata

##  IV Using Bridge Data in Imitation Learning

As a proof-of-concept to illustrate the utility of bridge datasets for
boosting generalization in robot learning, we will present experimental
results for an imitation-based approach that utilizes this data, although the
data could also be used with a variety of other robotic learning algorithms
such as offline RL and model-based planning.

Incorporating bridge data. While a variety of transfer learning methods have
been proposed in the literature for combining datasets from distinct domains,
we found that a simple joint training approach is effective for deriving
considerable benefit from bridge data. For each of the scenarios outlined in
Section III-A, we take the user-provided demonstrations in the target domain
and combine them with the entire bridge dataset for training. Since the sizes
of these datasets are significantly different, we rebalance the datasets by
weighting each datapoint, as discussed at the end of this section. Imitation
learning then proceeds normally, simply training the policy with supervised
learning on the combined dataset using the architecture described in the
following paragraph. It is also possible to incorporate bridge data in other
ways, for example by pretraining and finetuning. We found pretraining to be
significantly less effective than joint training in our experiments, a finding
that is consistent with prior works [21], but we emphasize that bridge
datasets can be combined with target domain data in a variety of ways. Policy
architecture. We use task-conditioned behavioral cloning (BC) with an
additional task-id input to the policy, which is used to distinguish tasks
during training and testing. In some cases, a task cannot be uniquely
determined by only observing the input image, and a one-hot vector
representing the task will solve this issue. The images are first fed into a
34-layer ResNet [26] and the resulting feature maps are passed through a
spatial softmax [27, 28], which extracts a set of spatial positions of the
relevant features. The spatial features are then concatenated with the one-hot
task-id vector, and are fed into 3 layers of fully-connected networks by which
the final action prediction is produced. During training, for a batch of
training data containing tuples of task ids, images, and ground-truth actions,
the network is trained by minimizing the standard $\ell_{2}$-error between the
ground-truth actions and the predicted actions given by the policy provided
the task id and the image observation as the input.

Training details. Since the amount of target domain data is usually
significantly less than the amount of bridge data, we rebalance the two
datasets during training. In the matching behaviors and zero-shot transfer
with target support scenarios, the ratio between the number of trajectories in
the bridge and target data is roughly 10:1, and we rebalance the data such
that 70% of the dataset is bridge data and 30% is target domain data. In the
“boosting generalization of new tasks" scenario the imbalance is more severe,
roughly 60:1, and so we rebalance such that 90% of the dataset is bridge data
and 10% is target domain data. Lower rebalancing ratios of bridge data and
target domain data tend to produce overfitting when the amount of target
domain data is as low as 50 demonstrations.

##  V Experimental Results

Our experimental evaluation aims to study how well bridge data can facilitate
generalization in scenarios (1), (2), and (3), as outlined in Section III-A.
We utilize the bridge dataset described in Section III-B. We evaluate
generalization on a set of new target domains with limited target domain data
for each of the generalization scenarios, and compare the performance of
learned policies with and without bridge data. Videos of the experiments are
included in the supplementary materials and on the project webpage, which we
encourage the reader to view to get a clearer sense for the diversity of the
tasks: https://sites.google.com/view/bridgedata

Quantitative metrics. All quantitative evaluations use 10 trials per
task,varying object positions and distractors on every trial and varying the
position of the robot relative to the environment every 5 trials. This ensures
that all test configurations are unique and different from any condition seen
in training, providing a measurement of generalization performance for the
policy. When the experiments in toy kitchen 1-3 and toy sink 1-3 were
conducted, the bridge dataset only comprised 4700 trajectories. Other
experiments use the full dataset with 7200 trajectories total.

Figure 7: Comparisons of joint training with bridge data (blue) and other
approaches for each type of scenario. The black vertical lines on the average
success rate bar denote the standard error of the mean across different tasks
for that scenario. Left: Performing joint training on bridge and target data
leads to improved performance, here the task is included both in the bridge
and target dataset. Middle: Using target domain data from other tasks helps
transferring tasks from the bridge dataset to the target domain. Right: Joint
training with the bridge data and a target task that is _not_ contained in the
bridge dataset enables significant generalization improvement compared to only
training on the target task alone. Tasks with an asterisk (*) uses objects
that are not part of the bridge dataset. Figure 8: Examples of successful
trajectories performed by the policy jointly trained with prior data and
target domain data. Left: put pot in sink (scenario 1); middle: put carrot on
plate (scenario 2); Right: wipe plate with sponge (scenario 3).

Scenario (1): transfer with matching behaviors. Figure 7 (left) shows results
for the transfer learning with matching behaviors scenario, where the user
provides some data for a set of tasks in the target domain (which are also
present in the bridge data), and we evaluate whether including bridge data
during training improves performance and generalization. For comparison, we
include the performance of the policy when trained _only_ on the target domain
data, without bridge data (Target Domain Only), a baseline that uses only the
bridge data without any target domain data (Direct Transfer), as well as
baseline that trains a single-task policy on data in the target domain only
(Single Task). The Toy Kitchen 2 (tk2) target domain has 6 tasks, and Toy Sink
3 (ts3) has 10 tasks, each with 50 demonstrations.

As can be seen in the results, jointly training with the bridge data leads to
significant gains in performance ($66\%$ success averaged over tasks) compared
to the direct transfer ($14\%$ success), target domain only ($28\%$ success)
and the single task ($18\%$ success) baseline. This is not surprising, since
this scenario directly augments the training set with additional data of the
_same_ tasks, but it still provides a validation of the value of including
bridge data in training (for a qualitative example see Figure 8, left).

Scenario (2): zero-shot transfer with target support. In the next experiment,
we evaluate tasks in the target domain for which the user did _not_ provide
any data. Instead, the user only collected data for _other_ tasks in the
target domain. This experiment evaluates whether bridge data can be used to
“import” tasks into the target domain. We provide a qualitative example for
this scenario in Figure 8 middle, which shows an experiment where we transfer
the “put carrot on plate" task into the Toy Sink 1 target domain using the
bridge data and target domain data consisting of 10 _other_ tasks. Due to
space constraints, We provide a visualization of these other tasks on the
project webpage.

Since there is no target domain data for these tasks, we cannot compare to a
baseline that does not use bridge data at all, since such a baseline would
have no data for these tasks. However, we do include the “direct transfer”
baseline, which utilizes a policy trained only on the bridge data. Note that
this comparison is non-trivial: it is not at all clear a priori that target
domain data for _other_ tasks should boost transfer performance of tasks that
are only present in the bridge data. The results, shown in Figure 7 (middle),
indicate that the jointly trained policy which obtains $44\%$ success averaged
over tasks indeed attains a very significant increase in performance over
direct transfer ($30\%$ success), suggesting that the zero-shot transfer with
target support scenario offers a viable way for users to “import” tasks from
the bridge dataset into their domain.

Scenario (3): boosting generalization of new tasks. The last generalization
scenario, which most directly evaluates the _bridge data hypothesis_ , aims to
study how well bridge data can boost the generalization of entirely new tasks
in the target domain, which are not present in the bridge data. To study this
question, we collected data for 10 different unique tasks in 4 different
environments and excluded them from the bridge data to simulate a user
collecting their own unique task in their new target environment. Figure 8
right illustrates one of these scenarios, where we collected 50 demonstrations
for the “wipe place with sponge" task in the the real kitchen 1 target domain.
_Neither_ data from the target domain nor this task or this object are present
in the bridge data. After jointly training with both bridge and target data we
obtain a significant generalization boost when running the policy in the
target domain, compared to a policy trained on only the single-task target
domain data. Direct transfer is impossible here, because the bridge data does
not contain this task. The results are presented in Figure 7 (right), and show
that training jointly with the bridge data leads to significant improvement on
6 out of 10 tasks across three evaluation environments, leading to $50\%$
success averaged over tasks, whereas single task policies attain around $22\%$
success – a 2$\times$ improvement in overall performance (the asterisks denote
in which experiments the objects are _not_ contained in the bridge data). The
significant improvements obtained from including the bridge data suggest that
bridge datasets can be a powerful vehicle for boosting generalization of new
skills, and that a single shared bridge dataset can be utilized across a range
of domains and applications. Of course, structural similarity between
environments and tasks is important, and all of these evaluations use other
toy kitchen or sink setups. We expect the applicability of a bridge dataset to
increase as the breadth of domains and tasks in the dataset increases. When
does bridge data help? In Figure 9 we provide a list of example scenarios
where the bridge data helps and where it does not (the first 7 rows). More
qualitative results, including videos of these tasks and additional
discussion, are provided on the project website due to space constraints:
tinyurl.com/b258nvdu. Qualitatively, we observed that the tasks that most
consistently benefit from the inclusion of bridge data contain objects that
visually resemble those seen in the bridge data (e.g., there are gains for
‘put pear in bowl,’ where the pear resembles the vegetables in the bridge
data, but no gains in ‘flip orange pot upright,’ since the orange pot looks
very different from any container in the bridge data), contain behavior that
physically is related to behavior seen in the bridge data (e.g., in ‘put
detergent in dry rack,’ the bridge data helps since the motion resembles the
pick-and-place motions in the prior data, whereas in ‘open box flaps’ bridge
data does not help since the type of pushing motions involved in this task are
very rare in the bridge data), and take place in domains that are visually and
structurally related to those in the bridge data (e.g., the bridge data helps
with ‘wipe plate with sponge’ in Figure 8 in a real kitchen, but does not help
with ‘pick screwdriver from tool chest task,’ since the scene does not
resemble the toy kitchens in the bridge data). Unfortunately, it is difficult
to provide a more precise and formal treatment of when transfer learning
succeeds in general, though we expect this would be an exciting direction for
future research.

##  VI Conclusion

We show how a large, diverse bridge dataset can be leveraged to improve
generalization in robotic learning. Our experiments demonstrate that including
bridge data when training skills in a new domain can improve performance
across a range of scenarios, both for tasks that are present in the bridge
data and, perhaps surprisingly, entirely new tasks. This means that bridge
data may provide a generic tool to improve generalization in a user’s target
domain. In addition, we showed that bridge data can also function as a tool to
_import_ tasks from the prior dataset to a target domain, thus increasing the
repertoires of skills a user has at their disposal in a particular target
domain. This suggests that a large, shared bridge dataset, like the one we
have released, could be used by different robotics researchers to boost the
generalization capabilities and the number of available skills of their
imitation-trained policies.

Both our experimental evaluation and our technical approach do have a number
of limitations. While we carefully set up our experiments to reflect a likely
real-world usage scenario, where the target domain is distinct from the bridge
data (i.e., to reflect what would happen if _someone else_ were to use our
bridge data for their robot in their lab), we still only evaluate in a few
distinct settings, namely in 5 different environments at UC Berkeley.

However, our imitation learning results do illustrate the benefits of diverse
bridge data, and we hope that by releasing our dataset to the community, we
can take a step toward generalizing robotic learning and make it possible for
anyone to train robotic policies that readily generalize to varied
environments without repeatedly collecting large and exhaustive datasets.

Task |  Tar. Env |  Joint Train |  Single Task |  Potential reason for no gain with bridge data
---|---|---|---|---
turn faucet lever (1) | tk2 | 0% | 0% |  | The faucet in tk2 has very different
---
appearance from the other faucets
Pickup pan from stove (2) | ts1 | 0% | N/A | Not enough target domain data and prior data for this task
Put spoon into pot (2) | tk2 | 0% | N/A | Not enough target domain data and prior data for this task
flip orange pot upright (3) | tk2 | 50% | 60% |  | There is no orange pot in prior dataset,
---
only metal pots in prior dataset
open box flaps (3) | tk2 | 10% | 10% | Boxes and pushing motions do not occur in prior data
take lid off pot (3) | ts3 | 60% | 60% | Only 100 demos involving lids in prior dataset.
pick up screw driver (3) | toolchest | 0% | 0% |  | The toolchest and screwdrivers are
---
visually very different from prior data
put pot or pan in sink (1) | tk2 | 90% | 50% |
put carrot on plate (2) | ts1 | 40% | N/A |
Wipe plate w/ sponge (3) | k1 | 70% | N/A |
put pear in bowl (3) | tk2 | 50% | 10% |
put brush in pot (3) | ts3 | 90% | 0% |
put detergent dry rack (3) | ts3 | 80% | 10% |
lift bowl (3) | tk2 | 70% | 50% |
Figure 9: Comparison of scenarios where usage of the bridge data helps
performance and where it does not. Scenarios where usage of bridge data does
not help are marked in red font. The type of transfer setting is denoted by
the number in brackets after the task description.

## References

  * [1] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” _Advances in neural information processing systems_ , vol. 25, pp. 1097–1105, 2012.
  * [2] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” _arXiv preprint arXiv:1810.04805_ , 2018.
  * [3] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in _Conference on Computer Vision and Pattern Recognition_ , 2009.
  * [4] T. Yu, S. Kumar, A. Gupta, S. Levine, K. Hausman, and C. Finn, “Gradient surgery for multi-task learning,” _arXiv preprint arXiv:2001.06782_ , 2020\.
  * [5] D. Kalashnikov, J. Varley, Y. Chebotar, B. Swanson, R. Jonschkowski, C. Finn, S. Levine, and K. Hausman, “Mt-opt: Continuous multi-task robotic reinforcement learning at scale,” _arXiv preprint arXiv:2104.08212_ , 2021\.
  * [6] S. Dasari, F. Ebert, S. Tian, S. Nair, B. Bucher, K. Schmeckpeper, S. Singh, S. Levine, and C. Finn, “Robonet: Large-scale multi-robot learning,” _arXiv preprint arXiv:1910.11215_ , 2019.
  * [7] C. Finn, T. Yu, T. Zhang, P. Abbeel, and S. Levine, “One-shot visual imitation learning via meta-learning,” in _Conference on Robot Learning_. PMLR, 2017, pp. 357–368.
  * [8] Y. Duan, M. Andrychowicz, B. C. Stadie, J. Ho, J. Schneider, I. Sutskever, P. Abbeel, and W. Zaremba, “One-shot imitation learning,” _arXiv preprint arXiv:1703.07326_ , 2017.
  * [9] J. Ho and S. Ermon, “Generative adversarial imitation learning,” _arXiv preprint arXiv:1606.03476_ , 2016.
  * [10] T. Yu, C. Finn, A. Xie, S. Dasari, T. Zhang, P. Abbeel, and S. Levine, “One-shot imitation from observing humans via domain-adaptive meta-learning,” _arXiv preprint arXiv:1802.01557_ , 2018.
  * [11] Y. Liu, A. Gupta, P. Abbeel, and S. Levine, “Imitation from observation: Learning to imitate behaviors from raw video via context translation,” in _International Conference on Robotics and Automation (ICRA)_ , 2018.
  * [12] P. Sermanet, C. Lynch, Y. Chebotar, J. Hsu, E. Jang, S. Schaal, S. Levine, and G. Brain, “Time-contrastive networks: Self-supervised learning from video,” in _2018 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2018, pp. 1134–1141.
  * [13] A. Ghadirzadeh, X. Chen, W. Yin, Z. Yi, M. Bjorkman, and D. Kragic, “Human-centered collaborative robots with deep reinforcement learning,” _IEEE Robotics and Automation Letters_ , 2020.
  * [14] S. Tian, S. Nair, F. Ebert, S. Dasari, B. Eysenbach, C. Finn, and S. Levine, “Model-based visual planning with self-supervised functional distances,” _arXiv preprint arXiv:2012.15373_ , 2020.
  * [15] T. Zhang, Z. McCarthy, O. Jow, D. Lee, X. Chen, K. Goldberg, and P. Abbeel, “Deep imitation learning for complex manipulation tasks from virtual reality teleoperation,” in _2018 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2018, pp. 5628–5635.
  * [16] P. Sharma, L. Mohan, L. Pinto, and A. Gupta, “Multiple interactions made easy (mime): Large scale demonstrations data for imitation,” in _Conference on robot learning_. PMLR, 2018, pp. 906–915.
  * [17] A. Mandlekar, Y. Zhu, A. Garg, J. Booher, M. Spero, A. Tung, J. Gao, J. Emmons, A. Gupta, E. Orbay, S. Savarese, and L. Fei-Fei, “Roboturk: A crowdsourcing platform for robotic skill learning through imitation,” in _Conference on Robot Learning_ , 2018.
  * [18] A. Mandlekar, J. Booher, M. Spero, A. Tung, A. Gupta, Y. Zhu, A. Garg, S. Savarese, and L. Fei-Fei, “Scaling robot supervision to hundreds of hours with roboturk: Robotic manipulation dataset through human reasoning and dexterity,” _arXiv:1911.04052_ , 2019.
  * [19] L. Pinto and A. Gupta, “Supersizing self-supervision: Learning to grasp from 50k tries and 700 robot hours,” in _international conference on robotics and automation (ICRA)_. IEEE, 2016\.
  * [20] C. Finn and S. Levine, “Deep visual foresight for planning robot motion,” in _2017 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2017, pp. 2786–2793.
  * [21] S. Levine, P. Pastor, A. Krizhevsky, J. Ibarz, and D. Quillen, “Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection,” _The International Journal of Robotics Research_ , vol. 37, no. 4-5, pp. 421–436, 2018.
  * [22] D. Kalashnikov, A. Irpan, P. Pastor, J. Ibarz, A. Herzog, E. Jang, D. Quillen, E. Holly, M. Kalakrishnan, V. Vanhoucke, _et al._ , “Scalable deep reinforcement learning for vision-based robotic manipulation,” in _Conference on Robot Learning_. PMLR, 2018, pp. 651–673.
  * [23] F. Ebert, C. Finn, S. Dasari, A. Xie, A. Lee, and S. Levine, “Visual foresight: Model-based deep reinforcement learning for vision-based robotic control,” _arXiv preprint arXiv:1812.00568_ , 2018.
  * [24] A. Zeng, S. Song, J. Lee, A. Rodriguez, and T. Funkhouser, “Tossingbot: Learning to throw arbitrary objects with residual physics,” _IEEE Transactions on Robotics_ , vol. 36, no. 4, pp. 1307–1319, 2020.
  * [25] S. Young, D. Gandhi, S. Tulsiani, A. Gupta, P. Abbeel, and L. Pinto, “Visual imitation made easy,” _arXiv e-prints_ , pp. arXiv–2008, 2020.
  * [26] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Conference on Computer Vision and Pattern Recognition_ , 2016.
  * [27] C. Finn, X. Y. Tan, Y. Duan, T. Darrell, S. Levine, and P. Abbeel, “Deep spatial autoencoders for visuomotor learning,” in _International Conference on Robotics and Automation (ICRA)_ , 2016.
  * [28] S. Levine, C. Finn, T. Darrell, and P. Abbeel, “End-to-end training of deep visuomotor policies,” _The Journal of Machine Learning Research_ , vol. 17, no. 1, pp. 1334–1373, 2016.


# Learning to Check Contract Inconsistencies

Shuo Zhang1, Junzhou Zhao1, Pinghui Wang111footnotemark: 1, Nuo Xu1,
Yang Yang1, Yiting Liu1, Yi Huang2, Junlan Feng2
Corresponding Author

###### Abstract

Contract consistency is important in ensuring the legal validity of the
contract. In many scenarios, a contract is written by filling the blanks in a
precompiled form. Due to carelessness, two blanks that should be filled with
the same (or different) content may be incorrectly filled with different (or
same) content. This will result in the issue of contract inconsistencies,
which may severely impair the legal validity of the contract. Traditional
methods to address this issue mainly rely on manual contract review, which is
labor-intensive and costly. In this work, we formulate a novel Contract
Inconsistency Checking (CIC) problem, and design an end-to-end framework,
called Pair-wise Blank Resolution (PBR), to solve the CIC problem with high
accuracy. Our PBR model contains a novel BlankCoder to address the challenge
of modeling meaningless blanks. BlankCoder adopts a two-stage attention
mechanism that adequately associates a meaningless blank with its relevant
descriptions while avoiding the incorporation of irrelevant context words.
Experiments conducted on real-world datasets show the promising performance of
our method with a balanced accuracy of $94.05\%$ and an F1 score of $90.90\%$
in the CIC problem.

##  1 Introduction

A contract is a legally binding agreement that recognizes and governs the
rights and duties of the parties to the agreement. Correctly composing
contracts is crucial to ensure its legal validity. In many real-world
scenarios, a standard contract is prepared by filling blanks in a precompiled
form. Due to carelessness, two blanks that should be filled with the same (or
different) content may be incorrectly filled with different (or same) content.
This will result in _contract inconsistencies_ , which may severely impair the
legal validity of the contract.

Contract review is widely used by companies to check contract inconsistencies.
However, contract review is labor-intensive and costly. Big companies have to
hire tens of thousands of lawyers to conduct contract review, and it is
estimated that Fortune Global $2000$ and Fortune $1000$ companies spend about
$\$35$ billion annually to review and negotiate contracts (Strom 2019).
Therefore, it is desired to design methods to automate the contract review
process.

In this work, we study the Contract Inconsistency Checking (CIC) problem, and
propose an end-to-end model to solve the CIC problem with high accuracy. As
far as we know, the CIC problem has not yet been studied in the AI community
and no effective solution exists.

We consider contract inconsistencies caused by incorrectly filling blanks in a
precompiled form. For example, in Figure 1, the two blanks with red background
actually refer to the same item for sale, but incorrectly filled with
different content. The two blanks with yellow background actually refer to two
different prices (i.e., one with tax and the other without tax), but
incorrectly filled with the same price. Our goal is to find methods that can
automatically detect such contract inconsistencies with high accuracy.

Figure 1: Examples of contract inconsistencies. The two blanks with red
background refer to the same item for sale but incorrectly filled with
different content. The two blanks with yellow background refer to two
different prices (one with tax and the other without tax) but incorrectly
filled with the same price.

A straightforward method to solve the CIC problem is to build a set of rules
for each contract, and these rules regularize the allowed content for each
blank. Rule-based methods have been used for other contract-related tasks such
as checking obligation violations in contracts (Governatori and Milosevic
2005), verifying electronic contracts (Abdelsadiq, Molina-Jimenez, and
Shrivastava 2011), and contract formalization (Joanni and Ratiu 2018).
However, the rules have to be built manually by experts, and they do not scale
well to diverse types of contracts.

Coreference resolution (CR) methods (Sukthanker et al. 2020) aim to identify
words or phrases that refer to the same real-world entity in a document. At
first glance, CR methods may solve the CIC problem if we view blanks as words
or phrases, and hence CR methods can be used to answer whether two blanks
refer to the same concept. However, blanks are indeed not words or phrases
which can be encoded using well studied word2vec methods, while blanks are
meaningless empties and have no semantic meaning at all. Existing CR methods
fail in modeling blanks in our case.

In this work, we propose a Pair-wise Blank Resolution (PBR) framework to solve
the CIC problem with high accuracy. We formulate the CIC problem as a pair-
wise binary classification problem. For a pair of blanks in the contract
document, our PBR model adopts the Siamese architecture (Bromley et al. 1994;
Reimers and Gurevych 2019) to encode each blank separately through the same
blank encoder and then predict whether they should be filled with the same
content or not. Blank modeling is challenging since it is hard to perform a
semantic comparison to associate a meaningless blank to its relevant
descriptions. In our PBR framework, we propose a novel BlankCoder that extends
the Transformer (Vaswani et al. 2017) encoder architecture to address the
above issue. BlankCoder adopts a two-stage modeling strategy where a blank is
first initialized with the more related local context words and then updated
by recurrently incorporating relevant information from the global context
words. In this way, related descriptions of the blank would be fully
introduced and the irrelevant ones would be ignored, yielding an informative
blank representation.

To evaluate our method on the CIC task, we build a large-scale Chinese
contract dataset of $281$ contract documents including $299,621$ training
samples (blank pairs). The English contract dataset for element extraction
released by Chalkidis et al. (2017) is also used, and we view each element as
a filled blank. The experimental results show that our method significantly
and consistently outperforms all baseline methods with a promising performance
of the balanced accuracy of $94.05\%$ and F1 of $90.90\%$.

Our contributions are summarized as follows:

(1) We formulate the Contract Inconsistency Checking (CIC) problem. As far as
we know, this problem has not yet been studied in the AI community.

(2) We propose a novel Pair-wise Blank Resolution (PBR) framework to address
the CIC problem. In PBR, we propose a BlankCoder that extends the Transformer
encoder architecture to efficiently model meaningless blanks.

(3) We collected and labeled a large-scale Chinese contract corpus for CIC.
The experimental results show the promising performance of our PBR method.

##  2 Related Work

Our work is mainly related to three lines of recent researches: automatic
contract analysis, coreference resolution, and blank modeling.

Existing automatic contract analysis methods mainly assist legal professionals
by information extraction. Early ML-based attempts performed sentence-level
classification on clause patterns (Indukuri and Krishna 2010) and service
exceptions (Gao, Singh, and Mehra 2011). Recent DL-based methods focus on
fine-grained intra-document classification of contract elements (Chalkidis et
al. 2019), obligations and prohibitions (Chalkidis, Androutsopoulos, and
Michos 2018), and insurance policies (Sun et al. 2019), and cross-document
search of relevant clauses (Guo et al. 2020). Previous works do not perform a
further comparison on the retrieved related sentences, leaving the detailed
checking process to lawyers. In this paper, we automate the CIC process in a
fully data-driven and end-to-end manner that significantly speeds up the
manual review process.

Coreference resolution (CR) aims to identify the words or phrases (mentions)
that refer to the same real-world entity. Existing methods can be classified
into three broad categories of mention-pair, entity-mention, and ranking
models. Mention-pair models (Wiseman et al. 2015) predict the coreference
label for every two mentions. Entity-mention models (Clark and Manning 2016)
directly model an entity by clustering mentions. Ranking models (Lee et al.
2017) were further introduced to model the degree of coreference. CIC can be
view as a modified CR task that aims to identify the blanks that refer to the
same content. However, CIC is much more challenging since the blanks are
meaningless empties that can not be addressed with CR methods.

Blank modeling has been investigated in Text Infilling and Zero Pronoun
Resolution (ZPR). In text infilling, a blank is usually treated as an out-of-
vocabulary token and modeled with sequence models such as BiLSTM (Fedus,
Goodfellow, and Dai 2018) and Transformer (Zhu, Hu, and Xing 2019). However,
these methods encode a blank with all its context words that contain
irrelevant noise descriptions. Similar to CR, ZPR aims to identify words that
co-refer with a gap (an omitted pronoun). To encode the gap, Yin et al. (2016)
designed a CenteredLSTM that focuses on the more related local words. In their
later work (Yin et al. 2018), self-attention was further utilized to enhance
the performance. Recently, Aloraini et al. (2020) adopted BERT (Devlin et al.
2019) to encode the gap with its nearest two words. Though able to avoid
incorporating irrelevant descriptions, ZPR methods would negligent faraway
relevant descriptions.

##  3 Contract Inconsistency Checking Problem

We formulate the Contract Inconsistency Checking (CIC) problem as a pair-wise
binary classification problem, i.e., given a pair of blanks occurred in the
contract document, we want to predict whether they should be filled with the
same content or not.

Formally, let $B$ denote a set of blanks in the contract document. Each blank
$b\in B$ is included in a surrounding sentence
$s=\\{w_{1},w_{2},\ldots,b,\ldots\\}$ consisting of $|s|-1$ words and one
blank111If a sentence in the contract document contains multiple blanks, we
construct the surrounding sentence $s$ by deleting the other blanks and only
keep the blank of interest and all the words in the original sentence.. Given
two blanks $b_{i},b_{j}\in B$, the CIC problem aims at predicting their
consistency relation $r_{ij}\in\\{0,1\\}$, with $r_{ij}=1$ meaning that the
two blanks $b_{i}$ and $b_{j}$ should be filled with the same content; and
$r_{ij}=0$ otherwise.

##  4 Pair-wise Blank Resolution Framework

In this section, we propose a Pair-wise Blank Resolution (PBR) framework to
solve the CIC problem. The overall structure of our PBR framework is
illustrated in Figure 2.

$b_{i}\in B$$b_{j}\in B$BlankCoderBlankCoderptweight
sharing$\mathbf{b}_{i}\in\mathbb{R}^{d}$$\mathbf{b}_{j}\in\mathbb{R}^{d}$$|\mathbf{b}_{i}-\mathbf{b}_{j}|$vector
concatenationbinary classifier$\hat{r}_{ij}$ Figure 2: The PBR framework.

###  4.1 Overview of the PBR Framework

The PBR framework is inspired by Coreference Resolution (CR) (Sukthanker et
al. 2020). CR aims at identifying words or phrases that refer to the same
real-world entity in a document. However, as we explained previously, blanks
are different from words or phrases because blanks have no semantic meaning at
all. The key challenge for CIC is to represent the meaningless blanks in a
contract document.

As illustrated in Figure 2, the proposed PBR framework adopts the Siamese
architecture with a novel BlankCoder model to encode each blank, and predicts
the consistency relation of the blank pair by a binary classifier. The
proposed BlankCoder is a Transformer variant for blank modeling that could
easily generalize to other tasks such as text infilling.

###  4.2 BlankCoder

The Transformer architecture (Vaswani et al. 2017) has shown a powerful
semantic modeling capability on various natural language processing tasks
including the relevant task of text infilling (Zhu, Hu, and Xing 2019). We
extend the Transformer encoder architecture, and propose BlankCoder to address
the challenge of modeling meaningless blanks.

In BlankCoder, in order to obtain a good blank embedding, each blank is made
to fully utilize its context information in the surrounding sentence using a
two-stage modeling strategy, i.e., the blank is first initialized based on the
local context information (e.g., words in a local region of the blank), and
then refined and updated based on the global context information (e.g., words
outside of the local region in the surrounding sentence).

Figure 3 depicts the architecture of BlankCoder, which mainly consists of
three modules: 1) multi-head context attention, which associate relevant
context words, 2) local visible pooling, which initiates the blank
representation with local keywords, and 3) global update, which refine and
update the blank representation with related context words.

Surrounding Sentence Representation Multi-Head Context
Attentionpt$S\in\mathbb{R}^{d\times n}$Local Visible
Poolingpt$H\in\mathbb{R}^{d\times n}$$+$$+$ptRelativePositionEncodingMulti-
Head Blank-Context AttentionGated Sequential
Update$N\times$pt$H\in\mathbb{R}^{d\times
n}$pt$\mathbf{b}^{(0)}\in\mathbb{R}^{d}$globalupdateBlank Embedding
$\mathbf{b}\in\mathbb{R}^{d}$pt$\mathbf{b}^{(N)}\in\mathbb{R}^{d}$ (a)
BlankCoder

(b) Local visible pooling and global update

Figure 3: The architecture of BlankCoder. It extends the Transformer encoder
architecture to model meaningless blanks. First, we perform local visible
pooling to build an initial blank embedding $\mathbf{b}^{(0)}$ with the more
related local context information. Second, we perform global update to
recurrently refine the initial blank embedding by gathering and injecting
relevant descriptions from the global context words.

In the following, we focus on describing how to obtain the embedding of a
blank $b\in B$.

#### Surrounding Sentence Representation

Given a blank $b\in B$ and its surrounding sentence $s$ consisting of $n$
words and one blank (i.e., $|s|=n+1$), we embed the $i$-th word $w_{i}\in s$
to a $d$-dimension vector $\mathbf{e}_{i}\in\mathbb{R}^{d}$. Embeddings
$\\{\mathbf{e}_{1},\ldots,\mathbf{e}_{n}\\}$ are stacked up to form a matrix
$S\in\mathbb{R}^{d\times n}$, which will be the representation of the
surrounding sentence. Embedding vector $\mathbf{e}_{i}$ is the sum of three
different embedding vectors capturing different semantic aspects.

The first embedding vector is the word embedding, denoted by
$\mathbf{e}_{i}^{\mathrm{word}}\in\mathbb{R}^{d}$, which is obtained by using
a pre-trained word2vec model. Word embedding $\mathbf{e}_{i}^{\mathrm{word}}$
only contains the linguistic information of word $w_{i}$.

The second embedding vector is the positional embedding, denoted by
$\mathbf{e}_{i}^{\mathrm{pos}}\in\mathbb{R}^{d}$, which incorporates the order
information of word $w_{i}$ in surrounding sentence $s$. We calculate
$\mathbf{e}_{i}^{\mathrm{pos}}$ using the same approach in the vanilla
Transformer model (Vaswani et al. 2017).

The third embedding vector is the segmentation embedding, denoted by
$\mathbf{e}_{i}^{\mathrm{seg}}\in\mathbb{R}^{d}$, which distinguishes the
preceding and following context words of the blank. This can be viewed that
the blank separates the surrounding sentence into two segments, and each
segment has one segmentation embedding. We adopted the learned segmentation
embeddings as implemented in BERT (Devlin et al. 2019).

Following BERT, the final embedding of the $i$-th word $w_{i}\in s$ is
calculated by
$\mathbf{e}_{i}\triangleq\mathbf{e}_{i}^{\mathrm{word}}+\mathbf{e}_{i}^{\mathrm{pos}}+\mathbf{e}_{i}^{\mathrm{seg}}$.

#### Multi-Head Context Attention

To adequately incorporate relevant descriptions for blank modeling, it’s
essential to first model contextual information and correlate relevant words
in the surrounding sentence. To this end, in the BlankCoder, we first adopt
the powerful multi-head self-attention mechanism as implemented in the
Transformer encoder for contextual encoding.

The Transformer encoder takes the embedding matrix $S$ as input, and updates
the embedding matrix layer by layer. The $i$-th layer outputs a matrix
$H^{(i)}\in\mathbb{R}^{d\times n}$ by

$H^{(i)}=\texttt{FFN}(\texttt{MultiHeadAtt}(H^{(i-1)})),\quad H^{(0)}=S.$

Here, FFN represents a feed-forward neural network, and MultiHeadAtt is the
multi-head version self-attention, i.e.,

$\displaystyle\texttt{SelfAtt}(S)=V\cdot\mathrm{softmax}(\frac{K^{\mathsf{T}}Q}{\sqrt{d_{k}}})$
$\displaystyle Q=W_{q}S,\quad K=W_{k}S,\quad V=W_{v}S.$

Here $Q\in\mathbb{R}^{d_{k}\times n},K\in\mathbb{R}^{d_{k}\times n}$, and
$V\in\mathbb{R}^{d\times n}$ denote the query, key and value matrix,
respectively. $W_{q}\in\mathbb{R}^{d_{k}\times d}$,
$W_{k}\in\mathbb{R}^{d_{k}\times d}$, and $W_{v}\in\mathbb{R}^{d\times d}$ are
learnable parameters. We refer to (Vaswani et al. 2017) for more details on
the Transformer encoder.

The output of the last Transformer encoder layer, denoted by
$H\in\mathbb{R}^{d\times n}$, is a better representation of the surrounding
sentence, that incorporates the linguistic information, position information,
and context information of the surrounding sentence. In the following, we
discuss how to obtain the blank embedding using the surrounding sentence
embedding $H$.

#### Local Visible Pooling

Because a blank belongs to its surrounding sentence, a straightforward way to
obtain the blank embedding is to sum the embeddings of all the words in the
surrounding sentence, i.e., sum the columns of $H$. However, words in the
surrounding sentence may have different importance for describing the blank,
and simply summing them ignores their difference and may include too much
noise.

Intuitively, the words that are close to the blank should be more related to
the blank than words that are faraway in the surrounding sentence. Similar
observation has been reported in other language modeling tasks (Yin et al.
2016; Aloraini and Poesio 2020). Therefore, we propose to sum embeddings of
these closer words to obtain an initial blank embedding, which will be further
refined. We refer to this step as local visible pooling, as illustrated in
Figure 3.

Ideally, we want to find a vector $\bm{\alpha}\in\mathbb{R}^{n}$ such that it
has larger values for words in a small local region around blank $b$ and
almost zero values for words outside of this region. Then, blank $b$’s initial
embedding $\mathbf{b}^{(0)}\in\mathbb{R}^{d}$ can be represented as

$\mathbf{b}^{(0)}=H\bm{\alpha}.$

We refer to $\bm{\alpha}$ as the pooling weights, and we discuss how to obtain
pooling weights $\bm{\alpha}$ In the following.

We first formally define this local region. Assume blank $b$ is between the
$i$-th and $(i+1)$-th word in the surrounding sentence, $0\leq i\leq
n$.222Here $i=0$ (or $i=n$) means that the blank is at the beginning (or end)
of the surrounding sentence. We define the $k$-visible zone of blank $b$ as
the set of (at most) $2k$ closest positions around $b$, i.e.,

$\mathrm{zone}_{k}\triangleq\\{\max(1,i-k+1),\ldots,\min(n,i+k)\\}.$

In our design, only words in this $k$-visible zone have non-zero pooling
weights, and even words in the $k$-visible zone may have different pooling
weights, depending on how informative the word could describe the blank. We
propose a masked self-attention to calculate $\bm{\alpha}$, i.e.,

$\bm{\alpha}=\mathrm{softmax}(\tanh(WH)^{\mathsf{T}}\mathbf{q}_{w}+\mathbf{u}).$

Here $\mathbf{u}\in\mathbb{R}^{n}$ is a mask vector and $\mathbf{u}_{i}=0$ if
$i\in\mathrm{zone}_{k}$; $\mathbf{u}_{i}=-\infty$ otherwise.
$\mathbf{q}_{w}\in\mathbb{R}^{d_{w}}$ and $W\in\mathbb{R}^{d_{w}\times d}$
will be learned during training. That is, surrounding sentence representation
$H$ is first fed to a one-layer perceptron to obtain its hidden
representation. Then we measure the similarity between the hidden
representation and vector $\mathbf{q}_{w}$. $\mathbf{q}_{w}$ can be viewed as
a fixed query that queries important words over a set of words represented by
the hidden representation. Therefore, our design will assign those words that
are close and informative to the blank larger pooling weights than faraway and
meaningless words in the surrounding sentence.

• Remark. The blank $b$ now has an initial embedding $\mathbf{b}^{(0)}$, and
words already have embedding matrix $H$. At this time, we suggest adding the
relative position embeddings (RPEs) to these embeddings to incorporate the
relative position information between words and the blank. RPEs are directly
adapted from the PEs in Transformer: the blank’s RPE is simply
$\mathbf{PE}_{0}$; if two words have the same distance $x$ to the blank, then
the two words have the same RPE, i.e., $\mathbf{PE}_{x}$. We will slightly
abuse the notations and still use $\mathbf{b}^{(0)}$ and $H$ to denote the
embeddings that have been added RPEs.

#### Global Update

The local visible pooling step only uses blank $b$’s $k$-visible zone in the
surrounding sentence to obtain an initial embedding $\mathbf{b}^{(0)}$. Note
that some words outside of the $k$-visible zone may be also informative and
useful for describing the blank. We propose a global update step that uses
words in the entire surrounding sentence to further refine the initial
embedding $\mathbf{b}^{(0)}$. After global update, blank embedding will
incorporate more contextual information in the surrounding sentence, and hence
will be more representative than $\mathbf{b}^{(0)}$.

The global update step refines initial embedding $\mathbf{b}^{(0)}$ in a
recurrent manner, and we denote the blank embedding after $t$-th refinement by
$\mathbf{b}^{(t)}$. In the following, we describe how to refine
$\mathbf{b}^{(t)}$ and obtain $\mathbf{b}^{(t+1)}$, at the $t$-th global
update step.

Given the current blank representation $\mathbf{b}^{(t)}$ and surrounding
sentence embedding $H$, we want to find words in the surrounding sentence that
are correlated to the blank. This can be achieved by computing a correlation
score between the blank and a word, and we propose to compute the correlation
score in the following way, i.e.,

$\bm{\beta}^{(t)}=\sigma\left(\frac{K\mathbf{q}^{(t)}}{\sqrt{d_{k}}}\right)\,\,\text{where}\,\,\mathbf{q}^{(t)}=W_{uq}\mathbf{b}^{(t)},K=W_{uk}H$

Here $\sigma$ denotes a sigmoid function, $W_{uq}\in\mathbb{R}^{d_{k}\times
d}$ maps vector $\mathbf{b}^{(t)}$ to a query vector,
$W_{uk}\in\mathbb{R}^{d_{k}\times d}$ maps matrix $H$ to a key matrix, and
$\bm{\beta}^{(t)}\in\mathbb{R}^{n}$ is a scaled inner-product vector measuring
the correlation score between the blank and each word in the surrounding
sentence. In practice, we use a multi-head mechanism to calculate
$\bm{\beta}^{(t)}$ which will capture similarities from different aspects,
similar to Transformer.

$\bm{\beta}^{(t)}=\texttt{MultiHeadScore}(\mathbf{b}^{(t)},H)$

Note that we do not normalize elements of $\bm{\beta}^{(t)}$ jointly by a
softmax operation, as is used in Transformer. Instead, we normalize elements
of $\bm{\beta}^{(t)}$ individually by a sigmoid operation because it is
possible that none of the words in the surrounding sentence is informative for
describing the blank. We refer to the above processing by multi-head blank-
context attention.

Score vector $\bm{\beta}^{(t)}$ will be used as a memory gate to guard whether
we should memorize the corresponding vector in $H$ or not, i.e.,

$\mathbf{m}^{(t)}=H\bm{\beta}^{(t)}$

where $\mathbf{m}^{(t)}\in\mathbb{R}^{d}$ is the memorized embedding vectors
in $H$ and those memorized vectors are correlated to the blank. Hence
$\mathbf{m}^{(t)}$ is considered to be informative for describing the blank
and should be used to refine $\mathbf{b}^{(t)}$. To this end, we propose to
apply the classic GRU (Cho et al. 2014) which takes $\mathbf{m}^{(t)}$ as
input and $\mathbf{b}^{(t)}$ as hidden state, i.e.,

$\mathbf{b}^{(t+1)}=\texttt{GRU}(\mathbf{m}^{(t)},\mathbf{b}^{(t)}).$

The above procedure will repeat $N$ times. Thus the initial blank embedding
$\mathbf{b}^{(0)}$ will be refined recurrently. We refer to the processing as
the gated sequential update. The final output $\mathbf{b}^{(N)}$ will be
considered as the final embedding of blank $b$, i.e.,
$\mathbf{b}\triangleq\mathbf{b}^{(N)}$. The multi-head blank-context attention
operation and gated sequential update operation are illustrated in Figure 3.

###  4.3 Classifier

Armed with the elaborately designed BlankCoder, we are now able to predict the
consistency relation between two blanks following the routing in Figure 2.

Given two blanks $b_{i}$ and $b_{j}$, we first obtain their embeddings
$\mathbf{b}_{i}$ and $\mathbf{b}_{j}$ using the BlankCoder, respectively. Then
we concatenate $\mathbf{b}_{i}$, $\mathbf{b}_{j}$, and also a comparative term
$|\mathbf{b}_{i}-\mathbf{b}_{j}|$ as the representation of the input blank-
pair. We adopt a feed-forward neural network-based binary classifier to
predict the consistency relation $\hat{r}_{i,j}$, i.e.,

$\hat{r}_{i,j}=\texttt{FFN}([\mathbf{b}_{i}:\mathbf{b}_{j}:|\mathbf{b}_{i}-\mathbf{b}_{j}|])\in[0,1].$

Here $:$ denotes vector concatenation, $|\cdot|$ denotes element-wise
difference, and FFN is a feed-forward neural network that the middle layer
activation functions are ReLUs and the last layer activation function is
sigmoid. The comparative term works as an implicit contrastive loss that
forces consistent blanks to have similar representations and vice versa.

During training, we choose the Focal Loss (Lin et al. 2017) as the objective
to address the class imbalance issue in the datasets.

$\mathrm{FL}(\hat{r}_{i,j},r_{i,j})=\begin{cases}-\alpha(1-\hat{r}_{i,j})^{\gamma}\log(\hat{r}_{i,j}),&r_{i,j}=1,\\\
-(1-\alpha)\hat{r}_{i,j}^{\gamma}\log(1-\hat{r}_{i,j}),&r_{i,j}=0.\end{cases}$

Here, $r_{i,j}\in\\{0,1\\}$ is the ground-truth consistency relation, $\alpha$
and $\gamma$ are the balanced factor and focusing parameter of the focal loss,
respectively.

##  5 Experiments

In this section, we conduct experiments on two real-world contract datasets to
evaluate our method.

###  5.1 Contract Datasets

• Chinese Contracts We have collected $246$ open source business contract
templates and $35$ real contracts from a company. These contracts are written
in Chinese, and cover categories such as investment, lease, and labor.
Annotation details of templates are reported in our technical appendix.

• ICAIL Contracts by Chalkidis et al. (2017). These contracts are written in
English, and they were used for the task of tagging contract elements. We
treat these extracted elements as filled blanks. The contracts are anonymized
where the words are replaced with numbers for privacy concerns.

For each contract document, we collect all the blank pairs, and obtain their
consistency relation by comparing the filled content. This allows us to build
large training datasets. However, the class labels are imbalanced. For the
Chinese Contracts, the ratio between positive samples and negative samples is
$1:59$, and for the ICAIL Contracts, the ratio is $1:48$. The statistics of
these datasets are summarized in Table 1.

contract dataset | # contracts | # blank-pairs | pos : neg
---|---|---|---
Chinese Contracts | $281$ | $299,621$ | $1:59$
ICAIL Contracts | $1,526$ | $67,765$ | $1:48$
Table 1: Data statistics | Chinese Contracts | ICAIL Contracts
---|---|---
model | AUC | accuracy | precision | recall | F1 | MCC | AUC | accuracy | precision | recall | F1 | MCC
BiLSTM | $98.26$ | $92.58$ | $88.24$ | $86.11$ | $87.16$ | $86.96$ | $96.06$ | $84.86$ | $79.22$ | $70.11$ | $74.39$ | $74.03$
Transformer | $94.45$ | $88.51$ | $79.47$ | $77.36$ | $78.40$ | $78.04$ | $93.37$ | $83.09$ | $74.36$ | $66.67$ | $70.30$ | $69.82$
Transformer-seg | $97.15$ | $92.32$ | $85.15$ | $86.90$ | $86.01$ | $85.78$ | $95.26$ | $83.99$ | $77.78$ | $68.39$ | $72.78$ | $72.40$
CenteredLSTM | $98.35$ | $92.68$ | $88.96$ | $86.59$ | $87.76$ | $87.56$ | $96.15$ | $85.71$ | $78.62$ | $71.84$ | $75.08$ | $74.66$
AttnLSTM | $98.62$ | $92.85$ | $90.83$ | $85.51$ | $88.09$ | $87.93$ | $96.19$ | $85.76$ | $80.92$ | $70.69$ | $75.46$ | $75.16$
CorefBERT | $93.06$ | $90.20$ | $53.85$ | $81.58$ | $64.87$ | $65.61$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$
PBR | 98.73 | 94.05 | 93.74 | 88.22 | 90.90 | 90.77 | 96.25 | 86.01 | 81.22 | 72.08 | 76.38 | 76.09
%Improvement | $0.11$ | $1.29$ | $3.20$ | $1.88$ | $3.19$ | $3.23$ | $0.06$ | $0.29$ | $0.37$ | $0.33$ | $1.22$ | $1.24$
Table 2: Model evaluation results (%). Note that the accuracy above stands for
the balanced accuracy.

###  5.2 Settings

#### Evaluation Metrics.

We use AUC, precision, recall, F1 score, balanced accuracy, and Matthews
correlation coefficient (MCC) (Boughorbel, Jarray, and El-Anbari 2017) as
evaluation metrics. Balanced accuracy is defined as the average of recall
obtained in each class. MCC is a correlation coefficient between the predicted
and ground truth binary classifications, and it has a value between $-1$ and
$+1$, with $+1$ representing a perfect prediction. Note that because the
datasets are highly imbalanced, we thus focus on comparing balanced measures:
F1 and MCC.

#### Baselines

We compare our method with the following approaches in text infilling and zero
pronoun resolution.

  * •

BiLSTM (Graves, Mohamed, and Hinton 2013) is a widely adopted bidirectional
RNN in text infilling (Fedus, Goodfellow, and Dai 2018; Liu et al. 2019),
where a blank is treated as an out-of-vocabulary token in the sentence and
modeled together with the context words.

  * •

Transformer (Vaswani et al. 2017) is a SOTA attention based language
representation model. We adopt the vanilla Transformer encoder and perform
blank modeling in the same way as in BILSTM.

  * •

Transformer-seg (Zhu, Hu, and Xing 2019) is a segment-aware Transformer for
text infilling, where the segments are obtained by splitting the sentence at
the blanks.

  * •

CenteredLSTM (Yin et al. 2017) is a zero pronoun modeling method that adopts
two RNNs with one encoding the preceding context sequentially and the other
encoding the following context reversely. A blank is represented with the
concatenation of the two last hidden vectors.

  * •

AttnLSTM (Yin et al. 2018) extends CenteredLSTM with a self-attention
mechanism to generate segment representations that are concatenated to encode
the blank.

  * •

CorefBERT (Aloraini and Poesio 2020) is a zero pronoun modeling method that
adopts a pre-trained BERT (Devlin et al. 2019) for contextual encoding and
represents a blank by averaging the embeddings of its nearest two words.

Metrics | AUC | accuracy | precision | recall | F1 | MCC
---|---|---|---|---|---|---
PBR | $98.73$ | $94.05$ | $93.74$ | $88.22$ | $90.90$ | $90.77$
-no local | $97.76$ | $92.31$ | $81.51$ | $87.86$ | $84.57$ | $84.36$
-no update | $98.14$ | $92.95$ | $87.81$ | $86.11$ | $86.95$ | $86.74$
-no cmp | $98.38$ | $93.43$ | $93.57$ | $86.96$ | $90.14$ | $90.04$
Table 3: Ablation analysis on the Chinese Contracts.

#### Implementation Details

We implement all the benchmarks using Pytorch on a server equipped with 2
Nvidia Tesla V100 GPUs, each with 32GB memory. For the Chinese Contracts, we
adopt the Chinese word2vec embeddings released by Li et al. (2018). For the
ICAIL Contracts, the word2vec embeddings are provided by Chalkidis et al.
(2017). We perform the hyper-parameter search to find the best combinations
for all the models. The details are shown in the appendix.333Codes available
at https://github.com/ShuoZhangXJTU/CIC

###  5.3 Model Evaluation

We compare the performance of different models, and show the results in Table
2. Note that CorefBERT is not applicable on the ICAIL Contracts due to data
anonymization.

We observe that our PBR framework outperforms the others in terms of all
evaluation metrics. Specifically, for balanced accuracy, PBR achieves
$0.29\sim 1.29\%$ improvements, for F1 score, PBR achieves $1.22\sim 3.19\%$
improvements, and for MCC, PBR achieves $1.24\sim 3.23\%$ improvements.

Compared to the vanilla Transformer encoder and its segment-aware version, our
BlankCoder in PBR also has significant improvement. The balanced accuracy is
improved by $1.87\sim 6.26\%$, the F1 score is improved by $4.95\sim 15.94\%$,
and MCC is improved by $5.09\sim 16.31\%$. This demonstrates the effectiveness
of the two-stage semantic modeling approach in our BlankCoder.

We also observe a general performance decline of all the benchmarks and a
limited performance gain of our PBR framework on the ICAIL Contracts. We
attribute this to data anonymity and the ambiguous blank pair samples. In the
ICAIL dataset, the sentences are not split to fit the task of tagging contract
elements. Due to its anonymity, we have to specify a region as a pseudo-
sentence, which would contain incomplete or redundant sentences that hinder
the performance. Besides, ambiguous blank pairs where both the same and
different contents are allowed can introduce extra noise that impairs the
performance. Details are illustrated in our technical appendix due to space
limitations.

Figure 4: Hyper-parameter sensitivity

###  5.4 Ablation Experiments

To further investigate the performance of the two-stage blank modeling
strategy, we conducted ablation experiments on PBR (see Table 3). Similar
results on the ICAIL Contracts are omitted because of space limitations.

To verify the effectiveness of the local visible pooling layer, we build a no-
local version PBR (i.e., “-no local”), which directly assigns the whole
sentence as the local visible zone. To evaluate the importance of the word-
level semantic reasoning, we remove global update (i.e., “-no update”). To
evaluate the effectiveness of the comparative term (i.e.,
$|\mathbf{b}_{i}-\mathbf{b}_{j}|$), we build a base PBR with no comparative
term in the classifier (i.e., “-no cmp”).

In Table 3, we observe that both the local visible pooling and global update
are important components of the PBR framework. The local visible pooling is
more critical since it determines if there exists noise information in the
initial embedding. We also observe that the absence of the comparative term in
the classifier leads to performance decrease, which implies the importance of
contrastive supervision.

###  5.5 Hyper-parameter Sensitivity

In this section, we study how hyper-parameters in BlankCoder affect the
performance, i.e., the length of local visible zone $k$, and the number of the
stacked global update blocks $N$. The results on the Chinese Contracts are
shown in Figure 4. We omit similar results on the ICAIL Contracts due to space
limitations.

In Figure 4, we show five different lines where each line denotes the
performance (F1) using different $N$ and $k$. Fixing $k$, we observe that the
stack of multiple global update blocks could consistently improve the
performance, which demonstrates the effectiveness of the recurrent update
process. Fixing $N$, we observe that the performance increases with the size
of the local region at first and then decreases. This is due to the fact that
a minimal local region does not contain enough related information, while a
large local region would introduce irrelevant noise words. To reach better
performance, a proper local visible zone shall be determined for the
BlankCoder.

Figure 5: The attention visualization on a case example in a real-word sales
contract.

###  5.6 Case Study: Attention in BlankCoder

To intuitively show the effectiveness that PBR can extract representative
features, we compare the attention of the Transformer encoder and our
BlankCoder. For simplicity, we average the attention weights for all the
Transformer encoder layers and omit the context attention layers of our
BlankCoder.

Figure 5 shows a case example in a real-word sales contract with the blank
referring to the final payment amount, where darker red shades indicate larger
attention weights. For the Transformer encoder, we see that irrelevant
descriptions like “days” and “delivery” are incorrectly highlighted, which
would lead to biased blank representation that hinders further consistency
checking. Whereas in our BlankCoder, the blank is first initialized with
related local keywords (e.g., “dollars”) and then recurrently updated with
other relevant descriptions (e.g., “final” and “ payment”), yielding an
accurate and expressive blank representation without noise information.

##  6 Conclusion and Future Work

In this work, we formulate the Contract Inconsistency Checking (CIC) problem,
an automatic contract analysis task with significant practical importance, and
we propose a novel end-to-end Pair-wise Blank Resolution (PBR) framework to
predict the consistency relation for every two blanks with high accuracy. In
PBR, we extend the Transformer encoder architecture and propose BlankCoder, an
off-the-shelf effective blank modeling method that could easily generalize to
other tasks such as text infilling. Extensive experiments show that our model
can significantly and consistently outperform existing baselines, yielding a
promising balanced accuracy of $94.05\%$ and an F1 score of $90.90\%$. In the
future, we plan to consider more complex cases (e.g., ambiguous blank pairs)
and explore more complex consistency checking scenarios that require logical
reasoning.

##  7 Acknowledgments

The research presented in this paper is supported in part by National Key R&D
Program of China (2018YFC0830500), National Natural Science Foundation of
China (61922067, U1736205, 61902305), MoE-CMCC “Artifical Intelligence”
Project (MCM20190701), Natural Science Basic Research Plan in Shaanxi Province
of China (2019JM-159), Natural Science Basic Research Plan in Zhejiang
Province of China (LGG18F020016).

## References

  * Abdelsadiq, Molina-Jimenez, and Shrivastava (2011) Abdelsadiq, A.; Molina-Jimenez, C.; and Shrivastava, S. 2011.  A High Level Model Checking Tool for Verifying Electronic Contracts.  In _School of Computing Science Technical Report Series_.
  * Aloraini and Poesio (2020) Aloraini, A.; and Poesio, M. 2020.  Cross-lingual Zero Pronoun Resolution.  In _LREC_.
  * Boughorbel, Jarray, and El-Anbari (2017) Boughorbel, S.; Jarray, F.; and El-Anbari, M. 2017.  Optimal classifier for imbalanced data using Matthews Correlation Coefficient metric.  _PLOS ONE_ 12(6): 1–17.
  * Bromley et al. (1994) Bromley, J.; Guyon, I.; LeCun, Y.; Säckinger, E.; and Shah, R. 1994.  Signature verification using a siamese time delay neural network.  In _NIPS_.
  * Chalkidis, Androutsopoulos, and Michos (2017) Chalkidis, I.; Androutsopoulos, I.; and Michos, A. 2017.  Extracting Contract Elements.  In _ICAIL_.
  * Chalkidis, Androutsopoulos, and Michos (2018) Chalkidis, I.; Androutsopoulos, I.; and Michos, A. 2018.  Obligation and prohibition extraction using hierarchical rnns.  In _arXiv:1805.03871_.
  * Chalkidis et al. (2019) Chalkidis, I.; Fergadiotis, M.; Malakasiotis, P.; and Androutsopoulos, I. 2019.  Neural Contract Element Extraction Revisited.  In _NeurIPS Workshop Document Intelligence_.
  * Cho et al. (2014) Cho, K.; van Merrienboer, B.; Gulcehre, C.; Bahdanau, D.; Bougares, F.; Schwenk, H.; and Bengio, Y. 2014.  Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation.  In _EMNLP_.
  * Clark and Manning (2016) Clark, K.; and Manning, C. D. 2016.  Improving coreference resolution by learning entity-level distributed representations.  In _arXiv:1606.01323_.
  * Devlin et al. (2019) Devlin, J.; Chang, M.-W.; Lee, K.; and Toutanova, K. 2019.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.  In _NAACL-HLT_.
  * Fedus, Goodfellow, and Dai (2018) Fedus, W.; Goodfellow, I.; and Dai, A. M. 2018.  MaskGAN: Better text generation via filling in the _.  In _arXiv:1801.07736_.
  * Gao, Singh, and Mehra (2011) Gao, X.; Singh, M. P.; and Mehra, P. 2011.  Mining Business Contracts for Service Exceptions.  _IEEE Transactions on Services Computing_ 5(3): 333–344.
  * Governatori and Milosevic (2005) Governatori, G.; and Milosevic, Z. 2005.  Dealing with Contract Violations: Formalism and Domain Specific Language.  In _EDOC_.
  * Graves, Mohamed, and Hinton (2013) Graves, A.; Mohamed, A.-r.; and Hinton, G. 2013.  Speech recognition with deep recurrent neural networks.  In _IEEE ASSP_.
  * Guo et al. (2020) Guo, H.; An, B.; Guo, Z.; and Su, Z. 2020.  Deep Semantic Compliance Advisor for Unstructured Document Compliance Checking.  In _IJCAI_.
  * Indukuri and Krishna (2010) Indukuri, K. V.; and Krishna, P. R. 2010.  Mining E-contract Documents to Classify Clauses.  In _COMPUTE_.
  * Joanni and Ratiu (2018) Joanni, A.; and Ratiu, D. 2018.  Formalization of RAM Contracts for Advanced Consistency and Completeness Checking.  In _ESREL_.
  * Lee et al. (2017) Lee, K.; He, L.; Lewis, M.; and Zettlemoyer, L. 2017.  End-to-end neural coreference resolution.  In _arXiv:1707.07045_.
  * Li et al. (2018) Li, S.; Zhao, Z.; Hu, R.; Li, W.; Liu, T.; and Du, X. 2018.  Analogical Reasoning on Chinese Morphological and Semantic Relations.  In _ACL_.
  * Lin et al. (2017) Lin, T.-Y.; Goyal, P.; Girshick, R.; He, K.; and Dollár, P. 2017.  Focal loss for dense object detection.  In _ICCV_.
  * Liu et al. (2019) Liu, D.; Fu, J.; Liu, P.; and Lv, J. 2019.  TIGS: An inference algorithm for text infilling with gradient search.  In _arXiv:1905.10752_.
  * Reimers and Gurevych (2019) Reimers, N.; and Gurevych, I. 2019.  Sentence-bert: Sentence embeddings using siamese bert-networks.  In _arXiv:1908.10084_.
  * Strom (2019) Strom, R. 2019.  This Startup Is Coming for $7 Billion in Contract Review.  https://news.bloomberglaw.com/us-law-week/this-startup-is-coming-for-7-billion-in-contract-review.
  * Sukthanker et al. (2020) Sukthanker, R.; Poria, S.; Cambria, E.; and Thirunavukarasu, R. 2020.  Anaphora and coreference resolution: A review.  _Information Fusion_ 59: 139–162.
  * Sun et al. (2019) Sun, L.; Zhang, K.; Ji, F.; and Yang, Z. 2019.  Toi-CNN: A solution of information extraction on Chinese insurance policy.  In _NAACL_.
  * Vaswani et al. (2017) Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, Ł.; and Polosukhin, I. 2017.  Attention is all you need.  In _NeurIPS_.
  * Wiseman et al. (2015) Wiseman, S. J.; Rush, A. M.; Shieber, S. M.; and Weston, J. 2015.  Learning anaphoricity and antecedent ranking features for coreference resolution.  In _ACL-NLP_.
  * Yin et al. (2016) Yin, Q.; Zhang, W.; Zhang, Y.; and Liu, T. 2016.  A deep neural network for Chinese zero pronoun resolution.  In _arXiv:1604.05800_.
  * Yin et al. (2017) Yin, Q.; Zhang, Y.; Zhang, W.; and Liu, T. 2017.  Chinese zero pronoun resolution with deep memory network.  In _EMNLP_.
  * Yin et al. (2018) Yin, Q.; Zhang, Y.; Zhang, W.; Liu, T.; and Wang, W. Y. 2018.  Zero pronoun resolution with attention-based neural network.  In _ACL_.
  * Zhu, Hu, and Xing (2019) Zhu, W.; Hu, Z.; and Xing, E. 2019.  Text infilling.  In _arXiv:1901.00158_.