# Enhanced Sliding Window Superposition Coding for Industrial Automation
††thanks: The paper is sponsored in part by the National Key R&D Program of
China No. 2020YFB1806605, by the National Natural Science Foundation of China
under Grants 62022049 and 62111530197, and by Hitachi.

Bohang Zhang, Zhaojun Nan, Sheng Zhou, Zhisheng Niu  Department of Electronic
Engineering
Beijing National Research Center for Information Science and Technology,
Tsinghua University, Beijing 100084, China
Emails<EMAIL_ADDRESS><EMAIL_ADDRESS>sheng.zhou@,
<EMAIL_ADDRESS>
###### Abstract

The introduction of 5G has changed the wireless communication industry.
Whereas previous generations of cellular technology are mainly based on
communication for people, the wireless industry is discovering that 5G may be
an era of communications that is mainly focused on machine-to-machine
communication. The application of Ultra Reliable Low Latency Communication in
factory automation is an area of great interest as it unlocks potential
applications that traditional wired communications did not allow. In
particular, the decrease in the inter-device distance has led to the
discussion of coding schemes for these interference-filled channels. To meet
the latency and accuracy requirements of URLLC, Non-orthogonal multiple access
has been proposed but it comes with associated challenges. In order to combat
the issue of interference, an enhanced version of Sliding window superposition
coding has been proposed as a method of coding that yields performance gains
in scenarios with high interference. This paper examines the abilities of this
coding scheme in a broadcast network in 5G to evaluate its robustness in
situations where interference is treated as noise in a factory automation
setting. This work shows improvements of enhanced sliding window superposition
coding over benchmark protocols in the high-reliability requirement regions of
block error rates $\approx 10^{-6}$.

##  I Introduction

With the release of 5G and the subsequent rollout of this network, the
wireless industry is beginning to realize that 5G may not be as much an era
for interpersonal communication but rather for communication between machines.
The applications of Ultra-Reliable Low Latency Communications (URLLC) are of
vital importance as it aims to create a balance between both reliability and
latency. The requirements of URLLC is usually $10^{-4}\text{to}10^{-6}$ Block
Error Rate (BLER) at 1-10ms latency. Traditional methods of improving
communications usually come at the cost of reliability or latency, i.e HARQ
can improve upon the reliability of the communications link at the cost of
latency [1] while simply shortening the error correcting code will decrease
latency at the cost of error rate [2]. With this proliferation of URLLC
applications, the factory automation scenario appears to be an area of great
interest as it combines URLLC with the rise of machine-based communication. In
particular, coding schemes are important for the attainment of URLLC as coding
can determine the fundamental limits of the system. Research by others has
identified LDPC as a good benchmark when dealing with the factory automation
setting [4].

Most of the previous works are mainly dealing with channels where individuals
have considered a system under Orthogonal Multiple Access (OMA). This may not
be a suitable scheme for URLLC as one major source of delay in communications
systems is the multiple access scheduling. As such, work by others has
considered the applications of Non-Orthogonal Multiple Access (NOMA) for URLLC
[5] [6]. This application of NOMA also extends into the factory automation
setting as well[7]. A major challenge of NOMA is the coding scheme used to
deal with the interference from other uses and work having been done on
evaluating associated coding schemes [8]. On the issue of interference,
Sliding Window Superposition Coding (SWSC) had been proposed by others for use
inside the interference channel[9] [10]. Whereas previous works of SWSC are
mainly focused on the achievable rate at a Block Error Rate of 0.1, this work
will focus on regions of high-reliability requirement where message error
rates are $\approx 10^{-4}$ to $10^{-6}$. Another challenge of URLLC and using
LDPC is the issue of error floor [11] at short block length [12]. This work
will take this into account when constructing associated simulations to offer
a complete comparison.

The main contributions of this work are as the following:

  1. 1.

Propose a new decoding method for SWSC called Enhanced SWSC (ESWSC) that is
able to address the issue of error propagation within SWSC to increase
performance. This is done while maintaining the same coding efficiency as
SWSC.

  2. 2.

Evaluate the performance of SWSC and ESWSC through simulations and observe the
impacts of block length, number of blocks, and block ratio $\alpha$ (for
ESWSC) on the overall performance of the codes.

This work will first propose an improved decoding method for SWSC called
Enhanced SWSC (ESWSC) in Section II. The benchmark protocols will be LDPC and
a modified version of LDPC (mLDPC). These protocols will also be outlined in
Section II. The simulation parameters are discussed in Section III where the
results are also analyzed. The conclusion in Section IV will summarize the
results obtained in the previous sections.

##  II System Model

An indoor factory automation situation is used where the information for $N$
receivers is sent in a broadcast manner for downlink. The downlink message is
sent through the standard 5G specifications on the data plane [13]. The
broadcasted message contains the information intended for all receivers and
each round of broadcast is considered successful if all of the $N$ receivers
are able to decode their respective messages. The proposed algorithm of ESWSC
will be compared against benchmark SWSC and versions of LDPC.

###  II-A SWSC

For $N$ total information packets that need to be sent, $a_{i}$ is an
individual information packet while $a=(a_{1},...,a_{N})$ denotes all the
information bits that are sent. $b_{i}$ is the coded bits after $a_{i}$ has
gone through error correcting code encoding with $b=(b_{1},...,b_{N})$ denotes
all the bits after error correcting code. $c_{i1}$ and $c_{i2}$ are the first
and second half of bits in $b_{i}$ respectively so
$b_{i}=c_{i1}\mathbin{\|}c_{i2}$ with $c_{1}=(c_{11},...,c_{N1})$ and
$c_{2}=(c_{12},...,c_{N2})$ denoting the first and second halves of $b$
respectively. For simplicity’s sake, the example in Fig. 1 shows SWSC of 2
layers with $d_{i1}$ and $d_{i2}$ denoting the location within the layer of
superposition coding but this can work for any number of layers. The bits in
$d_{11}$ will be a known sequence called $c_{\text{clean}}$. For each
superposition location pair of $d_{i1}$ and $d_{i2}$, the layers can be
modulated together to form $e_{i}$ with $e=[e_{1},...,e_{N}]$ is the message
that is sent through the channel.

Algorithm 1 SWSC Encoding Scheme

1: Assume $N$ original information packets are to be sent with each packet
being $a_{i}$ .

2: $a_{i}$ becomes $b_{i}$ from error correcting code.

3: Split $b_{i}$ into $c_{i1}$ and $c_{i2}$

4:

5: for $i\in 1\rightarrow N$ do

6: if $i=1$ then

7: Place a known sequence $c_{\text{clean}}$ in $d1_{1}$ location

8: Place $c_{11}$ and $c_{12}$ into $d_{21}$ and $d_{12}$ respectively

9: else

10: Place $c_{i2}$ and $c_{i2}$ into $d_{i2}$ and $d_{(i+1)1}$ respectively

11: end if

12: bits in $d_{i1}$ and $d_{i2}$ are modulated to form $e_{i}$.

13: end for

Figure 1: SWSC Encoding

The decoding of SWSC is the reverse of all the steps in the encoding process.
To start, $\hat{e_{1}}$ will decode into $\hat{d_{11}}$ and $\hat{d_{12}}$
where $\hat{e_{i}}$ is the received message after $e_{i}$ goes through the
wireless channel. $\hat{c_{11}}$ will have a higher chance of being right due
to the $c_{clean}$ in $\hat{d_{11}}$. The existence of a known message will
make decoding much easier to complete. Next, $\hat{e_{2}}$ will decode into
$\hat{c_{12}}$ and $\hat{c_{21}}$ respectively. $\hat{c_{11}}$ and
$\hat{c_{12}}$ will combine to form $\hat{b_{1}}$ which can obtain
$\hat{a_{1}}$. $\hat{a_{1}}$ can form $\tilde{b}_{1}$ and therefore
$\tilde{c_{12}}$ which can serve as the new $\hat{c}_{\text{clean}}$ sequence
that is used for further sliding window decoding. This sequence is repeated in
the manner outlined in Algorithm 2 and shown in Fig. 2.

Algorithm 2 SWSC Decoding Scheme

1: $\hat{e_{i}}$ is the received message after $e_{i}$ goes through channel

2: for $i\in 1\rightarrow N$ do

3: if $i=1$ then

4: $\hat{e_{1}}$ gets back bits in $\hat{d_{i1}}$ and $\hat{d_{i2}}$ through
demodulation

5: The bits in $\hat{d}_{11}$ is a known sequence $c_{\text{clean}}$. This
will make the bits in $\hat{d}_{12}$ easier to decode.

6: Demodulate $\hat{e}_{2}$ to get the bits in $\hat{d}_{21}$ and
$\hat{d}_{22}$

7: Use $\hat{c}_{11}$ from $\hat{d}_{12}$ and $\hat{c}_{12}$ from
$\hat{d}_{21}$ to construct $\hat{b}_{1}$

8: Run $\hat{b}_{1}$ through error correcting code to obtain information
packet $\hat{a}_{1}$

9: $\hat{a}_{1}$ becomes $\tilde{b}_{1}$ through error correcting code

10: $\tilde{b}_{1}$ splits into $\tilde{c}_{11}$ and $\tilde{c}_{12}$

11: $\tilde{c}_{12}$ is placed in $\tilde{d}_{21}$ which serves as the new
clean sequence $\hat{c}_{\text{clean}}$

12: else

13: $\hat{e}_{i}$ gets back bits in $\hat{d}_{i1}$ and $\hat{d}_{i2}$

14: For any $\hat{e}_{i}$, the sequence in $\hat{d}_{i1}$ location is assumed
to be $\hat{d}_{\text{clean}}$ from the previous coding step. This will make
the bits in $\hat{d}_{i2}$ easier to decode

15: Demodulate $\hat{e}_{(i+1)}$ to get the bits placed in $\hat{d}_{(i+1)1}$
and $\hat{d}_{(i+1)2}$

16: Use $\hat{c}_{i1}$ from $\hat{d}_{(i)2}$ and $\hat{c}_{i2}$ from
$\hat{d}_{(i+1)1}$ to construct $\hat{b}_{i}$

17: Run $\hat{b}_{i}$ through error correcting code to obtain information
packet $\hat{a}_{i}$

18: $\hat{a}_{i}$ becomes $\tilde{b}_{i}$ through error correcting code

19: $\tilde{b}_{i}$ forms $\tilde{c}_{i1}$ and $\tilde{c}_{i2}$

20: $\tilde{c}_{i2}$ placed in $\tilde{d}_{(i+1)1}$ to be used as new clean
sequence $\hat{c}_{\text{clean}}$ for the next sliding window

21: end if

22: end for

Figure 2: SWSC Decoding

The major advantage that SWSC offers over current protocols is the ability to
take advantage of the mutual information between the pieces of the code word
from sliding window blocks. This non-i.i.d nature of adjacent blocks allows
for better decoding as the coding chain is constantly able to create a ”clean”
message that can aid in the decoding of future blocks.

###  II-B ESWSC

Both the strength and the weakness of SWSC lie in the non-i.i.d nature of the
adjacent information packets during decoding. The successful decoding of the
current message is dependent on the previous message being decoded
successfully. This has an issue of error propagation as the unsuccessful
decoding of a single package results in the rest of the code being decoded
incorrectly. In order to combat this, ESWSC is proposed as an alternative
method to decode the SWSC coding chain. While the first portion of the message
is decoded in the same manner. A ratio of blocks $0\leq\alpha\leq 0.5$
starting from the back of the SWSC will be decoded in reverse. The $c_{N2}$
sequence of bits will be decoded first aided by the clean message located at
$d_{(N+1)2}$ before being combined with $c_{N1}$ to go through error
correcting code. This is shown in Fig. 3. When $\alpha=0$, the code will be
decoded in the same manner as SWSC while $\alpha=0.5$ will be the ideal method
to maximize decoding success. The reason for $\alpha$ to have a range of
values is that some of the information packets may be latency limited within
the frame. This requires a set ordering of the SWSC and $\alpha$ can be
adjusted accordingly. In general, most situations are not overall latency
constrained but this work takes into account the effect of $\alpha$ in its
simulations.

Figure 3: ESWSC Decoding

###  II-C Message Error Rate

The Message Error Rate (MER) can be calculated by finding out
$P(a\neq\hat{a})$. In order to evaluate the performance of ESWSC and SWSC
under this definition, a number of probabilities will be used. The first
probability is the probability of success for the first information packet to
be successfully decoded during SWSC or ESWSC decoding
$P_{\text{clean}}=P(a_{1}=\hat{a_{1}})$. For any other information packets
$a_{i}$, the probability of successfully decoding it given the previous
message is decoded successfully is given by
$P_{\text{assumed}}=P(a_{i}=\hat{a_{i}}|a_{(i-1)}=\hat{a}_{(i-1)})$. As a
function of BLER, the MER looks like:

$\text{MER}=1-{(1-\text{BLER})}^{N}$ (1)

The BLER equivalent for some MER values are shown below in Table I.

TABLE I: Approximate BLER values for MER values MER | N=20 | N=50 | N=100
---|---|---|---
$10^{-1}$ | $\approx 5.25\times 10^{-3}$ | $\approx 2.10\times 10^{-3}$ | $\approx 1.05\times 10^{-3}$
$10^{-2}$ | $\approx 5.00\times 10^{-4}$ | $\approx 2.00\times 10^{-4}$ | $\approx 1.00\times 10^{-4}$
$10^{-3}$ | $\approx 5.00\times 10^{-5}$ | $\approx 2.00\times 10^{-5}$ | $\approx 1.00\times 10^{-5}$
$10^{-4}$ | $\approx 5.00\times 10^{-6}$ | $\approx 2.00\times 10^{-6}$ | $\approx 1.00\times 10^{-6}$

For a total number of packets $N$, the probability of all information packets
being correct for SWSC is given by:

$P_{\text{SWSC}}=P_{\text{clean}}\times{(P_{\text{assumed}})}^{(N-1)}$ (2)

At the same time, the expression for the performance of ESWSC is given by:

$\begin{split}P_{\text{ESWSC}}={(P_{\text{clean}}\times{(P_{\text{assumed}})}^{(N/2-1)})}^{2}\\\
={(P_{\text{clean}})}^{2}\times{(P_{\text{assumed}})}^{(N-2)}\end{split}$ (3)

Since it can be known that

$P_{\text{clean}}\geq P_{\text{assumed}}$ (4)

Therefore

$P_{\text{ESWSC}}\geq P_{\text{SWSC}}$ (5)

For the purposes of MER

$\text{MER}_{\text{ESWSC}}\leq\text{MER}_{\text{SWSC}}$ (6)

###  II-D LDPC Baselines

The current standard of 5G dictates the use of LDPC code for the data channel
and this shall be used as a baseline comparison [13]. The i.i.d nature of LDPC
coding makes the modulation stacking order of the code irrelevant so the LDPC
will be constructed in a stacked manner that is similar to SWSC but without
the sliding window offset to serve as a comparison. When comparing with ESWSC,
extra caution needs to be placed on the fact that $\alpha$ will lead to a
difference in total latency constraint. This means that the portion of code
within the $\alpha$ constraint can be combined together to form a longer block
length LDPC code in order to take better advantage of Shannon theorems. This
mLPDC allows for a fair comparison with ESWSC as the latency constraints
between the methods will be identical for information packets within the
message. This is a modification on the works of others [14] who had
demonstrated that concatenation of blocks in the short block length region
will yield to improvement in error performance when keeping the overall
latency the same.

##  III Simulations

A simple simulation will be set up where a number of receivers are scattered
around a centralized controller. The channel parameters are calculated based
on NYUSIM [15]. The number of required channels was randomly generated using
the default parameters of an indoor simulation scenario as shown in Tables II
and III. A total of $N=\text{100}$ blocks are generated and an appropriate
number of sensors are selected randomly through a uniform distribution for
$N\leq 100$ as required. The exact values of the simulator are shown in Table
II and Table III.

TABLE II:  NYUSIM Channel Parameters Value Name | Value
---|---
Scenario | InH
Frequency (GHz) | 28
Barometric Pressure (mbar) | 1013.25
Humidity (%) | 50
RF Bandwidth (MHZ) | 800
Temperature(°C) | 20
Polarization | Co-Pol
Environment | NLOS
T-R Separation Distance Lower Bound(m) | 10
T-R Separation Distance Upper Bound (m) | 50
User Terminal Height (m) | 1.5
Base Station Height (m) | 3
Number of RX Locations | 100
TX Power (dBm) | 30
Spatial Consistency | Off
Human Blockage Parameters | Off
TABLE III: NYUSIM Antenna Properties Value Name | Value
---|---
TX Array Type | ULA
RX Array Type | ULA
Number of TX Antenna Elements Nt | 4
Number of RX Antenna Elements Nr | 4
TX Antenna Spacing (in $\lambda$) | 0.5
RX Antenna Spacing (in $\lambda$) | 0.5
Number of TX Antenna Elements Per Row Wt | 1
Number of RX Antenna Elements Per Row Wt | 1
TX Antenna Azimuth HPBW (°) | 10
RX Antenna Azimuth HPBW (°) | 10
TX Antenna Elevation HPBW (°) | 10
RX Antenna Elevation HPBW (°) | 10

For the coding parameters themselves. A standard 5G downlink model will be
used with the following values being selected for default coding parameters as
shown in Table IV. As the downlink 5G data channel requires the usage of LDPC,
it will be the coding scheme that is selected and appropriate CRC attachments
are all following the associated 5G standard[13]. In order to ensure the same
coding efficiency, the LDPC-based coding schemes will be using a coding rate
that is lower than SWSC-based ones that adjusts according to the exact
situations to ensure the total number of bits sent for LDPC-based schemes will
$\geq$ than those of the SWSC based ones. This is due to the additional half-
block length that is needed for SWSC-based schemes. The following default
parameters are used except when changes are needed for results.

TABLE IV: 5G Downlink Parameters Value Name | Value
---|---
Transport Block Length (bits) | 200
Rate of Code (/1024) | 460
SNR (dB) | 16
$\alpha$ | 0.5
$N$ | 100
Modulation | QPSK

The optimal way to achieve MIMO (in this case a $4\times 4$ channel) channel
precoding and decoding is through Singular Value Decomposition (SVD) [16].

The simulations are completed 250 times with the default parameters shown in
Table IV and the result can be observed below in Fig. 4. This can serve as the
baseline for comparisons when changing coding parameters.

Figure 4: Default Simulation Results

###  III-A The Impact of Code Block Length

When the block length is changed from 200 to 100 bits, the error rate of the
various codes changes as seen in Fig. 5. There is an increase in the success
rate of the code at SNR=14 where the requirements for URLLC is not as apparent
and where MER is actually lower for shorter block length code. This result is
similar to the results obtained by others in which they observed that shorter
block length may lead to improvements for LDPC at certain length regions [12].
But when attention is focused on the very low MER regions required for URLLC,
the impacts of long block length code show how it is able to achieve a lower
MER at the same SNR.

Figure 5: Code Block Length=100

###  III-B The Impact of Number of Blocks $N$

The number of coding blocks has the greatest impact on error performance as
the message is only considered successful if all the messages are decoded
correctly. This shortening of the number of blocks leads to a noticeable
decrease in error rate performance. The overhead of half a block that exists
for SWSC-based methods did not lead to as much of a performance change as one
might expect. This matches with the assumptions made in the original SWSC
paper [10] as it did not find a need to take into account the additional half-
block length of code. The very small number of blocks shown in Fig. 6 shows a
slight improvement over a medium number of blocks shown in Fig. 7 which
performs slightly better than those in the benchmark with a large number of
devices in Figure 4.

Figure 6: N=20

Figure 7: N=50

###  III-C The Impact of Block Ratio $\alpha$

The $\alpha$ value is an important factor that can impact the results of the
code. In order to ensure the requirements of URLLC are met for all information
packets within the message, the $\alpha$ value is an adjustable value that can
be modified for mMLC and ESWSC. The lowering of this value will lead to a
decrease in error performance as observed in the following figure. The overall
MER improvement from $\alpha=0.5$ to $\alpha=0.2$ is about 4-7% in the regions
of low-reliability requirements as shown in Fig. 8. The drop is more evident
in the high-reliability region on the log graph. It can be observed that in
the low-reliability region, the mMLC outperforms ESWSC while the reverse is
true for the high-reliability region. This suggests that ESWSC is able is a
good choice for when there are high-reliability requirements like URLLC
industrial automation.

Figure 8: Values of $\alpha$ at SNR=14 dB for low reliability, SNR=16dB for
high reliability

##  IV Conclusion

In conclusion, it can be observed that SWSC and ESWSC may not always yield the
best results in all success rate regions and will usually struggle against
their benchmark LDPC counterparts. In the regions of low MER rate that
qualifies for URLLC, ESWSC, and SWSC are usually shown to have better results
than their LDPC and mLDPC counterparts. When the number of blocks decrease,
all the coding schemes tended towards better performance as expected. In cases
of decreased block length, the SWSC-based coding schemes were able to maintain
a slight performance improvement in the high-reliability requirement regions
while losing out in the low-reliability requirement region. This trend
continues into the comparisons with a varying $\alpha$ value as well where
ESWSC outperforms mLPDC in the high-reliability region of BLER $\approx
10^{-6}$ but falls short in low-reliability requirement regions of BLER
$\approx 10^{-3}$.

## References

  * [1] L. Vangelista and M. Centenaro, “Performance evaluation of harq schemes for the internet of things,” _Computers_ , vol. 7, no. 4, 2018. [Online]. Available: https://www.mdpi.com/2073-431X/7/4/48
  * [2] M. Bennis, M. Debbah, and H. V. Poor, “Ultrareliable and low-latency wireless communication: Tail, risk, and scale,” _Proceedings of the IEEE_ , vol. 106, no. 10, pp. 1834–1853, 2018.
  * [3] 3GPP, “Remaining Issues on URLLC Data Channel Coding,” 3rd Generation Partnership Project (3GPP), Discussion and Decision 1904849, 04 18, version R1-1904849. [Online]. Available: https://portal.3gpp.org/desktopmodules/Specifications/Specification Details.aspx?specificationId=2996
  * [4] L. Fanari, E. Iradier, I. Bilbao, R. Cabrera, J. Montalban, and P. Angueira, “Comparison between different channel coding techniques for ieee 802.11be within factory automation scenarios,” _Sensors_ , vol. 21, no. 21, 2021. [Online]. Available: https://www.mdpi.com/1424-8220/21/21/7209
  * [5] M. Amjad and L. Musavian, “Performance analysis of noma for ultra-reliable and low-latency communications,” in _2018 IEEE Globecom Workshops (GC Wkshps)_ , 2018, pp. 1–5.
  * [6] Y. Liu, Y. Deng, M. Elkashlan, A. Nallanathan, and G. K. Karagiannidis, “Analyzing grant-free access for urllc service,” _IEEE Journal on Selected Areas in Communications_ , vol. 39, no. 3, pp. 741–755, 2021.
  * [7] J. Montalban, E. Iradier, P. Angueira, O. Seijo, and I. Val, “Noma-based 802.11n for industrial automation,” _IEEE Access_ , vol. 8, pp. 168 546–168 557, 2020.
  * [8] A. Munari, F. Clazzer, O. Simeone, and Z. Utkovski, “Grant-free access for iot in beyond-5g systems: The potential of receiver diversity,” in _2020 2nd 6G Wireless Summit (6G SUMMIT)_ , 2020, pp. 1–5.
  * [9] L. Wang, Y.-H. Kim, C.-Y. Chen, H. Park, and E. Şaşoğlu, “Sliding-window superposition coding: Two-user interference channels,” _IEEE Transactions on Information Theory_ , vol. 66, no. 6, pp. 3293–3316, 2020.
  * [10] H. Park, Y.-H. Kim, and L. Wang, “Interference management via sliding-window superposition coding,” in _2014 IEEE Globecom Workshops (GC Wkshps)_ , 2014, pp. 972–976.
  * [11] P. Lee, L. Dolecek, Z. Zhang, V. Anantharam, Borivoje, and M. J. Wainwright, “Error floors in ldpc codes: Fast simulation, bounds and hardware emulation,” in _2008 IEEE International Symposium on Information Theory_ , 2008, pp. 444–448.
  * [12] T. Bai, C. Xu, R. Zhang, A. F. Al Rawi, and L. Hanzo, “Performance of harq-assisted ofdm systems contaminated by impulsive noise: Finite-length ldpc code analysis,” _IEEE Access_ , vol. 7, pp. 14 112–14 123, 2019.
  * [13] 3GPP, “Release 15 Description; Summary of Rel-15 Work Items,” 3rd Generation Partnership Project (3GPP), Technical Report (TR) 21.915, 10 2019, version 21.915.
  * [14] G. Durisi, T. Koch, and P. Popovski, “Toward massive, ultrareliable, and low-latency wireless communication with short packets,” _Proceedings of the IEEE_ , vol. 104, no. 9, pp. 1711–1726, 2016.
  * [15] S. Ju, O. Kanhere, Y. Xing, and T. S. Rappaport, “A millimeter-wave channel simulator nyusim with spatial consistency and human blockage,” in _2019 IEEE Global Communications Conference (GLOBECOM)_ , 2019, pp. 1–6.
  * [16] J. Andersen, “Array gain and capacity for known random channels with multiple element arrays at both ends,” _IEEE Journal on Selected Areas in Communications_ , vol. 18, no. 11, pp. 2172–2178, 2000.


∎

11institutetext: All authors are from Monash University
11email: {mei.lee, soon.layki, siew.eu-gene<EMAIL_ADDRESS>

# An Annotated Commodity News Corpus for Event Extraction

Meisin Lee1 Lay-Ki Soon  Eu-Gene Siew  Ly Fie Sugianto

(Received: date / Accepted: date)

###### Abstract

Commodity news contain a wealth of information; one of the key information is
the analysis of recent commodity price movements along with notable events
that led to that movement. Through event extraction, useful information can be
extracted from commodity news. Besides the details of price movements, causal
relation between events and commodity price movement can be extracted as well.
To facilitate future research in this area, we present a new dataset with the
following information identified and annotated: (i) entities (both nominal and
named), (ii) events (trigger words and argument roles), (iii) event metadata:
modality, polarity and intensity, and (iv) event-event relations. These
information, when extracted, can be very beneficial for understanding the
correlation of current world affairs and commodity price movement, which can
then be used for commodity price prediction. To the best of our knowledge,
this is the first corpus that is annotated with elements which are crucial for
event extraction from commodity news.

###### Keywords:

Commodities News Price Prediction Annotated Dataset Event Extraction

##  1 Introduction

Financial markets are sensitive to breaking news on economic events.
Specifically for crude oil markets, it is observed in (Brandt and Gao, 2019)
that news about macroeconomic fundamentals and geopolitical events affect the
price of the commodity. Therefore accurate and timely automatic identification
of events in news items is crucial for making timely trading decisions.
Commodity news contain a wealth of information. Generally, news of this genre
contain summaries or recent updates of the commodity in focus. The analyses by
the reporter or journalist of past events provide a good distillation of world
events that are truly causal to the movement of commodity prices.
Understanding the correlation between events and commodity price movement can
be useful for other downstream tasks such as for the purpose of learning event
sequence (also known as scripts in (Schank and Abelson, 2013)) and for
predicting commodity price movement. Commodity news typically contain these
few key information: (i) analysis of recent commodity price movements (up,
down or flat), (ii) a retrospective view of notable causal event(s) that led
to the movement, (iii) forecast or forward-looking analysis of supply-demand
situation as well as projected commodity price targets, and (iv) events
metadata.

Below are five sentences taken from a number of commodity news articles,
illustrating the various types of information found in the commodity corpus:

U.S. crude stockpiles soared(E1) by 1.350 million barrels in December from a
mere 200 million barrels to 438.9 million barrels, due to this oversupply(E2)
crude oil prices plunged(E3) more than 50% as Tuesday.111this example will be
used throughout to the paper to illustrate the annotation details. Brent crude
oil climbed(E4) towards $112 a barrel on Thursday on fears that a slower
global economy will demand less(E5) fuels than anticipated. Oil is cruising
higher(E6) after OPEC+ refused to cut(E7) supplies. Crude oil futures fell(E8)
in Asian trading on Wednesday after the International Energy Agency
trimmed(E9) its demand forecast for the fourth quarter of this year. It is
expected that London-traded crude future to drop(E10) to the $78 level by the
beginning of the second quarter since OPEC is considering a reduction in its
supply cuts(E11).

The types of information can be categorized into the following groups of key
information:

  1. i

Analysis of recent commodity price movement \- Events E1, E3, E4, E6, E8, and
E10 are recent commodity price movements.

  2. ii

Retrospective view of notable causal event(s) that led to commodity price
movement \- Events E2, E5, E7, E9, and E11 are causal events that led to the
the corresponding price movement. Details of causal event relation is covered
in Section 4.2.5.

  3. iii

Forecast or forward-looking analysis of supply-demand situation or price
targets \- Event E9 is an example of forecast of demand, while event E10 is an
example of anticipated commodity price target.

  4. iv

Event Metadata:

     * •

Modality: Event E5 is an anticipated event that has yet to take place.

     * •

Polarity: Event E7 did not take place since it is being refused.

     * •

Intensity: Event E11 is an event that has taken place but intensity is
reduced.
Detailed description of Event Metadata is covered in Section 4.2.4

Specifically for this genre of text, it is insufficient to conduct Event
Detection without extracting event arguments as well. To illustrate this
point, consider event E3 in example sentence (1), trigger word "plunged", it
is incomplete to just identify the event movement-down-loss without knowing
"what" plunged. In commodity news, possible events are ‘price plunged’,
‘supply plunged’ and ‘demand plunged’. As seen sentence (1), the sentence
consists of two movement events (E1 and E3). To disambiguate and to extract
events accurately, it is important to extract event arguments accurately as
well.

After an extensive search of conference papers and journals, we have come to a
conclusion that there is no publicly available annotated commodity news corpus
that is suitable for event extraction. Our contribution to facilitate future
research in this area is a new annotated commodity222At this stage, we target
only commodity news relating to crude oil dataset, characterising 18 event
types (Appendix C), with 21 entity types (Appendix B) along with each events’
metadata: polarity, modality and intensity. Each event type has its own set of
argument roles, in total there are 20 argument roles (Appendix D). This paper
describes the annotation methodology used in the commodity news corpus and
makes the annotated corpus available for academic research
purpose333https://github.com/meisin/Commodity-News-Corpus.

In this paper, information about the corpus, including data collection and
annotation procedure are laid out in Section 3. Annotation methodologies for
Entities and Events (Event Trigger, Event Arguments, Event Metadata and Event
Relation) are covered in detail in Section 4. Annotation evaluation and
challenges are stated in Section 5. Potential uses of the corpus are covered
in Section 6; it is then followed by Conclusion in Section 7.

##  2 Related Work

The annotation methodologies presented here are conceived based on the notions
proposed by canonical programs such as ACE2005 (Automatic Content Extraction),
TAC-KBP Event track shared task and ERE (Entities, Relations and Events). An
extensive comparison has been made in (Aguilar et al, 2014), where authors
analyzed and provided a summary of the different annotation approaches.
Subsequently there were a number of works that expanded earlier annotation
standards, such as in (O’Gorman et al, 2016), authors introduced the Richer
Event Description (RED) corpus and methodologies that annotate entities,
events, times, entities relations (coreference and partial coreference), and
events relations (temporal, causal, and sub-events). On the other hand,
authors in (Thompson et al, 2017) introduced a more comprehensive way of
annotating event metadata, which is known as meta-knowledge in the paper,
covering modality, subjectivity, source, polarity and specificity of the
event. Their schema was tailored for news events. In (Araki et al, 2018), the
authors presents methodologies for annotating events and event relations
across different domains, as an extension to formalization focusing on domain-
specific event ontology as seen in ACE2005 and TAC-KBP.

###  2.1 Financial Events

In the domain of Finance and Economics, it is found that annotated datasets
for the task of event extraction are rather scarce. Most of the available
datasets are on company-related events and are used mainly for extracting
company events for company stock price prediction. Examples of such datasets
are:

  * •

Dataset released in (Malik et al, 2011) is a dataset focused on financial
events found in European corporate press releases for events such as
dividends, profits announcements, and other quantitative events (non-
quantitative events such as mergers and acquisition are excluded). The paper
proposed the use of statistical classifiers aided by rules to extract these
events.

  * •

Dataset in (Hogenboom et al, 2013) is a dataset consisting of ten different
company-specific financial events; they are announcements regarding CEOs,
presidents, products, competitors, partners, subsidiaries, share values,
revenues, profits, and losses. Along with the dataset, authors presented an
event detection framework basing on knowledge-bases and patterns and relying
on domain ontology. The authors used rule-sets or domain ontology knowledge-
bases that are mostly or entirely created by hand.

  * •

Dataset in (Jacobs et al, 2018) is a dataset consisting of ten types of
company-economic events which were manually annotated: Buy ratings, Debt,
Dividend, Merger & acquisition, Profit, Quarterly results, Sales volume, Share
repurchase, Target price, Turnover. Event detection was done via the
supervised classification approach on this dataset.

  * •

In (Dor et al, 2019), the authors proposed using Wikipedia sections to extract
weak labels for sentences describing company events to overcome the constraint
of using predefined event taxonomies.

  * •

Authors in (Yang et al, 2018) presented a system that can automatically
generate a large scale labeled data and extract financial events at document-
level from Chinese Financial News. The experiments were carried out on four
types of financial events: Equity Freeze, Equity Pledge, Equity Repurchase,
and Equity Overweight.

All existing resources listed above are focused on company events and they
cannot be directly used in our work because company-related events are
essentially different from macro-economic and geo-political events found in
commodity news. After an extensive search for commodity news related
resources, we came across RavenPack’s444RavenPack is an analytics provider for
financial services. Among their services are finance and economic news
sentiment analysis. More information can be found on their page:
https://www.ravenpack.com/ crude oil dataset. This dataset is available
through subscription at the Wharton Research Data Services (WRDS). It is made
up of news headlines and a corresponding sentiment score generated by
Ravenpack’s own analytic engine. Unfortunately this dataset is not suitable
for the task of event extraction as it only contains sentiment score without
any annotation on events. However, Ravenpack’s event taxonomy on crude oil-
related events proves to be a useful resource in helping us define our own
event taxonomy. Details of event taxonomy is covered in Section 4.2.2.

##  3 Corpus Description

In this section, we describe the corpus and the data collection process in
detail.

###  3.1 Data Collection

The commodity news articles were collected from the following news
agencies555Due to copyright issues, original articles are not released.
Instead URL links to each articles are provided along with their corresponding
annotation file.:

  * •

https://www.investing.com/commodities/crude-oil-news

  * •

https://www.reuters.com/news/economy

  * •

https://www.cnbc.com

  * •

https://www.hellenicshippingnews.com/

  * •

https://oilprice.com/Latest-Energy-News/

  * •

https://www.marketwatch.com

  * •

https://www.marketpulse.com

  * •

https://www.fxempire.com/news/

From the pool of commodity news articles, about 150 pieces of news articles
were chosen based on their headlines. This ensures we create a balanced
dataset with each event type represented as evenly as possible to avoid
imbalanced data. More information about event types is included in Section
4.2.2.

###  3.2 Annotation Procedure

The dataset is annotated using Brat rapid annotation tool (Stenetorp et al,
2012), a web-based tool for text annotation. The annotated version of Sentence
(1) using Brat is shown in Figure 1.

Figure 1: Annotation using Brat annotation tool: (i) entities (both nominal
and named) are annotated with entity types listed above the respective words
(in various colours except green), (ii) events trigger words are also
annotated (in green), and (iii) entities are linked to their respective event
trigger through arches, argument roles these entities play in linked events
are listed on the arches.
Note: Event metadata: modality, polarity and intensity, and event-event
relations are not shown in this diagram.

The annotation process is designed to have high inter-annotator agreement
(IAA). One of the criteria is that annotators should possess basic knowledge
in business, finance, and economics. As such, annotators were chosen from a
pool of students from Monash School of Business. Annotators were then given
annotation training and provided with clear annotation schemas and examples.
Every piece of text was duly annotated by two annotators independently.

The annotation was done based on the following layers:

  * •

Layer 1: Identify and annotate entity mentions.

  * •

Layer 2: Annotate events by identifying event triggers.

  * •

Layer 3: Using event triggers as anchors, identify and link surrounding entity
mentions to their respective events. Annotate the argument roles each entity
mention plays with respect to the events identified.

  * •

Layer 4: Annotate event metadata: modality, polarity and intensity. Each
Modality, Polarity and Intensity label will be supported by the existence a
corresponding cue word.

  * •

Layer 5: Annotate event-event relations.

After each layer, an adjudicator assessed the annotation and evaluated inter-
annotator agreement before finalizing the annotation. For cases where there
are annotation discrepancies, the adjudicator will act as the tie-breaker to
decide on the final annotation. Once finalized, annotators then proceed with
the next layer. This is done to ensure no accumulation of the previous layer’s
errors in the subsequent layers of annotation.

##  4 Annotation Methodologies

As described in Section 3.2, annotation is done layer by layer at the sentence
level. This section describes our definition of events and principles for
annotation of entity mentions and events. Annotation of events is further
divided into (i) annotating entity mentions, (ii) annotating event triggers,
(iii) linking entity mentions to their respective events and identifying the
argument roles each entity mention plays, (iv) assigning the right metadata
labels to Polarity, Modality and Intensity, and (iv) annotating event-event
relations.

###  4.1 Entity Mention

An entity mention is a reference to an object or a set of objects in the
world, including named entities, nominal entities, and pronouns. For
simplicity and convenience, values and temporal expressions are also
considered as entity mentions in this work. There are 21 entity types
identified and annotated in the dataset. Nominal entities relating to Finance
and Economics are annotated. Apart from crude oil-related terms, below here
are some examples of nominal entities found in the Commodity News Corpus and
was duly annotated:

  1. 1.

attributes : price, futures, contract, imports, exports, consumption,
inventory, supply, production, usage, deliveries

  2. 2.

economic entity : economic growth, economy, market(s), economic outlook,
growth, dollar, green back, employment data

The list of entity types and the corresponding list of entities (both nominal
and named entities) are listed in Appendix , the list of entity types are
listed in Appendix BSeB.

###  4.2 Events

Events are defined as ‘specific occurrences’, involving ‘specific
participants’666https://www.ldc.upenn.edu/sites/www.ldc.upenn.edu/files/english-
events-guidelines-v5.4.3.pdf. The occurrence of an event is marked by the
presence of an event trigger. In addition to identifying triggers, all of the
participants of each event are also identified. An event’s participants are
Entities that are involved in that Event. Details and rules for identifying
event triggers are described in Section 4.2.1, while details for identifying
Event Arguments are covered in Section 4.2.3.

####  4.2.1 Event Triggers

Our annotation of Event Triggers is aligned to TAC-KBP Event Guidelines and
also to (Mitamura et al, 2015) where an event trigger (known as event nugget
in the shared task) can be either a single word (main verb, noun, adjective,
adverb) or a continuous or discontinuous multi-word phrase. Here are some
examples found in the dataset:

  * •

Verb: Houti rebels attacked Saudi Arabia.

  * •

Noun: The government slapped sanctions against its petroleum….

  * •

Adjective: A fast growing economy has…

  * •

Multi-verb: The market bounced back….

We have defined a set of 18 oil-related event types as our annotation scope.
Event types and the corresponding list of example trigger words are listed out
in Table 7 in Appendix C.

####  4.2.2 Event Taxonomy

According to (Brandt and Gao, 2019) who analyzed the Ravenpack’s sentiment
score of each event type and oil price, events that move commodity prices are
geo-political, macro-economic and commodity supply and demand in nature. The
list of Ravenpack’s Event Taxonomy as analyzed by (Brandt and Gao, 2019) is
found in Appendix A. Based on Ravenpack’s event taxonomy, 18 types of events
coupled with event arguments can be further grouped into the following
categories:

  * •

geo-political: Geo-political tension, civil unrest, embargo / sanctions, trade
tensions and other forms of geo-political crisis (event details and example
sentence in Appendix C.1).

  * •

macro-economic: US employment data, economic / gross domestic product (GDP)
growth, economic outlook, growth forecast (event details and example sentence
in Appendix C.2).

  * •

commodity supplies and demand: Demand / supply decrease, demand / supply
increase, action taken to increase demand / supply and action taken to
increase demand / supply (event details and example sentence in Appendix C.3
and C.4).

  * •

commodity price movement: price increase, price decrease, price forecast
(event details and example sentence in Appendix C.5).

####  4.2.3 Event Arguments

After event triggers and entity mentions are annotated, they need to be linked
up to form events. An event contains an event trigger and a set of event
arguments. Details of events and their corresponding list of arguments are
found in Appendix D. Referring to Figure 2, the event trigger soared is linked
to seven entity mentions via arches. The argument role of each entity mention
is labeled on each arch respectively, while entity types are labeled in
various colours on top of each entity span. This information is also
summarized in Table 1.

Figure 2: The first event from Figure 1 is re-drawn here to show the details
clearly. Event trigger (soared) is linked to its event arguments with argument
roles displayed on the arches. Table 1: List of Event Arguments of example
shown in Figure 2

.  Argument Role Entity Type Sentence Supplier Country [U.S.] crude…..soared
by… Item Commodity U.S. [crude]…..soared by… Attribute Financial-Attribute
U.S. crude [stockpiles] soared by… Difference Quantity ….soared by [1.350
million barrels]… Reference time Date ….soared by ….in December… Initial value
Quantity …soared by…from a mere 200 million barrels… Final value Quantity
…soared by…to 438.9 million barrels.

####  4.2.4 Event Metadata: Event Polarity, Modality and Intensity

After events are identified, they are also assigned a label each for Event
Polarity, Modality, and Intensity. Cue words for Polarity, Modality, and
Intensity are identified and annotated. The presence of these cue words acts
as supporting evidence when assigning event metadata labels. Each metadata is
described in detail in the following subsections.

##### POLARITY

(Possible labels for Event Polarity are: POSITIVE and NEGATIVE)
An event has the value POSITIVE unless there is an explicit indication that
the event did not take place, in which case NEGATIVE is assigned.

OPEC countries refused to cut_caused-movement-down-loss oil supplies.
Polarity: NEGATIVE (cue: refused)

OPEC’s efforts prevented oil prices from rising_movement-up-gain much.
Polarity: NEGATIVE (cue: prevented)

##### MODALITY

(Possible labels for Event Modality are: ASSERTED and OTHER)
Event modality determines whether the event represents a “real” occurrence.
ASSERTED is assigned if the author or speaker refers to it as though it were a
real occurrence, and OTHER otherwise. OTHER covers believed events,
hypothetical events, commanded and requested event, threats, proposed events,
discussed events, desired events, promised events, and other unclear
construct.

The market expects US to sanction_embargo Iran.
Modality: OTHER (cue: expects)

Analysts were anticipating oil inventories to fall_movement-down-loss by
800,000.
Polarity: OTHER (cue: anticipating)

##### INTENSITY

(Possible labels for Event Intensity are: NEUTRAL, INTENSIFIED and EASED)
Event intensity is a new event metadata, specifically created for this work to
better represent events found in this corpus. Oftentimes, events described in
Commodity News are about the intensity of an existing event, whether the event
is further intensified or eased.

Examples of events that are INTENSIFIED:

…could hit Iraq ’s output and deepen a supply shortfall_undersupply.
Intensity: INTENSIFIED (cue: deepen)

Examples of events that EASED:

Libya ’s civil strife_civil-unrest has been eased by potential peace talks.
Intensity: EASED (cue: eased) The event strife (civil unrest) in sentence (11)
is not an event with negative polarity because the event has actually taken
place but with reduced intensity. INTENSITY label is used to capture the
interpretation accurately, showing that the civil unrest event has indeed
taken place but with less intensity.

With these three event metadata information, we can annotate and capture all
essential information about an event. To further illustrate this point,
consider the list of examples of complex events below:

Delay a planned easing of output cutscaused-movement-down-loss
Polarity: NEGATIVE (cue: delay)
Modality: OTHER (cue: planned)
Intensity: EASED (cue: easing)

In order to end the global crisis, OPEC may hesitate to implement a planned
loosening of output curbs_caused-movement-down-loss.
Polarity: NEGATIVE (cue: hesitate)
Modality: OTHER (cue: planned)
Intensity: EASED (cue: loosening)

Oil prices rose to $110 a barrel on rumours of a renewed strife_civil-unrest.
Polarity: POSITIVE
Modality: OTHER (cue: rumours)
Intensity: INTENSIFIED (cue: renewed)

####  4.2.5 Event Relation

In (Araki et al, 2018) and in (O’Gorman et al, 2016), various event-event
relations are identified and annotated. For this work, however, only the
causal-preconditions relation is annotated. This is because, among the
relationships, the one that is of interest is the cause-effect relationship of
events, i.e. the events that caused commodity price movement. Similar to
(Araki et al, 2018) and (O’Gorman et al, 2016), two types of causal-
preconditions identified here are:

##### Explicit causal-preconditions

: There are two types of explicit causality: (i) one with explicit connective:
with the verbs such as “because” which have obvious causal meanings, and (ii)
one with ambiguous connectives with causal meaning such as “due to”. This is
illustrated in sentence (15) with causal connectives underlined.

Prices have fallen_movement-down-loss from $115 due to an oil glut_oversupply,
prices rebounded_movement-up-gain aided by sharp fall_movement-down-loss in
U.S. oil drilling.

##### Implicit causal-preconditions

: Implicit causality is often expressed in absence of causal connectives in
the text, or there is only one entity of the cause or effect. For simplicity,
in this work, both prior events and simultaneous events are considered as
implicit causal-precondition, as shown in sentence (16) and (17):

Oil price dropped_movement-down-loss as supply increased_movement-down-loss.

Oil prices pushed to new high_position-high amid supply worries_negative-
sentiment.

###  4.3 Data Augmentation

To increase the size of the annotated dataset, we introduced more variance
through data augmentation: (i) trigger word replacement and (ii) event
argument replacement).

####  4.3.1 Trigger word replacement

FrameNet was utilized to augment available data and to generate both diverse
and valid examples. FrameNet777https://framenet.icsi.berkeley.edu shares with
ACE/ERE a goal of capturing information about events and relations in text.
Authors in (Aguilar et al, 2014) pointed out that all events, relations, and
attributes that were represented by ACE2005 or ERE and TAC-KBP standards can
be mapped to FrameNet representations through some adjustments, as shown in
Table 2:

Table 2: The mapping between ACE schemas and FrameNet frames ACE Schema |  FrameNet Frame
---|---
Event Trigger |  Lexical Unit
Event Arguments |  Frame Elements
Argument Role |  Name of Frame Elements
Event Type |  Frame
Entity |  Entity

A number of annotated sentences were selected, and their trigger words were
replaced with words (known as lexical units in FrameNet) of the same frame in
FrameNet. The idea is to replace the existing trigger word with another valid
trigger word while maintaining the same semantic meaning (in FrameNet’s term -
maintaining the same frame). Event trigger replacement is illustrated in
sentence (18):

The benchmark for oil prices advanced 1.29% to $74.71.
Potential trigger word replacements are surged, rose, appreciated, climbs,
etc.

####  4.3.2 Event argument replacement

Apart from data augmentation through trigger word replacement, at the same
time, we have also replaced event arguments. The replacement candidates were
chosen from a pool of candidates of the same entity type and the same argument
role within the pool of existing annotations, as illustrated in sentence (19).

…..after civil-unrest in Libya…
Potential event argument replacements from existing similar arguments from the
corpus are Iraq, Nigeria, Persian Gulf, Ukraine and etc.

###  4.4 Analysis

We have considered all annotation standards and their annotation methodologies
and we have strived to align to those established by canonical programs.
However, the annotation methodologies presented in this paper are adapted and
changed to cater to special characteristics found in commodity news. For
instance, under Event Metadata, Tense and Genericity found in ACE and ERE
standards are dropped from our annotation scope while the new metadata -
Intensity is introduced. With accurate annotations, quality information can be
extracted for other downstream tasks such as those listed in Section 6.

The commodity news dataset introduced here contains, in total, 3,950 events,
about 8,850 entities and about 990 causal-precondition event-event relation.
Events distribution is shown is Table 3. Event Polarity, Modality and
Intensity class distribution are presented in Table 4.

Table 3: Event type distribution and sentence level counts Event type | Type ratio | # sentence instances
---|---|---
1\. Cause-movement-down-loss | 13.35% | 524
2\. Cause-movement-up-gain | 2.23% | 88
3\. Civil-unrest | 2.53% | 100
4\. Crisis | 0.76% | 30
5\. Embargo | 3.75% | 148
6\. Geopolitical-tension | 1.70% | 67
7\. Grow-strong | 6.03% | 238
8\. Movement-down-loss | 22.69% | 896
9\. Movement-flat | 1.52% | 60
10\. Movement-up-gain | 22.13% | 874
11\. Negative-sentiment | 4.79% | 189
12\. Oversupply | 2.63% | 104
13\. Position-high | 3.82% | 151
14\. Position-low | 3.11% | 123
15\. Prohibiting | 1.06% | 42
16\. Shortage | 1.04% | 41
17\. Slow-weak | 5.47% | 216
18\. Trade-tensions | 1.39% | 55
Total |  | 3949
Table 4: Event Polarity, Modality and Intensity Distribution Polarity | Type ratio | # sentence instances
---|---|---
POSITIVE | 96.51% | 3811
NEGATIVE | 3.49% | 138
Modality | Type ratio | # sentence instances
ASSERTED | 81.24% | 3208
OTHER | 18.76% | 741
Intensity | Type ratio | # sentence instances
NEUTRAL | 87.59% | 3459
EASED | 7.47% | 295
INTENSIFIED | 4.94% | 195

##  5 Evaluation and Annotation Challenges

In total, the two annotators have annotated the same set of 150 documents and
achieve a Cohen’s Kappa score of 0.787. This is very close to the near-perfect
agreement range of [0.81, 0.99].

The main challenge of event annotation is ambiguities on ‘eventiveness’. Here
are examples where information is conveyed in an indirect way that makes it
difficult to pinpoint any clear-cut events:

Then spent the rest of the week trying to defend those gains as market
optimism ……

Oil prices felt pressure on Tuesday from news that OPEC will slowly taper
output cuts. In cases above where an event trigger is absent, we do not
annotate any event even though the event is conveyed in an implicit manner.

The second challenge is the need for domain expertise to annotate the text. In
many cases, it is imperative for annotators to understand financial and macro-
economic terms and concepts to interpret the text accurately and annotate
events accordingly. For instance, sentences containing macro-economic terms
such as contango, quantitative easing, and backwardation will require
annotators to have finance and economics domain knowledge. To ensure that the
annotators are knowledgeable with the domain knowledge, we recruited our
annotators from a pool of undergraduate students from the School of Business.

The third challenge is the difference in interpreting special concepts that
determines how these should be annotated. For example:

  * •

The word outlook, should it be interpreted as forecast? Or, should it be
considered as a cue word for event modality?

  * •

If events surrounding US employment data are annotated, then what about
unemployment? Should this be treated as employment data but negated using
negative polarity?

  * •

How should double negation be treated? For example, ‘failed attempt to prevent
a steep drop in oil prices’, both failed and prevent are considered negative
polarity cue words, creating a double negation situation.

For these non-straight forward cases, each one was handled on a case-by-case
basis where the adjudicator discussed each situation with the annotators to
seek consensus before finalizing an agreed annotation.

##  6 Uses of the Corpus

The main motivation for creating this annotated dataset is to contribute to
resource building in the domain of finance and economics and to further
facilitate research in this domain. Given the list of key information found in
commodity news as described in Section 1, the annotated information can
potentially be used in the following tasks:

  1. 1.

Event extraction: One obvious use of this annotation is for event extraction.

  2. 2.

Event relation extraction: Causal relation between events are useful in
understanding the event chain, or known as script-learning in (Schank and
Abelson, 2013). This can be used to predict subsequent events given the
occurrence of an event.

  3. 3.

Nominal entity recognition in financial or macro-economics text: A number of
nominal entities relating to global macro-economic, geo-political and supply-
demand situation were identified and duly annotated. Full list of nominal
entities is found in Appendix B. This resource can be used for training a
nominal entity recognition model as a complement component to existing Named
Entity Recognition (NER) functionality. This will be help in processing, and
analysing financial and economic text more effectively.

  4. 4.

Polarity, modality and intensity classification: Given that commodity corpus
has abundant events with various combinations of polarity, modality and
intensity (see sentence (12), (13) and (14) in Section 4.2.4), this dataset is
a good resource for event polarity, modality and intensity classification
task.

  5. 5.

Potentially expandable to other types of commodities: Event extraction model
built using this dataset can potentially, through domain adaptation, be used
for other commodities (such are gold, silver, palm oil and etc.) that are
broadly influenced by the same types of global events as crude oil.

##  7 Conclusion

Event extraction in the domain of finance and economics at the moment are
limited to only company-related events. To contribute to the building of
resources in this domain, we have presented a new commodity news dataset with
the following information annotated: (i) Entity mentions, (ii) Events
(triggers and argument roles), (iii) Event Metadata (Polarity, Modality, and
Intensity) and (iv) Event relations. We have also shared methodologies of how
these information were annotated. Along with the dataset, we have also
presented the various uses of the dataset.

There are a number of avenues for future work. The main piece that can be
further explored is to expand the annotation scope to cover more event types.
Next, this work can also be expanded to cover more event-event relations such
as event co-reference, sub-event, event-sequence, and contradictory event
relation. Besides that, the current sentence-level annotation can be extended
to cater for event relations spanning multiple sentences, so that event
extraction and relation extract can be done at the document level.

###### Acknowledgements.

Wharton Research Data Services (WRDS) was used in preparing this publication.
This service and the data available thereon constitute valuable intellectual
property and trade secrets of WRDS and/or its third-party suppliers.

##  Appendix A Ravenpack’s Event Taxonomy

RavenPack is an analytics provider for financial services. Among their
services are finance and economic news sentiment analysis. Ravenpack has over
5000+ taxonomy categories and uses proprietary event matching methods to
generate labels for their own news stream classifier. Based on news articles,
they also calculate sentiment scores for commodity price prediction. Based on
the analysis done by (Brandt and Gao, 2019), Ravenpack’s Event taxonomy is
grouped into three broader classes: (i) Geo-political news, (ii) Macro-
economic news, and (iii) Oil supply and demand news.

Table 5: Categories of RavenPack event taxonomy grouped into three broader
classes as presented in (Brandt and Gao, 2019)

.  Geo-political News: Terrorism War & Conflict Civil unrest Natural disasters
Government Macro-economic News: Sovereign Debt Public finance Retail sales
Consumer confidence Housing Interest rates Treasury yield Durable goods orders
Consumer spending Recession Economic growth GDP growth CPI PPI Trade balance
Exports Foreign exchange Employment Private credit Oil supply and demand:
Crude oil supply Crude oil demand Price Target Drilling & pipeline accident

##  Appendix B Entity Types

Table 6: List of Entity Types Entity Type |  Examples
---|---
1\. Commodity |  oil, crude oil, Brent, West Texas Intermediate (WTI), fuel, U.S Shale, light sweet crude, natural gas
2\. Country** |  Libya, China, U.S, Venezuela, Greece
3\. Date** |  1998, Wednesday, Jan. 30, the final quarter of 1991, the end of this year
4\. Duration** |  two years, three-week, 5-1/2-year, multiyear, another six months
5\. Economic Item |  economy, economic growth, market, economic outlook, employment data, currency, commodity-oil
6\. Financial attribute |  supply, demand, output, production, price, import, export
7\. Forecast target |  forecast, target, estimate, projection, bets
8\. Group |  global producers, oil producers, hedge funds, non-OECD, Gulf oil producers
9\. Location** |  global, world, domestic, Middle East, Europe
10\. Money** |  $60, USD 50
11\. Nationality** |  Chinese, Russian, European, African
12\. Number** |  (any numerical value that does not have a currency sign)
13\. Organization** |  OPEC, Organization of Petroleum Exporting Countries, European Union, U.S. Energy Information Administration, EIA
14\. Other activities |  (free text)
15\. Percent** |  25%, 1.4 percent
16\. Person** |  Trump, Putin (and other political figures)
17\. Phenomenon |  (free text)
18\. Price unit |  $100-a-barrel, $40 per barrel, USD58 per barrel
19\. Production Unit |  170,000 bpd, 400,000 barrels per day, 29 million barrels per day
20\. Quantity |  1.3500 million barrels, 1.8 million gallons, 18 million tonnes
21\. State or province** |  Washington, Moscow, Cushing, North America

Entity type marked with ** in Table 6 are made up of mostly Named Entities.
These named entities are identitical to the NER-tagging in Stanford CoreNLP.

##  Appendix C Event Types

The list of annotated events are aligned to Ravenpack’s event taxanomy
categeories which is made publicly available. There are 18 event types
identified and annotated, the event types are listed in Table 7 below:

Table 7: List of Event Types Event Type |  Example Trigger Word(s)
---|---
1\. Cause-movement-down-loss |  cut, trim, reduce, disrupt, curb, squeeze, choked off
2\. Cause-movement-up-gain |  boost, revive, ramp up, prop up, raise
3\. Civil-unrest |  violence, turmoil, fighting, civil war, insurgent attacks, conflicts
4\. Crisis |  crisis, crises
5\. Embargo |  embargo, sanction
6\. Geopolitical-tension |  war, clashes, tensions, deteriorating relationship
7\. Grow-strong |  grow, picking up, boom, recover, expand, strong, rosy, improve, solid
8\. Movement-down-loss |  fell, down, less, drop, tumble, collapse, plunge, downturn, slump, slide, decline
9\. Movement-flat |  unchanged, flat, hold, no change, maintained
10\. Movement-up-gain |  up, gain, rise, surge, soar, swell, increase, rebound
11\. Negative-sentiment |  worries, concern
12\. Oversupply |  ample supply, glut, oversupply, bulging stock level, excess supplies
13\. Position-high |  high, highest, peak, highs
14\. Position-low |  low, lowest, lows, trough
15\. Prohibiting |  ban, bar, prohibit
16\. Shortage |  shortfall, shortage, under-supplied
17\. Slow-weak |  slow, weak, tight, lackluster, falter, weaken, bearish, slowdown, crumbles
18\. Trade-tensions |  price war, trade war, trade tensions, economic fallout, trade dispute

As described in Section 1, events are not fully represented if we just carry
out Event Detection focusing on only event triggers. The events listed in
Table 6 can be grouped into the following categories by coupling event
arguments with event triggers:

###  C.1 Geo-political News

  1. 1.

Civil Unrest (Civil-unrest) - Violence or turmoil within the oil producing
country

  2. 2.

…..a fragile recovery in Libyan supply outweighed fighting in Iraq ……

  3. 3.

…….a backdrop of the worst strife in Iran this decade….

  4. 4.

Embargo (Embargo, Prohibiting) - Trade or other commercial activity of the
commodity is banned.

  5. 5.

….. and sanctions against Iran.

  6. 6.

…..prepared to impose “ strong and swift ” economic sanctions on Venezuela…..

  7. 7.

Geo-political Tension (Geo-political-tension) - Political tension between oil-
producing nation with other nations.

  8. 8.

….. heightened tensions between the West and Russia…..

  9. 9.

….. despite geopolitical war in Iraq , Libya and Ukraine.

  10. 10.

Trade Tension (Trade-tensions) - Trade-related tension between oil-producing
and oil-consuming nations.

  11. 11.

….. escalating global trade wars, especially between the US and China.

  12. 12.

…. showing that OPEC is not ready to end its trade tensions……

  13. 13.

Other forms of Crisis (Crisis)

    1. (a)

A time of intense difficulty, such as other forms of unspecified crisis that
do not fall into any of the above category

    2. (b)

…. Ukraine declared an end to an oil crisis that has ………

    3. (c)

Financial / Economic Crisis (which can be grouped under Macro-economic News)

    4. (d)

….since the 2014/15 financial crisis as …….

###  C.2 Macro-economic News

  1. 1.

Strong Economy / GDP growth / US Employment (Grow-strong) - Strong or growing
economy / GDP of a nation; applicable to indicate strong status of US
Employment Data 888US Employment data is used as an indicator of the world
largest’s economic health.

  2. 2.

as strong U.S. employment data…..

  3. 3.

Weak or Contracting Economy / GDP / US Employment (Slow-weak) - Weakening or
contracting economy / GDP of a nation; applicable to indicate the weakening of
US employment data.

  4. 4.

……. concerns over slowing global growth…..

  5. 5.

U.S. employment data contracts with the euro zone…..

  6. 6.

Bearish technical view or outlook (Negative-sentiment) - Bearish sentiment or
outlook

  7. 7.

But in a market clouded by uncertainties…..

  8. 8.

….supply concerns would ease even more ……

###  C.3 Commodity Supplies including exports

  1. 1.

Oversupply (Oversupply) - Situation where production goes into surplus.

  2. 2.

….. the region surplus of supply…..

  3. 3.

….the market is still working off the gluts built up…..

  4. 4.

Shortage (Shortage) - Situation where demand is more than supply.

  5. 5.

…. increase a supply shortage from chaotic Libya….

  6. 6.

……and there is no shortfall in supply , the minister added.

  7. 7.

Supply increase (Movement-up-gain) - Situation where supply increased.

  8. 8.

….further increases in U.S. crude production…..

  9. 9.

The rise in production is definitely benefiting the United States….

  10. 10.

Action taken to increase supply (Cause-Movement-Up-gain) - Deliberate action
to increase supply.

  11. 11.

The IEA boosted its estimate of production from ExxonMobil to 1.8 million bpd
in July 4 holiday weekend.

  12. 12.

…..urged the kingdom to ramp up production…..

  13. 13.

Supply decrease (Movement-down-loss)- Situation where supply decreased. UAE ’s
production has almost halved in two years to 31.6 million bpd… …fears that
global supplies will drop due to Washington ’s sanctions on the OPEC member
nation.

  14. 14.

Action taken to decrease supply (ause-movement-down-loss) - Deliberate action
to decrease supply. ……by slashing production by almost three quarters in the
1980s…. …..an announcement by Iran that it would cut its production last week.

###  C.4 Commodity Demand including imports

  1. 1.

Demand increase (Movement-up-gain) - Situation where demand increased. It
expects consumption to trend upward by 1.05 million bpd , below 40,000 bpd
from July . ….as more seasonal demand kicks in due to colder weather.

  2. 2.

Action taken to increase demand (Caused-movement-up-gain) - Deliberate action
taken to increase demand. IEA tried to boost global oil demand by
introducing……

  3. 3.

Demand decrease (Movement-down-loss) - Situation where demand decreased.
….onto a market reeling from falling demand because of the virus outbreak.
….when global demand growth for air conditioning collapses from its summer
peak….

  4. 4.

Action taken to decrease demand (Caused-movement-down-loss)- Deliberate action
taken to decrease demand. The pandemic has zapped demand to a level never seen
before…

###  C.5 Commodity Price Movement

Commodity price here includes spot price, futures and futures contract

  1. 1.

Price increase (Movement-up-gain)- Situation where commodity price rises. Oil
price rose $105 a barrel on March…. …oil prices have jumped as much as 20
percent since June.

  2. 2.

Price decrease (Movement-down-loss) - Situation where commodity price drops.
The drop in oil prices to their lowest in two years….. Oil prices declined
back the final quarter of 1991 to 87 cents….

  3. 3.

Price movement flat (Movement-flat) - Situation where no or little change to
commodity price. Contango spread in Brent is steady at 15 cents per barrel…..
U.S. crude is expected to hold around $105 per barrel.

  4. 4.

Price position (Position-high, Position-low) - Describes the position of the
current commodity price. Oil price remained close to four-year highs…. Oil
slipped more than 20% to its lowest level in two years on 1980s….

###  C.6 Change in Forecasted value

  1. 1.

Increase forecast target (Caused-movement-up-gain) - Forecasted / target is
raised, possible values are price target, growth target, demand and supply
target. The IMF earlier said it increased its 2019 global economic growth
forecast to 3.30%. The International Monetary Fund doubled its global growth
forecast for 2013…..

  2. 2.

Price target /forecast decrease (Caused-movement-down-loss) - Forecasted /
target is lowered, possible values are commodity price target, growth target,
demand and supply target. Germany’s Bundesbank this week halved its 2015
growth forecasts for Europe ’s largest economy to 1 percent. OPEC also lowered
forecast global demand for its crude oil….

##  Appendix D Event Arguments

###  D.1 Movement-down-loss, Movement-up-gain, Movement-flat

Example sentence:  [Globally] [crude oil] [futures] surged [$2.50] to [$59 per
barrel] on [Tuesday].

Role |  Entity Type |  Argument Text
---|---|---
Type |  Nationality, Location |  globally
Place |  Country, Group, Organization, Location, State or province, Nationality |
Supplier_consumer |  Organization, Country, State_or_province, Group, Location |
Reference_point_time |  Date |  Tuesday
Initial_reference_point |  Date |
Final_value |  Percentage, Number, Money, Price_unit, Production_unit, Quantity |  $59 per barrel
Initial_value |  Percentage, Number, Money, Price_unit, Production_unit, Quantity |
Item |  Commodity, Economic_item |  crude oil
Attribute |  Financial_attribute |  futures
Difference |  Percentage, Number, Money, Production_unit, Quantity |  $2.50
Forecast |  Forecast_target |
Duration |  Duration |
Forecaster |  Organization |

###  D.2 Embargo

Example sentence: The [Trump administration] imposed a “strong and swift”
economic sanctions on [Venezuela] on [Thursday].

Role |  Entity Type |  Argument Text
---|---|---
Imposer |  Organization, Country, Nationality, State or province, Person, Group, Location |  Trump administration
Imposee999Not formally a word, but used here as a shorter version of “Party whom the action was imposed on |  Organization, Country, Nationality, State or province, Group |  Venezuela
Reference_point_time |  Date |  Thursday

###  D.3 Prohibiting

Example sentence: [Congress] banned most [U.S.] [crude oil] [exports] on
[Friday] after price shocks from the 1973 Arab oil embargo.

Role |  Entity Type |  Argument Text
---|---|---
Imposer |  Organization, Country, Nationality, State or province, Person, Group, Location |  Congress
Imposee |  Organization, Country, Nationality, State or province, Group |  U.S.
Item |  Commodity, Economic_item |  crude oil
Attribute |  Financial_attribute |  exports
Reference_point_time |  Date |  Friday
Activity |  Other_activities |

###  D.4 Caused-movement-down-loss, Caused-movement-up-gain

Example sentence: The [IMF] earlier said it reduced its [2018] [global]
[economic growth] [forecast] to [3.30%] from a [July] forecast of [4.10%].

Role |  Entity Type |  Argument Text
---|---|---
Type |  Nationality, Location |  global
Place |  Country, Group, Organization, Location, State or province, Nationality |  West African, European
Supplier_consumer |  Organization, Country, State_or_province, Group, Location |
Reference_point_time |  Date |  2018
Initial_reference_point |  Date |  July
Final_value |  Percentage, Number, Money, Price_unit, Production_unit, Quantity |  3.30%
Initial_value |  Percentage, Number, Money, Price_unit, Production_unit, Quantity |  4.10%
Item |  Commodity, Economic_item |  economic growth
Attribute |  Financial_attribute |
Difference |  Percentage, Number, Money, Production_unit, Quantity |
Forecast |  Forecast_target |  forecast
Duration |  Duration |
Forecaster |  Organization |  IMF

###  D.5 Shortage

Example Sentence: Oil reserves are within “acceptable” range in most oil
consuming countries and there is no shortage in [oil] [supply] [globally], the
minister added.

Role |  Entity Type |  Argument Text
---|---|---
Place |  Country, State or province, Location, Nationality |  Congress
Item |  Commodity |  crude oil
Attribute |  Financial_attribute |  exports
Type |  Location |  globally
Reference_point_time |  Date |

###  D.6 Oversupply

Example sentence: [Forecasts] for an [crude] oversupply in [West African] and
[European] [markets] [early June] help to push the Brent benchmark down more
than 20% January.

Role |  Entity Type |  Argument Text
---|---|---
Place |  Country, Group, Organization, Location, State or province, Nationality |  West African, European
Reference_point_time |  Date |  this year
Item |  Commodity |  crude
Attribute |  Financial_attribute |  markets
Difference |  Production_unit |
Forecast |  Forecast_target |  forecasts

###  D.7 Position-high, Position-low

Example sentence: The IEA estimates that U.S. crude oil is expected to seek
higher ground until reaching a [5-year] peak in [late April] of about [17
million bpd].

Role |  Entity Type |  Argument Text
---|---|---
Reference_point_time |  Date |  late April
Initial_reference_point |  Date |
Final_value |  Percentage, Number, Money, Price_unit, Production_unit, Quantity |  17 million bpd
Initial_value |  Percentage, Number, Money, Price_unit, Production_unit, Quantity |
Item |  Commodity, Economic_item |
Attribute |  Financial_attribute |
Difference |  Percentage, Number, Money, Production_unit, Quantity |
Duration |  Duration |  5-year

###  D.8 Geo-political Tension

Example sentence:  Deteriorating relations between [Iraq] and [Russia] [first
half of 2016] ignited new fears of supply restrictions in the market. Trade
Tension

Role |  Entity Type |  Argument Text
---|---|---
Participating_countries |  Country, Group, Organization, Location, State or province, Nationality |  U.S., China
Reference_point_time |  Date |  early June

###  D.9 Slow-weak, Grow-strong

Example sentence: [U.S.] [employment data] strengthens with the euro zone.

Role |  Entity Type |  Argument Text
---|---|---
Type |  Nationality, Location |
Place |  Country, Group, Organization, Location, State or province, Nationality |  U.S.
Supplier_consumer |  Organization, Country, State_or_province, Group, Location |
Reference_point_time |  Date |
Initial_reference_point |  Date |
Final_value |  Percentage, Number, Money, Price_unit, Production_unit, Quantity |
Initial_value |  Percentage, Number, Money, Price_unit, Production_unit, Quantity |
Item |  Commodity, Economic_item |  employment data
Attribute |  Financial_attribute |
Difference |  Percentage, Number, Money, Production_unit, Quantity |
Forecast |  Forecast_target |
Duration |  Duration |
Forecaster |  Organization |

###  D.10 Civil Unrest

Example sentence: The drop in oil prices to their lowest in two years has
caught many observers off guard, coming against a backdrop of the worst
violence in [Iraq] [this decade].

Role |  Entity Type |  Argument Text
---|---|---
Place |  Country, State or province, Location, Nationality |  Iraq
Reference_point_time |  Date |  this decade

###  D.11 Crisis

Example Sentence: Asia ’s diesel consumption is expected to recover this year
at the second weakest level rate since the [2014] [Asian] [financial] crisis.

Role |  Entity Type |  Argument Text
---|---|---
Place |  Country, State or province, Location, Nationality |  Asian
Reference_point_time |  Date |  this year
Item |  Commodity, Economic_item |  financial

###  D.12 Negative Sentiment

Example sentence: Oil futures have dropped due to concern about softening
demand growth and awash in crude.

Note: Negative Sentiment is a special type of event, where majority of the
time it contains just the trigger words such as concerns, worries, fears and 0
event arguments.

## References

  * Aguilar et al (2014) Aguilar J, Beller C, McNamee P, Van Durme B, Strassel S, Song Z, Ellis J (2014) A comparison of the events and relations across ace, ere, tac-kbp, and framenet annotation standards. In: Proceedings of the Second Workshop on EVENTS: Definition, Detection, Coreference, and Representation, pp 45–53
  * Araki et al (2018) Araki J, Mulaffer L, Pandian A, Yamakawa Y, Oflazer K, Mitamura T (2018) Interoperable annotation of events and event relations across domains. In: Proceedings 14th Joint ACL - ISO Workshop on Interoperable Semantic Annotation, Association for Computational Linguistics, Santa Fe, New Mexico, USA, pp 10–20
  * Brandt and Gao (2019) Brandt MW, Gao L (2019) Macro fundamentals or geopolitical events? a textual analysis of news events for crude oil. Journal of Empirical Finance 51:64–94
  * Dor et al (2019) Dor LE, Gera A, Toledo-Ronen O, Halfon A, Sznajder B, Dankin L, Bilu Y, Katz Y, Slonim N (2019) Financial event extraction using wikipedia-based weak supervision. In: Proceedings of the Second Workshop on Economics and Natural Language Processing, pp 10–15
  * Hogenboom et al (2013) Hogenboom A, Hogenboom F, Frasincar F, Schouten K, Van Der Meer O (2013) Semantics-based information extraction for detecting economic events. Multimedia Tools and Applications 64(1):27–52
  * Jacobs et al (2018) Jacobs G, Lefever E, Hoste V (2018) Economic event detection in company-specific news text. In: The 56th Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics, pp 1–10
  * Malik et al (2011) Malik HH, Bhardwaj VS, Fiorletta H (2011) Accurate information extraction for quantitative financial events. In: Proceedings of the 20th ACM international conference on information and knowledge management, pp 2497–2500
  * Mitamura et al (2015) Mitamura T, Yamakawa Y, Holm S, Song Z, Bies A, Kulick S, Strassel S (2015) Event nugget annotation: Processes and issues. In: Proceedings of the The 3rd Workshop on EVENTS: Definition, Detection, Coreference, and Representation, pp 66–76
  * O’Gorman et al (2016) O’Gorman T, Wright-Bettner K, Palmer M (2016) Richer event description: Integrating event coreference with temporal, causal and bridging annotation. In: Proceedings of the 2nd Workshop on Computing News Storylines (CNS 2016), Association for Computational Linguistics, Austin, Texas, pp 47–56
  * O’Gorman et al (2016) O’Gorman T, Wright-Bettner K, Palmer M (2016) Richer event description: Integrating event coreference with temporal, causal and bridging annotation. In: Proceedings of the 2nd Workshop on Computing News Storylines (CNS 2016), pp 47–56
  * Schank and Abelson (2013) Schank RC, Abelson RP (2013) Scripts, plans, goals, and understanding: An inquiry into human knowledge structures. Psychology Press
  * Stenetorp et al (2012) Stenetorp P, Pyysalo S, Topić G, Ohta T, Ananiadou S, Tsujii J (2012) Brat: a web-based tool for nlp-assisted text annotation. In: Proceedings of the Demonstrations at the 13th Conference of the European Chapter of the Association for Computational Linguistics, pp 102–107
  * Thompson et al (2017) Thompson P, Nawaz R, McNaught J, Ananiadou S (2017) Enriching news events with meta-knowledge information. Language Resources and Evaluation 51(2):409–438
  * Yang et al (2018) Yang H, Chen Y, Liu K, Xiao Y, Zhao J (2018) DCFEE: A document-level Chinese financial event extraction system based on automatically labeled training data. In: Proceedings of ACL 2018, System Demonstrations, Association for Computational Linguistics, Melbourne, Australia, pp 50–55