# AffectiveNet: Affective-Motion Feature Learning for Micro Expression
Recognition

Monu Verma, Santosh Kumar Vipparthi, and Girdhari Singh Monu Verma, Santosh
Kumar Vipparthi and Girdhari Singh are with Vision Intelligence Lab at
Department of Computer Science and Engineering, Malaviya National Institute of
Technology, Jaipur, India (Email<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>

###### Abstract

Micro-expressions are hard to spot due to fleeting and involuntary moments of
facial muscles. Interpretation of micro emotions from video clips is a
challenging task. In this paper we propose an affective-motion imaging that
cumulates rapid and short-lived variational information of micro expressions
into a single response. Moreover, we have proposed an AffectiveNet: affective-
motion feature learning network that can perceive subtle changes and learns
the most discriminative dynamic features to describe the emotion classes. The
AffectiveNet holds two blocks: MICRoFeat and MFL block. MICRoFeat block
conserves the scale-invariant features, which allows network to capture both
coarse and tiny edge variations. While MFL block learns micro-level dynamic
variations from two different intermediate convolutional layers. Effectiveness
of the proposed network is tested over four datasets by using two experimental
setups: person independent (PI) and cross dataset (CD) validation. The
experimental results of the proposed network outperforms the state-of-the-art
approaches with significant margin for MER approaches.

###### Index Terms:

Affective-Motion Imaging, multi scale features, AffectiveNet, micro expression
recognition.

##  1 Introduction

Micro expressions (ME) have rich source of information to reveal the true
emotions of a person. MEs originate in high stake situations, when a person is
trying to repress his/her genuine feelings within manifested expressions
(macro expressions). Usually, macro expression active on a face for 4 to 5
seconds that can be perceived easily. whereas, MEs are active for 1/30 to 1/25
seconds. Since MEs are short-lived and fleeting in nature, it is hard to
differentiate them through naked eyes. Earlier, trained persons were able to
spot micro-expressions but achieve less than 50$\%$ accuracy. Analysis of real
emotions from video clips has a wide range of applications such as: depression
analysis, police interrogation, law enforcement, multi-media entertainment,
clinical diagnosis etc.

In literature many feature extraction methods (handcrafted / traditional
feature descriptors) were proposed as spatiotemporal LBP with integral
projection (STLBP-IP) [1] and FDM [2] to encode spatial and temporal changes
from the ME video sequences. However, handcrafted feature descriptors were
focused only on superficial features and failed to capture sufficient features
of MEs.

Nowadays, with the advent of technology, deep learning models [3, 4, 5] have
gained popularity in solving various computer vision tasks like image
classification, semantic segmentation, face authorization, biometric, and many
more. Recently, some deep models [6, 7, 8, 9] are proposed to deal with micro-
expression recognition. However, most of the existing methods use combination
of CNN and RNN or CNN and LSTM to extract the spatial and temporal features,
respectively. These methods first capture the spatial features from the frames
and then these are fed to RNN or LSTM to fetch the temporal features. Thereby
these approaches failed to establish relationship between spatial and temporal
features occurring simultaneously in frames and lead to degrading the
performance.

Inspired from the literature [4, 5], a novel affective-motion feature learning
method is proposed to learn and classify the features of micro expressions.
The AffectiveNet has ability to capture spatial and temporal features
simultaneously from the affective-motion images. The contributions of the
proposed approach are summarized as follows:

Figure 1: Visualization of (a) input video (V) with k frames, (b) Motion
Images (MI) generated by multiplying input frames with coefficients Frame
weights (Fw) and (c) Affective-Motion Images (AMI) representing both
appearance and motion that occurred between the frames in a 2d image.

  1. 1.

We propose an affective-motion imaging that summarizes the spatial structure
features with temporal variations into one image instance.

  2. 2.

We propose an AffectiveNet for micro-expression recognition by introducing two
blocks: MICRoFeat and MFL blocks. MICRoFeat block has been proposed to enhance
learning capability of the network by capturing multi-scale features.MFL block
has been designed to increase the discriminability of the network as it is
able to learn micro-level features.

  3. 3.

The effectiveness of the proposed AffcetiveNet is examined by adopting two
validation schemes: person independent and cross dataset over for benchmark
datasets and compared with state-of-the-art MER approaches.

##  2 Literature Review

Feature extraction is an essential part of MER task. Wang et al. [12]
introduced a tensor independent color space model (TICS) by representing the
image sequences in 4D structure such as: 2D structure represents the spatial
texture patterns, 3rd dimension for momentary variation features and 4th
dimension describes RGB color components to spot the micro-expressions.
Furthermore, they extended thier work and proposed sparse tensor canonical
correlation method [13] to analyse the micro expressions movements. Happy et
al. [14] proposed a fuzzy histogram-based optical flow orientation technique
(FHOFO) to capture temporal features of the micro-expressions. Wang et al.
[15] introduced the main directional maximal difference (MDMD) to capture the
facial expressive movements by extracting the maximal magnitude difference in
between the optical flow directions.

Recently, the adoption of deep learning networks of VGG Net [3], ResNet [4]
and MobileNet [5] have created a tremendous take-off in the field of computer
vision. The literature on MER shows that, convolutional neural networks (CNN)
based models also achieve impressive results up to some extent.

Furthermore, an evolutionary search is being applied to detect the disparities
between the frames of micro expressions. Wang et al. [6] introduced a micro
attention module in resnet [4] that mainly focused on expressive regions which
included most of the action units. To capture the action units they utilized
the transfer learning from macro to micro expressions.khor et al. [16] adopted
CNN network and long short-term memory (LSTM) to learn the Spatio-temporal
information for each image frame. Wang et al. [7] utlized the CNN model for
visual feature extraction and LSTM for sequence learning between the frames to
spot the micro expressions. Moreover, Li et al. [10] introduced a 3d flow CNN
network, which incorporated optical flow information with CNN network to
learned deep features of minute variation responsible to spot micro expression
class. xia et al. [8] proposed a recurrent convoloution neural network to
capture the features of subtle changes occurred between image sequences.Liong
et al. [17] also utilized optical flow to represents flow variations between
frames and feed them to three parallelly connected CNN layer streams that
learn the salient features of micro-expressions and classify them accordingly.
Xia et al. [9]proposed a extended recurrent convolution network to extract the
spatial-temporal deformations of micro-expression sequence by considering
appearance and geometrical information, respectively .

Figure 2: The detailed architecture of the proposed Affective Network for
micro expression recognition. Figure 3: The feature maps of happy image
produced at each convolutional layer of MICRoFeat block.

##  3 Proposed Method

Micro-expressions appear only in the few frames of a video due to fleeting and
short-lived nature. Therefore, interpretation of the content in a video and
spotting micro-expressions between the frames is a challenging task. In
literature the state-of-art MER systems apply complex algorithms to represent
the adequate video content . Moreover, all benchmark datasets hold variant
size video sequences, thus most of the state-of-the-art approaches utilized
the time interpolation to normalize the dataset. It may lose or alter the
domain knowledge of micro-expressions by shearing or filling holes in between
the frames. To address these issues, in this paper we have proposed affective-
motion imaging (AMI). Affective-motion image represents video content into a
single instance by preserving high stake active dynamics of micro expressions.
Hence, we have used an AffectiveNet learn the dynamics of micro-expressions
and interprets the relevant emotion class.

###  3.1 Affective-Motion Imaging

Inspired from the literature [11] in this paper we introduced affective-motion
imaging (AMI). AMI interprets the content of the video by focusing on the
facial moving regions and compress that into a single instance. Therefore,
affective-motion image implies movements in a still image by summarizing
spatial and temporal dynamics of the whole video frames. To construct a single
image instance from video sequences, we estimate the motion between the frames
and allocated ranks to video frames by using a ranking function. Let LR is a
ranking function, which updates the Rank of frames by using Eq. 1-2.

$LR[1,i]=\frac{(2\times I[1,i])-k}{I[1,i]}$ (1) $I[1,i]=[i,i+1,i+2,...k]$ (2)

where $I$ represents as index matrix with $i\in{1,2,..k}$ and $k$ implies the
total number of frames extracted from the video . Furthermore, frame weight
$Fw(i)$ is assigned to each frame by using Eq. 3.

$Fw(i)=\sum_{j-1}^{k-i}{LR[1,j]}$ (3)

Moreover, motion images are computed by utilizing Eq. 4.

$MI_{i}=\nu_{i}\times Fw(i)$ (4)

where $\nu_{i}\in\nu$ represents the $i^{th}$ frame of the video $\nu$.
Specifically, frame weights analyze the motions between the frames in a video
and quantify it with the help of ranking function. Further, frames are
amplified by multiplying frame weight coefficient named as motion images.
Motion images magnify the temporal changes and abbreviated uniform
information, as shown in Fig. 1. Finally, Affective-motion image is computed
by merging all motion images such as.

$AMI=\sum_{i}^{k}{MI_{i}}$ (5)

Samples of affective motion images are demonstrated in Fig.1. From Fig. 1, it
is clearly visible that the affective motion images successfully preserve the
influencing dynamics of micro-expressions within single frame. Moreover,
affective motion images abbreviated uniform information and help to protrude
nonuniform variations, highlighted in red blocks those play decision making
role in MER. Further to learn effective features of micro-expressions
affective motion images are forwarded to the AffectiveNet.

Figure 4: The feature maps generated from two emotion classes a) Anger and b) Happy, at 1st level convolutional layers of Affective Network. The region of interest (red block) shows that AffectiveNet is able to differentiate between two expression classes (inter-class). TABLE I: Recognition Accuracy Comparison on CASME-I and CSME-II Datasets. *This result is from the corresponding original paper and H, S, D, R, T, P, N, O, Sa, F stands for Happy, Surprise, Disgust, Repression, Tense, Positive, Negative, Others, Sad, Fear. AffectiveNet-2 represents results evaluated by following experimental setup used in STRCNN [9]. Method | Task | CASME-I | CASME-II
---|---|---|---
STLBP-IP*[1] | $\left(H,S,D,R,O\right)$ | N$/$A | 59.91
FDM* [2] | $\left(D,R,S,T\right)$ | 56.14 | 45.93
3D-Flow* [10] | $\left(H,S,D,R,T\right)$ | 55.44 | 59.11
TICS* [12] | $\left(P,N,S,O\right)$ | 61.86 | 61.11
FHOFO* [14] | $\left(P,N,S,O\right)$ | 65.99 | 55.86
CNN-LSTM* [16] | $\left(H,S,D,R,O\right)$ | 60.98 | N$/$A
MicroAtt* [6] | $\left(A,D,F,H,Sa,S,O\right)$ | N/A | 65.90
Sp-RCNN* [8] | $\left(P,N,S,O\right)$ | 63.20 | 65.80
STRCNN* [9] | $\left(P,N,S,O\right)$ | N$/$A | 56.00
ResNet-50 [4] | $\left(P,N,S,O\right)$ | 25.04 | 32.12
MobileNet [5] | $\left(P,N,S,O\right)$ | 33.77 | 30.25
Af-Net-KS-1 | $\left(P,N,S,O\right)$ | 56.48 | 45.64
Af-Net-KS-2 | $\left(P,N,S,O\right)$ | 60.26 | 49.58
Af-Net-LFC | $\left(P,N,S,O\right)$ | 56.51 | 53.62
Af-Net-WoMFL | $\left(P,N,S,O\right)$ | 57.94 | 60.17
Af-Net-$3\times 3$ | $\left(P,N,S,O\right)$ | 59.32 | 54.12
Af-Net-$1\times 1$ | $\left(P,N,S,O\right)$ | 56.53 | 43.88
AffectiveNet-1 | $\left(P,N,S,O\right)$ | 66.99 | 61.58
AffectiveNet-2 | $\left(P,N,S,O\right)$ | 72.64 | 68.74

###  3.2 Affective Network

In this paper we have proposed a portable CNN model affective motion feature
learning (AffectiveNet) that learns the salient features of micro-expression
by capturing momentary changes from the affective motion images. AffectiveNet
mainly comprises of two blocks: multi-receptive feature preservative
(MICRoFeat) block and microfeature learning (MFL) block as shown in Fig. 2.

####  3.2.1 MICRoFeat Block

Micro-level variations can be captured through affective-motion images, where
the expressive regions may spread from small region to extensive regions. The
micro-level expression variations are clearly depicted in Fig. 1. Although
these changes are imperceptible but have a high impact in identifying the
micro-expressions. Therefore, a robust CNN network that can elicit both coarse
and detailed texture features are needed to acquire sufficient knowledge for
adequate emotion classification. In literature, it has been confirmed that
inferior variations like eyebrow lift, check crinkles, forehead wrinkles,
glabella, chin, eyelid and lip lines can be captured through small sized
convolutional filters, while the abstract changes like eyes, nose, mouth and
lip shapes tend to respond with large sized filter. However, most of the CNN
based models like VGG Net [3], ResNet [4] and MobileNet [5] hold uniform-sized
filters. Thus, these networks degrade the performance of micro-expression
recognition as they fail to acquire enough feature variations from affective-
motion images. Therefore, in this paper, we have introduced MICRoFeat block to
extract the detailed expressive features from the affective-motion images. The
MICRoFeat block has ability to capture detailed features from small regions to
extensive regions, by applying four convolutional (Conv) layers with multi-
scale filters as $3\times 3$, $5\times 5$, $7\times 7$ and $11\times 11$. Let
$I(u,v)$ be an input image and $\varepsilon_{S}^{x,N}\\{\cdot\\}$ represents
conv function, where, $S$ implies for stride, $N$ is depth, $x$ stands for the
size of filter. Then, output of Conv layers with multi-scale filters are
computed by Eq. (6-7).

$Fm_{i}^{*}=\varepsilon_{1}^{p(i),16}\\{I(u,v)\\}$ (6)

where, $i={1,2,3,4}$, represents the each multi-scale conv layers and

$p(i)=[3,5,7,11]$ (7)

Further, feature maps $Fm_{i}^{*}$ of each layer are forwarded to the next
aligned encapsulated feature (EncapFeat) blocks. EncapFeat block imposes two
Conv layers with different scales as $3\times 3$ and $5\times 5$ to express
the edge variations of each muscle movement (those provokes facial
expressions) by extracting coarse to fine edge variations. Furthermore,
resultant feature maps are refined by employing $3\times 3$ Conv layer.
Resultant feature maps $Fm_{i}^{1}$ of the EncapFeat are computed by using
Eq.(8-10).

$Fm_{i}^{1}=EncapFeat\\{Fm_{i}^{*}\\}$ (8)
$EncapFeat\\{Fm_{i}^{*}\\}=\varepsilon_{2}^{3,64}\\{f_{1}\\{Fm_{i}^{*}\\}+f_{2}\\{Fm_{i}^{*}\\}\\}$
(9) $f_{k}\\{Fm_{i}^{*}\\}=\varepsilon_{2}^{2k+1,32}\\{Fm_{i}^{*}\\}$ (10)

Moreover, response of each EncapFeat block is coupled and forwarded to next
down-sampled Conv layers such as.

$MICRoFeat=\\{Fm_{1}^{1}\|Fm_{2}^{1}\|Fm_{3}^{1}\|Fm_{4}^{1}\\}$ (11)

where, $\|$, represents the concat operation. The effectiveness of the
MICRoFeat block is depicted in Fig. 3, where red highlighted boxes represent
the expressive regions. From the Fig. 3, it is clear that small $\left(3\times
3,5\times 5\right)$ and large $\left(7\times 7,11\times 11\right)$ sized
filters are able to extract minute and high-level variations, respectively.

####  3.2.2 MFL block

Inspired from the residual concept [4], we introduced a MFL module. The main
aim of this module is to refine the micro-expression regions in two parallel
stages as shown in the Fig.2. In stage 1, low-level features of widespread
expressive regions are learned through one FC layer. These low-level features
increase the learning capability of the network. Similarly, in stage 2, micro
variation in the high-level feature are forwarded parallelly to FC network.
Further, resultant features of laterally connected FC layers are fused to
capture micro-level variations in an expressive region. The lower layer
features are effective for identifying variations in small regions. Thus,
fusion of these features improve the discriminable capability between the
inter and intra-class variations. Moreover, MFL block increases the learning
capability with minimum number of parameters for AffectiveNet as compared to
existing state-of-the-art approaches. Let FC represents the fully connected
layer and concat implies for the depth concatenation function. Then, output
feature vector $Fv$ is computed by Eq. (12-16).

$Fv=FC^{4}\left(\beta\\{Fv^{1}\|Fv^{2}\\}\right)$ (12)
$Fv^{1}=FC^{32}\left(Fm^{2}\right)$ (13) $Fv^{2}=FC^{32}\left(Fm^{3}\right)$
(14) $Fm^{2}=\varepsilon_{2}^{3,184}\\{\beta\left(MICRoFeat\right)\\}$ (15)
$Fm^{3}=\varepsilon_{2}^{3,196}\\{\varepsilon_{2}^{3,128\\{Fm^{2}\\}}\\}$ (16)

Where, $\beta$ represents the batch normalization (BN) function. BN is
incorporated in proposed network to deal with the issue of divergence in
feature distribution that occurs due to disproportion of image sets: training
and testing data. Thereby, BN improves the strength of AffectiveNet by
normalizing the feature responses of the preceding contact layer by
subtracting batch mean and dividing by the standard deviation as follows:

$p_{k}=\omega\bar{q_{k}}+\phi\beta\left(\bar{q_{k}}\right)$ (17)
$\bar{q}=\frac{q_{k}-n_{B}}{\sqrt{S_{B}^{2}+\epsilon}}$ (18)

where,$q_{k}$ is the mini-batch size and $B=\\{q_{1},q_{2},...,q_{N}\\}$ are
the learnable parameters. $\omega$ and $S-{B}$ implies the mean and standard
deviation of the batch as calculated using Eq. (19-20).

$n_{B}=\frac{1}{N}\sum_{k=1}^{N}\left(q_{k}\right)$ (19)
$S_{B}=\frac{1}{N}\sum_{k=1}^{N}\left(q_{i}-n_{B}\right)^{2}$ (20)

The capability of the AffectiveNet to control intra-class (anger and happy)
variations is depicted in Fig. 4. Thus, we can conclude that, proposed model
is able to differentiate between two expression classes (inter-class
variation) within an active patch. Active patches are highlighted by red color
boxes.

TABLE II: Recognition Accuracy Comparision on CASME2, SAMM AND CI2CII, CI2C2, CI2S, CII2CI, CII2C2, CII2S, S2CI, S2CII, S2C2 for PIE and CDE Experiments, Respectively. Here, PIE, CDE and CSM are stands for Person Independent Experiment, Cross Data Experiments and CASME, respectively. Method | Exp. | PIE | CDE
---|---|---|---
Training | CASME2 | SAMM | CASME-I | CASME-II | SAMM
Testing | CSM-II | CSM2 | SAMM | CSM-I | CSM2 | SAMM | CSM-I | CSM-II | CSM2
MicroAtt[6] | N/A | 48.50 | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A
STRCNN[9] | N/A | 54.45 | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A
ResNet-50[4] | 46.58 | 36.50 | 10.67 | 12.50 | 10.69 | 10.81 | 23.54 | 8.81 | 20.15 | 16.60 | 29.06
MobileNet [5] | 35.05 | 40.20 | 11.86 | 4.00 | 14.46 | 23.78 | 6.40 | 18.24 | 14.05 | 12.64 | 8.00
AffectiveNet-1 | 52.86 | 47.46 | 46.25 | 13.08 | 26.42 | 46.49 | 23.55 | 32.08 | 26.00 | 25.90 | 48.26
AffectiveNet-2 | 61.20 | 58.12 | - | - | - | - | - | - | - | - | -

##  4 Experimental Results and Analysis

###  4.1 Database Prepossessing

To test the effectiveness of AffectiveNet, we have utilized four benchmark
datasets: CASME-I [18], CASME-II [18], CASME2 [19] and SAMM [20]. These
datasets are prepared to analyze the candid expressions under various
challenges like illumination variations, subjects with different artifacts,
ethnicity variations, age differences, gender inequalities etc.

####  4.1.1 CASME-I

The Chinese Academy of Sciences Micro-expression (CASME) [18] dataset
comprises of 19 participants’ spontaneous micro-expressions. The dataset
samples are labeled with eight emotion classes as: contempt, tense, disgust,
happiness, surprise, fear, sadness and repression with onset, peak and offset
frame tags. However, in CASME-I dataset, some emotions like fear, sadness and
contempt include very few samples and some of the emotion labels are
ambiguous. Thus, most of the existing approaches [1, 2] dropped these emotion
classes to balance the inequality issue in datasets. Recently, some methods
[12, 14] have created new emotion classes by merging the existing emotions as
positive, negative, surprise and other. In our experimental setup we have
utilized the merged emotion classes and finally gathered 187 affective motion
images as: positive: 9, negative: 50, surprise: 21 and others: 106.

Figure 5: Confusion matrices of AffectiveNet for 4-class expression
classification for a) CASME-I, b) CASME-II, c) SAMM d) CASME2 and for e)
CII2CI, f) CII2C2, g) CII2S, in PIE and CDE setups, respectively.

####  4.1.2 CASME-II

The CASME-II [18] elicits 26 participants’ micro expressions in a well-
arranged laboratory with normal lighting to avoid the problem of illumination
variation. Each frame is annotated with one of seven emotions as disgust,
fear, happiness, other, repression, sadness and surprise. Similar to CASME-I,
we have also converted CASME-II dataset into four categories and collected 251
affective-motion images as positive: 31, negative: 72, surprise: 25 and
others: 126.

####  4.1.3 CASME2

The CASME2 [19] includes 22 subjects’ (6 female and 16 male) expressions
captured at 30 fps with $640\times 480$ resolution. The dataset has been
annotated with three emotion classes: happy, anger and disgust based on AUs,
self-decision of participants and emotion-evoking videos. In our experimental
setup, we have selected a total 345 image sequences: anger-102, happy-155 and
disgust-88 of micro expressions.

####  4.1.4 SAMM

The SAMM [20] dataset include 159 micro-expressions of 29 subjects with the
largest ethnicity variations. The dataset is labeled with eight identified
categories of expressions$:$ other, disgust, happiness, contempt, fear,
sadness, surprise and anger. Similar to CASME-I and CASME-II, SAMM dataset
also holds unequal data samples in emotion classes, thus we can combine
emotion classes and compile 159 affective-motion images as positive: 26,
negative:75, surprise:15 and others:43.

###  4.2 Experimental Setup

To evaluate the performance of the proposed method, we have chosen two sets of
experiments: Person independent experiments (PIE) and Cross dataset
experiments (CDE).

####  4.2.1 Person independent experiments

In literature [2, 9, 17], mainly two types of evaluation techniques are used
to validate the efficiency of MER systems: leave one video out (LOVO) and
leave one subject out (LOSO) cross validation. In LOVO one expression video is
used as testing set and remaining all videos are used for training set.
Therefore, LOVO is evaluating the performance of MER in person dependent
manner. Thus, LOVO is prone to subject biasing and does not validate the
performance of system in effective manner. Thus, in this paper, all
experiments are computed by using LOSO strategy. In LOSO, only one subject’s
expressions are involved in testing set and remaining all subject’s
expressions are used for training. This ensures robustness to unseen faces for
expression recognition.

####  4.2.2 Cross dataset experiments

In this paper we have utilized the cross-dataset experiments (CDE) setup to
evaluate the robustness and learnability of the AffectiveNet in cross domain.
In CDE setup a dataset is used to train a model and other dataset are used as
test set. In CDE, a different set of experiments are performed as follows.
CI2CII: where CASME-I dataset is used as training set and CASME-II is testing
set. Similarly, CI2C2, CI2S, CII2CI, CII2C2, CII2S, S2CI, S2CII and S2C2
experiments are conducted for other dataset combinations. Moreover, the
Performance of proposed method is measured using recognition accuracy
calculated by using Eq. 21.

$Recog.Acc.=\frac{Total\,no.\,of\,correctly\,predicted\,samples}{Total\,no.\,of\,samples}\times
100$ (21)

Figure 6: Qualitative representation of feature maps generated by existing
networks: ResNet, MobileNet and proposed network over different micro
expressions of four datasets: a) CASME-I: b) CASME-II: c) CASME2 and d) SAMM.
The red blocks validate that AffectiveNet is able to capture the furrow lines
more accurately as compared to ResNet and MobileNet.

Furthermore, to examine the effectiveness of AffectiveNet, we have compared
the proposed method with existing MER approaches by following two schemes.

  1. 1.

We trained existing conventional networks: ResNet [4], MobleNet [5] by
utilizing pre-trained weights over our experimental setup that ensure a fair
comparison between state-of-the-art and proposed method. However, in case of
other MER approaches [12, 14] we quoted the published results directly we
follow the similar experimental setup.

  2. 2.

Since, some of the recent approaches [7, 8, 9] and [17] etc. are follow
contrast experimental setup in terms of total number of samples, participants,
expression classes etc. or dropped some of emotion classes due to a smaller
number of images. Therefore to validate the effectiveness of proposed
AffectiveNet, we have compared with the existing state-of-the-art approaches
by following the experimental setup added in the [9].

However, In our experiments, we have augmented the generated affective-motion
images and create a large pool of data to avoid the problem of over-fitting in
training. Moreover, to train the network we have used SGD optimizer and
SoftMax loss function with 10-3 learning rate.

###  4.3 Quantitative Analysis

This section provides a comparative analysis of the obtained accuracy rates
between the existing and proposed network for both PIE and CDE experiments.

####  4.3.1 Person independent experiments

Recognition accuracy results over CASME-I, CASME-II, CASME2 and SAMM datasets
for existing state-of-art and affective approaches for PIE setup are tabulated
in Table-I, respectively. Specifically, for CASME-I, Proposed Network secures
33.22%, and 41.95% more accuracy as compared to MobileNet and ResNet
respectively. Moreover, AffectiveNet also outperforms the existing handcrafted
MER: TICS and FHOFO approaches by 5.13% and 1.00%, respectively. For CASME-II,
our network achieves 31.33% and 29.46% more accuracy as compared to MobileNet
and ResNet respectively. Furthermore, proposed model yields 0.47% and 5.72%
better accuracy rates as compared to TICS and FHOFO respectively. For CASME2
dataset, our proposed method secures 17.81% and 6.28% improvement over
MobileNet and ResNet respectively. Similarly, for SAMM dataset, AffectiveNet
outperforms the existing approaches MobileNet and ResNet by 7.26% and 10.96%
accuracy rates, respectively.

####  4.3.2 Cross dataset experiments

Comparative analysis results of the conventional CNN network and proposed
network for CDE setup are tabulated in Table-II. From the Table-II, it is
clear that, proposed model outperforms CDE experiment results and validates
the strength of proposed network. Particularly, AffectiveNet gains 35.58%,
0.58%, 15.73% and 34.39%, 9.08%, 11.96% more accuracy for CI2CII, CI2C2, CI2S
setups as compared to ResNet and MobileNet, respectively. Moreover,
AffectiveNet yields 35.68%, 0.01%, 23.27% and 22.71%, 17.15%, 13.84% better
accuracy rates for CII2CI, CII2C2 and CII2S experiments compared to ResNet and
MobileNet respectively. Similarly, for S2CI, S2CII and S2C2 proposed model
attains 5.85%, 9.3%, 19.2% and 22.71%, 13.26%, 40.25% as compared to ResNet
and MobileNet, respectively.

To analyze class-wise recognition accuracy (true positives and false
positives), we have presented the confusion matrices for all PIE and CDE
experiments in Fig. 5.

###  4.4 Qualitative Analysis

The learning capability of proposed network is compared with the state-of-the-
art networks are shown in Fig. 6. Fig. 6. demonstrates the three most
effective visual representations of different emotion classes as CASME-I:
disgust, CASME-II: Happy, CASME2: disgust and SAMM: Others. From figure, it is
clear that the response feature maps significantly assist in preserving the
dynamic variations in different expressive regions of the facial image. For
example, in Disgust: eyes, eyebrows, mouth regions; in happy: eyebrows, lips,
mouth and in others: eyes, mouth; give maximum affective response for related
facial expressions. Therefore, we conclude that AffectiveNet preserves more
relevant feature responses to outperform the existing CNN based networks
ResNet-50 and MobileNet for almost all emotion classes.

###  4.5 Computational Complexity

This section provides a comparative analysis of the computational complexity
between the existing and proposed networks. The total number of parameters
involved in each network are tabulated in Table-III. The proposed AffectiveNet
has only 2.3 million learnable parameters which are very less as compared to
other existing benchmark models like: MobileNet: 3.2M, VGG-16: 138M, VGG-19:
144M and ResNet: 11.7M. Moreover, proposed network architecture has fewer
depth channels and hidden layers as compared to former methods. Furthermore,
AffectiveNet takes only 8.3 MB memory storage which is very less as compared
to MobileNet: 25.3 MB, VGG-16: 515, VGG-19: 535 and ResNet: 44 MB.

Figure 7: The neuron visualization of responses for disgust emotion captured at 1st multiscale CNN layers of ablation experiments a) Af-Net-KS-1 b) Af-Net-KS-2, c) Af-Net-LFC and proposed AffectiveNet. TABLE III: Computational Complexity Analysis of AffectiveNet and Exitsing Networks. Network |  |  $\\#$ Parameters
---
(in millions)
|  $\\#$ Memory
---
(in megabytes)
| Speed
---
(in seconds)
VGG-16[3] | 138 | 515 | 17.8
VGG-19 [3] | 144 | 535 | 21.2
ResNet -50 [4] | 26 | 44 | 19
MobileNet [5] | 4.2 | 25.3 | 12
Af-Net-KS-1 | 2.3 | 8.5 | 8.5
Af-Net-KS-2 | 2.5 | 9.4 | 8.6
Af-Net-LFC | 1.1 | 4.0 | 8.6
Af-Net-WoMFL | 1.0 | 3.4 | 4.5
Af-Net-$3\times 3$ | 2.1 | 7.8 | 5.4
Af-Net-$1\times 1$ | 2.1 | 8.1 | 5.1
AfffectiveNet-1 | 2.2 | 8.3 | 8.7

###  4.6 Ablation Study

In order to investigate the deep insights of AffectiveNet, we have conducted
six more supplementary experiments for ablation study as represented in
Table-I. This section mainly focuses on examining the effect of different
kernel sizes in EncapFeat block,linearly connected fully connected layer in
MFL block, effect of different filter sizes and MFL block. First, we have
examined the impact of two large kernel sizes like $\left(5\times 5,7\times
7\right)$ and $\left(7\times 7,11\times 11\right)$instead of $\left(3\times
3,5\times 5\right)$ in EncapFeat block named as Af-Net-KS-1. and Af-Net-KS-2,
respectively. Therefore, we have observed that smaller size kernels are more
preferable for micro-expression recognition. Kernels with large scale ignore
the minute transitional information which is quite important in micro
expression. From the Fig. 3 and 7 it is clear that kernel size $\left(11\times
11\right)$ skips the micro edge variations and preserved only high-level
edges. Second, we have analyzed the effect of linearly connected FC layers in
MSF blocks in proposed method, named as Af-Net-LFC. Af-Net-LFC fails to learns
the pertinent features and degrades the performance of network. Third, we
computed results by dropping the MFL block named as af-Net-WoMFL to
investigate the role of MFL block in learning of dicriminative variations of
micro expressions. Further, we have examined the effect of multi-scale filters
by replacing all filters with $\left(3\times 3\right)$ named as af-
Net-$\left(3\times 3\right)$. Finally, to analyse the effects of
$\left(1\times 1\right)$ sized filters, we execute the AffectiveNet by
replacing $\left(3\times 3\right)$ with $\left(1\times 1\right)$ in MICRoFeat
block.Quantitative results, represents in Table-I validates the performance of
AffectiveNet over other supplementary results. Moreover. Thus, by observing
ablation studies experimental results, we can conclude that our proposed model
has generated best results as compared to other combinations.

##  5 Conclusion

This paper presents an AffectiveNet: affective-motion feature learning for
micro-expression recognition. First, we computed single instance responses of
the affective-motion images from micro expression sequences which preserves
the facial movements into one instance. Further, the generated single instance
is processed through the AffectiveNet to estimate the networks performance. In
AffectiveNet two blocks are introduced MICRoFeat and MFL, to learn the micro
expression features. MICRoFeat block holds multi-scale filters as $3\times 3$,
$5\times 5$,$7\times 7$ and $11\times 11$ to extract the comprehensive and
detailed edge variations from affective images. Thus, MICRoFeat block is
responsible to capture edge variations from small regions to extensive
regions. While, MFL block incorporats two-stage FC layers to more
discriminative features of micro expressions and allows network to define the
disparities between emotion classes. Moreover, the proposed network has a
small number of parameters that reduce the training and testing time of the
MER system. The effectiveness of system is evaluated on the benchmark dataset
CASME-I, CASME-II, CASME2 and SAMM. It is evident from experimental results,
visual demonstration, complexity analysis and ablation study that AffectiveNet
has achieved better accuracy rates as compared to state-of-the-art approaches
for MER.

## Acknowledgment

This work was supported by the Science and Engineering Research Board (under
the Department of Science and Technology, Govt. of India) project
$\\#$SERB/F/9507/2017. The authors would like to thank our Vision Intelligence
lab group for their valuable support. We are also thankful to Shastri Indo-
Canadian Institute for their support in the form of SRS fellowship.

## References

  * [1] Y. Wang, J. See, R. C. W. Phan and Y. H. Oh, _Lbp with six intersection points: Reducing redundant information in lbp-top for micro-expression recognition_ , In Asian Conf. on Comput. Vis., pp. 525-537, 2014.
  * [2] F. Xu, J. Zhang and J. Z. Wang, _Microexpression identification and categorization using a facial dynamics map_ , IEEE Trans. on Affect. Comput., vol. 8, no. 2, pp. 254-267, 2017.
  * [3] K. Simonyan and A. Zisserman, _Very deep convolutional networks for large-scale image recognition_ , arXiv preprint arXiv: 1409.1556, 2014.
  * [4] K. He, X. Zhang, S. Ren and J. Sun, _Deep residual learning for image recognition_ , In Proc. IEEE conf. on Comput. Vis. Pattern Recognit., pp. 770-778, 2016.
  * [5] A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto and H. Adam, _Mobilenets: Efficient convolutional neural networks for mobile vision applications_ , arXiv preprint arXiv:1704.04861, 2017.
  * [6] C. Wang, M. Peng, T. Bi and T. Chen, _Micro-Attention for Micro-Expression recognition_ , arXiv preprint arXiv:1811.02360, 2018.
  * [7] S. J. Wang, B. J. Li, Y. J. Liu, W. J. Yan, X. Ou, X. Huang, F. Xu and X. Fu, _Micro-expression recognition with small sample size by transferring long-term convolutional neural network_ , Neurocomputing, Vol. 312, pp.251-262, 2018.
  * [8] Z. Xia, X. Feng, X. Hong and G. Zhao, _Spontaneous facial micro-expression recognition via deep convolutional network_ , In 2018 Eighth International Conference on Image Processing Theory, Tools and Applications (IPTA), pp. 1-6, 2018.
  * [9] Z. Xia, X. Hong, X. Gao, X. Feng and G. Zhao, _Spatiotemporal recurrent convolutional networks for recognizing spontaneous micro-expressions_ , IEEE Transactions on Multimedia, 2019.
  * [10] J. Li, Y. Wang, J. See and W. Liu,_Micro-expression recognition based on 3D flow convolutional neural network_ , Patter. Analy. An App., pp. 1-9, 2018.
  * [11] H. Bilen, B. Fernando, E. Gavves, A. Vedaldiand S. Gould, “Dynamic image networks for action recognition,” In Proc. IEEE Int. Conf. Comput. Vis. Pattern Recognit. pp. 3034-3042, 2016.
  * [12] S. J. Wang, W. J. Yan, X. Li, G. Zhao and X. Fu, _Micro-expression recognition using dynamic textures on tensor independent color space_ , In 22nd IEEE Conf. Pattern Recognit, pp. 4678-4683, 2014.
  * [13] S. J. Wang, W. J. Yan, T. Sun, G. Zhao and X. Fu, _Sparse tensor canonical correlation analysis for micro-expression recognition_ , Neurocomputing, Vol. 214, pp. 218-232, 2016.
  * [14] S. L. Happy and A. Routray, _Fuzzy histogram of optical flow orientations for micro-expression recognition_ , IEEE Trans. Affect. Comput., 2017.
  * [15] S. J. Wang, S. Wu, X. Qian, J. Li and X. Fu, _A main directional maximal difference analysis for spotting facial movements from long-term videos_ , Neurocomputing, Vol. 230, pp. 382-389, 2017.
  * [16] H. Q. Khor, J. See, R. C. W. Phan and W. Lin, _Enriched long-term recurrent convolutional network for facial micro-expression recognition_ , In 13th IEEE Conf. Auto. Face Gestur. Recognit. (FG 2018), pp. 667-674, 2018.
  * [17] S. T. Liong, Y. S. Gan, J. See and H. Q. Khor, _A Shallow Triple Stream Three-dimensional CNN (STSTNet) for Micro-expression Recognition System_ , arXiv preprint arXiv: 1902.03634, 2019.
  * [18] W. J. Yan, X. Li, S. J. Wang, G. Zhao, Y. J. Liu, Y. H. Chenand X. Fu, _CASME II: An improved spontaneous micro-expression database and the baseline evaluation_ , PloS one, vol. 9, no. 1, pp. 86041, 2014.
  * [19] F. Qu, S. J. Wang, W. J. Yan, H. Li, S. Wu and X. Fu, _CAS (ME) 2: a database for spontaneous macro-expression and micro-expression spotting and recognition_, IEEE Trans. Affect. Comput., 2017.
  * [20] A. K. Davison, C. Lansley, N. Costen, K. Tan and M. H. Yap, _Samm: A spontaneous micro-facial movement dataset_ , IEEE Trans. Affect. Comput., vol. 9, no. 1, pp. 116-129, 2018.

|  Monu Verma received her B. Tech degree in Computer Science and Engineering
from GEC Bikaner, India, in 2013. She received her M. tech degree in 2016 from
NIT Jalandhar, India. She is currently pursuing her Ph.D. with the Department
of Computer Science and Engineering, MNIT Jaipur, India. She is a life member
of Vision Intelligence Lab @ MNIT, Jaipur. Her research interests include
facial expression recognition, depression analysis, micro expression
recognition, hand posture classification and finger sign analysis, .
---|---
|  Santosh Kumar Vipparthi received his B.E. degree in Electrical and
Electronics Engineering from Andhra University, India. Further, he received
his M. Tech. and Ph. D. in Systems Engineering from IIT BHU, Varanasi, India.
Currently, he is an assistant professor in the Department of Computer Science
and Engineering, MNIT Jaipur, India. He leads Vision Intelligence Lab @ MNIT
with research focused important visual perception tasks such as object
detection, human emotion recognition, aberrant event detection, image
retrieval, Gesture recognition, Motion analysis, etc.
---|---
|  Girdhari Singh received the B.E. degree in Computer Engineering from
Amravati University, Maharastra India, in 1990. Afterwards, he received his MS
in Software Engineering from BITS Pilani, India in 1996. Further, He received
his Ph.D. in Computer Engineering from MNIT, Jaipur, India in 2009. Currently,
he is working as an associate professor in the department of computer science
and engineering, MNIT, Jaipur, Rajasthan, India. His major fields of research
are software engineering, intelligent systems image processing and machine
learning.
---|---


# NewsQuote: A Dataset Built on Quote Extraction and Attribution for Expert
Recommendation in Fact-Checking

Wenjia Zhang,1 Lin Gui, 2 Rob Procter, 1,3 Yulan He 1,2,3

###### Abstract

To enhance the ability to find credible evidence in news articles, we propose
a novel task of expert recommendation, which aims to identify trustworthy
experts on a specific news topic. To achieve the aim, we describe the
construction of a novel NewsQuote dataset consisting of 24,031 quote-speaker
pairs that appeared on a COVID-19 news corpus. We demonstrate an automatic
pipeline for speaker and quote extraction via a BERT-based Question Answering
model. Then, we formulate expert recommendations as document retrieval task by
retrieving relevant quotes first as an intermediate step for expert
identification, and expert retrieval by directly retrieving sources based on
the probability of a query conditional on a candidate expert. Experimental
results on NewsQuote show that document retrieval is more effective in
identifying relevant experts for a given news topic compared to expert
retrieval.111Our source code can be accessed at:
https://github.com/WenjiaZh/NewsQuote

##  1 Introduction

The rapid growth of misinformation in recent years has been the subject of
much attention from academia, journalists, political analysts and fact-
checking organisations and has prompted research into NLP-based techniques and
tools to support fact-checking work and evidence verification (Lazarski, Al-
Khassaweneh, and Howard 2021; Zeng, Abumansour, and Zubiaga 2021; Guo,
Schlichtkrull, and Vlachos 2022). Much of this research effort has been based
on a document-centric model of fact-checking work, where the end goal is to
provide the journalist or fact-checker with an (automated) ranked list of
documents relevant to the claim that they can then use as evidence for
determining its likely veracity (e.g., Zhao et al. (2023)).

Our recent research reveals that some fact-checkers use a expert-centric
model, whereby they search for credible and trustworthy experts who are
willing to be quoted (Procter et al. 2023). Finding such experts is a big
challenge and often journalists and fact-checkers aim to interview several
experts as relying solely on one source may not be considered as sufficiently
credible. In the case of contentious claims, they may also need to ensure
their reports are balanced (Procter et al. 2023).

There is thus an urgent need to develop a tool for journalists and fact-
checkers to search for experts based on their past record of being quoted by
news media and fact-checking organisations, and other trustworthy agencies. To
achieve this goal, we need to first automatically extract quotes and their
sources from news articles, and then second return a ranked list of experts
relevant to a query that then can be assessed by the journalist or fact-
checker. This can be formulated as two tasks: (1) quote extraction and
attribution, and (2) expert recommendation.

For the first task of quote extraction and attribution, most datasets were
built on literature narratives and limited in size due to the reliance on
manual annotation (Zhang, Black, and Sproat 2003; Elson and McKeown 2010;
Fernandes, Motta, and Milidiú 2011; Lee and Yeung 2016). But newswire has much
fewer monologues and dialogues than fiction (O’Keefe et al. 2012). Early work
relied on rule-driven frameworks and manually-defined linguistic patterns,
hence they mainly focused on direct quotes (Lee and Yeung 2016; Zhang and Liu
2021; Vaucher et al. 2021). Unlike play scripts or fiction, people quoted in
the news media are not limited to a list of fixed characters. In addition, the
constantly evolving stream of events reported in news articles and diverse
writing styles used by news media outlets make it difficult to identify
experts and extract quotes by relying on regular expressions.

For the second task of expert recommendation, much work has been conducted for
expert finding in academic research (Sun et al. 2015; Silva 2014; Wang et al.
2017), online communities (Yuan et al. 2020), and the enterprise field (Paul
2016; Askari, Verberne, and Pasi 2022). However, we are not aware of any work
searching for experts based on their track record of being quoted in news
articles.

Corpus | #Quotes | Indirect% | Entity | Data Source
---|---|---|---|---
StylisticsCorpus | 16,533 | 16 | ✗ | Fiction, Newspaper, Biographies
PARC3 | 19,712 | 72 | ✗ | Wall Street Journal
QuoteBank | 178 million | - | ✓ | News Articles
DirectQuote | 10,279 | 0 | ✓ | News Articles
NewsQuote | 24,031 | 81 | ✓ | News Articles
Table 1: Summary of large-scale (larger than 10,000) news-originated English
quotation corpora.

In this paper, we propose a semi-automatic approach to construct a news
quotation dataset, called NewsQuote, from the AYLIEN coronavirus
dataset222This data was aggregated, analyzed, and enriched by AYLIEN using the
AYLIEN’s News Intelligence Platform.
https://aylien.com/resources/datasets/coronavirus-
dataset,https://aylien.com/blog/free-coronavirus-news-dataset, which contains
over 1.5 million English news articles generated from around 440 global
sources. We utilise the semantic role labelling results of sentences in news
articles to extract the quote trigger verbs, subjects (i.e., sources) and
objects (i.e., quotes), and identify sources by their corresponding
DBpedia333https://www.dbpedia.org/ ontology class labels. The resulting
dataset contains both direct and indirect quotes, and also mixed quotes where
only part of the quotations is placed inside quotation marks. We introduce the
task of finding sources of evidence from news reports and present a set of
approaches for (1) identifying quotations and their sources from text; and (2)
recommending potential experts for given news topics. Our experimental results
illustrate the feasibility of using our constructed NewsQuote dataset for
developing an automated tool for searching and ranking subject-matter experts
for journalists and fact-checkers.

##  2 Related Work

#### Quotation Extraction and Attribution

Quotation extraction and attribution originated as a study of literary works
(Zhang, Black, and Sproat 2003), and now typically covers three sub-tasks:
identifying sources, extracting quotations, and attributing a quotation to its
source. In Table 1, we summarise several large-scale English quotations
datasets that are built on news articles. The StylisticsCorpus (Semino and
Short 2004) was designed for discourse presentation in written British
narratives. They opted for hard news (e.g., accidents, conflicts, and crimes)
(Bell 1991) as a part of the data source because of its circulation,
narrative, authenticity, and cultural prominence. Of the total data, 5407
occurrences came from the press. They classified these samples into speech,
writing, and thought. Then they divided each class into many presentation
categories, such as indirect, free indirect, direct, and free direct. The
PARC3 (Pareti 2016) project aims to fill the gap of the attribution relation
(AR). Their annotation scheme tagged three constitutive components of an AR:
source, cue, and content. They labeled the quote status as direct, indirect,
or mixed by the usage of quote marks, and looked into the depth of attribution
by the level of nesting. The inspiration for generating QuoteBank (Vaucher et
al. 2021) came from the tangled nature of contemporary news flow. Vaucher et
al. (2021) exploited duplicate reports in different media to learn the
patterns of quote-source pairs. Focusing on the attribution of direct
quotations, they proposed an end-to-end minimally supervised framework, named
Quobert, to extract and attribute quotations. Using Quobert, they generated
QuoteBank from the Spinn3r dataset (Burton et al. 2011), and linked source
entities to the Wikidata knowledge base. DirectQuote (Zhang and Liu 2021)
contains direct quotations manually annotated from online news media. Like
QuoteBank, each source can be linked to a Wikidata named entity to benefit
various downstream tasks.

Among the existing news quotation datasets, StylisticCorpus and PARC3 contain
both direct and indirect quotes, but do not originate from multi-platform news
stories, nor do they provide source-entity linking to Wikidata. The other two
datasets, QuoteBank and DirectQuote, have each of their sources linked to a
Wikidata named entity, but they only focus on direct quotes. In comparison,
our NewsQuote contains various types of quotes including direct, indirect and
mixed quotes where only part of the quotation is inside the quotation marks.
In addition, all sources have their DBpedia entity links.

#### Expert Finding

The core task in expert finding is to identify candidates with the required
expertise for a given query (Yuan et al. 2020). Therefore, solutions focus on
matching the demand of searchers and the experience of relevant experts. In
practise, this problem has expanded to different situations where various
factors were considered. Academic accounts for up to 65% expert finding
research (Husain et al. 2019). When looking for academic experts, attention is
given to topic relevance, expert quality, research connectivity (Sun et al.
2015; Silva 2014; Wang et al. 2017), as well as capacity limitation (Neshati,
Beigy, and Hiemstra 2014). Meanwhile, many expert finding systems are used on
online platforms, such as community question answering, social networks and
forums (Yuan et al. 2020; Faisal, Daud, and Akram 2017). In the enterprise
field, experts’ accessibility and productivity are considered to have
significant economic benefits (Silva et al. 2013; Paul 2016). In the medical
domain, when looking for the most suitable doctor for a particular patient,
the patient’s underlying conditions are of critical importance (Tekin, Atan,
and Van Der Schaar 2014). In lawyer finding, users may prefer candidates in
the same state or city, hence the physical location was emphasized (Askari,
Verberne, and Pasi 2022).

##  3 NewsQuote: Dataset Construction

In this section, we describe how we constructed the dataset, including details
of the data source, pre-processing steps performed, and test set annotation.
Example data entries and dataset statistics will be presented at the end.

### Data Collection

We built our NewsQuote dataset from the AYLIEN coronavirus dataset, published
between November 2019 and August 2020. We used the AYLIEN News
API444https://aylien.com/product/news-api to retrieve news articles. Apart
from text, each article is also accompanied with the meta data such as
authors, keywords, summary, source, publishing time, topical categories coded
by both the Interactive Advertising Bureau (IAB)
taxonomy555https://www.iab.com and the IPTC
NewsCodes666https://iptc.org/standards/newscodes/, as well as recognized
entities and entity links from DBpedia.

### Pre-processing

#### Data De-duplication

As the same news story may be posted by multiple sources and there were exact
duplicates in the original dataset, we removed news articles that are similar
to ones already been published. News articles were first sorted in
chronological order. News duplicates were then detected using a RoBERTa
classifier777https://huggingface.co/vslaykovsky/roberta-news-duplicates
trained with title-body pairs using semi-supervised learning (Rücklé, Moosavi,
and Gurevych 2019). For processing efficiency, the dataset was split into 16
equal-sized subsets. For each subset, titles and first sentence of news
summaries of temporally-ordered news articles were sequentially fed as input
to the RoBERTa classifier. Any duplicates were removed. After data de-
duplication, 158,325 news articles remained. The total number of source
platforms is 258, and as shown in Figure 1(b), the top 5 source platforms are:
Daily Mail, Yahoo, Seeking Alpha, Business Insider, Reuters.

#### Quote Trigger Word Filtering

For each of the selected articles, we segment the the main body into
sentences, and then use a pre-trained BERT-based semantic role labeling model
(Shi and Lin 2019) to extract verbs (or predicates), subjects, and objects. We
obtained a candidate verb list sorted by their occurrence frequencies. After
manually checking the candidate verbs with over 100 occurrences, we identified
352 quote trigger words that are more likely indicative of direct or indirect
quotes. The list of verbs are presented in our source code repository
888https://github.com/WenjiaZh/NewsQuote/blob/main/SelectedTriggerVerbs.csv.
Some of the verbs are clearly indicative of quotes, such as ‘ _said_ ’, while
others may not be associated with quotes in a traditional sense, for example,
‘ _tweet_ ’. After identifying the quote trigger words, we only kept the
sentences with at least one trigger word, one subject and one object. The
subject is regarded as a potential source and the object is considered as a
potential quotation. To ensure that the quotations are informative, we also
require that the length of the object should be more than three words.

#### Source and Quote Filtering

We required that the subject of a candidate sentence should be a person or an
organisation and therefore identified potential source entities via the
accompanying DBpedia ontology
labels999http://mappings.dbpedia.org/server/ontology/classes/ in the dataset.
Our selected ontology classes are shown in our source code repository
101010https://github.com/WenjiaZh/NewsQuote/blob/main/SelectedOntologyClasses.txt.Since
each entity could have more than one ontology class, we further removed
samples with sources labeled as _Location_ , _Place_ and _Country_. As the
same subject could have multiple mentions, we use DBPedia entity links for
entity resolution and normalisation. In addition, we required a named entity
to appear at least twice in the dataset. Finally, to avoid the sentence split
error, we required quotation marks to be paired in sentences that contain
direct quotes and mixed quotes.

Figure 1: Three types of quotes in our dataset. Sources are highlighted in
blue, trigger verbs are highlighted in red, and quotes are highlighted in
yellow.

### Test Set Annotation

Since in practice, given a topic, we can only identify experts based on their
previous quotes published in earlier news articles, we divide the dataset into
training, validation and testing sets by news articles publishing timestamps,
ensuring quote-source pairs in the validation and testing sets occurred later
than those in the training set. Figure 2 demonstrates the distribution of
quote-source pairs based on the publishing dates of their associated news
articles.111111There is no data between 2020-05-31 and 2020-06-21 in the
original dataset.

Figure 2: The distribution of quote-source pairs. The training set contains
samples released from 2020-01-19 to 2020-05-31, and the validation/testing set
contains samples released from 2020-06-21 to 2020-08-02.

To ensure data quality, samples in the test set were manually screened by one
annotator. We list five types of noise and corresponding examples appearing in
the raw test set in Table A1. Data falling into one of these noise categories
were removed from the test set.

### Dataset Statistics

Our data covers three categories of quotes, illustrated in Figure 1. In short,
direct quotations are placed inside quotation marks, while indirect quotations
are not, and a mix of direct and indirect quotations have only part of the
quotations placed inside quotation marks. We roughly estimated the weight of
each quotation type on the dataset by the number and position of quotation
marks: 81% for indirect quotes, 10% for direct quotes and 9% for mixed quotes.
In the test set, there are 1,867 (84%) indirect quotes, 215 (10%) mixed quotes
and 143 direct quotes (6%). Table 2 shows the statistics of our final
NewsQuote dataset. In summary, we have a total of 24,031 English source-quote
pairs with 3,246 sources from 258 global sources. More related statistics and
plots are presented in Appendix A.

| Test | Valid | Train
---|---|---|---
No. of samples | 2,236 | 2,082 | 19,713
No. of articles | 1,937 | 1,766 | 14,526
No. of source entities | 1,016 | 765 | 2,963
Avg. quote length | 28.38 | 29.16 | 28.99
No. of news sources | 180 | 178 | 252
No. of news categories | 470 | 440 | 629
Avg. keywords per article | 43.23 | 44.28 | 42.17
Table 2: The NewsQuote Dataset statistics. Figure 3: Illustrations of 5
approaches described in Section 5. Plot(a) describes the QA pipeline, the
sequence labelling and the Rule-based Quota Annotator used for quote-source
extraction. Plot(b) introduces the document retrieval approach for expert
recommendation, and plot(c) presents the expert retrieval approach for expert
recommendation

##  4 Task Definition

In our dataset, each sample $S_{i}$ consists of a context $c_{i}$, a quote-
source pair $(q,e)_{i}$, a list of keywords $k_{i}$ and metadata $m_{i}$. The
context contains 3 sentences, the main sentence where the source and quote
appeared, its preceding sentence and following sentence. Both keywords and
metadata are defined at the document level and are retrieved from the AYLIEN
coronavirus dataset. We propose the following two tasks on this NewsQuote
dataset:

Source and quote extraction is defined as automatically extracting the source-
quote pair $(q,e)_{i}$ from a given context $c_{i}$.

Expert recommendation involves suggesting a ranked list of experts given a
query, based on what they said in the past.

##  5 Approaches

We present approaches for source and quote extraction, and expert
recommendation. An overview of the approaches is illustrated in Figure 3.

### Source and Quote Extraction

We tackle the problem of extracting quote-source pairs using three approaches:
rule-based method, sequence labelling, and question answering.

#### Approach 1: Rule-based Quote Annotator

Regular-expression-like rules can be used to extract direct quotes. We run the
Quote Annotator 121212https://stanfordnlp.github.io/CoreNLP/quote.html from
Stanford CoreNLP (Manning et al. 2014) on our test sample sentences. It can
only extract direct quotes that are delimited by quotation marks.

#### Approach 2: Sequence Labelling

We can label each sample in our dataset with a 5-class BIO tagging scheme. The
source is annotated by ’B-S’ and ’I-S’, denoting whether the corresponding
token indicates the beginning of a source mention, or is inside a source
mention. Similarly, the quotation is annotated by ’B-Q’ and ’I-Q’, and all the
other tokens are marked by ’O’. We then fine tune a BERT-based token
classifier (Devlin et al. 2018) to identify sources and quotes from the
context.

#### Approach 3: Question Answering (QA) pipeline

We use a QA pipeline for source and quote extraction by asking two questions
in turn:

> Q1: Who is the source?
> Q2: What did [source] say?

During training, the [source] in Q2 is the gold standard answer for question
Q1. During inference, it is the extracted answer for Q1. The input context is
composed of a question, a left sentence, $l$, a main sentence, $s$ and a right
sentence, $r$. To extract the answer from the context, we fine-tuned the pre-
trained BERT-based extractive QA model (Devlin et al. 2018), where the input
takes the form:

> [CLS] Question [SEP] l [SEP] s [SEP] r [SEP]

### Expert Recommendation

We can formulate expert recommendation as a retrieval problem, that given a
query, we would like to retrieve sources who can comment on the topic
discussed in the query ranked by their relevance to the query. There are two
possible approaches, one is to use sources’ past quotes as documents and
perform _document retrieval_ and then return the sources of the retrieved
quotes as results, another is to perform _expert retrieval_ directly.

#### Approach 1: Document Retrieval

_Document retrieval_ aims to first retrieve relevant documents (i.e., the
context where a quote appears) given a query, and then extract the sources
from the documents as results. For document indexing, we experiment with a
sparse bag-of-words Lucene index and four kinds of dense transformer-encoded
Faiss indices via Pyserini131313https://github.com/castorini/pyserini. A BM25
ranking approach on the sparse index and a nearest-neighbor search on dense
indexes were then applied to return the top 10 most relevant documents for a
given query. Sources in the top 10 retrieved documents are then identified as
the recommended experts.

#### Approach 2: Expert Retrieval

_Expert retrieval_ directly retrieves sources based on the probability of a
query conditional on a given candidate source $P(q|e)$. Following the the
framework introduced by Balog, Azzopardi, and de Rijke (2009), we implemented
both candidate-based and document-based expert finding approaches.

Candidate-Based Expert Retrieval Assuming that each term in the query is
sampled identically and independently, also that the document and the expert
source candidate are conditionally independent, the candidate-based approach
estimates $P(q|e)$ by:

$\displaystyle P(q|e)=\prod_{t\in q}\\{(1-\lambda)(\sum_{d\in
D}p(t|d)p(d|e)+\lambda p(t)\\}^{n(t,q)},$
$\displaystyle\lambda=\frac{\beta}{\beta+n(e)},\quad\beta=\frac{\sum_{E}|\\{d:n(e,d)>0\\}|\cdot|d|}{|E|},$

where $\lambda$ is the smoothing parameter, $p(t|d)$, $p(d|e)$ and $p(t)$ are
the conditional probability of a term $t$ in document $d$, the conditional
probability of a document $d$ given source $e$, and the probability of term
$t$, respectively. Both $p(t|d)$ and $p(t)$ are estimated by maximum
likelihood. The probability $p(d|e)$ is set by a Boolean model, which will be
discussed later. $|d|$ is the average document length, $n(t,q)$ is the number
of times that a term $t$ appears in the query $q$, $n(e,d)$ is the occurrence
frequency of an expert $e$ appeared in the document $d$, and $n(e)$ is the
total number of occurrences in documents associated with the source $e$.

Document-Based Expert Retrieval The document-based expert retrieval approach
searches for sources via relevant document collection. This approach assumes
the conditional independence between the query and candidate, and estimates
the probability of a term $t$ in each document:

$\displaystyle P(q|e)=\sum_{d\in D}\\{\prod_{t\in q}((1-\lambda)p(t|d)+\lambda
p(t))^{n(t,q)}\\}p(d|e),$
$\displaystyle\lambda=\frac{\beta}{\beta+n(d)},\quad\beta=|d|,$

where $n(d)$ is the length of document $d$.

In both the candidate-based and document-based expert finding approaches, the
document-candidate associations, $p(d|e)$, is estimated by a simple Boolean
model, where it is set to 1, if $n(e,d)>0$, and 0, otherwise.

##  6 Experiments

### Experimental Setup

For the rule-based approach, we directly feed the raw sentences into the Quote
Annotator. To build the token classifier, we segment the input text into a
sequence of 512 tokens, and fine tune the model for 100 epochs with an initial
learning rate of 2e-7. For the extractive QA model, the maximum length of the
extracted answer is set to 30 when questioning sources and 512 when
questioning quotes. For the question about source, we train the model for 50
epochs with an initial learning rate of 2e-6. For the question about quote, we
train the model for 100 epochs with an initial learning rate of 2e-5.

For expert recommendation, we consider two types of documents: the main
sentence where a source/quote occurred, or the main sentence together with its
surrounding context (i.e., the preceding and following sentences). For the
query to be used for expert retrieval, we use either the title of a news
article, its keywords , or the first sentence of the summary. To further
remove interference, we eliminate the source name from the input query if
there is any. For the expert retrieval method, we take only the first $w$
words in the news article title (the keyword list or the first sentence of the
news summary) as the input query to reduce the running time. After validating
the value of $w$ between 1 and 10, we finally set $w=5$.

| Overall | Direct Quotes | Indirect Quotes | Mixed Quotes
---|---|---|---|---
| Macro F1 | Exact Match | Macro F1 | Exact Match | Macro F1 | Exact Match | Macro F1 | Exact Match
Rulesource | 5.76 | 5.62 | 50.58 | 49.65 | 0.214 | 0.214 | 24.11 | 23.26
Rulequote | 7.72 | 1.93 | 82.33 | 30.07 | 0.145 | 0.00 | 23.84 | 0.00
SLsource | 98.06 | 95.37 | 98.63 | 95.80 | 97.99 | 95.34 | 98.23 | 95.35
SLquote | 95.65 | 85.17 | 97.17 | 89.51 | 95.61 | 85.11 | 95.05 | 82.79
QAspeakr | 98.86 | 98.61 | 99.30 | 99.30 | 98.77 | 98.50 | 99.38 | 99.07
QAquote |  |  |  |  |  |  |  |
$~{}~{}_{w/~{}true~{}source}$ | 95.96 | 90.74 | 95.83 | 93.01 | 95.96 | 90.31 | 96.06 | 93.02
$~{}~{}_{w/~{}pred.~{}source}$ | 95.61 | 89.93 | 95.78 | 93.01 | 95.55 | 89.34 | 96.06 | 93.02
$~{}~{}_{w/~{}source~{}mask}$ | 93.92 | 85.84 | 96.53 | 90.21 | 93.56 | 85.11 | 95.28 | 89.30

Table 3: Results of source and quotation extraction on the test set. Rule $-$
the rule-based annotator, SL $-$ sequence labeling, QA $-$ the question
answering pipeline. The subscripts indicate the aim of the models, either for
${source}$ extraction or for ${quote}$ extraction. Under the QAquote,
‘${}_{w/~{}true~{}source}$’ is where we use the true source name when asking ”
_What did + [source] + say?_ ”, while ‘${}_{w/~{}pred.~{}source}$’ uses the
predicted source from the QAspeakr results, and ‘${}_{w/~{}source~{}mask}$’
uses the generic word ” _they_ ”.

### Evaluation Metrics

To measure model performances for quote extraction and attribution, we use two
metrics defined in SQuAD (Rajpurkar et al. 2016), the exact match and the
macro-averaged F1 score. Exact Match is equal to one if the predicted outcome
is completely identical to the ground truth, while (Macro-averaged) F1
measures the average overlap between predicted and ground truth answers at the
token-level.

For expert recommendation, we use two metrics commonly used in information
retrieval, the mean average precision (MAP) and the normalized discounted
cumulative gain (NDCG). Mean Averaged Precision is the average value of the
precision at the points where relevant documents are retrieved. Normalized
Discounted Cumulative Gain at K first discounts the gain scale at the $i$-th
rank position by $\frac{1}{\log_{2}(i)}$, then adds up the converted gain
scales up to rank $k$, and finally normalizes the result by the ideal ranking
order. In addition, we propose relaxed metrics where the retrieved expert is
considered relevant if it is in the same cluster as the true source. In the
construction of relaxed metrics, we opt for the top 100 most frequent source
DBpedia categories and use the binary vectors to embed sources141414In our
dataset, a source is assigned to 4 to 5 DBpedia categories on average.. We
then perform $k$-means clustering on the source embeddings. We empirically set
$k=40$ according to the cluster coherence and separation scores.

| Strict Metrics | Relaxed Metrics
---|---|---
| MAP | NDCG5 | NDCG10 | MAP | NDCG5 | NDCG10
DRsparse | 0.2903 | 0.2807 | 0.3590 | 0.4162 | 0.3925 | 0.5183
DRflat | 0.1481 | 0.1440 | 0.1939 | 0.2886 | 0.2714 | 0.3887
DRhnswpq | 0.1509 | 0.1473 | 0.1926 | 0.2966 | 0.2805 | 0.3956
DRhnsw | 0.1446 | 0.1406 | 0.1889 | 0.2865 | 0.2686 | 0.3850
DRpq | 0.1395 | 0.1363 | 0.1838 | 0.2739 | 0.2583 | 0.3734
ERcan | 0.1021 | 0.1106 | 0.1252 | 0.2306 | 0.2294 | 0.3135
ERdoc | 0.1205 | 0.1281 | 0.1418 | 0.2465 | 0.2412 | 0.3285

Table 4: Results of expert recommendation using quote context as document, and
news article keywords as query. In the first five rows, DR denotes the
document retrieval approach, and the subscripts represent 5 types of retrieval
indices mentioned in Section 5 Approach 1, Lucene sparse bag-of-words index,
Faiss flat index, Faiss HNSWPQ index, Faiss HNSW index, and Faiss PQ index.
ERcan is the candidate-based expert finding approach, and ERdoc is the
document-based expert finding approach. In document-based expert finding
approaches, the input query length is set to 5 keywords.

### Experimental Results

We first present the results of the three quote extraction and attribution
methods described in Section 5, and subsequently present the evaluation
results for the two expert recommendation approaches introduced in Section 5.

#### Quote Extraction and Attribution

Table 3 presents the performance of the rule-based annotator, sequence
labeling and the QA pipeline on the test set. It is not surprising that the
rule-based quote annotator performs the worst as it can only extract direct
quotes using regular-expression-like rules. In our test set, only 337 out of
2225 samples were identified as containing quotes. On this subset, the rule-
based annotator gives a higher exact match score of 49.65 for sources compared
to quotes. But it performs much better for direct quote extraction in Macro F1
compared to source extraction. On the other two categories, indirect and mixed
quotes, the rule-based annotator essentially failed to produce any sensible
results. Sequence labeling gives much better results compared to the rule-
based annotator. We notice that in terms of exact match, quote extraction
appears to be nearly 10% lower than source extraction, showing that the model
struggled with longer answer extraction. For the three categories of quotes,
the model gives the best results for quote extraction on the direct quotes,
followed by the indirect quotes, and it performs the worst on the mixed
quotes. This is expected since mixed quotes are more complex to deal with
compared to the other two categories. The QA pipeline achieves the best
performance in both identifying sources and extracting the quotations. In
testing the QA pipeline’s quote extraction capabilities, we experimented with
three scenarios by using either: the true source name in the question for
quote, the predicted source from the results of QAsource, or masking the
source with the pronoun ’ _they_ ’ to completely remove the source information
from the question. Since the accuracy of our QA model for source
identification is already high, using the true or predicted source for the
question for quote extraction does not make much difference. However, if the
source information is lost, the quote extraction performance drops by nearly
2% in Macro F1 and over 4% in exact match.

#### Expert Recommendation

We show in Table 4 the expert recommendation results from using keywords of a
news article as query, and the context of quotes (the main sentence where the
source and quote occurred, together with the preceding and the following
sentences) as the document. It can be observed that the document retrieval
(DR) approaches generally outperform the expert retrieval (ER) approaches.
Among various document indexing strategies, using Lucene sparse bag-of-words
index (DRsparse) gives superior results compared to other dense transformer-
encoded Faiss indices. As expected, using the Relaxed Metrics where a
retrieved source is considered as relevant to the true source if they reside
in the same cluster, we obtain better results compared to the strict
metrics.151515Results using other document retrieval or expert retrieval
approaches based on different combinations of the formulation of documents and
queries are in Appendix C.

##  7 Challenges and Future Directions

We have presented our NewsQuote dataset, and introduced a new task, namely
expert recommendation in the field of journalism and fact-checking. Our
experiments confirmed the possibility of extracting quote-source pairs using a
question-answering pipeline as well as finding expert sources using document
retrieval and expert retrieval. Here, we outline some potential future
directions.

First, in the construction of our dataset, the quote trigger verbs are
manually selected from the most frequent group of verbs. On one hand, the
identified verb list does not cover all the possible verbs that are indicative
of quotations, such as those occurred less frequently or are not closely
related to the Covid topic. On the other hand, some verbs are ambiguous and
need to be contextualized to determine whether they are indeed the trigger
words. Although we removed disambiguous cases when examining the test set, it
is not practical to perform manual filtering on such large-scale data. Future
work could explore the possibility of leveraging other large-scale quote
corpora for training a model for the detection of quote trigger words. Also,
our dataset has been constructed from the news articles about the coronavirus.
In the future, this could be extended to cover a wide range of topics such as
business, technology, education, and politics.

Second, co-reference resolution will be vital for increasing the quote-source
attribution data as it is common to use pronouns to refer to previously
mentioned sources in news articles. Our preliminary experiments on co-
referencing resolution led to noisy quote-source attribution results. In the
future work, the content similarity and/or coherence between the quote
associated with a pronoun and a quote of a candidate source could be leveraged
to improve the co-reference resolution results.

Third, with the DBpedia links referred to as identifications of sources in our
dataset, external knowledge could be imported as evidence to enhance the
performance of expert recommendation.

Fourth, our framework makes it possible to build a quote-source library for
the newsroom that can help with veracity assessment, where summaries of the
comments made by each source, including who has quoted them, when and in
relation to which veracity check, can be made available to journalists and
fact-checkers, thereby reducing duplication of effort and supporting
collaboration.

Finally, it is important that journalists and fact-checkers do not become
over-reliant on tools such as the one we present here (i.e., fall victim to
so-called ’automation bias’). The results therefore need to be interpreted
with care and the final decision on which experts to approach should always
made by the journalist or fact-checker. It is therefore important that such
models provide evidence for their recommendations that can be assessed for
credibility and relevance by the user (Procter et al. 2023).

##  8 Conclusions

We have described the construction of a novel, large-scale dataset on quote-
source pairs retrieved from news articles. Our NewsQuote dataset comprises
direct quotations, indirect quotations and their combinations. The diversity
of quote types will encourage the development of more advanced approaches for
the challenging tasks of indirect and mixed quote extraction. Based on the
NewsQuote dataset, we have demonstrated that the QA pipeline is able to
achieve over 98% exact match for source extraction and close to 90% for quote
extraction. In addition, we have introduced the expert recommendation task and
shown that the document retrieval approach with sparse indexing gives the best
results compared to other dense retrieval approaches.

## Ethics Statement

All data we used are from open public sources. We have obtained a written
consent from the Aylien to download their data. As per the data owner’s
requirement, we will not directly share the downloaded data, instead, we will
share the download script and all pre-processing scripts so that others could
obtain the same dataset we used in the paper from the Aylien’s website.

## Acknowledgements

This work was supported in part by the EPSRC (grant no. EP/V048597/1). YH is
supported by a Turing AI Fellowship funded by the UKRI (grant no.
EP/V020579/2).

## References

  * Askari, Verberne, and Pasi (2022) Askari, A.; Verberne, S.; and Pasi, G. 2022.  Expert Finding in Legal Community Question Answering.  _arXiv preprint arXiv:2201.07667_.
  * Balog, Azzopardi, and de Rijke (2009) Balog, K.; Azzopardi, L.; and de Rijke, M. 2009.  A language modeling framework for expert finding.  _Information Processing & Management_, 45(1): 1–19.
  * Bell (1991) Bell, A. 1991.  _The language of news media_.  Blackwell Oxford.
  * Burton et al. (2011) Burton, K.; Java, A.; Soboroff, I.; et al. 2011.  The ICWSM 2011 Spinn3r Dataset.  In _In Proceedings of the Fifth Annual Conference on Weblogs and Social Media (ICWSM 2011)_.
  * Devlin et al. (2018) Devlin, J.; Chang, M.-W.; Lee, K.; and Toutanova, K. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_.
  * Elson and McKeown (2010) Elson, D. K.; and McKeown, K. R. 2010.  Automatic attribution of quoted speech in literary narrative.  In _Twenty-Fourth AAAI Conference on Artificial Intelligence_.
  * Faisal, Daud, and Akram (2017) Faisal, M.; Daud, A.; and Akram, A. 2017.  Expert Ranking using Reputation and Answer Quality of Co-existing Users.  _International Arab Journal of Information Technology (IAJIT)_ , 14(1).
  * Fernandes, Motta, and Milidiú (2011) Fernandes, W. P. D.; Motta, E.; and Milidiú, R. L. 2011.  Quotation extraction for portuguese.  In _Proceedings of the 8th Brazilian Symposium in Information and Human Language Technology_.
  * Guo, Schlichtkrull, and Vlachos (2022) Guo, Z.; Schlichtkrull, M.; and Vlachos, A. 2022.  A survey on automated fact-checking.  _Transactions of the Association for Computational Linguistics_ , 10: 178–206.
  * Husain et al. (2019) Husain, O.; Salim, N.; Alias, R. A.; Abdelsalam, S.; and Hassan, A. 2019.  Expert finding systems: A systematic review.  _Applied Sciences_ , 9(20): 4250.
  * Lazarski, Al-Khassaweneh, and Howard (2021) Lazarski, E.; Al-Khassaweneh, M.; and Howard, C. 2021.  Using nlp for fact checking: A survey.  _Designs_ , 5(3): 42.
  * Lee and Yeung (2016) Lee, J.; and Yeung, C. Y. 2016.  An Annotated Corpus of Direct Speech.  In _Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16)_ , 1059–1063. Portorož, Slovenia: European Language Resources Association (ELRA).
  * Manning et al. (2014) Manning, C.; Surdeanu, M.; Bauer, J.; Finkel, J.; Bethard, S.; and McClosky, D. 2014\.  The Stanford CoreNLP Natural Language Processing Toolkit.  In _Proceedings of 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations_ , 55–60. Baltimore, Maryland: Association for Computational Linguistics.
  * Neshati, Beigy, and Hiemstra (2014) Neshati, M.; Beigy, H.; and Hiemstra, D. 2014.  Expert group formation using facility location analysis.  _Information processing & management_, 50(2): 361–383.
  * O’Keefe et al. (2012) O’Keefe, T.; Pareti, S.; Curran, J. R.; Koprinska, I.; and Honnibal, M. 2012.  A sequence labelling approach to quote attribution.  In _Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning_ , 790–799.
  * Pareti (2016) Pareti, S. 2016.  PARC 3.0: A corpus of attribution relations.  In _Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16)_ , 3914–3920.
  * Paul (2016) Paul, S. A. 2016.  Find an expert: Designing expert selection interfaces for formal help-giving.  In _Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems_ , 3038–3048.
  * Procter et al. (2023) Procter, R.; Arana-catania, M.; He, Y.; Liakata, M.; Zubiaga, A.; Kochkina, E.; and Zhao, R. 2023.  Some Observations on Fact Checking Work with Implications for Computational Support.  In _Proceedings of the International AAAI Conference on Web and Social Media_. AAAI Press.
  * Rajpurkar et al. (2016) Rajpurkar, P.; Zhang, J.; Lopyrev, K.; and Liang, P. 2016.  SQuAD: 100,000+ Questions for Machine Comprehension of Text.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , 2383–2392. Austin, Texas: Association for Computational Linguistics.
  * Rücklé, Moosavi, and Gurevych (2019) Rücklé, A.; Moosavi, N. S.; and Gurevych, I. 2019.  Neural Duplicate Question Detection without Labeled Training Data.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , 1607–1617. Hong Kong, China: Association for Computational Linguistics.
  * Semino and Short (2004) Semino, E.; and Short, M. 2004.  _Corpus stylistics: Speech, writing and thought presentation in a corpus of English writing_.  Routledge.
  * Shi and Lin (2019) Shi, P.; and Lin, J. 2019.  Simple bert models for relation extraction and semantic role labeling.  _arXiv preprint arXiv:1904.05255_.
  * Silva (2014) Silva, A. T. P. 2014.  _A research analytics framework for expert recommendation in research social networks_.  Ph.D. thesis, City University of Hong Kong.
  * Silva et al. (2013) Silva, T.; Guo, Z.; Ma, J.; Jiang, H.; and Chen, H. 2013.  A social network-empowered research analytics framework for project selection.  _Decision Support Systems_ , 55(4): 957–968.
  * Sun et al. (2015) Sun, J.; Xu, W.; Ma, J.; and Sun, J. 2015.  Leverage RAF to find domain experts on research social network services: A big data analytics methodology with MapReduce framework.  _International Journal of Production Economics_ , 165: 185–193.
  * Tekin, Atan, and Van Der Schaar (2014) Tekin, C.; Atan, O.; and Van Der Schaar, M. 2014.  Discover the expert: Context-adaptive expert selection for medical diagnosis.  _IEEE transactions on emerging topics in computing_ , 3(2): 220–234.
  * Vaucher et al. (2021) Vaucher, T.; Spitz, A.; Catasta, M.; and West, R. 2021.  Quotebank: a corpus of quotations from a decade of news.  In _Proceedings of the 14th ACM International Conference on Web Search and Data Mining_ , 328–336.
  * Wang et al. (2017) Wang, Q.; Ma, J.; Liao, X.; and Du, W. 2017.  A context-aware researcher recommendation system for university-industry collaboration on R&D projects.  _Decision Support Systems_ , 103: 46–57.
  * Yuan et al. (2020) Yuan, S.; Zhang, Y.; Tang, J.; Hall, W.; and Cabotà, J. B. 2020.  Expert finding in community question answering: a review.  _Artificial Intelligence Review_ , 53(2): 843–874.
  * Zeng, Abumansour, and Zubiaga (2021) Zeng, X.; Abumansour, A. S.; and Zubiaga, A. 2021.  Automated fact-checking: A survey.  _Language and Linguistics Compass_ , 15(10): e12438.
  * Zhang, Black, and Sproat (2003) Zhang, J. Y.; Black, A. W.; and Sproat, R. 2003.  Identifying speakers in children’s stories for speech synthesis.  In _Eighth European Conference on Speech Communication and Technology_.
  * Zhang and Liu (2021) Zhang, Y.; and Liu, Y. 2021.  DirectQuote: A Dataset for Direct Quotation Extraction and Attribution in News Articles.  _arXiv preprint arXiv:2110.07827_.
  * Zhao et al. (2023) Zhao, R.; Arana-Catania, M.; Zhu, L.; Kochkina, E.; Gui, L.; Zubiaga, A.; Procter, R.; Liakata, M.; and He, Y. 2023.  PANACEA: An Automated Misinformation Detection System on COVID-19.  _arXiv preprint arXiv:2303.01241_.

(a) Top frequent news article categories. The count is the number of articles
in the corresponding category.

(b) Top news sources. The count is the number of articles published by the
corresponding news source.

(c) Top frequent sources in our quote-source pair dataset. The count is the
number of quotations came from the corresponding sources.

Figure A1:  Noise Type |  Example Text
---|---
Jumble text |  -=-=-=- +++lead-in-text Last August, Apple announced that it would [distribute special iPhones](https://www.
Incorrect labeling of the quote (only text in bold is marked as quote) |  Furthermore, CBS reported, citing current and former diplomats with insight into the situation, that Gunter since his nomination in May 2019 has created an increasingly ”untenable” working environment by ”flying into a rage” and changing deputy chiefs of mission at will.
Improper trigger verb |  ”These are warnings that have been inevitable from the very start and exactly the reason why ICE should have, and should continue to, release people, especially those who are medically vulnerable to COVID-19, to prevent a humanitarian disaster,” she said.
Improper source |  We are in professional corporate relations with various companies and this helps us in digging out market data that helps us generate accurate research data tables and confirms utmost accuracy in our market forecasting.
Not an affirmative statement |  Apple didn’t respond to a request for comment.
Table A1: Types of noisy samples removed from the test set.

##  Appendix A Data Statistics

Figure 1(a) presents 25 of the most frequent news article categories. One
average, each article has 3.92 category labels. The top five categories are:
’Personal Finance’, ’Stocks’, ’Business’, ’Law, Gov’t & Politics’, and
’Travel’.

Figure 1(b) shows 25 of the most common news sources. In our dataset, the
total number of source platforms is 258. The top 5 source platforms are: Daily
Mail, Yahoo, Seeking Alpha, Business Insider, Reuters.

Figure 1(c) lists 25 of the most frequent sources. in total we have 3246
sources. The top 5 sources are: Amazon, Apple, Google, Microsoft, Reuters.

|  |  | Strict Metrics | Relaxed Metrics
---|---|---|---|---
Document | Query | Method | MAP | NDCG5 | NDCG10 | MAP | NDCG5 | NDCG10
|  | DRsparse | 0.2903 | 0.2807 | 0.3590 | 0.4162 | 0.3925 | 0.5183
|  | DRflat | 0.1481 | 0.1440 | 0.1939 | 0.2886 | 0.2714 | 0.3887
|  | DRhnswpq | 0.1509 | 0.1473 | 0.1926 | 0.2966 | 0.2805 | 0.3956
Context | Keyword | DRhnsw | 0.1446 | 0.1406 | 0.1889 | 0.2865 | 0.2686 | 0.3850
|  | DRpq | 0.1395 | 0.1363 | 0.1838 | 0.2739 | 0.2583 | 0.3734
|  | ERcan | 0.1021 | 0.1106 | 0.1252 | 0.2306 | 0.2294 | 0.3135
|  | ERdoc | 0.1205 | 0.1281 | 0.1418 | 0.2465 | 0.2412 | 0.3285
|  | DRsparse | 0.1357 | 0.1367 | 0.1719 | 0.3112 | 0.2980 | 0.4048
|  | DRflat | 0.1321 | 0.1304 | 0.1670 | 0.2979 | 0.2800 | 0.3917
|  | DRhnswpq | 0.1206 | 0.1177 | 0.1538 | 0.2830 | 0.2659 | 0.3790
Context | Title | DRhnsw | 0.1311 | 0.1299 | 0.1659 | 0.2963 | 0.2787 | 0.3902
|  | DRpq | 0.1231 | 0.1223 | 0.1570 | 0.2931 | 0.2797 | 0.3873
|  | ERcan | 0.1062 | 0.1116 | 0.1289 | 0.2516 | 0.2485 | 0.3366
|  | ERdoc | 0.1111 | 0.1183 | 0.1320 | 0.2532 | 0.2518 | 0.3387
|  | DRsparse | 0.1498 | 0.1484 | 0.1884 | 0.3285 | 0.3115 | 0.4297
|  | DRflat | 0.1435 | 0.1389 | 0.1818 | 0.3125 | 0.2945 | 0.4089
|  | DRhnswpq | 0.1285 | 0.1254 | 0.1648 | 0.2969 | 0.2761 | 0.3953
Context | Summary | DRhnsw | 0.1419 | 0.1373 | 0.1803 | 0.3097 | 0.2907 | 0.4056
|  | DRpq | 0.1402 | 0.1418 | 0.1773 | 0.3103 | 0.2953 | 0.4098
|  | ERcan | 0.0918 | 0.0966 | 0.1157 | 0.2461 | 0.2419 | 0.3337
|  | ERdoc | 0.1014 | 0.1069 | 0.1235 | 0.2570 | 0.2542 | 0.3479
|  | DRsparse | 0.1356 | 0.1366 | 0.1730 | 0.3059 | 0.2938 | 0.4074
|  | DRflat | 0.0721 | 0.0724 | 0.0999 | 0.2279 | 0.2127 | 0.3238
|  | DRhnswpq | 0.0756 | 0.0746 | 0.1027 | 0.2501 | 0.2410 | 0.3458
Quote | Keyword | DRhnsw | 0.0713 | 0.0719 | 0.0984 | 0.2266 | 0.2124 | 0.3214
|  | DRpq | 0.0692 | 0.0684 | 0.0931 | 0.2161 | 0.2036 | 0.3047
|  | ERcan | 0.0685 | 0.0720 | 0.0815 | 0.1902 | 0.1902 | 0.2633
|  | ERdoc | 0.0552 | 0.0588 | 0.0682 | 0.1685 | 0.1647 | 0.2404
|  | DRsparse | 0.0955 | 0.0956 | 0.1204 | 0.2713 | 0.2614 | 0.3600
|  | DRflat | 0.0953 | 0.0958 | 0.1227 | 0.2744 | 0.2611 | 0.3729
|  | DRhnswpq | 0.0861 | 0.0865 | 0.1113 | 0.2683 | 0.2526 | 0.3654
Quote | Title | DRhnsw | 0.0943 | 0.0946 | 0.1209 | 0.2751 | 0.2620 | 0.3724
|  | DRpq | 0.0934 | 0.0947 | 0.1188 | 0.2674 | 0.2565 | 0.3615
|  | ERcan | 0.0644 | 0.0669 | 0.0785 | 0.1951 | 0.1935 | 0.2725
|  | ERdoc | 0.0558 | 0.0582 | 0.0678 | 0.1851 | 0.1832 | 0.2598
|  | DRsparse | 0.1021 | 0.1037 | 0.1306 | 0.2837 | 0.2739 | 0.3784
|  | DRflat | 0.1058 | 0.1074 | 0.1338 | 0.2902 | 0.2771 | 0.3895
|  | DRhnswpq | 0.0941 | 0.0911 | 0.1258 | 0.2912 | 0.2718 | 0.3933
Quote | Summary | DRhnsw | 0.1038 | 0.1056 | 0.1309 | 0.2882 | 0.2756 | 0.3866
|  | DRpq | 0.1023 | 0.1013 | 0.1293 | 0.2882 | 0.2750 | 0.3854
|  | ERcan | 0.0568 | 0.0585 | 0.0717 | 0.2002 | 0.1958 | 0.2778
|  | ERdoc | 0.0457 | 0.0473 | 0.0569 | 0.1850 | 0.1799 | 0.2627

Table A2: DR denotes the document retrieval approach, and the subscripts
represent 5 types of retrieval indices mentioned in Section 5 Approach 1,
Lucene sparse bag-of-words index, Faiss flat index, Faiss HNSWPQ index, Faiss
HNSW index, and Faiss PQ index. ERcan is the candidate-based expert finding
approach, and ERdoc is the document-based expert finding approach. In two
expert finding approaches the input query length is set to 5.

##  Appendix B Noisy Sample

Five types of noise and their corresponding examples that appeared in the raw
test set are listed in Table A1.

##  Appendix C Results: Expert Recommendation

Table A2 shows the experimental results of expert recommendation. Among all 6
document-query combinations, the document retrieval (DR) approaches outperform
the expert retrieval (ER) approaches. Among various document indexing
strategies, the Lucene sparse bag-of-words index (DRsparse) gives better
results compared to other dense transformer-encoded Faiss indices. Averagely,
experiments perform better when we use the context as our documents. We
believe that adjacent contexts can also contain information about the sources
and their quotations.