# Astronomical Images Quality Assessment with Automated Machine Learning

Olivier Parisot, Pierrick Bruneau, Patrik Hitzelberger
Luxembourg Institute of Science and Technology (LIST)
5 Avenue des Hauts-Fourneaux, 4362 Esch-sur-Alzette - Luxembourg
<EMAIL_ADDRESS>

###### Abstract

Electronically Assisted Astronomy consists in capturing deep sky images with a
digital camera coupled to a telescope to display views of celestial objects
that would have been invisible through direct observation. This practice
generates a large quantity of data, which may then be enhanced with dedicated
image editing software after observation sessions. In this study, we show how
Image Quality Assessment can be useful for automatically rating astronomical
images, and we also develop a dedicated model by using Automated Machine
Learning.

##  1 INTRODUCTION

Nowadays, Electronically Assisted Astronomy (EAA) is widely applied by
astronomers to observe deep sky objects (nebulae, galaxies, star clusters). By
capturing raw images directly from a digital camera coupled to a telescope and
applying lightweight image processing (fast alignment and stacking), this
approach allows to generate enhanced views of deep sky targets that can be
displayed in near real time on a screen (laptop, tablet, smartphone) (Figure
1).

Figure 1: EAA setups used to capture data. The first one is a Stellina
automated station, the second one is a 72/420 apochromatic refractor
complemented by a low-end SVBONY SV305 digital camera – connected to a laptop
and driven by a dedicated software.

EAA also enables observing faint deep sky targets in difficult outdoor
conditions, for example in geographical zones heavily impacted by light
pollution or during a night with Moon (it considerably affects the sky
background, often making it difficult the observations). Celestial objects
like nebulae and galaxies are almost invisible through direct observation in
an urban or suburban night sky; with EAA they become impressive and detailed
[Parisot et al., 2022]. In practice, hundreds of targets can be imaged – they
are listed in well-known astronomical catalogs (Messier, New General Catalog
(NGC), Index catalog (IC), Sharpless, Barnard) and described many books and
software [Zack et al., 2018].

Thus, a large quantity of images are handled by astronomers during such EAA
sessions: the targets are numerous, the observation conditions variable, which
means that each image is different. Quality may depend on many parameters
[Redfern, 2020], among them:

  * •

Instrument: aperture and focal ratio, optical quality, digital camera
sensitivity and read noise, tracking mount precision.

  * •

Setup installation: balance and stability of tripod, focusing, collimation.

  * •

Seeing conditions: light pollution weather (clouds, fog, wind), moon phase,
steadiness and transparency of the atmosphere.

Some of these conditions may vary during the same night, meaning that acquired
data may have very heterogeneous quality levels.

During night capture sessions, it is possible to visualize _on live_ the
target by doing a lightweight processing: on-the-fly raw frames alignment and
stacking, then quick cosmetic processing (in general, histogram stretching).
This type of on-the-fly processing is performed by software such as SharpCap
111https://www.sharpcap.co.uk. At the beginning, the stacked image is very
noisy, then with the accumulation of raw images, this image will become more
qualitative.

After night capture sessions (generally days after), raw images are then
heavily post-processed in dedicated editing / post-processing tools, allowing
to mitigate most of the issues (noise, aberrations removal, blur) and to
enhance the signal (contrast stretching, color correction) [Bracken, 2017,
Adake, 2022]. The calibration images play an important role in this phase.
However, all these tools are very complex, and only very experienced users can
really estimate the quality of the final processed images and thus improve it
in a relevant way.

In this paper, we propose to combine Image Quality Assessment (IQA) and
Automated Machine Learning (AutoML) to automatically rate astronomical RGB
images: it aims at guiding EAA sessions and then images post-processing.

The rest of this article is organized as follows. Firstly, related works are
described (Section 2). Then, an approach with dataset preparation, model
training and a prototype are detailed (Section 3). Finally, preliminary
results on astronomical images are presented (Section 4) and discussed
(Section 5). We conclude by opening some perspectives (Section 6).

##  2 RELATED WORKS

###  2.1 IQA

Quality of astronomical images is traditionally estimated through two methods:

  * •

Signal-to-noise ratio (SNR): ratio of the strength of the astronomical signal
to the level of the noise in an image. A higher SNR indicates that the image
is of higher quality.

  * •

Full Width at Half Maximum (FWHM): sharpness of the point sources in an image,
such as stars. A smaller FWHM indicates that the image is of higher quality.

An other popular measure is the highest magnitude of the faintest star/object
visible in the image: it needs precise astrometry to do the comparison between
the image and the known deep sky objects and stars present in celestial
catalogues [Hogg et al., 2008].

Recently, numerous IQA generic approaches were developed [Zhai and Min, 2020].
In this paper, we focus on No-reference (NR) and Blind methods to rate single
RGB images; among them we can list:

  * •

BRISQUE, efficient on natural scenes: a score between 0 (good quality) and 100
(poor quality) is computed [Mittal et al., 2012]. It works well to filter
really bad images by using a fixed threshold (i.e. 70).

  * •

Deep Learning methods like NIMA (Neural Image Assessment): technical and
aesthetic scores between 0 (bad) and 10 (good) [Talebi and Milanfar, 2018]. In
practice, this score is not efficient on low-light images [Parisot and
Tamisier, 2022].

  * •

Deep CNN-Based Blind Image Quality Predictor (DIQA) methodology: two models
provide scores between 0 (bad) and 10 (good) [Kim et al., 2019].

Naively, these generic methods may be used to filter very bad astronomical
images by using a value threshold, but it is not efficient in practice. Let’s
take the example of two different images of the M17 nebulae [Parisot et al.,
2023] analyzed with two Python tools ( _image-quality_ package
222https://pypi.org/project/image-quality/ and NIMA tensorflow model
333https://tinyurl.com/idealo-iqa). In this typical case, the BRISQUE and NIMA
evaluations are slightly better for the first image. However, the second image
has a better overall quality, especially regarding contrast, luminance and
noise.

Figure 2: First image of M17 (aka Omega Nebula) - BRISQUE score is 28.25 and
NIMA score is 4.52. The image lacks contrast, the stars are mixed with noise
and the nebulosity is not visible. Figure 3: Second image of M17 - BRISQUE
score is 28.28 and NIMA score is 4.17. The contrast is good, stars and
nebulosity are clearly distinct from noise.

We can note that a recent IQA method based on clustering was proposed to deal
with ground-based astronomical images captured by professional surveys
[Teimoorinia et al., 2020].

###  2.2 AUTOML

AutoML consists in generating and deploying Machine Learning models from an
input dataset with little or no configuration and coding effort [Hutter et
al., 2019]. The growing application of Machine Learning in a wide range of
fields has led to the design of frameworks facilitating the production of
readily actionable models. Let us consider the traditional pipeline for
traditional Machine Learning tasks:

  * •

Data preprocessing is required to adjust raw data to the specificity of
Machine Learning algorithms [Zelaya, 2019]: cleansing, feature selection,
sampling, transformation, etc.

  * •

A Machine Learning model architecture is selected and then trained by using
the prepared data [Raschka, 2018].

  * •

Depending on the algorithm, some hyper-parameters have to be optimized to
improve the accuracy of the model; in general, this is realized through
heuristics requiring heavy computation [Feurer and Hutter, 2019].

  * •

Model accuracy is evaluated by computing standard statistical tests (AUC,
Precision, Recall, F1, etc.) with a given strategy (holdout or cross-
validation).

In practice, all those steps are time-consuming and exposed to methodological
errors. AutoML platforms aim at systematizing the whole process in order to
launch it a number of times with various combinations: numerous pipelines are
tested, the obtained models are evaluated and the _most accurate_ one is
finally selected [Raschka, 2018].

##  3 APPROACH

Our approach consists in providing an image regressor model to rate the
quality of RGB astronomical images obtained during EAA sessions. The model
aims at taking into account the following criteria: contrast/luminance, noise
and sharpness. To this end, we have designed a set of images associated to a
defined rating, and the task consists in training a model to fit with this
score definition. AutoML allows to test automatically a multitude of
combinations – leading to models with various characteristics: sizes (i.e.
parameters count and feature map size, etc.) and accuracies. Then, these
models are tested through a process based on specific data augmentation
process, in order to select the most robust model (test-time augmentation
[Shorten and Khoshgoftaar, 2019]).

###  3.1 DATA PREPARATION

We have built a dataset with deep sky images and an associated rate (between 0
and 10, from bad to good), in a similar way to what is done in the DIQA
methodology [Kim et al., 2019].

As original sources, we have used:

  * •

Galaxy10 DECals Dataset containing 17736 256x256 pixels RGB galaxy images
[Leung and Bovy, 2018].

  * •

Nebula Dataset containing 1657 high-resolution images extracted from Wikimedia
commons [Ravi, 2020].

Then, we have prepared a set of ideal images, obtained after a long manual
treatment of initial images with different editing software that are efficient
to improve astronomical images: Siril, TopazLabs [Redfern, 2020]. For each
ideal image, a rating of 10 was assigned. We have produced a set of
transformed images by applying random degradations to modify noise level
(adding Gaussian & Poisson noise), sharpness (blurring, deforming stars) and
luminance/contrast (adding background level and gradient, reducing signal,
degrading color saturation).

Each transformed image is then rated using a value between 0 (bad quality) an
10 (good quality) . To determine a value, we have evaluated the impacts of
distortions on contrast/luminance, noise and sharpness – by comparing the
transformed image with the ideal image.

  * •

Contrast/luminance: we used Structural Similarity Index Measure (SSIM) because
it is efficient to compare difference of contrast and luminance [Aliakhmet and
James, 2019]: 1 is given to a perfectly similar image and 0 indicates no
similarity.

  * •

Noise: we used the normalized Noise Variance difference between the ideal and
the transformed image. Noise Variance is estimated through the Fast Estimation
method.

  * •

Sharpness: we used the normalized FWHM difference between the ideal and the
transformed image. FWHM is estimated through an heuristic based on stars
detection 444https://tinyurl.com/starsfinder.

We have defined the final rating associated to the transformed image as:

$10*(r_{con}-max(r_{noi},A)-max(r_{sha},B)$

This formula ensures that each starting criterion is taken into account in the
rating – the interest being to have an index to compare the images as a whole,
whatever the defect. The value $A$ and $B$ acts as maximum _malus_ associated
to noise and lack of sharpness – in practice we have empirically used 4 as
value for this two constants.

Thus, the constructed dataset contains several thousands of images of
different qualities and associated ratings, for instance: good (Figure 4),
medium-quality (Figure 5), very bad (Figure 6).

The image resolution (256x256) is a good compromise because it corresponds to
a standard amount of data for recent Deep Learning model architectures.

Figure 4: A good-quality 256x256 RGB image of Helix Nebula (NGC7293) – rating:
9. The contrast is good, the noise level is low. Figure 5: A 256x256 RGB image
with moderated quality of the Andromeda galaxy (M31) – rating: 6. Slight noise
and blur have been added to degrade the original image. Figure 6: A poor-
quality 256x256 RGB image of Orion Nebula (M42) in the built dataset – rating:
2. Background noise and strong motion blur have been added to degrade the
original image.

In the next section, we show how this dataset is then used to train a IQA
model able to rate astronomical RGB images.

###  3.2 TRAINING

To run AutoML, a Python prototype was implemented. Image processing is
realized with well-known open-source packages like _openCV_
555https://pypi.org/project/opencv-python/ and _scikit-images_
666https://pypi.org/project/scikit-image/.

We have used AutoKeras – an open source Python package based on Bayesian
optimization [Jin et al., 2023]. AutoKeras aims at building and fine-tuning
Machine Learning and Deep Learning models by only defining inputs and expected
outputs. Other AutoML solutions exist (like TPOT [Olson and Moore, 2016]), but
AutoKeras provides a native support for Image Regression models.

Behind the scenes, AutoKeras generates and launches numerous predefined
training pipelines on-the-fly (model architecture selection, preprocessing,
hyper-parameters setting, training, model evaluation and comparison). it will
notably check if normalization step of the input data will allow to obtain a
better model. It will also test a whole list of hyper-parameters such as drop-
our rate, activation function (ex: Relu, Sigmoid), optimization algorithm (ex:
ADAM, RMSprop), learning rate, etc. AutoKeras does not work randomly to find
the best configuration: it tests a number of pre-defined pipelines, then
refines the best configuration by making small mutations (as an evolutionary
algorithm would do) [Jin et al., 2023]. Step by step, the pipeline producing
the best model is thus refined up to a user-defined limit (i.e. number of
trials).

The computations were executed on a computing infrastructure with the
following hardware capabilities: 40 cores and 128 GB RAM (Intel(R) Xeon(R)
Silver 4210 CPU @ 2.20GHz) and NVIDIA Tesla V100-PCIE-32GB. CUDA
777https://developer.nvidia.com/cuda-zone and NUMBA 888http://numba.pydata.org
frameworks have been used in background to optimize the hardware usage during
images treatment(CPUs and GPUs).

After numerous experiments, we have run thousands of different pipelines by
combining variations of data preprocessing, optimizers usage, different hyper-
parameters values (Figure 7).

Figure 7: Console dashboard of AutoKeras during the executions of numerous
training pipelines. It is thus possible to monitor the execution of the
pipelines and to see which is the best model at time _t_.

Two models were shown to be worthy of interest after all the calculations:

  * •

The best one leds to a model embedding a _ResNet50_ model [He et al., 2016],
with 23 millions trainable parameters and 53 120 non-trainable parameters –
obtained with a ADAM optimizer (learning rate of 0.01). Its accuracy is 1.19
(Mean Squared Error, MSE) on the test dataset. The R squared value is 0.4 –
which is rather satisfying because we have here a regression on images.

  * •

The smaller model is based on _EfficientNetB1_ with 6.589.337 parameters [Tan
and Le, 2019] – obtained with a ADAM optimizer (learning rate of 0.00001). The
model accuracy was worse (MSE 1.25).

To observe the robustness of these two trained models on realistic data, we
tested them on a augmented test dataset – on images having additional
distortions – not present in the training set.

###  3.3 MODEL SELECTION ON AN AUGMENTED TEST SET

To obtain an additional large and realistic test dataset, we wrote several
Python scripts for realistic image augmentation, i.e. to reproduce defects
that are frequently found in astronomical images. For example, we have added
different types of noises (Gaussian, Poisson, Salt and Pepper), we have merged
the real signal with a realistic sky background [Bradley et al., 2016]. We
also have blurred images in a sophisticated way by using image augmentation
techniques based on Deep Learning – especially for motion blur [Jung, 2019].
In a similar way, we generated starless versions of the images, in order to
test the robustness of the model – this was realized with a dedicated Deep
Learning model 999https://www.starnetastro.com.

Both models are globally able to reproduce the defined rating – to make an
estimation between images with and without defects. However the IQA model
based on ResNet50 provides much better results, especially on the ratings of
very good or very bad images (around 0 or around 10).

Figure 8: Regression plot of expected (x axis) and computed ResNet50 IQA ratings (y axis) obtained on a part of the augmented dataset (i.e. 100 images). Table 1: Rating obtained with the best ResNet50 IQA model on a augmented test dataset built from 500 different 256x256 RGB images. The (mean, standard deviation, minimum, maximum) outputs of the model are given for each images set.  | ResNet50 ratings
---|---
| mean | std | min | max
No distortion | 9.2 | 0.6 | 3.6 | 10
Distortion | 3.6 | 0.9 | 1.1 | 6.9
Strong distortion | 0.7 | 0.6 | 0 | 3.6
Table 2: Rating obtained with the best EfficientNetB1 IQA model on a augmented test dataset built from 500 different 256x256 RGB images. The (mean, standard deviation, minimum, maximum) outputs of the IQA model are given for each images set.  | EfficientNetB1 ratings
---|---
| mean | std | min | max
No distortion | 8.9 | 1.8 | 1.2 | 10
Distortion | 5.5 | 2.8 | 0.7 | 10
Strong distortion | 2.9 | 3.2 | 0 | 10

Certainly, there are more powerful classification architectures than ResNet50
and EfficientNet. Nevertheless, AutoML allowed us to obtain optimized models
for the use-case presented in this paper.

In the next section, we detail the efficiency of the ResNet50 IQA model on
live stacked images captured during EAA sessions.

##  4 EXPERIMENTS

The model was then tested on data captured during EAA sessions with two
setups:

  * •

300 live stacked images obtained from 10 seconds sub-frames, with short total
integration time (from 20 to 30 minutes) by using a Stellina observation
station [Parisot et al., 2023]. Images correspond to different deep sky
objects (example: Messier 31, NGC4565, etc.).

  * •

100 live stacked images obtained from 5 seconds sub-frames, with short total
integration time (from 20 to 30 minutes) by using a 72/420 refractor and a
dedicated low-end astronomical digital
camera(101010https://tinyurl.com/sv305).

All these RGB images have a high resolution (3096x2048 for the first set,
2048x2048 for the second set). As the input of our IQA model are 256x256 RGB
images, all of them were split into patches and the overall IQA rating is the
mean of patch ratings (no overlap).

Table 3: Evaluation of live stacked images captured during EAA sessions with two setups: a Stellina observation station (300 images) and a 72/400 refractor coupled to a low-end digital camera (100 images). The (mean, standard deviation, minimum, maximum) outputs of the ResNet50 IQA model are listed. | ResNet50 IQA rating
---|---
| mean | std | min | max
Stellina | 5.8 | 1 | 2.6 | 9
72/400 refractor | 1.3 | 0.7 | 0.6 | 3.1

The computed ratings are representative of the image sets: in practice, the
Stellina observation station provides much better quality images than the
other setup (Table 3). They are useful to compare images too: the second M17
nebula image presented in Figure 2 has a rating which is 50% higher than the
rating of the first image (Figure 3).

Figure 9: Live stacked image of the M33 galaxy obtained with a SV305 camera
and a 72/400 refractor: high noise, malformed stars and insufficient contrast.
The ResNet50 IQA model rating is 1.49. Figure 10: Live stacked image of
Pleiades (M45) captured with a Stellina automated station: with little noise
and stars with a punctual appearance. The ResNet50 IQA model rating is 7.1 .

One of the interests of the approach is to measure the quality of the stacked
image obtained during a EAA capture session. In theory, an astronomical image
improves when accumulating integration time (i.e. by collecting as much data
as possible). Capturing data is time-consuming and sometimes challenging
(especially due to weather conditions), and the IQA estimation on the live
stacked image may help to capture only what is necessary to have a stacked
image with the desired quality (Figure 11). As the number of raw images
increases, it becomes more difficult to increase the quality of the stacked
image. Let’s take the example of the observation of NGC1499 with a Stellina
station: 711 images of 10 seconds of exposure time each
111111https://youtu.be/BTURaF9dTIU. The graph shows that the quality increases
strongly at the beginning of the capture, and less afterwards (Figure 11). The
score can thus be used to determine if it is still relevant to continue data
acquisition.

Figure 11: Evolution of IQA score during a EAA session for observing the
California Nebula (NGC1499). The more raw images accumulated, the better the
final stacked image quality.

Out of curiosity, we have tested the IQA model on two extreme cases of well-
known astronomical pictures. Although not typical use cases of our approach,
it gives us an overview of what our model produces on different images:

  * •

First Andromeda image captured by Isaac Roberts in (1899)
121212https://commons.wikimedia.org/wiki/File:Pic˙iroberts1.jpg: our IQA model
provides a rating of 2.56. Even if the image is incredible for its time, it is
far from the standards expected today: a poor score is logic.

  * •

Webb’s First Deep Field – a long-exposure image of the SMACS0723 galaxy
cluster captured vy the James Webb Space Telescope
131313https://tinyurl.com/webbdf: our IQA model computes a rating of 7.2 for
the part without the JWST’s typical diffraction spikes [Rigby et al., 2022].
The rating drops to 4.9 with these spikes because the model seems to consider
them as defects.

##  5 DISCUSSION

With our approach, we can observe that the image defects are really penalized
by the IQA model as we have chosen to mix potential distortions into the
training process. There is a drawback: this method is not able to list, grade
and locate precisely the defects that are most present in an image, and for
this we will have to go further (for instance, by combining with object
detection).

Another point concerns the execution time of the IQA model on large images.
Even if training is realized on a high performance computing platform, the
models use should be possible on normal computers with modest capabilities –
especially for on-the-fly evaluation of stacked images.

Let’s take the example of a 4096x4096 astronomical image: with no overlap, we
may need to evaluate 256 256x256 patches – it may take some time depending of
the hardware. To be efficient, one must try to minimize the number of
calculations required. In a pragmatic way, the following strategies can be
applied:

  * •

Decrease the resolution of the image to reduce the count of patches to
evaluate.

  * •

Estimate the IQA rating of a small relevant subset of patches – for instance
by ignoring dark patches or patches with low signal.

During our experiments, we have concluded that the second one provides better
results. Further performance optimizations will be realized after deep
analysis of model execution with dedicated tools [Jin and Finkel, 2020].

##  6 CONCLUSION

This paper presented an approach to automatically estimate the quality of
astronomical RGB images. A dedicated model was trained by using Automated
Machine Learning, and then tested on various image sets captured during
Electronically Assisted Astronomy sessions. A Python prototype was presented
and preliminary results were discussed. In future works, we will extend the
approach building on these current results to design and train additional
models that are both more sophisticated and interpretable, by relying on
eXplainable Artificial Intelligence.

Acknowledgments: This research was funded by the Luxembourg National Research
Fund (FNR), grant reference 15872557. Tests were realized on the LIST AIDA
platform, thanks to Raynald Jadoul and Jean-François Merche.

## REFERENCES

  * Adake, 2022 Adake, N. (2022).  Trust the process: An investigation into astrophotography.  In AIAA Southeastern Regional Student Conference.
  * Aliakhmet and James, 2019 Aliakhmet, K. and James, A. P. (2019).  Temporal g-neighbor filtering for analog domain noise reduction in astronomical videos.  IEEE Trans. on Circuits and Systems II: Express Briefs, 66(5):868–872.
  * Bracken, 2017 Bracken, C. (2017).  The deep-sky imaging primer.  Deep-sky Publishing.
  * Bradley et al., 2016 Bradley, L., Sipocz, B., Robitaille, T., Tollerud, E., Deil, C., Vinícius, Z., Barbary, K., Günther, H. M., Bostroem, A., Droettboom, M., et al. (2016).  Photutils: Photometry tools.  Astrophysics Source Code Library, pages ascl–1609.
  * Feurer and Hutter, 2019 Feurer, M. and Hutter, F. (2019).  Hyperparameter optimization.  In Automated Machine Learning, pages 3–33. Springer, Cham.
  * He et al., 2016 He, K., Zhang, X., Ren, S., and Sun, J. (2016).  Deep residual learning for image recognition.  In IEEE CVPR 2016, pages 770–778.
  * Hogg et al., 2008 Hogg, D. W., Blanton, M., Lang, D., Mierle, K., and Roweis, S. (2008).  Automated astrometry.  In Astronomical Data Analysis Software and Systems XVII, volume 394, page 27.
  * Hutter et al., 2019 Hutter, F., Kotthoff, L., and Vanschoren, J. (2019).  Automated machine learning: methods, systems, challenges.  Springer Nature.
  * Jin et al., 2023 Jin, H., Chollet, F., Song, Q., and Hu, X. (2023).  Autokeras: An automl library for deep learning.  Journal of Machine Learning Research, 24(6):1–6.
  * Jin and Finkel, 2020 Jin, Z. and Finkel, H. (2020).  Analyzing deep learning model inferences for image classification using OpenVINO.  In IPDPSW 2020, pages 908–911.
  * Jung, 2019 Jung, A. (2019).  Imgaug documentation.  Readthedocs. io, Jun, 25.
  * Kim et al., 2019 Kim, J., Nguyen, A.-D., and Lee, S. (2019).  Deep cnn-based blind image quality predictor.  IEEE Trans. on Neural Networks and Learning Systems, 30(1):11–24.
  * Leung and Bovy, 2018 Leung, H. W. and Bovy, J. (2018).  Deep learning of multi-element abundances from high-resolution spectroscopic data.  Monthly Notices of the Royal Astronomical Society.
  * Mittal et al., 2012 Mittal, A., Moorthy, A. K., and Bovik, A. C. (2012).  No-reference image quality assessment in the spatial domain.  IEEE Trans. on image processing, 21(12):4695–4708.
  * Olson and Moore, 2016 Olson, R. S. and Moore, J. H. (2016).  TPOT: A tree-based pipeline optimization tool for automating machine learning.  In Workshop on automatic machine learning, pages 66–74. PMLR.
  * Parisot et al., 2022 Parisot, O., Bruneau, P., Hitzelberger, P., Krebs, G., and Destruel, C. (2022).  Improving accessibility for deep sky observation.  ERCIM News, 2022(130).
  * Parisot et al., 2023 Parisot, O., Hitzelberger, P., Bruneau, P., Krebs, G., Destruel, C., and Vandame, B. (2023).  MILAN Sky Survey, a dataset of raw deep sky images captured during one year with a Stellina automated telescope.  Data in Brief, 48:109133.
  * Parisot and Tamisier, 2022 Parisot, O. and Tamisier, T. (2022).  Applying genetic algorithm and image quality assessment for reproducible processing of low-light images.  In IMPROVE 2022, pages 189–194.
  * Raschka, 2018 Raschka, S. (2018).  Model evaluation, model selection, and algorithm selection in machine learning.
  * Ravi, 2020 Ravi, A. (2020).  Nebula images.
  * Redfern, 2020 Redfern, G. I. (2020).  Astrophotography is Easy!: Basics for Beginners.  Springer.
  * Rigby et al., 2022 Rigby, J., Perrin, M., McElwain, M., Kimble, R., Friedman, S., Lallo, M., Doyon, R., Feinberg, L., Ferruit, P., Glasse, A., et al. (2022).  Characterization of jwst science performance from commissioning.  arXiv preprint arXiv:2207.05632.
  * Shorten and Khoshgoftaar, 2019 Shorten, C. and Khoshgoftaar, T. M. (2019).  A survey on image data augmentation for deep learning.  Journal of big data, 6(1):1–48.
  * Talebi and Milanfar, 2018 Talebi, H. and Milanfar, P. (2018).  Nima: Neural image assessment.  IEEE Trans. on Image Processing, 27(8):3998–4011.
  * Tan and Le, 2019 Tan, M. and Le, Q. (2019).  Efficientnet: Rethinking model scaling for convolutional neural networks.  In ICML, pages 6105–6114. PMLR.
  * Teimoorinia et al., 2020 Teimoorinia, H., Kavelaars, J., Gwyn, S., Durand, D., Rolston, K., and Ouellette, A. (2020).  Assessment of astronomical images using combined machine-learning models.  The Astronomical Journal, 159(4):170.
  * Zack et al., 2018 Zack, M., Gannon, A., and McRoberts, J. (2018).  Software and apps to help the suburban astronomer.  In Stargazing Under Suburban Skies, pages 341–350. Springer.
  * Zelaya, 2019 Zelaya, C. V. G. (2019).  Towards explaining the effects of data preprocessing on machine learning.  In ICDE 2019, pages 2086–2090. IEEE.
  * Zhai and Min, 2020 Zhai, G. and Min, X. (2020).  Perceptual image quality assessment: a survey.  Science China Information Sciences, 63(11):1–52.


# ArgFuse: A Weakly-Supervised Framework for Document-Level Event Argument
Aggregation

Debanjana Kar<EMAIL_ADDRESS>Sudeshna Sarkar Pawan Goyal

###### Abstract

Most of the existing information extraction frameworks Wadden et al. (2019);
Veyseh et al. (2020) focus on sentence-level tasks and are hardly able to
capture the consolidated information from a given document. In our endeavour
to generate precise document-level information frames from lengthy textual
records, we introduce the task of Information Aggregation or Argument
Aggregation. More specifically, our aim is to filter irrelevant and redundant
argument mentions that were extracted at a sentence level and render a
document level information frame. Majority of the existing works have been
observed to resolve related tasks of document-level event argument extraction
Yang et al. (2018a); Zheng et al. (2019a) and salient entity identification
Jain et al. (2020) using supervised techniques. To remove dependency from
large amounts of labeled data , we explore the task of information aggregation
using weakly-supervised techniques. In particular, we present an extractive
algorithm with multiple sieves which adopts active learning strategies to work
efficiently in low-resource settings. For this task, we have annotated our own
test dataset comprising of 131 document information frames and have released
the code and dataset to further research prospects in this new domain. To the
best of our knowledge, we are the first to establish baseline results for this
task in English. Our data and code are publicly available at
https://github.com/DebanjanaKar/ArgFuse

##  1 Introduction

Extraction of event argument information at a document level is an important
non-trivial task that requires a system to have advanced natural language
understanding capabilities. Most of the existing event-argument extraction
systems Nguyen et al. (2016); Luan et al. (2019a); Wadden et al. (2019);
Veyseh et al. (2020) pertain to a sentence-level focus, often circumventing to
capture information at a document-level. Among the few existing works that
have researched the task of document-level event argument extraction, we
observe that unsupervised techniques like Hamborg et al. (2019) lack the
capacity to identify complex argument mentions, while sophisticated supervised
mechanisms like that of Yang et al. (2018b); Zheng et al. (2019b) rely on
large amounts of annotated corpus and present domain-specific solutions.

Figure 1: Example document excerpt from our corpus highlighting the different
challenges of the aggregation task. The phrases highlighted in red, blue and
green denote the redundant, irrelevant and exclusive sentence-level arguments
respectively. The document level arguments are as reported at the end of the
example.

While supervised techniques may often produce highly accurate systems, in real
life, annotating a large corpus can be both very expensive and time consuming.
In order to surmount the existing shortcomings coupled with the challenging
scenario of data scarcity, we propose our model ArgFuse. ArgFuse focuses on
extraction of relevant and non-redundant event arguments at a document level
scope. The task of document level event argument extraction typically focuses
on extracting argument mentions associated with an event type or event trigger
from a document. It does not involve checking extracted arguments for
irrelevant and redundant mentions. Through our work, we also propose the
related task of Argument Aggregation which focuses on assessing extracted
arguments for irrelevance and redundancy to produce a precise aggregated
document-level information frame. Figure 1 provides an illustrative example of
our task while highlighting the different challenges the task presents.
The task presents itself with multiple challenges and avenues to explore. To
produce a precise document-level information frame, we focus on sieving
irrelevant and redundant sentence-level argument mentions. Irrelevant argument
mentions, as illustrated in Figure 1 refer to argument mentions that do not
contribute to the topical focus of the document. A category of irrelevant
arguments often encountered in real life news articles are past event records
mentioned in narratives to provide a comparative perspective to the reader,
much like the arguments in blue mentioned in Figure 1. While irrelevant
arguments are usually mentions that refer to past, future or unrelated events;
redundancy in arguments manifests in diverse forms. Redundant arguments can
either be i) duplicate argument mentions (e.g. last week), ii) arguments with
similar surface form (e.g. Indonesia, Indonesia’s main Island Java), iii) re-
worded (e.g. killed 41 people, death toll more than doubles to 41), or iv)
subsuming information of the other argument(s) (e.g. killed 41 people, 25
people killed). While the first two types of redundancy can be tackled by
simple heuristics, the detection of the remaining types of redundancy requires
implicit natural language understanding and coreference reasoning
capabilities. To filter such arguments effectively, we realised that
contextual information of a document is imperative. The argument mentions in
green illustrated in Figure 1 highlight arguments which impart unique
information with respect to the context of the document. These argument
mentions cannot be aggregated and are directly added to the output information
frame. We refer to such argument mentions in our work as exclusive argument
mentions.
Based on the different types of arguments we encounter, we propose an
extractive algorithm that aggregates sentence-level argument or entity
mentions to produce precise document-level information frames from lengthy
text articles effectively. In our work, we present an end-to-end framework to
extract events and arguments from English news articles and present an
aggregated information frame at the document level. Given that we introduce a
novel task with no prior labeled dataset, we present a weakly-supervised
algorithm to achieve our task with a good accuracy. Our contributions in this
work are two fold:

  1. 1.

We propose a novel task of document-level event-argument aggregation and
establish baseline for the same. We also release the first annotated test
dataset for this task with 131 aggregated document information frames.

  2. 2.

We propose a weakly supervised model to aggregate event-arguments at a
document-level. We deeply analyse the task, dataset and algorithm proposed in
this paper, thus highlighting areas of future research and development.

In the following sections, we discuss our dataset, algorithms and findings in
detail. Our analysis emphasizes on the importance of having document level
information extraction frameworks for the task of argument aggregation and we
invite the research community to further investigate this task.

##  2 Related Work

Event argument extraction is a well researched information extraction task
which has seen a lot of work at the sentence-level Wang et al. (2019); Wadden
et al. (2019); Nguyen et al. (2016); Luan et al. (2019b); Veyseh et al. (2020)
but a scarce amount of research has been carried out at the document level.
Recent literature on event argument extraction at a document level include the
works of Yang et al. (2018a) and Zheng et al. (2019a). While Zheng et al.
(2019a)’s work explores supervised transformer based techniques to extract
events and arguments from Chinese financial documents, Yang et al. (2018a)
employs Bi-LSTM based classifiers on a subset of the same dataset to extract
events first at the sentence level followed by a document level extraction
similar to our framework. Jain et al. (2020) employs a BiLSTM-CRF classifier
to finetune SciBERTBeltagy et al. (2019) on various document-level information
extraction tasks including the related task of salient entity clustering. All
of these methods employ supervised techniques which call for a large corpus of
annotated dataset, making it difficult to adopt to domains and tasks with no
labelled annotations. Hamborg et al. (2019) presents an unsupervised approach
of extracting document level information from news documents, but the
heuristics adopted in their work do not extend well to our task which involves
more complex argument mentions. The limited amount of research that exists in
this domain does not explore the task of aggregation in particular, where,
given a set of arguments referring to the same concept, the most informative
argument is selected to represent that knowledge. We propose a novel task and
present an end-to-end baseline solution to extract and aggregate document-
level arguments which presents a complete overview of the document without
minimal loss of information. In our work, we employ ranking strategies as part
of our aggregation process. Some of the classic works related to the task of
ranking text snippets are that of PageRank Page et al. (1998) and TextRank
Mihalcea and Tarau (2004).

##  3 Dataset

One of the main challenges that we faced was the unavailability of annotated
resources for this task. For our auxiliary task of sentence level event
argument extraction, we use the dataset adopted by Kar et al. (2021). The
dataset is available in five Indian languages but for this task, we only use
the English dataset. The dataset covers 32 event types at a fine grain level
and 12 event types at a coarse level. The dataset contains annotations for 14
argument types, but in our work we focus on 6 main argument types which are,
Time, Place, Casualties, After-Effect, Reason and Participant. In the sections
to follow we discuss regarding the scarcely available document-level annotated
resources and the details of the annotated dataset we release with this work.

###  3.1 Existing Document-Level IE Datasets

Information Extraction (IE) is a well-researched domain albeit mostly at the
sentence-level. Event-Argument Extraction, the IE task most related to the
task of aggregation has a number of well-documented and reliable datasets
annotated at the sentence level in different languages like ACE 2005 and TAC
KBP Mitamura et al. (2015) datasets. IE tasks with a document-level focus have
gained attention in recent times but there are hardly any document-level event
argument annotated datasets. We discuss two recent works that include
document-level event argument or entity mention annotations here; the RAMS
Ebner et al. (2020) and the SciREX Jain et al. (2020) dataset. The RAMS
dataset is not particularly document-level, but explores the task of
extracting argument roles beyond sentences. In a 5-sentence window of a news
article around each event trigger, they annotate the closest span for each
argument role. Their ontology consists of 139 event types and 65 argument
roles. The SciREX dataset is a comprehensive dataset comprising of document-
level annotations for a variety of IE tasks. The dataset consists of
annotations for related tasks like entity recognition and coreference on 438
scientific articles. However, none of these datasets provide consolidated
argument annotations for an entire document. To the best of our knowledge, we
are the first to introduce a document-level event-argument annotated dataset
in English which provides an aggregated overview of the document, that is, the
first annotated dataset for the task of Event-Argument Aggregation in English.

###  3.2 ArgFuse Dataset

Most of our work employs weakly-supervised techniques for curation of document
level information frames but for sound testing of our final model we manually
aggregate document-level arguments for each argument type from the 131 English
documents in the test set of the above mentioned dataset. Each information
frame for a document contains the event-type and the corresponding relevant
arguments for each argument type from the document. During curation of the
test set, we followed certain annotation guidelines defined for each argument
type. The guidelines contain detailed instructions for identifying relevant
arguments at a document level. For example, if the Time arguments of a
document mention different degrees of temporal expressions like day, month and
hour of the day, all the arguments are to be considered as relevant and
aggregation is not required. The dataset was curated by two research scholars
with good domain knowledge. We report the statistics of our dataset in Figures
2 and 3. In figure 2 we can observe the amount of redundancy and irrelevance
prevalent in the extracted sentence-level information. In figure 3, we observe
that although a number of argument roles in a document constitute of a single
relevant argument mention (referred to as Singles), a significant number of
argument roles constitute of multiple number of relevant argument mentions
(referred to as Multiples). This highlights the fact that the number of
relevant arguments for a particular argument role or type can be flexible and
the model should be able to accommodate that flexibility. We release the
manually annotated test set along with the annotation guidelines to further
research prospects in this novel
task.111https://github.com/DebanjanaKar/ArgFuse

Figure 2: Distribution of sentence-level and document-level argument mentions
in the annotated ArgFuse dataset. In the figure, merged refers to document-
level annotations and unmerged refers to sentence-level annotations.  Figure
3: Distribution of single and multiple argument mentions among the document-
level arguments in the annotated ArgFuse dataset. Singles refer to a category
of argument roles which constitute of a single argument mention and Multiples
refer to a category of argument roles which constitute of more than one
relevant argument mentions at a document-level.

##  4 Information Aggregation

In this section, we detail the approaches that were taken to build a weakly-
supervised argument aggregation framework. The framework primarily involves
two main modules: i) Sentence-level Information Extraction (IE), which
extracts the sentence-level arguments along with their event type, the ii)
Aggregation Unit, which renders document-level information frames. A general
overview of the pipeline is illustrated in Figure 4. We explain each of these
modules in detail in the sections to follow.

Figure 4: General overview of the complete argument aggregation framework.

###  4.1 Event Argument Extraction

Given a document $D$ of event type $E$, the objective of this sub-task is to
extract the argument label sequence $(y_{1},...,y_{n})$ for the corresponding
word sequence $(x_{1},...,x_{n})$, $n$ being the number of tokens in $D$. For
example, for a given document: “The flood waters destroyed 500 homes in Assam
…”, the corresponding label sequence would be: ‘O O O AFTER_EFFECTS_ARG
AFTER_EFFECTS_ARG AFTER_EFFECTS_ARG O PLACE_ARG …’. To ensure high accuracy
and low error propagation, we have adopted Kar et al. (2021)’s approach of
sentence-level event argument extraction using causal knowledge structures for
this sub-task. Kar et al. (2021)’s approach provides state-of-the-art results
on the INDEE dataset Maheshwari et al. (2020) and ensures efficient extraction
of the low resource, complex causal arguments like Reason and After-Effects
using the specially designed causality feature. The causality feature for each
event consists of words and phrases which are used frequently in a causal
context for particular event scenarios. The input document, concatenated with
the feature at either extremes is encoded using a fine-tuned BERT encoder and
each token is ultimately classified to one of the six argument types in the TO
format (adopted from Maheshwari et al. (2020)). While we adopt Kar et al.
(2021)’s approach in our work, the event extraction module can be easily
substituted with more suitable models in the future thus leveraging the
modular nature of our algorithm. Using Kar et al. (2021)’s approach, we
extract the sentence-level event arguments for our corpus with an F1 score of
$86.12\%$.

###  4.2 Aggregation

The aggregation unit is the primary module which identifies the most relevant
and informative arguments from a pool of sentence-level argument mentions at a
document-level scope. As illustrated in Figure 4, the aggregation unit
consists of i) the Relevance Filter, which sieves the irrelevant arguments,
ii) the Argument Ranking module, which ranks the arguments based on their
informativeness and iii) the Redundancy Filter, which sieves the redundant
arguments at a document level. The detailed architecture of the aggregation
unit is illustrated in Figure 5.

####  4.2.1 Relevance Filter

Sentence-level IE outputs have often been observed to contain arguments that
are not relevant to the document’s focus. The main task of the Relevance
Filter unit is to sieve such arguments. Given the extracted sentence level
arguments of a particular argument type along with the context of its
constituent document, the relevance filter proceeds to classify each argument
mention as relevant or not. Since we did not possess labelled samples for this
task, we manually annotated 500 training and 100 test instances. We observe
that identifying irrelevant instances is relatively easier with explicit
contextual and syntactic cues. Hence, on fine-tuning a ROBERTA-based
classifier for our subtask,even with a very limited number of training
instances, we obtain an F1 score of $85\%$. The performance metrics of the
relevance filter is detailed in Table 1.

Figure 5: Detailed illustration of ArgFuse. This figure depicts the flow of
control during inference with ArgFuse. The spans $(a_{i}..a_{i}+j)$ in the
relevance and redundancy filters refer to argument mention spans within
sentences in the input document.

####  4.2.2 Redundancy Filter

Detection of redundant arguments is comparatively more challenging compared to
the sub-task of relevance detection. While certain groups of argument mentions
are explicitly redundant (eg. duplicate mentions, substrings), other groups of
redundant argument mentions are more implicit in nature. To effectively
identify redundant arguments in a low-resource setting, we employ active
learning strategies. Given a pair of arguments $(a_{i},a_{j})$ along with the
context of it’s constituent document, we train a binary classifier to maximize
$P(y|a_{i},a_{j})$, where $y=1$, if $(a_{i},a_{j})$ are redundant else $y=0$.
To train a binary classifier for such a task, a large annotated corpus would
have been effective but in the absence of such a corpus, we adopt the
effective technique of active learning with $1045$ manually annotated seed
instances. After each epoch of active learning, $50$ most uncertain samples
are identified using the Monte Carlo estimation of error reduction Roy and
McCallum (2001). These samples are then manually annotated and transferred
from the pool of unannotated test samples to the list of annotated training
instances. This process is repeated until we do not see any further
improvement in the F1 Score. Based on the findings reported in Hu et al.
(2018), to avoid bias from the previous epoch, we fine-tune the pre-trained
BERT-based classifier on the entire annotated dataset for every run of active
learning. Once we have the necessary annotations derived from the multi-epoch
active learning session, we train our binary classifier using all the
annotated samples for $15$ epochs. Our relevance filter is evaluated in Table
1.

###  4.3 Inference

The steps followed during inference are illustrated in Figure 5. The process
starts by employing the trained Relevance filter to segregate relevant
argument mentions from irrelevant ones. The relevant argument mentions are
then ranked based on their informativeness. Given a list of argument-mentions
$(arg_{1},arg_{2},...,arg_{m});m$ being the total number of mentions in the
list of type $t$, our objective is to rank the mentions based on which
argument instance imparts greater knowledge about the document’s event. To
compare the informativeness of the arguments, we rank the arguments using the
unsupervised Biased TextRank Kazemi et al. (2020). Biased TextRank is formally
defined as:

$\displaystyle R(V_{i})=BiasWeight*(1-d)+d*R^{{}^{\prime}}(V_{i})$
$\displaystyle R^{{}^{\prime}}(V_{i})={\sum\limits_{V_{j}\in
In(V_{i})}\dfrac{w_{ij}}{\sum\limits_{V_{k}\in Out(V_{j})}w_{jk}}*R(V_{j})}$

where, $R(V_{i})$ is the score assigned to the vertex $V_{i}$, $In(V_{i})$
denotes the incoming and $Out(V_{i})$ denotes the outgoing edges from the
vertex $V_{i}$. The damping factor $d$ is set to 0.85. In our task, each of
the arguments in the list correspond to a vertex and the vertices are
connected by a weighted edge. The weight of each edge is determined by the
cosine similarity between two arguments $(arg_{i},arg_{j})$. The bias weight
is determined by calculating the cosine similarity between the bias text and
the argument text. The bias texts comprise of short text snippets defined for
each argument type along with the document and it’s event type. Each of the
arguments and the bias texts are encoded using the tuned embeddings from the
redundancy filter. Higher the score of the argument (or vertex), more
informative is the argument. The ranked list contains the arguments sorted in
a non-decreasing order based on their obtained Biased TextRank scores.
Each argument from the ranked list of argument mentions is compared
sequentially with the other arguments using the Redundancy filter. If any pair
of arguments are classified redundant, the argument with lower score is
discarded and the process is continued. To reduce loss of information further,
we adopt the following rules:

  1. 1.

If an argument type contains a single argument mention extracted at the
sentence level, the argument is added to the document-level information frame
directly.

  2. 2.

If all the mentions in a list of arguments of argument type $t$ are classified
irrelevant or redundant rendering a null set, we add the argument with the
highest score from the list to the document-level information frame.

Once all the sentence-level arguments of a document have been processed
through the above described modules, a precise document-level information
frame is rendered.

##  5 Experiments & Results

In this section we shall detail the execution details of the experiments and
analyse the results obtained.

###  5.1 Experiments

We have experimented with different encodings and classifiers in our work. For
the Sentence Level Event-Argument Extraction task we have encoded the text
using Huggingface’s Wolf et al. (2020) bert-base-multilingual-cased model pre-
trained on 104 languages222https://huggingface.co/bert-base-multilingual-cased
Devlin et al. (2019). For encoding text in the relevance filter, the
pretrained roberta-base- model333https://huggingface.co/roberta-baseLiu et al.
(2019) was used while for the redundancy filter, the pretrained bert-base-
model444https://huggingface.co/bert-base-cased was used. The ROBERTA-based
relevance filter was trained for $3$ epochs on $500$ training samples while
the redundancy filter was trained for $15$ epochs after retrieving required
annotations from $5$ epochs of active learning. The batch size for our
experiments was $8$. All our experiments were performed on a Tesla K40-C
server.

###  5.2 Metrics

To analyse the generated document-level information frames and report the
performance of the designed framework, we have adopted Precision, Recall and
F1-Score as the metrics of our choice. To calculate the above mentioned
metrics we count TP, FP and FN as follows:

  * •

True Positive (TP): When the detected argument exists in the true argument
list.

  * •

False Positive (FP):  When detected argument does not exist in the true
argument list.

  * •

False Negative (FN):  When argument from true argument list is not among the
detected arguments.

Modules | Macro | Micro
---|---|---
P | R | F1 | F1
Relevance Filter | 81 | 85 | 83 | 85
Redundancy Filter | 69 | 71 | 70 | 70
Table 1: Module-wise performance measure using F1-scores (in %). Macro & Micro denote the averaging scheme adopted for these metrics. Models | P | R | F1
---|---|---|---
GiveMe5W1H | 25.13 | 25.17 | 25.15
TextRank @ k = 1 | 65.63 | 50.33 | 56.97
TextRank @ k = 2 | 66.59 | 51.18 | 57.87
Biased TRank @ k = 1 | 70.48 | 56.73 | 62.86
Biased TRank @ k = 2 | 60.48 | 68.91 | 64.42
ArgFuse | 67.98 | 64.90 | 66.40
Table 2: Comparison of our models with the defined baselines. We can observe
that our model reports the best performance compared to the other solutions.
Figure 6: Comparison of Argument-Wise F1-Scores. Figure 7: Comparison of
sentence-level outputs with reduced document-level machine and human outputs.
Highlighted phrases refer to the arguments that differ from the document-level
gold standard. Phrases in red are the Place arguments while the ones in Purple
and Blue are the Casualties and the Participant arguments respectively.

###  5.3 Results

We present our findings and results for each module in Table 1. For evaluating
the results of our complete framework, we first prepare our reference text to
which the machine output will be compared. Each of the manually curated
information frames are presented as a sentence in the reference text in a
newline. The presented sentence contains the aggregated arguments for each
argument type separated by the comma delimiter. We compare our final framework
with three baseline models:

  * •

GiveMe5W1H Hamborg et al. (2019): is an unsupervised approach for extracting
document level phrases related to the six 5W1H questions (what, where, when,
who, why, and how) from English News Articles. We map the six questions to our
argument types so that we could run the system on our dataset and evaluate.

  * •

TextRank Mihalcea and Tarau (2004): We use the graph based ranking algorithm
to rank the sentence-level argument phrases for each argument type in a
document extracted using Kar et al. (2021)’s model. We select the top-k
arguments as the representative arguments for that argument type from the
document.

  * •

Biased TRank: Similar to the above described baseline, but instead of TextRank
we have used Biased TextRank Kazemi et al. (2020) to rank the extracted
sentence level arguments directly.

The final results are reported in Table 2.

##  6 Analysis

Figure 8: Comparison among various baseline models and ArgFuse. The
highlighted arguments are the ones that match with the gold standard output.

In this section, we present a thorough analysis of our findings and the novel
task that we introduce. We investigate the contributions as well as the
pitfalls of our framework and attempt to provide directions for improvement.
In the sections to follow, we analyse our framework both quantitatively and
qualitatively.

###  6.1 Quantitative Analysis

In this section we shall scrutinize the performance of each module in our
framework as well as analyze the overall performance. The module wise
performance is reported in Table 1. The final results are reported in Table 2.
We can observe that while the completely unsupervised method (GiveMe5W1H) has
a poorer extraction capacity, the other semi-supervised models exhibit a
stronger performance. We also observe that the approaches processing
contextual information like Biased TRank and our model ArgFuse report a much
higher performance thus highlighting the importance of contextual information
in this task. We find that while Biased TRank @ k = 1 reports the highest
precision score, Biased TRank @ k = 2 reports the highest recall score. The
stark difference between the precision and recall values of these two baseline
methods is reflective of the problem of fixing a suitable ’k’ value for the
task of aggregation. While the higher number of argument roles with a single
mention in our dataset (singles as illustrated in Figure 3) is favourable for
a higher precision value for Biased TRank @ k = 1, allowing the inclusion of
more arguments in the final document frame can be attributed to the high
recall score of Biased TRank @ k = 2. However, fixing the number of argument
mentions to be included in the final document frame increases the count of
false negatives for k = 1 and increases the number of false positives for k =
2. Our model presents a dynamic approach of including any number of relevant
and precise arguments for representing document-level information. ArgFuse
reports state-of-the-art results with a an acceptable balance between the
precision and recall values.
In Figure 6, the argument-wise results of the various baseline models along
with that of ArgFuse is presented. We can observe that for most of the
arguments, ArgFuse and Biased TRank report the highest performance, with
ArgFuse reporting better or comparable performance in 4 out the 6 arguments.
We find that ArgFuse reports comparatively poorer performance for arguments
like Reason and Participant which constitute the argument classes with least
number of samples (as depicted in Figure 2). Also, argument classes like
Reason mostly constitute of singles as depicted in Figure 3 and hence reports
a higher score using Biased TRank @ k = 1.

###  6.2 Qualitative Analysis

In this section we closely look at the failure cases for error analysis. In
Figure 7, we record two examples from our dataset with both human and machine
annotated information frames which are representative of the generic merits
and demerits of our model. In the first example, we notice that instead of
selecting “Baumgarten”, the mention containing the NERs “Austria” and
“Slovakia” is chosen. This is indicative of the bias of the ranking process
towards more popular named entities and might not favour mentions containing
rare named entities. In the second example, for the Place arguments, we
observe that “New York, New York City apartment building” are rightly deemed
redundant by our redundancy classifier, and the argument “New York” is
discarded. When the arguments “New York City apartment building, first floor
of a brick building and quickly spread upstairs” are evaluated, they are
incorrectly classified as redundant. When the argument mentions, such as the
above mentioned pair, are very similar in both their surface forms and
content, the model sometimes fails to capture their exclusivity in terms of
information. In case of the participant arguments, we observe in the second
example that the irrelevant argument “arsonist” is correctly discarded.
However, for both the Casualty arguments in the first example and the
Participant arguments in the second example, the relevant arguments are again
very similar in nature, and the model ranks the incorrect argument mention
over the correct ones. For example, in example 1, “18 people were injured” is
ranked higher than the other candidate argument mentions thus resulting in
some loss of information.

In Figure 8, we present the comparison between the outputs generated by
ArgFuse and some of the other baseline models. We observe that the outputs
retrieved from both GiveMe5W1H and Biased TRank present with irrelevant and
redundant content. The output of ArgFuse correlates the most with the gold
standard output for the document. This can be regarded to the explicit
relevance and redundancy checks in the ArgFuse algorithm which mines precise
document level information frames effectively.

##  7 Conclusion

In this paper, we present an extractive approach to aggregate sentence-level
argument or entity mentions to produce precise document-level information
frames from lengthy text articles effectively. With a very scarce amount of
work being conducted in the field of document-level IE, we develop and open-
source our dataset of aggregated argument mentions. To the best of our
knowledge, we present the baseline for the task of argument aggregation and
open-source our work for research. We closely analyse the merits and demerits
of the model and encourage scientists to build on the pitfalls discussed and
enhance the aggregation capabilities at a document level. For future work, we
want to analyse the model’s aggregation capabilities in crosslingual and
multilingual environments and extend the aggregation capabilities across
document boundaries. As explored in works like Piskorski et al. (2008); Ji and
Grishman (2008), extending ArgFuse to aggregate information from multiple news
sources can present with a very useful and practical use case.

## Acknowledgments

The work done in this paper is an outcome of the project titled “A Platform
for Cross-lingual and Multi-lingual Event Monitoring in Indian Languages”,
supported by IMPRINT-1, MHRD, Govt. of India, and MeiTY, Govt. of India.

## References

  * Beltagy et al. (2019) Iz Beltagy, Kyle Lo, and Arman Cohan. 2019.  Scibert: A pretrained language model for scientific text.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Ebner et al. (2020) Seth Ebner, Patrick Xia, Ryan Culkin, Kyle Rawlins, and Benjamin Van Durme. 2020\.  Multi-sentence argument linking.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 8057–8077, Online. Association for Computational Linguistics.
  * Hamborg et al. (2019) Felix Hamborg, Corinna Breitinger, and Bela Gipp. 2019.  Giveme5w1h: A universal system for extracting main events from news articles.  In _Proceedings of the 13th ACM Conference on Recommender Systems, 7th International Workshop on News Recommendation and Analytics (INRA 2019)_.
  * Hu et al. (2018) Peiyun Hu, Zachary C Lipton, Anima Anandkumar, and Deva Ramanan. 2018.  Active learning with partial feedback.  _arXiv preprint arXiv:1802.07427_.
  * Jain et al. (2020) Sarthak Jain, Madeleine van Zuylen, Hannaneh Hajishirzi, and Iz Beltagy. 2020.  SciREX: A challenge dataset for document-level information extraction.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7506–7516, Online. Association for Computational Linguistics.
  * Ji and Grishman (2008) Heng Ji and Ralph Grishman. 2008.  Refining event extraction through cross-document inference.  In _Proceedings of ACL-08: HLT_ , pages 254–262, Columbus, Ohio. Association for Computational Linguistics.
  * Kar et al. (2021) Debanjana Kar, Sudeshna Sarkar, and Pawan Goyal. 2021.  Event argument extraction using causal knowledge structures.
  * Kazemi et al. (2020) Ashkan Kazemi, Verónica Pérez-Rosas, and Rada Mihalcea. 2020.  Biased TextRank: Unsupervised graph-based content extraction.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pages 1642–1652, Barcelona, Spain (Online). International Committee on Computational Linguistics.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.
  * Luan et al. (2019a) Yi Luan, Dave Wadden, Luheng He, Amy Shah, Mari Ostendorf, and Hannaneh Hajishirzi. 2019a.  A general framework for information extraction using dynamic span graphs.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 3036–3046.
  * Luan et al. (2019b) Yi Luan, Dave Wadden, Luheng He, Amy Shah, Mari Ostendorf, and Hannaneh Hajishirzi. 2019b.  A general framework for information extraction using dynamic span graphs.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 3036–3046, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Maheshwari et al. (2020) Ayush Maheshwari, Hrishikesh Patel, Nandan Rathod, Ritesh Kumar, Ganesh Ramakrishnan, and Pushpak Bhattacharyya. 2020.  Tale of tails using rule augmented sequence labeling for event extraction.
  * Mihalcea and Tarau (2004) Rada Mihalcea and Paul Tarau. 2004.  TextRank: Bringing order into text.  In _Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing_ , pages 404–411, Barcelona, Spain. Association for Computational Linguistics.
  * Mitamura et al. (2015) Teruko Mitamura, Yukari Yamakawa, Susan Holm, Zhiyi Song, Ann Bies, Seth Kulick, and Stephanie Strassel. 2015.  Event nugget annotation: Processes and issues.  In _Proceedings of the The 3rd Workshop on EVENTS: Definition, Detection, Coreference, and Representation_ , pages 66–76, Denver, Colorado. Association for Computational Linguistics.
  * Nguyen et al. (2016) Thien Huu Nguyen, Kyunghyun Cho, and Ralph Grishman. 2016.  Joint event extraction via recurrent neural networks.  In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 300–309.
  * Page et al. (1998) L. Page, S. Brin, R. Motwani, and T. Winograd. 1998.  The pagerank citation ranking: Bringing order to the web.  In _Proceedings of the 7th International World Wide Web Conference_ , pages 161–172, Brisbane, Australia.
  * Piskorski et al. (2008) Jakub Piskorski, Hristo Tanev, Martin Atkinson, and Erik Van Der Goot. 2008.  Cluster-centric approach to news event extraction.  In _Proceedings of the 2008 Conference on New Trends in Multimedia and Network Information Systems_ , page 276–290, NLD. IOS Press.
  * Roy and McCallum (2001) Nicholas Roy and Andrew McCallum. 2001.  Toward optimal active learning through monte carlo estimation of error reduction.  _ICML, Williamstown_ , pages 441–448.
  * Veyseh et al. (2020) Amir Pouran Ben Veyseh, Tuan Ngo Nguyen, and Thien Huu Nguyen. 2020.  Graph transformer networks with syntactic and semantic structures for event argument extraction.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings_ , pages 3651–3661.
  * Wadden et al. (2019) David Wadden, Ulme Wennberg, Yi Luan, and Hannaneh Hajishirzi. 2019.  Entity, relation, and event extraction with contextualized span representations.  _arXiv preprint arXiv:1909.03546_.
  * Wang et al. (2019) Xiaozhi Wang, Ziqi Wang, Xu Han, Zhiyuan Liu, Juanzi Li, Peng Li, Maosong Sun, Jie Zhou, and Xiang Ren. 2019.  HMEAE: Hierarchical modular event argument extraction.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 5777–5783, Hong Kong, China. Association for Computational Linguistics.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020.  Transformers: State-of-the-art natural language processing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 38–45, Online. Association for Computational Linguistics.
  * Yang et al. (2018a) Hang Yang, Yubo Chen, Kang Liu, Yang Xiao, and Jun Zhao. 2018a.  DCFEE: A document-level Chinese financial event extraction system based on automatically labeled training data.  In _Proceedings of ACL 2018, System Demonstrations_ , pages 50–55, Melbourne, Australia. Association for Computational Linguistics.
  * Yang et al. (2018b) Hang Yang, Yubo Chen, Kang Liu, Yang Xiao, and Jun Zhao. 2018b.  Dcfee: A document-level chinese financial event extraction system based on automatically labeled training data.  In _Proceedings of ACL 2018, System Demonstrations_ , pages 50–55.
  * Zheng et al. (2019a) Shun Zheng, Wei Cao, Wei Xu, and Jiang Bian. 2019a.  Doc2EDAG: An end-to-end document-level framework for Chinese financial event extraction.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 337–346, Hong Kong, China. Association for Computational Linguistics.
  * Zheng et al. (2019b) Shun Zheng, Wei Cao, Wei Xu, and Jiang Bian. 2019b.  Doc2edag: An end-to-end document-level framework for chinese financial event extraction.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 337–346.