# The Additional Representative Images for Legacy (ARI-L) project for the ALMA
Science Archive

M. Massardi1,2, F. Stoehr3, G. J. Bendo4, M. Bonato1, J. Brand1, V. Galluzzi5,
F. Guglielmetti3, E. Liuzzo1, N. Marchili1, A. M. S. Richards4, K. L. J.
Rygl1, F. Bedosti1, A. Giannetti1, M. Stagni1, C. Knapic5, M. Sponza5, G. A.
Fuller4, T. W. B. Muxlow4 1INAF - Istituto di Radioastronomia - Italian ALMA
Regional Centre, via Gobetti 101, 40129 Bologna, Italy
2SISSA, Via Bonomea 265, 34136 Trieste, Italy
3European Southern Observatory (ESO), Karl-Schwarzschild-Str. 2, 85748
Garching bei München, Germany
4UK ALMA Regional Centre Node, Jodrell Bank Centre for Astrophysics,
Department of Physics and Astronomy,
The University of Manchester, Oxford Road, Manchester M13 9PL, UK
5INAF-Osservatorio Astronomico di Trieste - Italian Astronomical Archives, via
Tiepolo 11,34131 Trieste, Italy
<EMAIL_ADDRESS>
###### Abstract

The Additional Representative Images for Legacy (ARI-L) project is a European
Development project for ALMA Upgrade approved by the Joint ALMA Observatory
(JAO) and the European Southern Observatory (ESO), started in June 2019. It
aims to increase the legacy value of the ALMA Science Archive (ASA) by
bringing the reduction level of ALMA data from Cycles 2-4 close to that of
data from more recent Cycles processed for imaging with the ALMA Pipeline. As
of mid-2021 more than 150000 images have been returned to the ASA for public
use. At its completion in 2022, the project will have provided enhanced
products for at least 70% of the observational data from Cycles 2-4
processable with the ALMA Pipeline. In this paper we present the project
rationale, its implementation, and the new opportunities offered to ASA users
by the ARI-L products. The ARI-L cubes and images complement the much limited
number of archival image products generated during the data quality assurance
stages (QA2), which cover only a small fraction of the available data for
those Cycles. ARI-L imaging products are highly relevant for many science
cases and significantly enhance the possibilities for exploiting archival
data. Indeed, ARI-L products facilitate archive access and data usage for
science purposes even for non-expert data miners, provide a homogeneous view
of all data for better dataset comparisons and download selections, make the
archive more accessible to visualization and analysis tools, and enable the
generation of preview images and plots similar to those possible for
subsequent Cycles.

Keywords: instrumentation: interferometers – techniques: image processing –
Astronomical Data bases

##  1 Introduction: ARI-L project goals and context

Most of the scientific results of radio interferometry lie in the continuum
images and spectral line data cubes that are created by transforming the
observed visibility data (the observed brightness at a given angular scale)
from the Fourier domain to the image domain. However, the resulting products
are not unique. In the imaging process, resolution can, for example, be
traded-off against sensitivity by applying different visibility weighting
schemes. The possibility to obtain more than one valid image for any
visibility dataset has long justified the fact that calibrated visibilities
have been considered the final product of an observation. Typically, raw data
and, in some cases like ALMA, calibration scripts were stored and made
available through the archive interfaces for any user to reprocess them and
produce images.

The amount of observational astronomical data is growing exponentially. For
example, at ESO, the data-growth is approximately 37% per year (Stoehr 2019).
Radio astronomy, with facilities like LOFAR, JVLA, MWA and - of course - ALMA,
is one of the main drivers of the worldwide growth of astronomical data. Even
larger amounts of data are expected to arrive when the full SKA becomes
operational.

Astronomy, and in particular radio astronomy, has arrived in the era of ’big
data’ where the quantity of observational data is too large for astronomers to
analyse it without major assistance from data reduction software and full data
reduction pipelines. Consequently, in addition to significant investment in
user support through three ALMA Regional Centres (ARCs), ALMA’s design from
the very beginning included a CASA-based (McMullin et al. 2007) data reduction
pipeline***https://almascience.org/processing/science-pipeline that would
automatically perform the data calibration as well as the imaging step.

ALMA was the first radio astronomy facility to pledge that by the time the
full Science Operations stage is reached, the fundamental data product of the
Joint ALMA Observatory will be calibrated, deconvolved images and data cubes.
These data products are not unique because of the relatively large freedom in
parameter choices during the imaging process, but even in their generic form,
they provide a quick way for users to assess the data-quality, the content of
the data products and the interesting spatial and spectral regions. Depending
on the science case, the data products may also be used for scientific
analysis. For these reasons, the image products are delivered to the ALMA
users through the ALMA Science Archive (ASA).

From ALMA’s Early Science period up to Cycle 3, the imaging-dedicated part of
the ALMA pipeline was not available. The staff at the observatory and at the
ARCs manually performed the quality assessment (QA2) of the data before they
were delivered to the PIs (Petry et al. 2020). This manual procedure was
carried out for over 1800 ALMA projects. The manual imaging of each full data
set is very time consuming, so the analysis often focused on only a small
subset of a project’s calibrated data that was just large enough to assess the
quality. As a result, only a very small fraction of all raw data ($<10\%$) was
converted into images and image cubes. This fraction increased drastically
from Cycle 5 (i.e. from 2017) onwards when the ALMA Science Pipeline was used
nearly exclusively for QA2 data reduction.

Use of the archive has increased over the years, and the ASA is now considered
an important, widely-utilized resource; about 25% of all ALMA publications are
based either purely on archival data or use archival data in addition to PI
data (Stoehr et al. 2017).

The availability of deconvolved images and data cubes vastly speeds up
researchers’ data analysis process. Rather than having to download the raw
data, to identify the CASA versions to use, to run the calibration or
restoration script, and to modify and run the imaging script for hours or days
just to determine whether objects or spectral lines have been detected,
researchers can use previously-created products to make these types of
assessments in seconds.

In addition to allowing direct analysis by archival researchers, these
products also enable additional services. This includes the visualisation of
products using the remote visualisation tool
CARTA†††https://cartavis.github.io/, the creation of preview images that can
be displayed directly on the ALMA Science Archive query
interface‡‡‡https://almascience.org/asax, the automated post-analysis of the
ALMA data products (for example, with the ALMA Data mining ToolkIT
ADMIT§§§http://admit.astro.umd.edu/, Teuben et al. 2015, or the Keywords of
Astronomical FITS-Images Explorer KAFE, Burkutean et al. 2018), and the
ability to directly access the products through VO services like TAP, SIA2 and
DataLink, that allow for spectral multi-band comparisons, source
identifications, and catalogue reconstruction.

The Additional Representative Images for Legacy (ARI-L) project is an ALMA
Development project approved by JAO and ESO in December 2018 following an ALMA
development study approved by ESO and reported by Massardi et al. (2019). The
aim of ARI-L, as presented in this paper, is to use the ALMA Imaging Pipeline
(introduced in 2017) on the data from early observing Cycles (2-4) so as to
create, where possible, data products of the same completeness and quality as
ALMA is creating for new observations nowadays and to ingest them in the ASA.

Over its three years of activity, ARI-L will produce and ingest into the ASA a
uniform set of full data cubes and continuum images that cover at least 70% of
the data from Cycles 2-4 that can be processed with the ALMA Pipeline. These
cubes complement the existing QA2-generated image products.

The ARI-L project activity started in June 2019. The first data products were
delivered to JAO in November 2019, and the project has been ongoing since then
and has produced, assessed the quality of, and delivered more than 150000
images that can already be downloaded from the ASA. ARI-L products are
certified for quality and ingested into the archive as so-called “external
products”. The ARI-L project documentation and website as well as periodic
updates are publicly available through the ALMA science portal ¶¶¶
https://almascience.org/alma-data/aril.

In this paper, we present the project principles, the workflow, and the data
products (see Sect. 2). We then discuss some new opportunities offered to the
ALMA Science Archive users through ARI-L products (in Sect. 3), and we finally
(in Sect. 4) summarize the current and future project plans.

Table 1:  Number and properties of Member Observing Unit Sets (MOUS) for ALMA Cycles 2-4 that can be processed in the ARI-L project, including QA2-passed and -semipassed projects but excluding full-Stokes, VLBI, solar, and TP observations as well as projects that have already been processed with the Imaging Pipeline and are not in QA3 (see the footnote for definition) as of June 2021. Cycle | 2 | 3 | 4 | Total
---|---|---|---|---
ARI-L MOUS | 1085 | 1719 | 672 | 3476
% of observed MOUS | 82.3 | 83.5 | 32.0 | 63.5
Total raw data size [TB] | 22 | 66 | 50 | 138
Median size per MOUS [GB] | 11.2 | 16.8 | 20.3 | 14.7

##  2 The ARI-L Project workflow

###  2.1 ARI-L terms of reference

ALMA observations and products are organized hierarchically. Each project is
identified by a code with the form of ‘YYYY.C.nnnnn.T’ in which ‘YYYY’
corresponds to the year in which the project was submitted, ‘C’ corresponds to
the call, ‘nnnnn’ is the sequential number assigned to the proposal when it
was submitted, and ‘T’ is the proposal type. ARI-L only works on data from
Cycles 2, 3, and 4, i.e. project IDs that start with ’2013’, ’2015’ and
’2016’.

Cycles 0 and 1 only make up 2% and 8% of the raw data size of the first 5 ALMA
Cycles, respectively. The Cycle 0 and 1 data format and packaging have
differences with respect to the following Cycles and therefore require
dedicated procedures. However, most of the objects have been re-observed in
later Cycles using more antennas. Our study (Massardi et al. 2019) indicates
that it is preferable to focus on more recent Cycles. The Imaging Pipeline has
been used for ALMA products since early 2017 (during Cycle 4), so the products
from subsequent Cycles are already as complete as the pipeline processing
allows.

PIs specify their observations as a series of Science Goals. At operation
stage, this corresponds to one or more ‘Group Observing Unit Sets’, each of
which may be split in various ‘Member Observing Unit Sets’ (MOUSs) that
include the instrumental setting needed to reach the PI’s goals. Each MOUS
translates into ‘Scheduling Block’ commands during observations that, for
calibration purposes, are limited in execution time. Hence, multiple
repetitions (or ‘Execution Blocks’, which can be abbreviated as EBs) might be
necessary to reach the requested sensitivity. Each EB is calibrated
individually. Because calibrated data could be combined to generate images at
the MOUS level, the selection and analysis level for ARI-L is the MOUS. Table
1 lists the number of MOUSs that are being processed in the ARI-L project.

Only datasets that are accessible for public download and that have passed
ALMA quality control (either ’PASS’ or ’SEMIPASS’ in QA2∥∥∥An MOUS is
considered ’PASS’ in level 2 of the Quality Assurance procedure if after the
full calibration and generation of imaging products it reaches the conditions
of sensitivity and resolution requested by the PI in the proposal that
generated that dataset. ’SEMIPASS’ indicates that the dataset falls short of
meeting the PI-requested sensitivity and resolution, but are otherwise of good
quality (see ALMA Technical Handbook).) are selected for ARI-L processing.
This excludes data which are in QA3******Level 3 of the Quality Assurance
could be initiated by PIs in case of issues encountered after product
delivery. QA3 typically reactivates data access restrictions for MOUSs,
allowing only PIs and users with delegated access to download the data. and
data still covered by a proprietary period.

Furthermore, we select only datasets with observing modes that can be handled
by the imaging tasks of the ALMA Pipeline (hereafter defined as ‘Imaging
Pipeline’). Many of the observing modes that were considered non-standard when
observed (and thus required special care during calibration) can now be
processed by the Imaging Pipeline as common standard observing modes. However,
solar, full-Stokes, and VLBI observations are not reprocessed because they
cannot be handled by the Imaging Pipeline. Similarly, Total Power datasets are
also not reprocessed because the full data reduction (calibration and imaging)
for those data has already been completed.

The ARI-L products are intended for archive-mining purposes and as an overview
of the archived data content. They are not necessarily science-ready. However,
the ARI-L project also uses the Imaging Pipeline outside the range of goals
for which it has been commissioned, as it is the best tool currently available
to create homogeneous images of data from past Cycles.

For processing purposes, we sort the selected MOUS as an increasing function
of the expected run time of the datasets, processing the faster ones first.
Our tests (Massardi et al. 2019) demonstrated that a very rough run-time score
value for an MOUS can be constructed using a function of its data volume and
the maximum distance between the antennas during the observations (the maximum
baseline). The data volume is related to the number of EBs (defined by the
duration of the observing run and by the presence of multiple pointings, which
could be either in a mosaic or covering multiple targets) and to the number of
spectral channels; more time is needed to create images with larger data
volumes. The maximum distance between the antennas during the observations is
an indication of the resolution that can be achieved in the image, and more
time is needed to create higher-resolution images.

Processing parameters are typically applied in the same way as they are in
later Cycles. A README file is provided for each imaged MOUS summarising the
data processing and any deviations from standard procedures.

Figure 1: The decision tree and the different phases of the ARI-L project as
applied to each MOUS in Cycles 2-4.

###  2.2 ARI-L processing

The ARI-L products are generated by a Python-based workflow engine (the ARI-L
code). According to the above-stated terms of reference, for each processable
MOUS, the ARI-L code

  * •

downloads the raw data, auxiliary data (including the calibration scripts and
tables), and existing products from the ASA;

  * •

identifies the original CASA version used to generate the calibration scripts
that are stored in the ASA;

  * •

runs the scriptForPI to restore the calibrated Measurement Sets (MS);

  * •

adds additional flagging that was identified by the original QA2 data-
reducers;

  * •

uses the latest CASA version available at the start of processing (so far CASA
5.6.1-8) to generate the data-products with the Imaging Pipeline, extending
the processing to also include data-cubes for calibrator observations;

  * •

packs the calibrated MS for permanent storage;

  * •

renames the products to follow the ALMA naming standard;

  * •

compresses the auxiliary files;

  * •

generates a README file with a summary of the content of the MOUS;

  * •

runs an automatic comparison with the existing manually created QA2 products
(see Section 2.5);

  * •

catches processing errors and identifies classes of errors for later recovery;

  * •

stores the processing information in a relational database system.

After the ARI-L code processing, the data undergo the ARI-L quality assurance
step before they are delivered to ESO and then JAO for ingestion into the ASA.
The ingested files are cross-checked with the generated products, and, if they
match, the latter are deleted from the ARI-L processing area. A visual
representation is shown in Fig. 1

###  2.3 Restoring calibration

Packages of archived datasets include all the scripts used during the ALMA
Quality Assurance processes for data reduction and for checking whether the
sensitivity and resolution required by the PI have been reached. In the early
Cycles, all the MOUS were manually calibrated by experts from the ALMA
Regional Centres using scripts for calibration and imaging generated from a
standard template. Despite strictly following common checklists that guarantee
the quality of the final products, each analyst was free to modify the scripts
as they felt was necessary. For this reason, early Cycle scripts might differ
from each other.

The information needed for the ARI-L processing (e.g. the CASA version needed
for calibration) is retrieved automatically from the downloaded data. Then,
for the execution blocks (EB) of each MOUS, calibration is performed with an
appropriate CASA version using the prescriptions of the calibration scripts
included in the downloaded folder. No change or verification is applied to the
calibration stage except on rare occasions where problems had been identified
in early procedures.

As mentioned above, imaging scripts typically are written to produce example
images or data cubes that demonstrate that the requirements of the PI were
achieved in the observations. Hence, the QA2-produced imaging scripts included
in the script folder are not used except for extracting meaningful additional
flagging commands. This may happen in manual imaging when the QA2 analyst
realized that, at the end of the calibration, a portion of the data (i.e. some
channels, a time range, one or more antennas, some baselines, etc.) are still
significant outliers with respect to all other data and therefore decided to
flag the outlying data after the calibration but before imaging. After these
flagging commands are extracted, they are applied in the ARI-L procedure after
the calibration and before the imaging. This process generates the calibrated
measurement sets that we give as input to the Imaging Pipeline.

###  2.4 Imaging with the ALMA pipeline

The calibrated measurement sets are processed with the most recent version of
the Imaging Pipeline that has been tested for ARI-L purposes. In the first
year of ARI-L activities, we have used the version included in the CASA
release 5.6.1-8††††††We refer to the ALMA Science Pipeline User’s Guide
(https://almascience.org/processing/documents-and-tools/alma-science-pipeline-
users-guide-casa-5-6.1) for details on the parameters used.. The Pipeline task
sequence is the same as that implemented in later Cycles.

Imaging is performed for each source included in each MOUS. We image not only
the science targets but also the bandpass, phase, and check source
calibrators. For each target and calibrator in each MOUS, the ARI-L procedure,
after uv continuum subtraction, produces aggregate continuum images and multi-
frequency synthesis (mfs) continuum images and spectral cubes for each
spectral window. We take advantage of the improved continuum-finding abilities
of the recent pipeline to detect weak line emission, particularly in ACA 7-m
data. Consequently, the estimated noise per channel is ameliorated with
respect to previous CASA versions, allowing for a more robust S/N thresholds
derivation for the associated channels. The improved automasking is beneficial
for both emission and absorption line detection. Nonetheless, this
amelioration may still be inadequate in the case of weak calibrators; see the
CASA guide‡‡‡‡‡‡https://casaguides.nrao.edu/index.php/Automasking_Guide for
more information.

All ARI-L images are primary beam corrected. The corrected images cover a
region of diameter equal to the FWHM of the primary beam centered at the
pointing position. The Imaging Pipeline by default attempts automasking of the
images. Primary beam and mask maps are delivered together with the product
image.

For cubes, the channel resolution is defined by the native resolution of the
observations but also accounts for the nominal channel size having been
reduced by a factor $\sim 2$ by Hanning smoothing. We allowed the Imaging
Pipeline to set the optimal parameters and masks for each image.

The Briggs robust parameter is set to 0.5 for all data unless the Imaging
Pipeline indicates a different approach. In all cases, the robust parameter is
stored in the image header in the ROBUST keyword so that the applied value can
always be verified.

In case the size of the product file is larger than 180 GB, the Imaging
Pipeline has implemented a mitigation process by which only one-third of the
primary beam is imaged or the image resolution is smoothed to 3 pixels per
synthesized beam. In later Cycles, channel averaging might also be applied to
high spectral resolution data that produce cubes that are too large or to data
from mosaic observations where the size of the field made it impossible to
produce final images. In extreme cases, only a few fields and/or a few
spectral windows are imaged. We set a threshold size for mitigation 3 times
larger than the one used in later ALMA Cycles to allow for better completeness
of our products. In ARI-L Cycles 3 and 4 processed MOUS mitigation cases
constituted less than 1%; these data were checked on a case by case basis. In
case images of target sources or spectral windows are missing because of
mitigation a note is added to the ARI-L README.

The Imaging Pipeline produces a Weblog report that is delivered in the ARI-L
product folder and reports all the commands, parameters, and tasks used.
However, this Weblog is available only for the sources observed as science
targets, not for the calibrators. Every step listed in the Weblog also gets a
score that indicates its success with respect to common heuristics. In case
the pipeline cannot apply automasking, applies mitigation, or defines a
different set of weighting parameters, a warning flag in the Weblog is raised.
In case the resultant images are considered of good quality they are accepted.

This approach has been successfully tested on all the ALMA Cycles within the
scope of ARI-L and used thus far to generate mostly Cycle 3 and 4 products.
Some additional caveats in the usage of the Imaging Pipeline on Cycle 2 data
are described in the product README files whenever the procedure differs from
the one described above.

###  2.5 The ARI-L Project quality assurance (ARIQA)

The ARI-L products have a quality that is virtually always comparable to the
existing manually-created products from QA2. However, ARI-L offers complete
cubes for all the sources in the datasets, while the manually imaged products
often show only a small fraction of the available data (eg. only one or a few
sources or a limited number of channels).

All the ARI-L products are quality-checked by our team of expert data-analysts
before being ingested into the archive. The Additional Representative Images
Quality Assurance (ARIQA) procedure has four layers.

  * •

A check is performed to ensure that the ARI-L code was executed correctly. In
case of failures, ARI-L products will not be ingested into the ASA.

  * •

Another check is performed to verify that all the images that were expected
are produced. The causes of missing images are carefully identified. Decisions
about missing images are made on a case by case basis and, if necessary,
reported in the README.

  * •

The Weblog of the Imaging Pipeline products is reviewed to verify that the
pipeline has properly generated good quality images according to its
heuristics (i.e. all the ALMA Imaging Pipeline tasks have been executed with a
score larger than 0.90). Any discrepancy is analysed and, if the discrepancy
is not justified, the relative ARI-L products are not ingested into the ASA.

  * •

The portion of the data corresponding to what is available as QA2 products are
extracted and smoothed to the same resolution, and then the rms noise and
dynamic ranges are compared. Differences should be evaluated on the basis of
the processes used during QA2. In case the differences are not justified and
the resulting rms or dynamic range worsen by more than 30% or the imaged
structure is clearly wrong, the ARI-L products are not ingested into the ASA.

The ARIQA is not intended to be a repetition of QA2 and will never change the
QA2 outcome for an MOUS. It only evaluates the quality of the produced images
and relies on the calibration for each MOUS as it is done in the archived
scripts. For this reason, once the ARI-L code has successfully produced an
image product, any quality comparison will be done with the existing QA2
images and not in reference to the project’s requirements in sensitivity and
angular resolution. Nevertheless, ARI-L images have been through a distinct
and less manually-intensive QA process compared to the corresponding products
generated in the ALMA QA2 process. Image characteristics resulting from these
distinct processes are typically comparable but can vary, particularly when
manual adjustments were required for QA2 (e.g. self-calibration or
optimization of the continuum channel selection). Furthermore, we stress that
the ARIQA process ensures reliable imaging products for manually calibrated
data sets as well.

In cases where only some of the images of an MOUS can pass the ARIQA, we may
consider ingesting only those that passed ARIQA into the ASA even after
accurate evaluations and attempts to recover the other images during the
quality assurance process. If the products still have any issues after the
ARIQA but their quality is considered sufficient for delivery, a note is added
in the README file enclosed in the ARI-L product folder and they are
delivered.

###  2.6 The ARI-L products delivery and retrieval

For all the successfully processed MOUSs that pass quality control, the final
ARI-L calibrated measurement sets are stored in a dedicated storage system
outside the ASA that is hosted and maintained by the INAF-
IA2111www.ia2.inaf.it facility. The data are available to the user community
via the ARI-L webpage. The measurement sets will be stored for the whole
duration of the ARI-L project and for a minimum of 3 years after the
conclusion of the project. In this system, the calibrated dataset folder,
which is named after the original observing project and MOUS, includes
measurement sets for each calibrated EB.

The imaging products from MOUSs that passed the ARIQA process are delivered to
ESO to be ingested in the ASA. A README file is enclosed in the ARI-L product
folder. Its purpose is to trace the history and hierarchy of the dataset used
to generate all the product images, including:

  * •

a synthetic description of the purposes of the ARI-L products;

  * •

the description of the MOUS as taken from the archived QA folder;

  * •

notes on the ARI-L process (including at least the CASA version used for
processing);

  * •

a table describing the content of the EBs included in the MOUS (one per line):
sequential number of the EB; EB name; time range in MJD of the observations;
array (7-m, 12-m); number of antennas; frequency range for each spectral
window in GHz; spectral resolution for each spectral window in MHz; range of
angular scales covered in arcsec; first, second, and third quartile of the uv
range in metres;

  * •

a table summarising the properties for each source observation (science target
or calibrator; one per column) included in the MOUS: right ascension and
declination in degrees; sequential number of EB as in the previous table in
which the source is included; number of pointings (which differs from unity in
case of mosaics); largest angular scale and resolution of the observation (in
arcsec); intent of the observation of the source.

Each ARI-L product file in the ASA archive includes the ‘ari-l’ tag in the
name along with indications of the original project and MOUS numbers, the
spectral windows, and the source names, all following the more recent ALMA
file naming scheme. ARI-L image, README and Weblog files can be retrieved from
the ALMA Science Archive in the form of single files as ”Externally delivered
products” on the Request Handler download page; these files are directly
listed below the ALMA data.

###  2.7 The ARI-L failure analysis and recovery

MOUSs that failed ARI-L processing or ARIQA are temporarily quarantined. They
are then classified and analysed during dedicated recovery sessions. We
consider reprocessing each MOUS using the same or different Imaging Pipeline
parameters and attempt this at least once for each quarantined MOUS. If
successful, a note is added to the README with the final processing parameters
if they are different from the standard ones. This approach is important in
guaranteeing the homogeneity of our delivered products and our control on the
application of the Imaging Pipeline to old datasets, and it improves our
knowledge of the data processing as well.

QA2ARI-LQA2ARI-L

Figure 2: Comparisons of the QA2 and ARI-L products for the source
G345.49+1.47 in the MOUS uid://A001/X87a/X738 from the project 2016.1.00288.S.
The top row shows images of the emission at specific channels. The second and
third rows show the complete spectra extracted from the data available in the
ASA, with red arrows indicating the part of the spectrum that the images in
the top row are extracted from. In the QA2 products, images without continuum
subtraction are available for the first two spectral windows, and a continuum-
subtracted cube is available for only the first one. In the ARI-L products,
continuum-subtracted cubes are available for all 13 spectral windows, as shown
in the complete ARI-L spectrum and in the next figure

.

spw0spw1spw2spw3spw10spw7spw4spw5spw6spw8spw9spw11spw12 Figure 3: Spectral of
ARI-L products for the source G345.49+1.47 in the MOUS uid://A001/X87a/X738
from the project 2016.1.00288.S, presented in the previous figure: continuum-
subtracted cubes are available for all 13 spectral windows. Almost all of the
ARI-L spectra show clear spectral features. ARI-LARI-LQA2ARI-LQA2 Figure 4:
Images and spectra ingested into the ASA for the project 2016.1.00875.S MOUS
uid://A001/X88f/X11a (a spectral scan of a region of the Central Molecular
Zone of the Milky Way). The first two images show the QA2 and ARI-L products
for the same channel, which is indicated by the green arrow in the two spectra
shown below. The image on the right shows the structure associated with the
bright spectral feature indicated by the red arrow in the ARI-L cube spectrum;
this is absent in the QA2 cubes.

##  3 Examples of ARI-L products

The project officially started in June 2019 and is planned to finish after
three years. The project deployment is on schedule, most of the expected
products are already available for download from the ASA, and the relative
calibrated measurement sets can be requested on our website accessible through
the ALMA Science Portal******https://almascience.org/alma-data/aril.

Given the large number of science cases involved, it is clear that a
comprehensive overview of the scientific opportunities offered by the ARI-L
images is not possible and also beyond the scope of the current paper.
Massardi et al. (2019) list a few possible science cases. In the following, we
present just a few examples of the improvements and new applications that
ARI-L products allow.

The most relevant added-value that ARI-L brings to the ASA is the completion
of the images for the older Cycles. Figures 2, 3, and 4 show how the QA2
images are typically limited to only a portion of the data while ARI-L images
provide a more complete overview of the data content. In figure 2, we show
that the QA2 package includes images for 2 of the 13 spectral windows observed
for an O-type young stellar object (G345.49+1.47) to study kinematic
structures in dense, warm gas associated with star formation (Cesaroni et al.
2017, Moscadelli et al. 2019, Maud et al 2019). The complete spectral coverage
of the ARI-L cubes (see Fig. 3) shows the rich chemistry of this object. The
kinematic information from the line emission can be used to investigate the
properties of Keplerian disks or outflows and to constrain the mechanisms of
massive star formation. Figure 4 presents the spectral scan of a region of the
Central Molecular Zone of the Milky Way. The QA2 process verified the
achievement of PI requirements only for a portion of each spectral windows,
while the ARI-L spectra cover the entire frequency range. These cases are
examples of the improved information that the ARI-L images offer for spectral
analysis.

ARI-L also offers unprecedented easy access to data that can be used for the
investigation of light curves, spectral behaviour, variability and
morphological analysis of the ALMA calibrators. They are typically blazars
(Bonato et al. 2018, 2019), a class of AGNs characterized by strong
variability over a broad range of time scales in all wavebands (e.g. Ulrich et
al. 1997; Webb 2006; Aharonian et al. 2007, Tavecchio et al. 2011, Abdo et al.
2010; Galluzzi et al. 2017). The entire non-thermal continuum is believed to
originate mainly from a relativistic jet pointing close to our line of sight.
ARI-L images of calibrators provide an extraordinary amount of multi-epoch and
multi-frequency data in the poorly explored mm/sub-mm spectral region that can
shed light on the physical processes in action, such as particle acceleration
and emission mechanisms, relativistic beaming, the origins of flares, and the
size, structure, and location of the emitting regions.

Figures 5, 6, and 7 show some of the capabilities offered by ARI-L images for
studying the morphology and variability of blazars. We present images for
three ALMA calibrators with well-known resolved structure, namely PKS0521-365,
3C273, and PKS0637-752 (see also for comparison Meyer et al. 2017 and Agliozzo
et al. 2017 for PKS 0637-752, Hovatta et al. 2018 for 3C273, and Liuzzo et al.
2015 for PKS 0521-365).

For PK0637-752 we present some of the ARI-L continuum images for Cycle 3
projects observed in 2016 with average frequency of $\sim$105 GHz that have
been ingested into the archive thus far. We extracted light curves from the
core, jet and hot-spot regions that are similar to what could be used for
investigating the evolution of variability along the source substructures. We
also compared the core region light curves at $\sim$105 GHz and at $\sim$93
GHz with the data available in similar frequency ranges in the ALMA Calibrator
Manual*†*†*†https://almascience.eso.org/alma-data/calibrator-catalogue, which
provides data with very good time coverage but does not provide images.

Figure 5: ARI-L images of PKS 0521-365 (top) observed in project
2015.1.00503.S MOUS uid://A002/Xb0eb5c/X50 and 3C273 (bottom) observed in
project 2016.1.01340.S MOUS uid://A001/X87d/X9ed.
2016-01-072016-01-122016-01-242016-03-102016-03-262016-04-042016-04-162016-06-182016-12-31$\
\ Jet\ \ $$\ \ Core\ \ $$Hotspot$ Figure 6: ARI-L images of 105 GHz
observations of PKS 0637-752 from Cycle 3 projects observed in 2016 (with
labels indicating the observation date) and light curves of the hot-spot,
core, and jet regions (with error bars that include a calibration error equal
to 5% of the flux density). The various angular resolutions of the data
available in the archive make it possible to investigate the source structure.
This blazar is strongly core-dominated, and the second brightest component is
only 2 orders of magnitude fainter than the core in this band. The projects
used for this figure are 2015.1.01046.S, 2015.1.01195.S, 2015.1.00190.S
2015.1.00204.S, 2015.1.00196.S, 2015.1.01388.S, 2015.1.00697.S, and
2016.1.00193.S. Figure 7: The 2016 light curves extracted from the core region
of PKS 0637-752. Data from the ARI-L images at $\sim$93 and $\sim$105 GHz are
shown as blue and cyan diamonds, while the ALMA Calibrator catalogues values
available at 91.5 and 103.5 GHz are shown as red and orange squares.

Figure 8: Zeroth moments of the brightness distribution (corresponding to
integrated brightness) of the CO(5-4) line for the dusty obscured galaxy XID42
(D’Amato et al. 2020) obtained exploiting the QA2 (top left) and the rebinned
ARI-L (center) products for MOUS uid://A001/X340/X2c of the project
2015.1.01205.S. For comparison we present also the the difference between the
zeroth moments (top right), and the comparison of the line spectra

The ARI-L product images are produced at the native spectral resolution. Quite
often, spectral features cannot immediately be detected within ALMA data in
their native spectral resolution, in which case the CASA task IMREBIN could be
used to enhance spectral line signals above the noise. IMREBIN, in fact,
averages along the spectral axis of a cube by a given factor without
considering any difference among the signals in the different channels
*‡*‡*‡https://casa.nrao.edu/docs/taskref/imrebin-task.html.

In an interferometric dataset, visibilities for the same baseline B at
different frequencies, $\nu$, observe emission on different angular scales,
$\theta$, according to the relation $\theta\propto Bc/\nu$, where $c$ is the
speed of light.

On the one hand, when generating images at their native spectral resolution,
the different channels sample different angular scales, and the difference in
the angular scales is related to the difference in the frequencies sampled by
the channels. When contiguous channels are averaged together in the rebinning
task, channels with information about emission on different scales are
averaged together. Furthermore, different channels correspond to different
synthesized beam sizes, so even a point source appears larger at longer
wavelength. (Also, IMREBIN does not apply any smoothing before averaging.)

On the other hand, if visibilities from different channels are used in the
generation of an image with lower spectral resolution, then they contribute to
the definition of larger channels within the image cube that combine the
signals on all the sampled angular scales according to the chosen weighting
scheme.

The difference between the high-resolution rebinned products and the low-
resolution imaged products is more significant in case the Fourier transform
of the target brightness distribution is rapidly variable or if the overall
coverage of the Fourier domain is sparse, as the products might be more
sensitive to different visibility sampling.

It is always recommended to reproduce the images when specific spectral
resolutions are needed. Nonetheless, we investigated the differences between a
rebinned ARI-L image to an image created with clean or tclean using a lower
spectral resolution. For this analysis, we used observations of the CO(5-4)
line in the spectrum of an obscured AGN at redshift 2.94 (D’amato et al. 2020,
see fig. 8). CO lines are well-recognized tracers of molecular gas and
indirectly of star formation at any redshift (Kennicutt 1998, Carilli et al.
2013), with the highest transitions more significantly excited in denser and
more energetic environments like those surrounding the supermassive black
holes in AGN. There is increasing evidence of common evolutionary behaviour
between star formation and BHs, with both cosmic densities peaking at redshift
2-3 (Madau & Dickinson 2014, Gruppioni et al. 2013).

According to models of BH-star formation in-situ co-evolution (Lapi et al.
2018, Pantoni et al. 2019), dusty star forming galaxies should span regions
with sizes of 1-5 kpc at the peak of cosmic evolution. This corresponds to
0.12-0.6 arcseconds, which is easily achieved by ALMA in its intermediate
configurations, even at the lowest frequencies. In these data, the galaxies
might appear barely resolved (Pantoni et al. in prep, D’Amato et al. 2020).

CO lines in galaxies at the peak of cosmic star formation might be several
hundreds of km/s in width, corresponding to many tenths of MHz at
$\sim$100GHz. This would span a few tens of channels in ALMA data, even in
data with low spectral resolutions. However, the presence (or the absence) of
a spectral line in a target might determine its inclusion in selected samples,
so it is important to quickly establish if a line could be detected from
archived images.

Fig.8 presents the zeroth moment of the CO(5-4) spectral line for XID 42 as
obtained by imaging the target at the spectral resolution of 70 km/s, which is
what was available from the archived QA2 products (i.e. imaged by QA2
analysts). For comparison, the ARI-L image of the same target (calibrated with
the same script, but imaged with the Imaging Pipeline) was rebinned by a
factor 4 along the frequency axis to match its spectral resolution to the QA2
data as closely as possible. The rms noise in the zeroth moment images is 0.31
and 0.34 (mJy/beam) km/s respectively for the QA2 and rebinned ARI-L images.
At the position of the source, the maximum absolute difference between the two
is 0.28 (mJy/beam) km/s. When the resulting spectra are also compared, the
spectral lines are clearly similar within the noise variation, and the pattern
of noise (that, in an interferometric image, is not genuinely Poissonian but
determined by the visibility domain coverage that is common between the two
images) is similar.

Hence, the detection of a CO line in the rebinned ARI-L image is indicative of
the presence of the line in images produced with clean or tclean using coarser
resolution settings.

Similarly, first moment maps of spectral line distributions produced from
ARI-L images could be informative of the kinematic behaviour of the objects.
Figure 9 shows the moment images corresponding to the integrated brightness,
velocity distribution, and velocity dispersion that have been obtained from
the archived ARI-L image of the CO(6-5) line of the lensed galaxy SDP9. These
images clearly show the presence of a disk or an outflow in this $z=1.57$
lensed galaxy at the same significance levels that members of our
collaboration achieved a few years ago in a complete analysis of the archived
dataset (see fig. 5 in Massardi et al. 2018 for comparison).

Figure 9: Zeroth, first, and second moment images (corresponding to integrated
brightness, velocity distribution and velocity dispersion; top to bottom) of
the CO(6-5) line for the lensed galaxy SDP9 obtained using the ARI-L products
for MOUS uid://A001/X2de/X28 of the project 2015.1.00415.S (for more info see
Massardi et al. 2018).

##  4 Summary

The Additional Representative Images for Legacy (ARI-L) project is a European
Development project for ALMA Upgrade approved by JAO and ESO that officially
started in June 2019 and is currently on-going and well-ahead of schedule. The
project aims to increase the legacy value of the ASA by bringing the reduction
level of ALMA data from Cycles 2-4 close to the level of the more recent
Cycles processed with the ALMA Imaging Pipeline. Over a period of three years,
the ARI-L project will produce and ingest into the ASA a uniform set of full
data cubes and continuum images at native spectral resolution and average
weighting scheme, that cover at least 70% of the observational data from
Cycles 2-4 that can be processed with the ALMA Imaging Pipeline.

By the end of the project, we expect to have produced and delivered to the
ALMA Science Archive images for at least 2733 MOUS, which corresponds to a
minimum of 70% of the 3476 MOUS that we expect the current version of the
Imaging Pipeline could process. As of June 2021, more than 1800 of them have
been reprocessed, and more than 150000 images have been returned to the ASA
and are available to astronomers. These products can already be retrieved from
the ALMA Science Archive as ”Externally delivered products” on the Request
Handler download page; the ARI-L products are listed directly below the ALMA
data. As of June 2021, there have been more than 146800 downloads of a file
tagged ”ari-l” from the ASA, either as an individual file or included in
packages (ASA staff private communication; a single file can be downloaded
more than once). Furthermore, all the corresponding calibrated measurement
sets are stored on the INAF-IA2 facilities and can be requested through the
ARI-L project website https://almascience.eso.org/alma-data/aril where also
further details on the project status are available.

The ARI-L cubes and continuum images complement the much smaller QA2-generated
archival image products, which cover only a small fraction of the available
data for Cycles 2-4. ARI-L imaging products are highly relevant for many
science cases and significantly enhance the possibilities for exploiting
archival data. They facilitate archive access and data usage for science
purposes even for non-expert data miners, they provide a homogeneous view of
archival data for better dataset comparisons and download selections, they
make the archive more accessible to visualization and analysis tools, and they
enable the generation of preview images and plots similar to those that can be
created for subsequent Cycles.

## Acknowledgements

We thank Leonardo Testi, Tony Mroczkowski, Ciska Kemper, Carlos De Breuck, and
the ARI-L review committee chaired by Erich Schmid for the useful comments
they provided to our project. We thank Sandra Burkutean, Andrea Lapi, Quirino
D’Amato, Lara Pantoni, and Giovanni Sabatini for their contribution to the
discussion on the reported examples of science cases and to the project. We
thank the anonymous Referee for the constructive comments to our manuscript.

This paper makes use of the following ALMA data: ADS/JAO.ALMA#2016.1.00288.S,
ADS/JAO.ALMA#2016.1.00875.S,
ADS/JAO.ALMA#2015.1.00503.S
ADS/JAO.ALMA#2016.1.01340.S
ADS/JAO.ALMA#2015.1.01046.S,
ADS/JAO.ALMA#2015.1.01195.S,
ADS/JAO.ALMA#2015.1.00190.S,
ADS/JAO.ALMA#2015.1.00204.S,
ADS/JAO.ALMA#2015.1.00196.S,
ADS/JAO.ALMA#2015.1.01388.S,
ADS/JAO.ALMA#2015.1.00697.S,
ADS/JAO.ALMA#2016.1.00193.S,
ADS/JAO.ALMA#2015.1.01205.S, and
ADS/JAO.ALMA#2015.1.00415.S as examples of the Cycle 2–4 projects included in
the ARI-L processing lists. ALMA is a partnership of ESO (representing its
member states), NSF (USA) and NINS (Japan), together with NRC (Canada), MOST
and ASIAA (Taiwan), and KASI (Republic of Korea), in cooperation with the
Republic of Chile. The Joint ALMA Observatory is operated by ESO, AUI/NRAO and
NAOJ.

MM acknowledges the support from grant PRIN MIUR 2017 - 20173ML3WW_001.

MB acknowledges support from the Ministero degli Affari Esteri della
Cooperazione Internazionale - Direzione Generale per la Promozione del Sistema
Paese Progetto di Grande Rilevanza ZA18GR02.

## Data Availability

All the input raw data and the ARI-L product already made public are available
through the ALMA Science Archive query interface at
https://almascience.eso.org/asax/.

## References

## References

  * [1] Abdo A. A., Ackermann M., Ajello M., Antolini E., Baldini L., Ballet J., Barbiellini G., et al., 2010, ApJ, 722, 520. doi:10.1088/0004-637X/722/1/520
  * [2] Agliozzo C., Trigilio C., Pignata G., Phillips N. M., Nikutta R., Leto P., Umana G., et al., 2017, ApJ, 841, 130. doi:10.3847/1538-4357/aa72a1
  * [3] Aharonian F., Akhperjanian A. G., Bazer-Bachi A. R., Behera B., Beilicke M., Benbow W., Berge D., et al., 2007, ApJL, 664, L71. doi:10.1086/520635
  * [4] Bonato M., Liuzzo E., Giannetti A., Massardi M., De Zotti G., Burkutean S., Galluzzi V., et al., 2018, MNRAS, 478, 1512. doi:10.1093/mnras/sty1173
  * [5] Bonato M., Liuzzo E., Herranz D., González-Nuevo J., Bonavera L., Tucci M., Massardi M., et al., 2019, MNRAS, 485, 1188. doi:10.1093/mnras/stz465
  * [6] Burkutean S., Giannetti A., Liuzzo E., Massardi M., Rygl K., Brand J., Bedosti F., et al., 2018, JATIS, 4, 028001. doi:10.1117/1.JATIS.4.2.028001
  * [7] Carilli, C. L. & Walter, F. 2013, ARA&A, 51, 105. doi:10.1146/annurev-astro-082812-140953
  * [8] Cesaroni R., Sánchez-Monge Á., Beltrán M. T., Johnston K. G., Maud L. T., Moscadelli L., Mottram J. C., et al., 2017, A&A, 602, A59. doi:10.1051/0004-6361/201630184
  * [9] D’Amato, Q., Gilli, R., Vignali, C., et al. 2020, A&A, 636, A37. doi:10.1051/0004-6361/201936175
  * [10] Galluzzi V., Massardi M., Bonaldi A., Casasola V., Gregorini L., Trombetti T., Burigana C., et al., 2017, MNRAS, 465, 4085. doi:10.1093/mnras/stw3017
  * [11] Hovatta T., O’Sullivan S., Martí-Vidal I., Savolainen T., Tchekhovskoy A., 2019, A&A, 623, A111. doi:10.1051/0004-6361/201832587
  * [12] Kennicutt, R. C. 1997, Astrophysics and Space Science Library, 171
  * [13] Lapi A., Pantoni L., Zanisi L., Shi J., Mancuso C., Massardi M., Shankar F., et al., 2018, ApJ, 857, 22. doi:10.3847/1538-4357/aab6af
  * [14] Liuzzo E., Nagai H., Giovannini G., Mignano A., 2015, ASPC, 499, 129
  * [15] Loiacono, F., Decarli, R., Gruppioni, C., et al. 2021, A&A, 646, A76. doi:10.1051/0004-6361/202038607
  * [16] Madau, P. & Dickinson, M. 2014, ARA&A, 52, 415. doi:10.1146/annurev-astro-081811-125615
  * [17] Massardi M., et al., 2019, ALMA memo series, 614, http://library.nrao.edu/public/memos/alma/main/memo614.pdf
  * [18] Massardi M., Enia A. F. M., Negrello M., Mancuso C., Lapi A., Vignali C., Gilli R., et al., 2018, A&A, 610, A53. doi:10.1051/0004-6361/201731751
  * [19] Maud L. T., Cesaroni R., Kumar M. S. N., Rivilla V. M., Ginsburg A., Klaassen P. D., Harsono D., et al., 2019, A&A, 627, L6. doi:10.1051/0004-6361/201935633
  * [20] McMullin J. P., Waters B., Schiebel D., Young W., Golap K., 2007, ASPC, 376, 127
  * [21] Meyer E. T., Breiding P., Georganopoulos M., Oteo I., Zwaan M. A., Laing R., Godfrey L., et al., 2017, ApJL, 835, L35. doi:10.3847/2041-8213/835/2/L35
  * [22] Moscadelli L., Sanna A., Cesaroni R., Rivilla V. M., Goddi C., Rygl K. L. J., 2019, A&A, 622, A206. doi:10.1051/0004-6361/201834366
  * [23] Pantoni L., Lapi A., Massardi M., Goswami S., Danese L., 2019, ApJ, 880, 129. doi:10.3847/1538-4357/ab2adc
  * [24] Petry D., Stanke T., Biggs A., Díaz Trigo M., Guglielmetti F., Hatziminaoglou E., van Kampen E., et al., 2020, Msngr, 181, 16. doi:10.18727/0722-6691/5206
  * [25] Stoehr F., 2017, Msngr, 169, 53. doi:10.18727/0722-6691/5040
  * [26] Stoehr F., 2019, ASPC, 523, 387
  * [27] Tavecchio F., Becerra-Gonzalez J., Ghisellini G., Stamerra A., Bonnoli G., Foschini L., Maraschi L., 2011, A&A, 534, A86. doi:10.1051/0004-6361/201117204
  * [28] Teuben P., Pound M., Mundy L., Rauch K., Friedel D., Looney L., Xu L., et al., 2015, ASPC, 495, 305
  * [29] Ulrich M.-H., Maraschi L., Urry C. M., 1997, ARA&A, 35, 445. doi:10.1146/annurev.astro.35.1.445
  * [30] Urry C. M., Padovani P., 1995, PASP, 107, 803. doi:10.1086/133630
  * [31] Webb J. R., 2006, ASPC, 350, 3


# Defending Pre-trained Language Models
from Adversarial Word Substitution Without Performance Sacrifice

Rongzhou Bao, Jiayi Wang, Hai Zhao111 Corresponding author. This paper was
partially sup- ported by National Key Research and Development Pro- gram of
China (No. 2017YFB0304100), Key Projects of National Natural Science
Foundation of China (U1836222 and 61733011). This work was supported by Huawei
Noah’s Ask Lab
Department of Computer Science and Engineering, Shanghai Jiao Tong University
2 Key Laboratory of Shanghai Education Commission for Intelligent Interaction
and Cognitive Engineering, Shanghai Jiao Tong University, Shanghai, China
MoE Key Lab of Artificial Intelligence, AI Institute,
Shanghai Jiao Tong University, Shanghai, China
<EMAIL_ADDRESS>
{wangjiayi_102_23}<EMAIL_ADDRESS>

###### Abstract

Pre-trained contextualized language models (PrLMs) have led to strong
performance gains in downstream natural language understanding tasks. However,
PrLMs can still be easily fooled by adversarial word substitution, which is
one of the most challenging textual adversarial attack methods. Existing
defence approaches suffer from notable performance loss and complexities.
Thus, this paper presents a compact and performance-preserved framework,
Anomaly Detection with Frequency-Aware Randomization (ADFAR). In detail, we
design an auxiliary anomaly detection classifier and adopt a multi-task
learning procedure, by which PrLMs are able to distinguish adversarial input
samples. Then, in order to defend adversarial word substitution, a frequency-
aware randomization process is applied to those recognized adversarial input
samples. Empirical results show that ADFAR significantly outperforms those
newly proposed defense methods over various tasks with much higher inference
speed. Remarkably, ADFAR does not impair the overall performance of PrLMs. The
code is available at https://github.com/LilyNLP/ADFAR.

##  1 Introduction

Deep neural networks (DNNs) have achieved remarkable success in various areas.
However, previous works show that DNNs are vulnerable to adversarial samples
Goodfellow et al. (2015); Kurakin et al. (2017); Wang et al. (2021), which are
inputs with small, intentional modifications that cause the model to make
false predictions. Pre-trained language models (PrLMs) Devlin et al. (2019);
Liu et al. (2019); Clark et al. (2020); Zhang et al. (2020, 2019) are widely
adopted as an essential component for various NLP systems. However, as DNN-
based models, PrLMs can still be easily fooled by textual adversarial samples
Wallace et al. (2019); Jin et al. (2019); Nie et al. (2020); Zang et al.
(2020). Such vulnerability of PrLMs keeps raising potential security concerns,
therefore researches on defense techniques to help PrLMs against textual
adversarial samples are imperatively needed.

Different kinds of textual attack methods have been proposed, ranging from
character-level word misspelling Gao et al. (2018), word-level substitution
Alzantot et al. (2018); Ebrahimi et al. (2018); Ren et al. (2019); Jin et al.
(2019); Zang et al. (2020); Li et al. (2020); Garg and Ramakrishnan (2020),
phrase-level insertion and removal Liang et al. (2018), to sentence-level
paraphrasing Ribeiro et al. (2018); Iyyer et al. (2018). Thanks to the
discrete nature of natural language, attack approaches that result in illegal
or unnatural sentences can be easily detected and restored by spelling
correction and grammar error correction Islam and Inkpen (2009); Sakaguchi et
al. (2017); Pruthi et al. (2019). However, attack approaches based on
adversarial word substitution can produce high-quality and efficient
adversarial samples which are still hard to be detected by existing methods.
Thus, the adversarial word substitution keeps posing a larger and more
profound challenge for the robustness of PrLMs. Therefore, this paper is
devoted to overcome the challenge posed by adversarial word substitution.

Several approaches are already proposed to mitigate issues posed by
adversarial word substitution Zhou et al. (2019); Jia et al. (2019); Huang et
al. (2019); Cohen et al. (2019); Ye et al. (2020); Si et al. (2021). Although
these defense methods manage to alleviate the negative impact of adversarial
word substitution, they sometimes reduce the prediction accuracy for non-
adversarial samples to a notable extent. Given the uncertainty of the
existence of attack in real application, it is impractical to sacrifice the
original prediction accuracy for the purpose of defense. Moreover, previous
defense methods either have strong limitations over the attack space to
certify the robustness, or require enormous computation resources during
training and inference. Thus, it is imperatively important to find an
efficient performance-preserved defense method.

For such purpose, we present a compact and performance-preserved framework,
Anomaly Detection with Frequency-Aware Randomization (ADFAR), to help PrLMs
defend against adversarial word substitution without performance sacrifice.
Xie et al. (2018) show that introducing randomization at inference can
effectively defend adversarial attacks. Moreover, Mozes et al. (2020) indicate
that the usual case for adversarial samples is replacing words with their less
frequent synonyms, while PrLMs are more robust to frequent words. Therefore,
we propose a frequency-aware randomization process to help PrLMs defend
against adversarial word substitution.

However, simply applying a randomization process to all input sentences would
reduce the prediction accuracy for non-adversarial samples. In order to
preserve the overall performance, we add an auxiliary anomaly detector on top
of PrLMs and adopt a multi-task learning procedure, by which PrLMs are able to
determine whether each input sentence is adversarial or not, and not introduce
extra model. Then, only those adversarial input sentences will undergo the
randomization procedure, while the prediction process for non-adversarial
input sentences remains the same.

Empirical results show that as a more efficient method, ADFAR significantly
outperforms previous defense methods Ye et al. (2020); Zhou et al. (2019) over
various tasks, and preserves the prediction accuracy for non-adversarial
sentences. Comprehensive ablation studies and analysis further prove the
efficiency of our proposed method, and indicate that the adversarial samples
generated by current heuristic word substitution strategy can be easily
detected by the proposed auxiliary anomaly detector.

Figure 1: Frequency-aware randomization examples.

##  2 Related Work

###  2.1 Adversarial Word Substitution

Adversarial word substitution (AWS) is one of the most efficient approaches to
attack advanced neural models like PrLMs. In AWS, an attacker deliberately
replaces certain words by their synonyms to mislead the prediction of the
target model. At the same time, a high-quality adversarial sample should
maintain grammatical correctness and semantic consistency. In order to craft
efficient and high-quality adversarial samples, an attacker should first
determine the vulnerable tokens to be perturbed, and then choose suitable
synonyms to replace them.

Current AWS models Alzantot et al. (2018); Ebrahimi et al. (2018); Ren et al.
(2019); Jin et al. (2019); Li et al. (2020); Garg and Ramakrishnan (2020)
adopt heuristic algorithms to locate vulnerable tokens in sentences. To
illustrate, for a given sample and a target model, the attacker iteratively
masks the tokens and checks the output of the model. The tokens which have
significant influence on the final output logits are regarded as vulnerable.

Previous works leverage word embeddings such as GloVe Pennington et al. (2014)
and counter-fitted vectors Mrkšić et al. (2016) to search the suitable synonym
set of a given token. Li et al. (2020); Garg and Ramakrishnan (2020) uses BERT
Devlin et al. (2019) to generate perturbation for better semantic consistency
and language fluency.

###  2.2 Defense against AWS

For general attack approaches, adversarial training Goodfellow et al. (2015);
Jiang et al. (2020) is widely adopted to mitigate adversarial effect, but
Alzantot et al. (2018); Jin et al. (2019) shows that this method is still
vulnerable to AWS. This is because AWS models leverage dynamic algorithms to
attack the target model, while adversarial training only involves a static
training set.

Methods proposed by Jia et al. (2019); Huang et al. (2019) are proved
effective for defence against AWS, but they still have several limitations. In
these methods, Interval Bound Propagation (IBP) Dvijotham et al. (2018), an
approach to consider the worst-case perturbation theoretically, is leveraged
to certify the robustness of models. However, IBP-based methods can only
achieve the certified robustness under a strong limitation over the attack
space. Furthermore, they are difficult to adapt to PrLMs for their strong
reliance on the assumption of model architecture.

Two effective and actionable methods (DISP Zhou et al. (2019) and SAFER Ye et
al. (2020)) are proposed to overcome the challenge posed by AWS, and therefore
adopted as the baselines for this paper. DISP Zhou et al. (2019) is a
framework based on perturbation discrimination to block adversarial attack. In
detail, when facing adversarial inputs, DISP leverages two auxiliary PrLMs:
one to detect perturbed tokens in the sentence, and another to restore the
abnormal tokens to original ones. Inspired by randomized smoothing Cohen et
al. (2019), Ye et al. (2020) proposes SAFER, a novel framework that guarantees
the robustness by smoothing the classifier with synonym word substitution. To
illustrate, based on random word substitution, SAFER smooths the classifier by
averaging its outputs of a set of randomly perturbed inputs. SAFER outperforms
IBP-based approaches and can be easily applied to PrLMs.

###  2.3 Randomization

In recent years, randomization has been used as a defense measure for deep
learning in computer vision Xie et al. (2018). Nevertheless, direct extensions
of these measures to defend against textual adversarial samples are not
achievable, since the text inputs are discrete rather than continuous. Ye et
al. (2020) indicates the possibility of extending the application of the
randomization approach to NLP by randomly replacing the words in sentences
with their synonyms.

##  3 Method

###  3.1 Frequency-aware Randomization

Since heuristic attack methods attack a model by substituting each word
iteratively until it successfully alters the model’s output, it is normally
difficult for static strategies to defense such kind of dynamic process.
Rather, dynamic strategies, such as randomization, can better cope with the
problem. It is also observed that replacing words with their more frequent
alternatives can better mitigate the adversarial effect and preserve the
original performance. Therefore, a frequency-aware randomization strategy is
designed to perplex AWS strategy.

Figure 1 shows several examples of the frequency-aware randomization. The
proposed approach for the frequency-aware randomization is shown in Algorithm
1, and consists of three steps. Firstly, rare words with lower frequencies and
a number of random words are selected as substitution candidates. Secondly, we
choose synonyms with the closest meanings and the highest frequencies to form
a synonym set for each candidate word. Thirdly, each candidate word is
replaced with a random synonym within its own synonym set. To quantify the
semantic similarity between two words, we represent words with embeddings from
Mrkšić et al. (2016), which is specially designed for synonyms identification.
The semantic similarity of two words are evaluated by cosine similarity of
their embeddings. To determine the frequency of a word, we use a frequency
dictionary provided by FrequencyWords
Repository111https://github.com/hermitdave/FrequencyWords.

Algorithm 1 Frequency-aware Randomization

0: Sentence $X=\\{w_{1},w_{2},...,w_{n}\\}$, word embeddings $Emb$ over the
vocabulary $Vocab$

0: Randomized sentence $X_{rand}$

1: Initialization: $X_{rand}\leftarrow X$

2: Create a set $W_{rare}$ of all rare words with frequencies less than
$f_{thres}$, denote $n_{rare}=|W_{rare}|$.

3: Create a set $W_{rand}$ by randomly selecting $n*r-n_{rare}$ words
$w_{j}\notin W_{rare}$, where $r$ is the pre-defined ratio of substitution.

4: Create the substitution candidates set, $W_{sub}\leftarrow
W_{rare}+W_{rand}$, and $|W_{sub}|=n*r$.

5: Filter out the stop words in $W_{sub}$.

6: for each word $w_{i}$ in $W_{sub}$ do

7: Create a set $S$ by extracting the top $n_{s}$ synonyms using
$CosSim(Emb_{w_{i}},Emb_{w_{word}})$ for each word in $Vocab$.

8: Create a set $S_{freq}$ by selecting the top $n_{f}$ frequent synonyms from
$S$.

9: Randomly choose one word $w_{s}$ from $S$.

10: $X_{rand}\leftarrow$ Replace $w_{i}$ with $w_{s}$ in $X_{rand}$.

11: end for

###  3.2 Anomaly Detection

Applying the frequency-aware randomization process to every input can still
reduce the prediction accuracy for normal samples. In order to overcome this
issue, we add an auxiliary anomaly detection head to PrLMs and adopt a multi-
task learning procedure, by which PrLMs are able to classify the input text
and distinguish the adversarial samples at the same time, and not introduce
extra model. In inference, the frequency-aware randomization only applied to
the samples that are detected as adversarial. In this way, the reduction of
accuracy is largely avoided, since non-adversarial samples are not affected.

Zhou et al. (2019) also elaborates the idea of perturbation discrimination to
block attack. However, their method detects anomaly on token-level and
requires two resource-consuming PrLMs for detection and correction, while ours
detects anomaly on sentence-level and requires no extra models. Compared to
Zhou et al. (2019), our method is two times faster in inference speed and can
achieve better accuracy for sentence-level anomaly detection.

###  3.3 Framework

In this section, we elaborate the framework of ADFAR in both training and
inference.

####  3.3.1 Training

Figure 2: Framework of ADFAR in training.

Figure 2 shows the framework of ADFAR in training. We extend the baseline
PrLMs by three major modifications: 1) the construction of training data, 2)
the auxiliary anomaly detector and 3) the training objective, which will be
introduced in this section.

##### Construction of Training Data

As shown in Figure 2, we combine the idea of both adversarial training and
data augmentation Wei and Zou (2019) to construct our randomization augmented
adversarial training data. Firstly, we use a heuristic AWS model (e.g.
TextFooler) to generate adversarial samples based on the original training
set. Following the common practice of adversarial training, we then combine
the adversarial samples with the original ones to form an adversarial training
set. Secondly, in order to let PrLMs better cope with randomized samples in
inference, we apply the frequency-aware randomization on the adversarial
training set to generate a randomized adversarial training set. Lastly, the
adversarial training set and the randomized adversarial training set are
combined to form a randomization augmented adversarial training set.

##### Auxiliary Anomaly Detector

In addition to the original text classifier, we add an auxiliary anomaly
detector to the PrLMs to distinguish adversarial samples. For an input
sentence, the PrLMs captures the contextual information for each token by
self-attention and generates a sequence of contextualized embeddings
$\\{h_{0},\dots,h_{m}\\}$. For text classification task, $h_{0}\in R^{H}$ is
used as the aggregate sequence representation. The original text classifier
leverages $h_{0}$ to predict the probability that $X$ is labeled as class
$\hat{y_{c}}$ by a logistic regression with softmax:

$\displaystyle y_{c}$ $\displaystyle=Prob(\hat{y_{c}}|x),$ $\displaystyle={\rm
softmax}(W_{c}(dropout(h_{0}))+b_{c}),$

For the anomaly detector, the probability that $X$ is labeled as class
$\hat{y_{d}}$ (if $X$ is attacked, $\hat{y_{d}}=1$; if $X$ is normal,
$\hat{y_{d}}=0$) is predicted by a logistic regression with softmax:

$\displaystyle y_{d}$ $\displaystyle=Prob(\hat{y_{d}}|x),$ $\displaystyle={\rm
softmax}(W_{d}(dropout(h_{0}))+b_{d}),$

As shown in Figure 2, the original text classifier is trained on the
randomization augmented adversarial training set, whereas the anomaly detector
is only trained on the adversarial training set.

##### Training Objective

We adopt a multi-task learning framework, by which PrLM is trained to classify
the input text and distinguish the adversarial samples at the same time. We
design two parallel training objectives in the form of minimizing cross-
entropy loss: $loss_{c}$ for text classification and $loss_{d}$ for anomaly
detection. The total loss function is defined as their sum:

$\begin{split}loss_{c}&=-[y_{c}*{\rm log}(\hat{y_{c}})+(1-y_{c})*{\rm
log}(1-\hat{y_{c}})]\\\ loss_{d}&=-[y_{d}*{\rm
log}(\hat{y_{d}})+(1-y_{d})*{\rm log}(1-\hat{y_{d}})]\\\
Loss&=loss_{c}+loss_{d}\end{split}$

####  3.3.2 Inference

Figure 3: Framework of ADFAR in inference.

Figure 3 shows the framework of ADFAR in inference. Firstly, the anomaly
detector predicts whether an input sample is adversarial. If the input sample
is determined as non-adversarial, the output of the text classifier (Label A)
is directly used as its final prediction. If the input sample is determined as
adversarial, the frequency-aware randomization process is applied to the
original input sample. Then, the randomized sample is sent to the PrLM again,
and the second output of the text classifier (Label B) is used as its final
prediction.

##  4 Experimental Implementation

###  4.1 Tasks and Datasets

Experiments are conducted on two major NLP tasks: text classification and
natural language inference. The dataset statistics are displayed in Table 1.
We evaluate the performance of models on the non-adversarial test samples as
the original accuracy. Then we measure the after-attack accuracy of models
when facing AWS. By comparing these two accuracy scores, we can evaluate how
robust the model is.

Task | Dataset | Train | Test | Avg Len
---|---|---|---|---
Classification | MR | 9K | 1K | 20
SST2 | 67K | 1.8K | 20
IMDB | 25K | 25K | 215
Entailment | MNLI | 433K | 10K | 11
Table 1:  Dataset statistics.

##### Text Classification

We use three text classification datasets with average text lengths from 20 to
215 words, ranging from phrase-level to document-level tasks. SST2 Socher et
al. (2013): phrase-level binary sentiment classification using fine-grained
sentiment labels on movie reviews. MR Pang and Lee (2005): sentence-level
binary sentiment classification on movie reviews. We take 90$\%$ of the data
as training set and 10$\%$ of the data as test set as Jin et al. (2019). IMDB
Maas et al. (2011): document-level binary sentiment classification on movie
reviews.

##### Natural Language Inference

NLI aims at determining the relationship between a pair of sentences based on
semantic meanings. We use Multi-Genre Natural Language Inference (MNLI) Nangia
et al. (2017), a widely adopted NLI benchmark with coverage of transcribed
speech, popular fiction, and government reports.

Model | MR | SST2 | IMDB | MNLI
---|---|---|---|---
Orig. Acc. | Adv. Acc. | Orig. Acc. | Adv. Acc. | Orig. Acc. | Adv. Acc. | Orig. Acc. | Adv. Acc.
BERT | 86.2 | 16.9 | 93.1 | 39.8 | 92.4 | 12.4 | 84.0 | 11.3
BERT + Adv Training | 85.6 | 34.6 | 92.6 | 48.8 | 92.2 | 34.2 | 82.3 | 33.4
BERT + DISP | 82.0 | 42.2 | 91.6 | 70.4 | 91.7 | 82.0 | 76.3 | 35.1
BERT + SAFER | 79.0 | 55.4 | 91.3 | 75.6 | 91.3 | 88.1 | 82.1 | 54.7
BERT + ADFAR | 86.6 | 66.0 | 92.4 | 75.6 | 92.8 | 89.2 | 82.6 | 67.8
Table 2:  The performance of ADFAR and other defense frameworks using
BERT${}_{\text{BASE}}$ as PrLM and TextFooler as attack model. Orig. Acc. is
the prediction accuracy of normal samples and Adv. Acc. is the after-attack
accuracy of models when facing AWS. The results are based on the average of
five runs.

###  4.2 Attack Model and Baselines

We use TextFooler222https://github.com/jind11/TextFoolerJin et al. (2019) as
the major attack model for AWS. Moreover, we implement Ren et al. (2019) and
GENETIC Alzantot et al. (2018) based on the TextAttack Morris et al. (2020)
code base to further verify the efficiency of our proposed method.

We compare ADFAR with DISP Zhou et al. (2019) and SAFER Ye et al. (2020). The
implementation of DISP is based on the repository offered by Zhou et al.
(2019). For SAFER, we also leverage the code44footnotetext:
https://github.com/lushleaf/Structure-free-certified-NLP proposed by Ye et al.
(2020). Necessary modifications are made to evaluate these methods’
performance under heuristic attack models.

###  4.3 Experimental Setup

The implementation of PrLMs is based on
PyTorch333https://github.com/joey1993/bert-defender. We leverage,
BERT${}_{\text{BASE}}$ Devlin et al. (2019), RoBERTa${}_{\text{BASE}}$ Liu et
al. (2019) and ELECTRA${}_{\text{BASE}}$ Clark et al. (2020) as baseline
PrLMs. We use AdamW Loshchilov and Hutter (2018) as our optimizer with a
learning rate of 3e-5 and a batch size of 16. The number of epochs is set to
5. 55footnotetext: https://github.com/huggingface

For the frequency-aware randomization process, we set $f_{thres}=200$,
$n_{s}=20$ and $n_{f}=10$. In the adopted frequency dictionary, 5.5k out of
50k words have a frequency lower than $f_{thres}=200$ and therefore regarded
as rare words. $r$ is set to different values for training ($25\%$) and
inference ($30\%$) due to different aims. In training, to avoid introducing
excessive noise and reduce the prediction accuracy for non-adversarial
samples, $r$ is set to be relatively low. On the contrary, in inference, our
aim is to perplex the heuristic attack mechanism. The more randomization we
add, the more perplexities the attack mechanism receives, therefore we set a
relatively higher value for $r$. More details on the choice of these
hyperparameters will be discussed in the analysis section.

##  5 Experimental Results

###  5.1 Main results

Following Jin et al. (2019), we leverage BERT${}_{\text{BASE}}$ Devlin et al.
(2019) as baseline PrLM and TextFooler as attack model. Table 2 shows the
performance of ADFAR and other defense frameworks. Since randomization may
lead to a variance of the results, we report the results based on the average
of five runs. Experimental results indicate that ADFAR can effectively help
PrLM against AWS. Compared with DISP Zhou et al. (2019) and SAFER Ye et al.
(2020), ADFAR achieves the best performance for adversarial samples.
Meanwhile, ADFAR does not hurt the performance for non-adversarial samples in
general. On tasks such as MR and IMDB, ADFAR can even enhance the baseline
PrLM.

DISP leverages two extra PrLMs to discriminate and recover the perturbed
tokens, which introduce extra complexities. SAFER makes the prediction of an
input sentence by averaging the prediction results of its perturbed
alternatives, which multiply the inference time. As shown in Table 3, compared
with previous methods, ADFAR achieves a significantly higher inference speed.

Model | Parameters | Inference Time
---|---|---
BERT${}_{\text{BASE}}$ | 110M | 15.7ms (100%)
BERT${}_{\text{BASE}}$ \+ DISP | 330M | 38.9ms (247%)
BERT${}_{\text{BASE}}$ \+ SAFER | 110M | 27.6ms (176%)
BERT${}_{\text{BASE}}$ \+ ADFAR | 110M | 18.1ms (115%)
Table 3:  Parameters and Inference Time statistics. The inference time
indicate the average inference time for one sample in MR dataset using one
NVIDIA RTX3090.

###  5.2 Results with Different Attack Strategy

Since ADFAR leverages the adversarial samples generated by TextFooler Jin et
al. (2019) in training, it is important to see whether ADFAR also performs
well when facing adversarial samples generated by other AWS models. We
leverage PWWS Ren et al. (2019) and GENETIC Alzantot et al. (2018) to further
study the performance of ADFAR.

Attack | MR | SST2
---|---|---
BERT | +ADFAR | BERT | +ADFAR
Attack-Free | 86.2 | 86.6 | 93.1 | 92.3
PWWS | 34.2 | 74.2 | 54.3 | 80.5
Genetic | 21.3 | 70.4 | 38.7 | 72.2
TextFooler | 16.9 | 66.0 | 39.8 | 73.8
Table 4:  Performance of BERT and BERT with ADFAR when facing various AWS
models. The results are based on the average of five runs.

As shown is Table 4, the performance of ADFAR is not affected by different AWS
models, which further proves the efficacy of our method.

###  5.3 Results with Other PrLMs

Table 5 shows the performance of ADFAR leveraging RoBERTa${}_{\text{BASE}}$
Liu et al. (2019) and ELECTRA${}_{\text{BASE}}$ Clark et al. (2020) as PrLMs.
In order to enhance the robustness and performance of the PrLM, RoBERTa
extends BERT with a larger corpus and using more efficient parameters, while
ELECTRA applies a GAN-style architecture for pre-training. Empirical results
indicate that ADFAR can further improve the robustness of RoBERTa and ELECTRA
while preserving their original performance.

PrLM | MR | SST2
---|---|---
Orig. Acc. | Adv. Acc. | Orig. Acc. | Adv. Acc.
BERT | 86.2 | 16.9 | 93.1 | 39.8
+ADFAR | 86.6 | 66.0 | 92.3 | 73.8
RoBERTa | 88.3 | 30.4 | 93.4 | 37.4
+ADFAR | 87.2 | 71.0 | 93.2 | 77.6
ELECTRA | 90.1 | 33.6 | 94.2 | 40.4
+ADFAR | 90.4 | 71.2 | 95.0 | 83.0
Table 5:  Results based with various PrLMs.

##  6 Analysis

###  6.1 Ablation Study

ADFAR leverages three techniques to help PrLMs defend against adversarial
samples: adversarial training, frequency-aware randomization and anomaly
detection. To evaluate the contributions of these techniques in ADFAR, we
perform ablation studies on MR and SST2 using BERT${}_{\text{BASE}}$ as our
PrLMs, and TextFooler as the attack model. As shown in Table 6, the frequency-
aware randomization is the key factor which helps PrLM defense against
adversarial samples, while anomaly detection plays an important role in
preserving PrLM’s prediction accuracy for non-adversarial samples.

Model | MR | SST2
---|---|---
Orig. Acc. | Adv. Acc. | Orig. Acc. | Adv. Acc.
BERT | 86.2 | 16.9 | 93.1 | 39.8
\+ Adv | 85.6 | 34.6 | 92.6 | 48.8
\+ FR | 85.0 | 72.8 | 90.6 | 82.6
\+ AD | 86.6 | 66.0 | 92.3 | 73.8
Table 6:  Ablation study on MR and SST2 using BERT${}_{\text{BASE}}$ as PrLM,
and TextFooler as attack model. Adv represents adversarial training, FR
indicates frequency-aware randomization and AD means anomaly detection. The
results are based on the average of five runs.

###  6.2 Anomaly Detection

In this section, we compare the anomaly detection capability between ADFAR and
DISP Zhou et al. (2019). ADFAR leverages an auxiliary anomaly detector, which
share a same PrLM with the original text classifier, to discriminate
adversarial samples. DISP uses an discriminator based on an extra PrLMs to
identify the perturbed adversarial inputs, but on token level. For DISP, in
order to detect anomaly on sentence level, input sentences with one or more
than one adversarial tokens identified by DISP are regarded as adversarial
samples. We respectively sample 500 normal and adversarial samples from the
test set of MR and SST to evaluate the performance of ADFAR and DISP for
anomaly detection.

Table 7 shows the performance of ADFAR and DISP for anomaly detection.
Empirical results show that ADFAR can predict more precisely, since it
achieves a significantly higher $F_{1}$ score than DISP. Moreover, ADFAR has a
simpler framework, as its anomaly detector shares the same PrLM with the
classifier, while DISP requires an extra PrLM. The results also indicate that
the current heuristic AWS strategy is vulnerable to our anomaly detector,
which disproves the claimed undetectable feature of this very adversarial
strategy.

Method | MR | SST2
---|---|---
Precision | Recall | $F_{1}$ | Precision | Recall | $F_{1}$
DISP | 68.0 | 92.0 | 73.2 | 59.5 | 94.2 | 72.9
ADFAR | 90.1 | 84.0 | 86.9 | 88.0 | 90.0 | 88.9
Table 7:  Performance for anomaly detection.

###  6.3 Effect of Randomization Strategy

As the ablation study reveals, the frequency-aware randomization contributes
the most to the defense. In this section, we analyze the impact of different
hyperparameters and strategies adopted by the frequency-aware randomization
approach, in inference and training respectively.

####  6.3.1 Inference

The frequency-aware randomization process is applied in inference to mitigate
the adversarial effects. Substitution candidate selection and synonym set
construction are two critical steps during this process, in which two
hyperparameters ($r$ and $n_{s}$) and the frequency-aware strategy are
examined.

##### Selection of Substitution Candidates

The influence of different strategies for substitution candidate selection in
inference is studied in this section. The impact of two major factors are
measured: 1) the substitution ratio $r$ and 2) whether to apply a frequency-
aware strategy. In order to exclude the disturbance from other factors, we
train BERT on the original training set and fix $n_{s}$ to $20$. Firstly, we
alter the value of $r$ from $5\%$ to $50\%$, without applying the frequency-
aware strategy. As illustrated by the blue lines in Figure 4, as $r$
increases, the original accuracy decreases, while the adversarial accuracy
increases and peaks when $r$ reaches $30\%$. Secondly, a frequency-aware
strategy is added to the experiment, with $f_{thres}=200$. As depicted by the
yellow lines in Figure 4, both original and adversarial accuracy, the general
trends coincide with the non-frequency-aware scenario, but overall accuracy is
improved to a higher level. The highest adversarial is obtained when $r$ is
set to $30\%$ using frequency-aware strategy.

Figure 4:  Effect of the substitution ratio $r$ and the frequency-aware
strategy in substitution candidate selection during inference.

##### Construction of Synonym Set

The influence of different strategies for synonym set construction in
inference is evaluated in this section. The impact of two major factors are
measured: 1) the size of a single synonym set $n_{s}$ and 2) whether to apply
a frequency-aware strategy. In order to exclude the disturbance from other
factors, we train BERT on the original training set and fix $r$ to $30\%$ .
Firstly, we alter the value of $n_{s}$ from $5$ to $50$, without applying the
frequency-aware strategy. The resulted original and adversarial accuracy are
illustrated by the blue lines in Figure 5. Secondly, a frequency-aware
strategy is added to the experiment, with $n_{f}=50\%*n_{s}$. As depicted by
the yellow lines in Figure 5, the original accuracy and the adversarial
accuracy both peaks when $n_{s}=20$, and the overall accuracy is improved to a
higher level compared to the non-frequency-aware scenario.

Figure 5:  Effect of the size of synonym set $n_{s}$ and the frequency-aware
strategy in construction of synonym set.

####  6.3.2 Training

The frequency-aware randomization process is applied in training to augment
the training data, and hereby enables the PrLM to better cope with randomized
samples inference. Based on this purpose, the frequency-aware randomization
process in training should resemble the one in inference as much as possible.
Therefore, here we set an identical process for synonym set construction, i.e.
$n_{s}=20$ and $n_{f}=50\%*n_{s}$. However, for the substitution selection
process, to avoid introducing excessive noise and maintain the accuracy for
the PrLM, the most suitable substitution ratio $r$ might be different than the
one in inference. Experiments are conducted to evaluate the influence of $r$
in training. We alter the value of $r$ from $5\%$ to $50\%$. In Figure 6, we
observe that $r=25\%$ results in highest original and adversarial accuracy.

Figure 6:  Effect of the size of synonym set $n_{s}$ and the frequency-aware
strategy in construction of synonym set.

##  7 Conclusion

This paper proposes ADFAR, a novel framework which leverages the frequency-
aware randomization and the anomaly detection to help PrLMs defend against
adversarial word substitution. Empirical results show that ADFAR significantly
outperforms those newly proposed defense methods over various tasks.
Meanwhile, ADFAR achieves a remarkably higher inference speed and does not
reduce the prediction accuracy for non-adversarial sentences, from which we
keep the promise for this research purpose.

Comprehensive ablation study and analysis indicate that 1) Randomization is an
effective method to defend against heuristic attack strategy. 2) Replacement
of rare words with their more common alternative can help enhance the
robustness of PrLMs. 3) Adversarial samples generated by current heuristic
adversarial word substitution models can be easily distinguished by the
proposed auxiliary anomaly detector. We hope this work could shed light on
future studies on the robustness of PrLMs.

## References

  * Alzantot et al. (2018) Moustafa Alzantot, Yash Sharma, Ahmed Elgohary, Bo-Jhang Ho, Mani Srivastava, and Kai-Wei Chang. 2018.  Generating natural language adversarial examples.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 2890–2896, Brussels, Belgium. Association for Computational Linguistics.
  * Clark et al. (2020) Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020.  Electra: Pre-training text encoders as discriminators rather than generators.  In _International Conference on Learning Representations_.
  * Cohen et al. (2019) Jeremy Cohen, Elan Rosenfeld, and Zico Kolter. 2019.  Certified adversarial robustness via randomized smoothing.  In _Proceedings of the 36th International Conference on Machine Learning_ , pages 1310–1320.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Dvijotham et al. (2018) Krishnamurthy Dvijotham, Sven Gowal, Robert Stanforth, Relja Arandjelovic, Brendan O’Donoghue, Jonathan Uesato, and Pushmeet Kohli. 2018.  Training verified learners with learned verifiers.  _arXiv preprint arXiv:1805.10265_.
  * Ebrahimi et al. (2018) Javid Ebrahimi, Daniel Lowd, and Dejing Dou. 2018.  On adversarial examples for character-level neural machine translation.  In _Proceedings of the 27th International Conference on Computational Linguistics_ , pages 653–663, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
  * Gao et al. (2018) J. Gao, J. Lanchantin, M. L. Soffa, and Y. Qi. 2018.  Black-box generation of adversarial text sequences to evade deep learning classifiers.  In _2018 IEEE Security and Privacy Workshops (SPW)_ , pages 50–56.
  * Garg and Ramakrishnan (2020) Siddhant Garg and Goutham Ramakrishnan. 2020.  BAE: BERT-based adversarial examples for text classification.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 6174–6181, Online. Association for Computational Linguistics.
  * Goodfellow et al. (2015) Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. 2015.  Explaining and harnessing adversarial examples.  _arXiv preprint arXiv:1905.07129_.
  * Huang et al. (2019) Po-Sen Huang, Robert Stanforth, Johannes Welbl, Chris Dyer, Dani Yogatama, Sven Gowal, Krishnamurthy Dvijotham, and Pushmeet Kohli. 2019.  Achieving verified robustness to symbol substitutions via interval bound propagation.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 4083–4093, Hong Kong, China. Association for Computational Linguistics.
  * Islam and Inkpen (2009) Aminul Islam and Diana Inkpen. 2009.  Real-word spelling correction using Google Web 1T 3-grams.  In _Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing_ , pages 1241–1249, Singapore. Association for Computational Linguistics.
  * Iyyer et al. (2018) Mohit Iyyer, John Wieting, Kevin Gimpel, and Luke Zettlemoyer. 2018.  Adversarial example generation with syntactically controlled paraphrase networks.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 1875–1885, New Orleans, Louisiana. Association for Computational Linguistics.
  * Jia et al. (2019) Robin Jia, Aditi Raghunathan, Kerem Göksel, and Percy Liang. 2019.  Certified robustness to adversarial word substitutions.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 4129–4142, Hong Kong, China. Association for Computational Linguistics.
  * Jiang et al. (2020) Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Tuo Zhao. 2020.  SMART: Robust and efficient fine-tuning for pre-trained natural language models through principled regularized optimization.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 2177–2190, Online. Association for Computational Linguistics.
  * Jin et al. (2019) Di Jin, Zhijing Jin, Joey Tianyi Zhou, and Peter Szolovits. 2019.  Is bert really robust? natural language attack on text classification and entailment.  _arXiv preprint arXiv:1907.11932_.
  * Kurakin et al. (2017) Alexey Kurakin, Ian Goodfellow, and Samy Bengio. 2017.  Adversarial examples in the physical world.  _arXiv preprint arXiv:1607.02533_.
  * Li et al. (2020) Linyang Li, Ruotian Ma, Qipeng Guo, Xiangyang Xue, and Xipeng Qiu. 2020.  BERT-ATTACK: Adversarial attack against BERT using BERT.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 6193–6202, Online. Association for Computational Linguistics.
  * Liang et al. (2018) Bin Liang, Hongcheng Li, Miaoqiang Su, Pan Bian, Xirong Li, and Wenchang Shi. 2018\.  Deep text classification can be fooled.  _Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence_.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_.
  * Loshchilov and Hutter (2018) Ilya Loshchilov and Frank Hutter. 2018.  Fixing weight decay regularization in adam.
  * Maas et al. (2011) Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011.  Learning word vectors for sentiment analysis.  In _Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies_ , pages 142–150, Portland, Oregon, USA. Association for Computational Linguistics.
  * Morris et al. (2020) John Morris, Eli Lifland, Jin Yong Yoo, Jake Grigsby, Di Jin, and Yanjun Qi. 2020\.  TextAttack: A framework for adversarial attacks, data augmentation, and adversarial training in NLP.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 119–126, Online. Association for Computational Linguistics.
  * Mozes et al. (2020) Maximilian Mozes, Pontus Stenetorp, Bennett Kleinberg, and Lewis D. Griffin. 2020\.  Frequency-guided word substitutions for detecting textual adversarial examples.  _arXiv preprint arXiv:2004.05887_.
  * Mrkšić et al. (2016) Nikola Mrkšić, Diarmuid Ó Séaghdha, Blaise Thomson, Milica Gašić, Lina Rojas-Barahona, Pei-Hao Su, David Vandyke, Tsung-Hsien Wen, and Steve Young. 2016\.  Counter-fitting word vectors to linguistic constraints.  _arXiv preprint arXiv:1603.00892_.
  * Nangia et al. (2017) Nikita Nangia, Adina Williams, Angeliki Lazaridou, and Samuel R Bowman. 2017.  The repeval 2017 shared task: Multi-genre natural language inference with sentence representations.  In _RepEval_.
  * Nie et al. (2020) Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2020.  Adversarial NLI: A new benchmark for natural language understanding.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4885–4901, Online. Association for Computational Linguistics.
  * Pang and Lee (2005) Bo Pang and Lillian Lee. 2005.  Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales.  In _Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL’05)_ , pages 115–124, Ann Arbor, Michigan. Association for Computational Linguistics.
  * Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014.  GloVe: Global vectors for word representation.  In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 1532–1543, Doha, Qatar. Association for Computational Linguistics.
  * Pruthi et al. (2019) Danish Pruthi, Bhuwan Dhingra, and Zachary C. Lipton. 2019.  Combating adversarial misspellings with robust word recognition.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 5582–5591, Florence, Italy. Association for Computational Linguistics.
  * Ren et al. (2019) Shuhuai Ren, Yihe Deng, Kun He, and Wanxiang Che. 2019.  Generating natural language adversarial examples through probability weighted word saliency.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 1085–1097, Florence, Italy. Association for Computational Linguistics.
  * Ribeiro et al. (2018) Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2018.  Semantically equivalent adversarial rules for debugging NLP models.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 856–865, Melbourne, Australia. Association for Computational Linguistics.
  * Sakaguchi et al. (2017) Keisuke Sakaguchi, Matt Post, and Benjamin Van Durme. 2017.  Grammatical error correction with neural reinforcement learning.  In _Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_ , pages 366–372, Taipei, Taiwan. Asian Federation of Natural Language Processing.
  * Si et al. (2021) Chenglei Si, Zhengyan Zhang, Fanchao Qi, Zhiyuan Liu, Yasheng Wang, Qun Liu, and Maosong Sun. 2021.  Better robustness by more coverage: Adversarial training with mixup augmentation for robust fine-tuning.
  * Socher et al. (2013) Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013.  Recursive deep models for semantic compositionality over a sentiment treebank.  In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing_ , pages 1631–1642, Seattle, Washington, USA. Association for Computational Linguistics.
  * Wallace et al. (2019) Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, and Sameer Singh. 2019.  Universal adversarial triggers for attacking and analyzing NLP.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 2153–2162, Hong Kong, China. Association for Computational Linguistics.
  * Wang et al. (2021) Wenqi Wang, Run Wang, Lina Wang, Zhibo Wang, and Aoshuang Ye. 2021.  Towards a robust deep neural network in texts: A survey.
  * Wei and Zou (2019) Jason Wei and Kai Zou. 2019.  EDA: Easy data augmentation techniques for boosting performance on text classification tasks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 6382–6388, Hong Kong, China. Association for Computational Linguistics.
  * Xie et al. (2018) Cihang Xie, Jianyu Wang, Zhishuai Zhang, Zhou Ren, and Alan Yuille. 2018.  Mitigating adversarial effects through randomization.  _arXiv preprint arXiv:1711.01991_.
  * Ye et al. (2020) Mao Ye, Chengyue Gong, and Qiang Liu. 2020.  SAFER: A structure-free approach for certified robustness to adversarial word substitutions.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 3465–3475, Online. Association for Computational Linguistics.
  * Zang et al. (2020) Yuan Zang, Fanchao Qi, Chenghao Yang, Zhiyuan Liu, Meng Zhang, Qun Liu, and Maosong Sun. 2020.  Word-level textual adversarial attacking as combinatorial optimization.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 6066–6080, Online. Association for Computational Linguistics.
  * Zhang et al. (2020) Zhuosheng Zhang, Yuwei Wu, Hai Zhao, Zuchao Li, Shuailiang Zhang, Xi Zhou, and Xiang Zhou. 2020.  Semantics-aware bert for language understanding.
  * Zhang et al. (2019) Zhuosheng Zhang, Yuwei Wu, Junru Zhou, Sufeng Duan, Hai Zhao, and Rui Wang. 2019\.  Sg-net: Syntax-guided machine reading comprehension.
  * Zhou et al. (2019) Yichao Zhou, Jyun-Yu Jiang, Kai-Wei Chang, and Wei Wang. 2019.  Learning to discriminate perturbations for blocking adversarial attacks in text classification.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 4904–4913, Hong Kong, China. Association for Computational Linguistics.