# The RATAN-600 multi-frequency catalogue of blazars – BLcat

Yu. Sotnikova,1 T. Mufakharov,1,2 R. Udovitskiy,1 M. Mingaliev,1,2,3 T.
Semenova,1 A. Erkenov,1 N. Bursov,1 A. Mikhailov,1 Yu. Cherepkova1
1Special Astrophysical Observatory of RAS, Nizhny Arkhyz, 369167 Russia
2Kazan Federal University, Kazan, 420008 Russia
3Institute of Applied Astronomy of RAS, St. Petersburg, 191187 Russia E-mail:
<EMAIL_ADDRESS>
(Received 17 August 2022. Accepted 9 September 2022.
Published in Astrophysical Bulletin Vol. 77, No.4, pp.397–410 (2022)
DOI: 10.1134/S1990341322040149)

###### Abstract

In this paper we present the RATAN-600 multi-frequency catalogue of blazars,
an updated version of the BLcat: the RATAN-600 multi-frequency catalogue of BL
Lacertae objects. The main novelty in the catalogue is an extension of the
sample with flat-spectrum radio quasars (FSRQs), thus currently it contains
more than 1700 blazars of different types. The main feature of the BLcat is a
compilation of radio continuum data for blazars based on the RATAN-600 quasi-
simultaneous measurements at frequencies of 1.2, 2.3, 4.7, 7.7/8.2, 11.2, and
21.7/22.3 GHz. We additionally supplement the catalogue with the radio data
from external sources to provide an opportunity to more complete study of
radio spectra and radio light curves. For the convenience of users, we
developed tools to calculate the spectral index, variability index, and radio
luminosity. We briefly describe basic radio properties of blazar subsamples of
the catalogue: spectral classification, spectral indices, flux density
variability, and radio luminosity.

###### keywords:

galaxies: active – radio continuum: galaxies – quasars: general – BL Lacertae
objects: general– catalogues

††pubyear: 2022††pagerange: The RATAN-600 multi-frequency catalogue of blazars
– BLcat–References

##  1 Introduction

Blazars are radio-loud active galactic nuclei (AGNs) with their jets pointed
in the direction to the observer (Urry & Padovani, 1995; Padovani et al.,
2017). After their discovery, they became interesting as the objects of
extreme physical conditions with rapid and violent variability at different
timescales. Later, in addition to that, the interest to their investigation
was triggered by the fact that they are the most common sources of
$\gamma$-ray emission in the sky (Ajello et al., 2020). Recently, they were
discovered as the possible sources of PeV-energy neutrino (IceCube
Collaboration et al., 2018). For a comprehensive study of blazars and the
connection between the radio emission and neutrinos in them, it is important
to conduct multi-band high-cadence long-term observations (Kovalev et al.,
2020; Plavin et al., 2020; Hovatta et al., 2021). The Whole Earth Blazar
Telescope (WEBT, Villata et al. 2002) is a collaboration which organises such
campaigns on blazar monitoring, from radio to $\gamma$-rays. In the radio
bands, the greatest contribution was made by several observatories: the long-
term regular monitoring of brightest AGNs at 37 GHz by the Metsähovi Radio
Observatory (Salonen et al., 1987; Nieppola et al., 2009), the 15 GHz blazar
monitoring program by the Owens Valley Radio Observatory (Richards et al.,
2011), the three-frequency (4.8–14.5 GHz) monitoring by the University of
Michigan Radio Astronomy Observatory (Aller et al., 1985), the 1.4 GHz NRAO
VLA Sky Survey (NVSS, Condon et al. 1998), the 8.4 GHz observations with the
Very Large Array (VLA, Healey et al. 2007), and the 2–4 GHz Very Large Array
Sky Survey (VLASS, Lacy et al. 2020). However, long-term light curves are
usually sparsely sampled, and most blazars have been observed mainly within
multi-frequency flare activities monitoring campaigns (Hovatta & Lindfors,
2019). For example, in the Roma-BZCAT Multi-frequency Catalogue of Blazars
(Massaro et al., 2015) more than 20% of them have less than 20 radio
measurements (Fig. 1).

Figure 1: The number of literature radio measurements for the Roma-BZCAT
blazars (3561 objects). About 200 blazars with more than 400 measurements were
excluded for a clear view.

Among the mentioned radio telescopes and surveys, the RATAN-600 radio
telescope stands out for its capability of obtaining six-frequency
measurements (1–22 GHz) simultaneously. AGNs, including blazars, have been
regularly monitored with RATAN-600 since 2005, and this data formed the basis
for the first version of the BLcat111https://www.sao.ru/blcat/ (Mingaliev et
al., 2014), the catalogue of “RATAN-600 multi-frequency data for the BL
Lacertae objects”. It contained more than 300 BL Lac objects and candidates.
In this paper, we present the electronic “RATAN-600 multi-frequency catalogue
of blazars”, which is an update of the BLcat. In this new BLcat edition 1.3
(September 2021), we present several major changes to the previous version:
firstly, the catalogue was enlarged with new types of blazars, namely blazars
of an uncertain type (Blaz.un.t) and flat-spectrum radio quasars (FSRQs), and
now it contains more than 1700 blazars; secondly, the BL Lac objects from the
first version of the catalogue were supplemented by new RATAN-600 flux density
measurements; and thirdly, the radio data from external databases and the
literature are given along with the RATAN-600 data.

The paper is organised as follows. The properties of the sample are given in
Section 2; the description of the RATAN-600 observations and characteristics
of the antenna and receivers are given in Section 3.1; in Section 3.2 the
external data added to the catalogue are described; in Section 4 we present
the general organisation, the structure of the catalogue, and available tools.
Section 5 describes radio properties of blazar populations of the catalogue.
The summary of the paper is given in Section 6.

##  2 The sample

The list of BLcat blazars is based on the Roma-BZCAT catalogue by Massaro et
al. (2009), which is the largest collection of blazars selected from different
multi-band surveys. Its 5th edition (Massaro et al., 2015) contains 3561
objects, all of them provided with the radio band data from the NVSS (Condon
et al., 1998), the FIRST survey (White et al., 1997), or the SUMMS (Mauch et
al., 2003). The BZCAT blazar classification is based in general on the
spectral energy distribution (SED) showing strong non-thermal emission over
the entire electromagnetic spectrum and the presence of evidence for
relativistic beaming. When available, the data for $\gamma$-rays, soft and
hard X-rays, microwaves, and the $R$-band magnitudes are also given.

The BLcat catalogue originally included 306 BL Lacertae objects from the large
Metsähovi BL Lac sample (Nieppola et al., 2006) and contained the RATAN-600
measurements in 2005–2014 (Mingaliev et al., 2014, 2017b). Later we included
to the catalogue the FSRQ blazars from Roma-BZCAT within the declination range
from $-34\degr$ to $+49\degr$ with an NVSS flux density limit of $S_{1.4}\geq
100$ mJy. In comparison with the first version of the catalogue (Mingaliev et
al., 2014) current edition contains almost six times more objects, the
population is mainly extended because of FSRQ type of blazars (Table 1).

Figure 2: The Hammer–Aitoff projection (in equatorial coordinates) of the sky
distribution of the objects from the BLcat catalogue. The size of the circles
corresponds to the flux density level at 1.4 GHz, and the colour indicates the
redshift value.

Figure 3: Redshift distribution. FSRQs are marked by green colour, BL Lac’s
are grey, and blazars of uncertain type are marked by yellow colour.

The BLcat aims to fully represent the RATAN-600 long-term measurements of
blazars; an external radio data are being systematically included in the
catalogue. At present, the catalogue contains more than 1700 blazars, which is
almost half of all the Roma-BZCAT sources. We adopted the blazar
classification, redshifts, and $R$-band magnitudes from Roma-BZCAT. The BL
Lacertae are subclassified as BL Lacertae–type objects, BL Lac candidates, and
BL Lac–galaxy dominated sources according to Roma-BZCAT. Fig. 2 shows the
distribution of the objects from the catalogue in the sky, with the redshifts
designated by the colour, and the source’s flux densities shown by the marker
size.

The redshift distribution for different blazar populations is shown in Fig. 3.
The median value of $z$ is 1.34 for FSRQs, 0.58 for blazars uncertain type,
and 0.27 for BL Lacs. FSRQs exhibit larger scatter in a range of redshift,
from 0.09 to 5.29.

Table 1:  Blazars subclasses in the first and in the current editions of the BLcat. N - is the number of sources. Class |  $N$ (%)
---|---
2014 | 2022
FSRQ | 7 | (2.3) | 1097 | (62.2)
BL Lac | 220 | (71.9) | 416 | (23.6)
BL Lac candidates | 43 | (14.1) | 32 | (1.8)
BL Lac galaxy-dominated | – | 92 | (5.2)
Blazars of uncertain type | 36 | (11.8) | 128 | (7.2)
All | 306 |  | 1765 |

##  3 CATALOGUE’S CONTENTS

###  3.1 RATAN-600 Data

The observations of blazars with RATAN-600 have been carried out since 2005,
and part of the measurements were published in the electronic catalogues
Mingaliev et al. (2008); Mingaliev et al. (2012a); Mingaliev et al. (2015);
Mufakharov et al. (2015a, b); Mingaliev et al. (2017a); Sotnikova et al.
(2019b). RATAN-600 radio continuum spectra are measured simultaneously at
several frequencies on a time scale of 3–5 minutes (Korolkov & Pariiskii,
1979; Parijskij, 1993). We used two radiometric complexes working at three
(4.7, 11.2, 21.7/22.3 GHz) and six (1.2, 2.3, 4.7, 7.7/8.2, 11.2, 21.7/22.3
GHz) frequencies (Tsybulev, 2011; Tsybulev et al., 2018). The parameters of
the radiometers and RATAN antenna system are given in Table 2. The
observations were processed using the automated data reduction system
(Udovitskiy et al., 2016), which is based on the Flexible Astronomical Data
Processing System (FADPS) standard data reduction software developed by
Verkhodanov (1997).

The following flux density secondary calibrators were used: 3C48, 3C138,
3C147, 3C161, 3C286, 3C295, 3C309.1, and NGC 7027. The flux density scale is
based on the measurements by Baars et al. (1977) and Perley & Butler (2013,
2017), which are in good agreement with each other and differ only within the
measurement errors. We also used the traditional RATAN-600 flux density
calibrators: J0240$-$23, J1154$-$35, J1347$+$12, and J0521$+$16 (Sotnikova et
al., 2019a). The measurements of the calibrators were corrected for angular
size and linear polarisation according to the data from Ott et al. (1994) and
Tabara & Inoue (1980).

A standard error of the flux density includes an uncertainty of the RATAN-600
calibration curve (about 2–10%) and the antenna temperature measurement error.
The systematic uncertainty of the absolute flux density scale, which is 3–10%
at different frequencies (Baars et al., 1977), is excluded in the total error.
The median values of the standard error are 3–10% at 11.2, 8.2, and 4.7 GHz,
and 7–20% at 2.3, 1.2, and 22.3 GHz.

At present, the number of RATAN observations is non-uniform over the whole
BLcat list, it varies from 1 up to more than 100. The average number of
observations is 17. Most blazars, about 65%, have been observed on average 5
times. About 10% of blazars have 15–45 RATAN-600 observational epochs; 22% –
45–70 epochs, and 4% – 70–115 epochs.

Table 2: The RATAN-600 continuum radiometer parameters: the central frequency $f_{0}$, the bandwidth $\Delta f_{0}$, the detection limit for point sources per transit $\Delta F$. ${FWHM}_{\rm{RA}\times\rm{Dec}}$ is the angular resolution along RA and Dec. calculated for ${\rm Dec}=0\degr$ $f_{0}$, |  $\Delta f_{0}$, |  $\Delta F$, | $FWHM_{\rm{RA}\times\rm{Dec}}$
---|---|---|---
GHz | GHz | mJy/beam | $~{}~{}\times$ |
$21.7/22.3$ | $2.5$ | $50$ |  $0\aas@@fstack{\prime}17$ $~{}~{}\times$ | $1\aas@@fstack{\prime}8$
$11.2$ | $1.4$ | $15$ |  $0\aas@@fstack{\prime}34$ $~{}~{}\times$ | $3\aas@@fstack{\prime}9$
$7.7/8.2$ | $1.0$ | $10$ |  $0\aas@@fstack{\prime}47$ $~{}~{}\times$ | $5\aas@@fstack{\prime}2$
$4.7$ | $0.6$ | $8$ |  $0\aas@@fstack{\prime}85$ $~{}~{}\times$ | $9\aas@@fstack{\prime}5$
$2.25$ | $0.08$ | $40$ |  $1\aas@@fstack{\prime}64$ $~{}~{}\times$ | $19\aas@@fstack{\prime}2$
$1.25$ | $0.08$ | $200$ |  $3\aas@@fstack{\prime}0$ $~{}~{}\times$ | $32\aas@@fstack{\prime}4$

###  3.2 External Data

Figure 4: The surveys and observatories with the most contribution to the
blazar radio continuum data in BLcat.

The new BLcat edition collects external radio continuum data from
CATS222https://www.sao.ru/cats/, the Astrophysical CATalogs support System
database (Verkhodanov et al., 1997, 2005, 2009), available on the Special
Astrophysical Observatory website, from NED333http://ned.ipac.caltech.edu, the
NASA/IPAC Extragalactic Database, and from the VizieR Information
System444https://vizier.u-strasbg.fr/viz-bin/VizieR (Ochsenbein et al., 2000).
The data used for the construction of broadband radio continuum spectra and
cover the frequency range from tens of MHz up to hundreds of GHz and time
period of 30–40 years. The total list with references is given in Table 3.
Fig. 4 presents the distribution of main of surveys or monitoring programs
contributing to the blazar radio data.

Table 3:  External catalogs and programs with radio data included in the current edition Catalog/survey | Observatory/telescope | Frequency range, GHz | Refs
---|---|---|---
87GB, GB6 | GBT | 2.5, 4.85, 8.2 | [1, 2, 3, 4]
| UMRAO | 4.8, 8, 14.5 | [5]
F-GAMMA | OVRO | 10.8, 15 | [6, 7]
GLEAM | MWA | 0.072–0.231 | [8]
F-GAMMA | Effelsberg | 2.64–43 | [9]
AT20G, PACO | ATCA | 20, 4.5–40 | [10, 11]
| Metsähovi | 22, 37, 87 | [12, 13, 14]
VLASS | VLA | 3 | [15]
| VLBA | 1.4, 5 | [16]
| RATAN-600 | 1–22 | [17, 18, 19, 20]
| IRAM | 90, 142, 230 | [21]
| GBT, Parkes | 2.7, 5 | [22]
NVSS | VLA | 1.4 | [23]
TGSS | GMRT | 0.15 | [24]
PMN | Parkes | 4.85, 8.4 | [25, 26]
| Arecibo | 0.32, 0.43 | [27]
| GBT | 0.61, 0.88, 1.4 | [27]
TXS | UTRAO | 0.365 | [28]
Refs: [1, 2, 3, 4] — Lazio et al. (2001); Altschuler (1986), Gregory & Condon
(1991); Becker et al. (1991); [5] — Aller et al. (1985); [6, 7] — Richards et
al. (2011); Seielstad et al. (1983); [8] — Hurley-Walker et al. (2017); [9] —
Angelakis et al. (2019); [10, 11] — Massardi et al. (2011, 2008); [12, 13, 14]
— Teraesranta et al. (1998); Terasranta et al. (1992); Teräsranta et al.
(2004); [15, 16] — Gordon et al. (2020); Taylor et al. (1996); [17, 18, 19,
20] — Mingaliev et al. (2017b, 2012b); Kovalev et al. (1999); Kiikov et al.
(2002); [21] — Reuter et al. (1997); [22] — Kuehr et al. (1981); [23] — Condon
et al. (1998); [24] — Intema et al. (2017); [25, 26] — Wright et al. (1990,
1996); [27] — Mitchell et al. (1994); [28] — Douglas et al. (1996).

External data collected by different instruments are often inhomogenous. For
example, interferometric observations have higher angular resolution than our
RATAN-600 telescope, thus leading to underestimations of source’s flux
densities compared to RATAN-600 measurements. On the other hand, measurements
from single-dish telescopes with beams larger than those of RATAN-600 may be
impacted by confusion with nearby sources, leading to overestimates of
source’s flux densities. Noting this we listed and classified all the
catalogues from CATS, separating interferometric from single-dish data. We
marked interferometric and single-dish measurements with different colours and
they are optionally allowed to be drawn in cumulative radio spectrum.

Table 4: The first ten rows given as an example of the structure and information stored in the main table of the BLcat catalogue edition 1.3 (September 2021) $N$ | Epochs | Source | RA | Dec | $z$ | Rmag | $S_{4.7}$, Jy | $P_{4.7}$, W Hz-1 | Blazar
---|---|---|---|---|---|---|---|---|---
name | (J2000.0) | (J2000.0) | type
(1) | (2) | (3) | (4) | (5) | (6) | (7) | (8) | (9) | (10)
1 | 2 | 5BZQJ0001$-$1551 | 00:01:05 |  $-$15:51:06 | 2.044 | 18.1 | 0.20 | $3.23\times 10^{27}$ | FSRQ
2 | 6 | 5BZQJ0001$+$1914 | 00:01:08 |  $+$19:14:34 | 3.1 | 21.6 | 0.13 | $3.04\times 10^{27}$ | FSRQ
3 | 3 | 5BZBJ0001$-$0746 | 00:01:18 |  $-$07:46:26 | – | 17.9 | 0.18 | – | BL Lac
4 | 6 | 5BZBJ0001$-$0011 | 00:01:21 |  $-$00:11:39 | 0.462 | 19.6 | 0.05 | $3.26\times 10^{25}$ | BL Lac
5 | 46 | 5BZBJ0002$-$0024 | 00:02:57 |  $-$00:24:47 | 0.523 | 19.7 | 0.09 | $8.55\times 10^{25}$ | BL Lac
6 | 1 | 5BZBJ0004$-$1148 | 00:04:04 |  $-$11:48:57 | – | 18.8 | 0.78 | – | BL Lac
7 | 2 | 5BZQJ0004$+$4615 | 00:04:16 |  $+$46:15:18 | 1.81 | 20.4 | 0.24 | $2.35\times 10^{27}$ | FSRQ
8 | 2 | 5BZQJ0005$-$1648 | 00:05:17 |  $-$16:48:04 | 0.78 | 18.4 | 0.15 | $3.01\times 10^{26}$ | FSRQ
9 | 1 | 5BZQJ0005$+$0524 | 00:05:20 |  $+$05:24:10 | 1.9 | 16.2 | 0.14 | $1.39\times 10^{27}$ | FSRQ
10 | 66 | 5BZQJ0005$+$3820 | 00:05:57 |  $+$38:20:15 | 0.229 | 17.6 | 0.50 | $6.35\times 10^{25}$ | FSRQ
Column designations: (1)—the entry number, (2)—the number of RATAN-600
observing epochs, (3)—the source name, (4)—right ascension (J 2000, hh:mm:ss),
(5)—declination (J 2000, dd:mm:ss), (6)—redshift, (7)—the $R$-band magnitude
from USNO-B1, (8)—the average flux density at 4.7 GHz measured based on the
RATAN-600 observations, (9)—the average radio luminosity at 4.7 GHz measured
based on the RATAN-600 observations, (10)—the blazar type based on the BZCAT
classification: BL Lac, BL Lac candidate (BL Lac cand.), BL Lac-galaxy
dominated (BL Lac.g.dom.), FSRQ, or a blazar of an uncertain type
(Blaz.un.t.).

##  4 General organisation of the catalogue

Figure 5: Multi-frequency radio spectra of the BL Lac object J0022$+$06,
derived from the catalogue. There are 72 RATAN-600 observing epochs for the
object from 2006.07 to 2021.01 (colored). The external radio data are shown by
the grey circles.

Figure 6: Multi-frequency light curves of the BL Lac object J0022$+$06,
derived from the catalogue.

The catalogue is published as an interactive online tool to view and access
the data at https://www.sao.ru/blcat/. The main page contains a list of
objects with their characteristics placed in columns: the number of RATAN-600
observing epochs for each object, name of the object (in the BZCAT5 format
where it is possible), the J 2000 right ascension (RA) and declination (Dec)
coordinates, the redshift, the $R$-band magnitude, the averaged flux density
at 4.7 GHz, the averaged luminosity at 4.7 GHz, and the type of the blazar
(based on the Roma-BZCAT 5th edition). An example of the main table contents
is given in Table 4. Each column can be sorted, and the columns containing the
number of observing epochs, the redshift, the $R$-band magnitude, the flux
density, the radio luminosity, and the blazar type also have distribution
statistics. The radio data of any object/s can be accessed by clicking on the
“Data explorer” option. It opens a new window with the RATAN-600 data,
represented by the flux densities obtained at three to six frequencies quasi-
simultaneously, the radio spectrum, and the light curves (an example is shown
in Figs. 5 and 6). The radio spectrum could be supplemented with external data
(“Show external data”), which can be exported right there (“Export external
data”) in the format of a text file containing the blazar name, the frequency,
the flux density, and the name of a catalogue or a bibliographic code to
identify the literature.

Table 5:  The median and average values of the redshift $z$, flux density $S_{4.7}$, and radio luminosity $P_{4.7}$ at 4.7 GHz; here $N$ is the number of objects, the standard deviation for the average values is given in parentheses Blazar type | $z$ |  $S_{4.7}$, Jy |  $P_{4.7}$, W Hz-1
---|---|---|---
$N$ | average | median | $N$ | average | median | $N$ | average | median
FSRQ | 1097 | 1.47 (0.79) | 1.34 | 1097 | 0.59 (1.29) | 0.32 | 1065 | 5.0 (11.2)$\times 10^{29}$ | 1.9$\times 10^{29}$
BL Lac | 324 | 0.38 (0.35) | 0.27 | 539 | 0.27 (0.42) | 0.11 | 252 | 3.2 (9.7)$\times 10^{28}$ | 1.9$\times 10^{27}$
Blaz.un.t. | 113 | 0.77 (0.71) | 0.58 | 128 | 0.79 (2.33) | 0.31 | 105 | 2.0 (2.5)$\times 10^{29}$ | 3.5$\times 10^{28}$

We provide spectral and variability indices, automatically calculated in
different frequency intervals. The spectral index $\alpha$ defined from the
power law $S_{\nu}\sim\nu^{\alpha}$, where $S_{\nu}$ is the flux density at
frequency $\nu$, and $\alpha$ is a spectral slope. It is calculated using the
standard formula:

$\alpha=\dfrac{\log S_{2}-\log S_{1}}{\log{\nu}_{2}-\log{\nu}_{1}},$ (1)

where $S_{1}$ and $S_{2}$ are the flux densities at frequencies $\nu_{1}$ and
$\nu_{2}$, respectively.

The variability index is estimated using the formula adopted from Aller et al.
(1992):

${V}_{S}=\dfrac{(S_{i}-\sigma_{i})_{\rm max}-(S_{i}+\sigma_{i})_{\rm
min}}{(S_{i}-\sigma_{i})_{\rm max}+(S_{i}+\sigma_{i})_{\rm min}},$ (2)

where $S_{\rm max}$ and $S_{\rm min}$ are the maximum and minimum values of
the flux density over all epochs of observations, while $\sigma_{\rm max}$ and
$\sigma_{\rm min}$ are their root-mean-square errors. The spectral index can
be calculated both for a single RATAN-600 observing epoch and for the entire
period of observation as an averaged index. Also we provide the estimates of
the monochromatic radio luminosity calculated at 4.7 GHz following the
equation:

$P_{4.7}=4\pi D_{L}^{2}S_{4.7}(1+z)^{-\alpha-1},$ (3)

where $S_{4.7}$ is the measured flux density at 4.7 GHz, $z$ is the redshift,
$\alpha$ is the average spectral index at 4.7 GHz, and $D_{L}$ is the
luminosity distance. For calculation of luminosity distances, we used the
$\Lambda$CDM cosmology with $H_{0}=67.74$ km s-1 Mpc-1, $\Omega_{m}=0.3089$,
and $\Omega_{\Lambda}=0.6911$ (Planck Collaboration, 2016). The uncertainties
of the parameters ($\alpha$, ${V}_{S}$, and $P_{4.7}$) are calculated as the
square root of the sum of the squared errors of the parameters included in the
formula.

Table 6:  Theaverage values of spectral indices at 11.2–22.3, 8.2–11.2, 4.7–8.2, 2.3–4.7, and 1.2–2.3 GHz for three blazar types. The standard deviation for average spectral indices is indicated in parentheses Blazar type | $\alpha_{11.2-22.3}$ | $\alpha_{8.2-11.2}$ | $\alpha_{4.7-8.2}$ | $\alpha_{2.3-4.7}$ | $\alpha_{1.2-2.3}$
---|---|---|---|---|---
FSRQ |  $-$0.09 (0.41) |  $-$0.22 (0.44) |  $-$0.13 (0.45) |  $+$0.16 (0.38) |  $-$0.11 (0.52)
BL Lac |  $+$0.03 (0.46) |  $-$0.21 (0.54) |  $-$0.17 (0.52) |  $-$0.17 (0.60) |  $-$0.46 (0.64)
Blaz.un.t. |  $-$0.10 (0.48) |  $-$0.31 (0.49) |  $-$0.21 (0.56) |  $-$0.01 (0.55) |  $-$0.22 (0.70)

The content of the main table, measured RATAN-600 flux densities, and
calculated parameters ($\alpha$, ${V}_{S}$, and $P_{4.7}$) could be downloaded
(e.g., as a CSV format text file) using the “Export main table and RATAN-600
data” button. On the website there is also the “Help” button, where the
description of the catalogue, data, and all the tools are given.

##  5 THE RADIO PROPERTIES OF BLAZARS IN THE CATALOGUE

This section briefly describes some radio properties of the blazar subsamples
in the catalogue. The spectral indices and radio variability were calculated
based on observed data obtained using one instrument and at four–six
frequencies simultaneously, which minimizes the influence of measurement
inhomogeneities on the result. The median and average values of the redshift,
flux density, and radio luminosity at 4.7 GHz for FSRQs, BL Lacs, and Blazars
of uncertain type are presented in Table 5. The obtained radio properties of
the subsamples reflect a wide range of blazar parameters and statistical
differences between them.

###  5.1 Spectral Index

We analysed the averaged radio spectra of blazars compiled from the RATAN-600
quasi-simultaneous measurements over a time period from 2005 to 2022. The
spectral indices $\alpha$ were calculated using Equation 1. Their median and
average values at five frequency intervals are listed in Table 6 for three
blazar subsamples. The median values of $\alpha_{8.2-22.3}$ are $-0.15$,
$-0.08$, and $-0.18$ for the FSRQ, BL Lac, and Blaz.un.t. types respectively.
For $\alpha_{2.3-4.7}$ the median values are equal to $+0.14$, $-0.08$, and
$+0.01$ respectively.

Figure 7 presents the spectral type distribution in a convenient form of a
two-colour diagram in which the low-frequency spectral index is plotted
against the high-frequency index. The green square corresponds to the spectral
index range from $-0.5$ to $0.5$ (“blazar box”). The light green color
indicates the area of ultra-steep spectra with $\alpha\leq-1.1$. The other
areas are marked according to the spectral classification criteria from Table
7.

The vast majority of blazars have a flat (876, 50.5%) or rising (476, 27.5%)
radio spectrum. We found a small part of blazars (26, 1.5%) having peaked
radio spectra with a turnover in their average radio spectrum (Table 7) and
spectral indices $\alpha_{\rm low}\geq+0.5$ and $\alpha_{\rm high}\leq-0.5$,
which correspond to the slopes of classical gigahertz-peaked spectrum (GPS)
sources (de Vries et al., 1997). A very small number of these blazars (2.2%)
have ultra-steep radio spectra with $\alpha\leq-1.1$.

Figure 7: Two-colour diagram. The green square indicates the spectral index range from $-0.5$ to $0.5$, the blue lines mark the zero spectral indices, and the ultra-steep spectrum sources area is presented with a light green colour. FSRQs are marked by green colour, BL Lacs are grey, and blazars of an uncertain type are marked by black colour. Table 7: Spectral types in the sample. N - is the number of sources. Type | Criteria |  $N$ (%)
---|---|---
Flat | $-0.5\leq\alpha\leq 0$ | 876 | (50.5)
Peaked |  $\alpha_{\rm low}\geq+0.5$, $\alpha_{\rm high}\leq-0.5$ | 26 | (1.5)
Rising | $\alpha>0$ | 476 | (27.5)
Upturn |  $\alpha_{\rm low}<0$, $\alpha_{\rm high}>0$ | 109 | (6.3)
Steep | $-1.1<\alpha<-0.5$ | 208 | (12)
Ultra-steep | $\alpha\leq-1.1$ | 39 | (2.2)

###  5.2 Variability

The variability indices $V_{S}$ (Equation 2) were calculated using quasi-
simultaneous RATAN-600 measurements at frequencies of 2.3, 4.7, 7.7/8.2, 11.2,
and 21.7/22.3 GHz. The $V_{S}$ distributions for different subclasses are
shown in Fig. 8. The average values of $V_{S}$ for each type are shown in
Table 8. The higher variability indices was found at 22.3 GHz. On average, the
BL Lac blazars are more variable at all frequencies. The average and median
$V_{S}$ values at 22.3 GHz span from 0.07 to 0.30 with maximum values of 1.77,
0.90, and 0.89 for FSRQ, BL Lac, and Blaz.un.t.respectively. The maximum
variability indices 0.90–1.77 were estimated for FSRQ and BL Lac blazars at
4.7–22.3 GHz. For the Blaz.un.t. subsample the maximum variability indices are
slightly lower at almost all frequencies, $V_{\rm max}$=0.53–0.89.

Table 8:  The average values of the variability indices $V_{S}$ calculated at five frequencies, 2.3, 4.7, 8.2, 11.2 and 22.3 GHz, for three blazar types. The standard deviation for average variability indices is indicated in parentheses Blazar type | $V_{22.3}$ | $V_{11.2}$ | $V_{8.2}$ | $V_{4.7}$ | $V_{2.3}$
---|---|---|---|---|---
FSRQ | 0.28 (0.21) | 0.23 (0.23) | 0.10 (0.14) | 0.23 (0.20) | 0.15 (0.12)
BL Lac | 0.30 (0.23) | 0.27 (0.20) | 0.17 (0.18) | 0.25 (0.25) | 0.17 (0.15)
Blaz.un.t. | 0.26 (0.22) | 0.26 (0.20) | 0.15 (0.14) | 0.15 (0.14) | 0.16 (0.14)

Figure 8: The variability index distributions at 22.3, 11.2, 8.2, 4.7, 2.3,
and 1.2 GHz for the subclasses of blazars from the catalogue.

##  6 Summary

We present the online “RATAN-600 multi-frequency catalogue of blazars” (BLcat)
that summarises radio continuum measurements of more than 1700 objects in the
frequency range from tens of MHz up to hundreds of GHz. The quasi-simultaneous
data at 1.2–22.3 GHz are represented by the RATAN-600 measurements since 2005,
and additional literature data from external sources cover a time period of
30–40 years.

The catalogue provides estimation of flux densities, spectral indices,
variability, and radio luminosity for the large blazars list. It is a useful
tool for analysing the flare activity and synchrotron radio spectra of AGNs.
The catalogue is regularly updated with new RATAN-600 and literature
measurements.

The users who will make use of the RATAN-600 multi-frequency catalogue of
blazars in a publication are kindly requested to acknowledge the source of the
data by referencing this paper.

## Acknowledgements

This work is supported in the framework of the national project “Science” by
the Ministry of Science and Higher Education of the Russian Federation under
the contract 075-15-2020-778. The observations were carried out with the
RATAN-600 scientific facility. Observations with RATAN-600 are supported by
the Ministry of Science and Higher Education of the Russian Federation. This
research has made use of the CATS database, operated at SAO RAS, Russia. This
research has made use of the Roma-BZCAT NASA/IPAC Extragalactic Database
(NED), which is operated by the Jet Propulsion Laboratory, California
Institute of Technology, under contract with the National Aeronautics and
Space Administration. This research has made use of the SIMBAD database,
operated at CDS, Strasbourg, France

## References

  * Ajello et al. (2020) Ajello M., et al., 2020, ApJ, 892, 105
  * Aller et al. (1985) Aller H. D., Aller M. F., Latimer G. E., Hodge P. E., 1985, ApJS, 59, 513
  * Aller et al. (1992) Aller M. F., Aller H. D., Hughes P. A., 1992, ApJ, 399, 16
  * Altschuler (1986) Altschuler D. R., 1986, A&AS, 65, 267
  * Angelakis et al. (2019) Angelakis E., et al., 2019, A&A, 626, A60
  * Baars et al. (1977) Baars J. W. M., Genzel R., Pauliny-Toth I. I. K., Witzel A., 1977, A&A, 61, 99
  * Becker et al. (1991) Becker R. H., White R. L., Edwards A. L., 1991, ApJS, 75, 1
  * Condon et al. (1998) Condon J. J., Cotton W. D., Greisen E. W., Yin Q. F., Perley R. A., Taylor G. B., Broderick J. J., 1998, AJ, 115, 1693
  * Douglas et al. (1996) Douglas J. N., Bash F. N., Bozyan F. A., Torrence G. V., Wolfe C., 1996, AJ, 111, 1945
  * Gordon et al. (2020) Gordon Y. A., et al., 2020, Research Notes of the American Astronomical Society, 4, 175
  * Gregory & Condon (1991) Gregory P. C., Condon J. J., 1991, ApJS, 75, 1011
  * Healey et al. (2007) Healey S. E., Romani R. W., Taylor G. B., Sadler E. M., Ricci R., Murphy T., Ulvestad J. S., Winn J. N., 2007, ApJS, 171, 61
  * Hovatta & Lindfors (2019) Hovatta T., Lindfors E., 2019, New Astronomy Reviews, 87, 101541
  * Hovatta et al. (2021) Hovatta T., et al., 2021, A&A, 650, A83
  * Hurley-Walker et al. (2017) Hurley-Walker N., et al., 2017, MNRAS, 464, 1146
  * IceCube Collaboration et al. (2018) IceCube Collaboration et al., 2018, Science, 361, 147
  * Intema et al. (2017) Intema H. T., Jagannathan P., Mooley K. P., Frail D. A., 2017, A&A, 598, A78
  * Kiikov et al. (2002) Kiikov S. O., Mingaliev M. G., Stolyarov V. A., Stupalov M. S., 2002, Bulletin of the Special Astrophysics Observatory, 54, 5
  * Korolkov & Pariiskii (1979) Korolkov D. V., Pariiskii I. N., 1979, Sky & Telesc., 57, 324
  * Kovalev et al. (1999) Kovalev Y. Y., Nizhelsky N. A., Kovalev Y. A., Berlin A. B., Zhekanis G. V., Mingaliev M. G., Bogdantsov A. V., 1999, A&AS, 139, 545
  * Kovalev et al. (2020) Kovalev Y. A., et al., 2020, Advances in Space Research, 65, 745
  * Kuehr et al. (1981) Kuehr H., Witzel A., Pauliny-Toth I. I. K., Nauber U., 1981, A&AS, 45, 367
  * Lacy et al. (2020) Lacy M., et al., 2020, PASP, 132, 035001
  * Lazio et al. (2001) Lazio T. J. W., Waltman E. B., Ghigo F. D., Fiedler R. L., Foster R. S., Johnston K. J., 2001, ApJS, 136, 265
  * Massardi et al. (2008) Massardi M., et al., 2008, MNRAS, 384, 775
  * Massardi et al. (2011) Massardi M., Bonaldi A., Bonavera L., López-Caniego M., de Zotti G., Ekers R. D., 2011, MNRAS, 415, 1597
  * Massaro et al. (2009) Massaro E., Giommi P., Leto C., Marchegiani P., Maselli A., Perri M., Piranomonte S., Sclavi S., 2009, A&A, 495, 691
  * Massaro et al. (2015) Massaro E., Maselli A., Leto C., Marchegiani P., Perri M., Giommi P., Piranomonte S., 2015, Ap&SS, 357, 75
  * Mauch et al. (2003) Mauch T., Murphy T., Buttery H. J., Curran J., Hunstead R. W., Piestrzynski B., Robertson J. G., Sadler E. M., 2003, MNRAS, 342, 1117
  * Mingaliev et al. (2008) Mingaliev M. G., Sotnikova Y. V., Bursov N. N., Kardashev N. S., Larionov M. G., 2008, VizieR Online Data Catalog, p. J/AZh/84/387
  * Mingaliev et al. (2012a) Mingaliev M. G., Sotnikova Y. V., Torniainen I., Tornikoski M., Udovitskiy R. Y., 2012a, VizieR Online Data Catalog, pp J/A+A/544/A25
  * Mingaliev et al. (2012b) Mingaliev M. G., Sotnikova Y. V., Torniainen I., Tornikoski M., Udovitskiy R. Y., 2012b, A&A, 544, A25
  * Mingaliev et al. (2014) Mingaliev M. G., Sotnikova Y. V., Udovitskiy R. Y., Mufakharov T. V., Nieppola E., Erkenov A. K., 2014, A&A, 572, A59
  * Mingaliev et al. (2015) Mingaliev M. G., Sotnikova Y. V., Mufakharov T. V., Erkenov A. K., Udovitskiy R. Y., 2015, VizieR Online Data Catalog (other), 0330, J/other/AstBu/70
  * Mingaliev et al. (2017a) Mingaliev M. G., et al., 2017a, VizieR Online Data Catalog, p. J/AN/338/700
  * Mingaliev et al. (2017b) Mingaliev M., et al., 2017b, Astronomische Nachrichten, 338, 700
  * Mitchell et al. (1994) Mitchell K. J., Dennison B., Condon J. J., Altschuler D. R., Payne H. E., O’Dell S. L., Broderick J. J., 1994, ApJS, 93, 441
  * Mufakharov et al. (2015a) Mufakharov T. V., Mingaliev M. G., Sotnikova Y. V., Ya. V. N., Erkenov A. K., 2015a, VizieR Online Data Catalog, p. J/MNRAS/450/2658
  * Mufakharov et al. (2015b) Mufakharov T. V., Sotnikova Y. V., Erkenov A. K., Mingaliev M. G., 2015b, VizieR Online Data Catalog (other), 0330, J/other/AstBu/69
  * Nieppola et al. (2006) Nieppola E., Tornikoski M., Valtaoja E., 2006, A&A, 445, 441
  * Nieppola et al. (2009) Nieppola E., Hovatta T., Tornikoski M., Valtaoja E., Aller M. F., Aller H. D., 2009, AJ, 137, 5022
  * Ochsenbein et al. (2000) Ochsenbein F., Bauer P., Marcout J., 2000, A&AS, 143, 23
  * Ott et al. (1994) Ott M., Witzel A., Quirrenbach A., Krichbaum T. P., Standke K. J., Schalinski C. J., Hummel C. A., 1994, A&A, 284, 331
  * Padovani et al. (2017) Padovani P., et al., 2017, A&ARv, 25, 2
  * Parijskij (1993) Parijskij Y. N., 1993, IEEE Antennas and Propagation Magazine, 35, 7
  * Perley & Butler (2013) Perley R. A., Butler B. J., 2013, ApJS, 204, 19
  * Perley & Butler (2017) Perley R. A., Butler B. J., 2017, ApJS, 230, 7
  * Planck Collaboration (2016) Planck Collaboration 2016, A&A, 594, A13
  * Plavin et al. (2020) Plavin A., Kovalev Y. Y., Kovalev Y. A., Troitsky S., 2020, ApJ, 894, 101
  * Reuter et al. (1997) Reuter H. P., et al., 1997, A&AS, 122, 271
  * Richards et al. (2011) Richards J. L., et al., 2011, ApJS, 194, 29
  * Salonen et al. (1987) Salonen E., et al., 1987, A&AS, 70, 409
  * Seielstad et al. (1983) Seielstad G. A., Pearson T. J., Readhead A. C. S., 1983, PASP, 95, 842
  * Sotnikova et al. (2019a) Sotnikova Y. V., Kovalev Y. A., Erkenov A. K., 2019a, Astrophysical Bulletin, 74, 497
  * Sotnikova et al. (2019b) Sotnikova Y. V., Mufakharov T. V., Majorova E. K., Mingaliev M. G., Udovitskiy R. Y., Bursov N. N., Semenova T. A., 2019b, VizieR Online Data Catalog (other), 0330, J/other/AstBu/74
  * Tabara & Inoue (1980) Tabara H., Inoue M., 1980, A&AS, 39, 379
  * Taylor et al. (1996) Taylor G. B., Vermeulen R. C., Readhead A. C. S., Pearson T. J., Henstock D. R., Wilkinson P. N., 1996, ApJS, 107, 37
  * Teraesranta et al. (1998) Teraesranta H., et al., 1998, A&AS, 132, 305
  * Terasranta et al. (1992) Terasranta H., et al., 1992, A&AS, 94, 121
  * Teräsranta et al. (2004) Teräsranta H., et al., 2004, A&A, 427, 769
  * Tsybulev (2011) Tsybulev P. G., 2011, Astrophysical Bulletin, 66, 109
  * Tsybulev et al. (2018) Tsybulev P. G., Nizhelskii N. A., Dugin M. V., Borisov A. N., Kratov D. V., Udovitskii R. Y., 2018, Astrophysical Bulletin, 73, 494
  * Udovitskiy et al. (2016) Udovitskiy R. Y., Sotnikova Y. V., Mingaliev M. G., Tsybulev P. G., Zhekanis G. V., Nizhelskij N. A., 2016, Astrophysical Bulletin, 71, 496
  * Urry & Padovani (1995) Urry C. M., Padovani P., 1995, PASP, 107, 803
  * Verkhodanov (1997) Verkhodanov O. V., 1997, Astronomical Data Analysis Software and Systems VI, A.S.P. Conference Series, 125, 46
  * Verkhodanov et al. (1997) Verkhodanov O. V., Trushkin S. A., Chernenkov V. N., 1997, Baltic Astronomy, 6, 275
  * Verkhodanov et al. (2005) Verkhodanov O. V., Trushkin S. A., Andernach H., Chernenkov V. N., 2005, Bulletin of the Special Astrophysics Observatory, 58, 118
  * Verkhodanov et al. (2009) Verkhodanov O. V., Trushkin S. A., Andernach H., Chernenkov V. N., 2009, Data Science Journal, 8, 34
  * Villata et al. (2002) Villata M., et al., 2002, Mem. Soc. Astron. Italiana, 73, 1191
  * White et al. (1997) White R. L., Becker R. H., Helfand D. J., Gregg M. D., 1997, ApJ, 475, 479
  * Wright et al. (1990) Wright A. E., Wark R. M., Troup E., Otrupcek R., Hunt D., Cooke D. J., 1990, Publ. Astron. Soc. Australia, 8, 261
  * Wright et al. (1996) Wright A. E., Griffith M. R., Hunt A. J., Troup E., Burke B. F., Ekers R. D., 1996, ApJS, 103, 145
  * de Vries et al. (1997) de Vries W. H., Barthel P. D., O’Dea C. P., 1997, A&A, 321, 105


# Code Compliance Assessment as a Learning Problem

Neela Sawant  Amazon
Bengaluru, India
<EMAIL_ADDRESS>Srinivasan H. Sengamedu  Amazon
Seattle, USA
<EMAIL_ADDRESS>
###### Abstract

Manual code reviews and static code analyzers are the traditional mechanisms
to verify if source code complies with coding policies. However, these
mechanisms are hard to scale. We formulate _code compliance assessment_ as a
machine learning (ML) problem, to take as input a natural language policy and
code, and generate a prediction on the code’s compliance, non-compliance, or
irrelevance. This can help scale compliance _classification_ and _search_ for
policies not covered by traditional mechanisms.

We explore key research questions on ML model formulation, training data, and
evaluation setup. The core idea is to obtain a joint code-text embedding space
which preserves compliance relationships via the vector distance of code and
policy embeddings. As there is no task-specific data, we re-interpret and
filter commonly available software datasets with additional pre-training and
pre-finetuning tasks that reduce the semantic gap.

We benchmarked our approach on two listings of coding policies (_CWE_ and
_CBP_). This is a zero-shot evaluation as none of the policies occur in the
training set. On _CWE_ and _CBP_ respectively, our tool _Policy2Code_ achieves
classification accuracies of (59%, 71%) and search MRR of (0.05, 0.21)
compared to CodeBERT with classification accuracies of (37%, 54%) and MRR of
(0.02, 0.02). In a user study, 24% Policy2Code detections were accepted
compared to 7% for CodeBERT.

###### Index Terms:

code embeddings, natural language analysis

##  I Introduction

Coding policies are natural language rules or best practices on diverse topics
like security, exception handling, and input validation. Code compliance
assessment, e.g., verifying that source code is compliant with coding policies
is an important but non-trivial problem. Consider two policies from the well-
known _Common Weakness Enumeration_ (CWE) listing [1].

  1. 1.

CWE-396111https://cwe.mitre.org/data/definitions/396.html \- “Catching overly
broad exceptions promotes complex error handling code that is more likely to
contain security vulnerabilities”. Now consider this code snippet:

⬇

private static byte[] decrypt(byte[] src, byte[] key) throws RuntimeException
{

try {

SecureRandom sr = new SecureRandom();

DESKeySpec dks = new DESKeySpec(key);

…

Cipher cipher = Cipher.getInstance(DES);

cipher.init(Cipher.DECRYPT_MODE, securekey, sr);

return cipher.doFinal(src);

} catch (Exception e) {

throw new RuntimeException(e);

}

}

This code is non-compliant with CWE-396 as the ‘catch (Exception e)’ statement
broadly covers all exception types. Instead, specific exceptions such as
NoSuchAlgorithmException or NullPointerException should be used.

  2. 2.

CWE-197222https://cwe.mitre.org/data/definitions/197.html \- “Truncation
errors occur when a primitive is cast to a primitive of a smaller size and
data is lost in the conversion”. Now consider this code snippet:

⬇

public static byte[] toByteArray(InputStream input, long size) throws
IOException {

if (size > Integer.MAX_VALUE) {

throw new IllegalArgumentException(""Size greater than Integer max value: "" +
size);

}

return toByteArray(input, (int) size);

}

This code is compliant with CWE-197 because the casting operation _‘(int)
size’_ maps variable _size_ from primitive _long_ to another primitive _int_
of smaller size, and this operation is protected against truncation errors via
the preceding _if_ comparison on integer maximum value.

Code compliance assessment requires a deep understanding of program behavior
and associated policies. This task is subsequently carried out via manual code
reviews and static code analysis. However, both approaches are hard to scale
to increasing and evolving coding policies across diverse programming
languages and frameworks. The global market size for static code analyzers is
projected to reach US $2002 million by 2027, from US $748.1 million in 2020,
at a cumulative annual growth rate of 15.1% during 2021-2027 [2]. Increased
automation can help with this goal.

Our objective is to explore how machine learning can help scale code
compliance assessment - _“Can we automatically label a code as compliant or
non-compliant (or irrelevant) given any natural language policy input?”_ This
has two use-cases: (a) _classification_ \- detecting non-compliant codes for
policies not covered by manual reviews and static analyzer rules, and (b)
_search_ \- finding compliant and non-compliant code examples that will serve
as test cases in developing new static analysis rules. While there is prior
work in detecting buggy codes [3, 4, 5, 6] as well as natural language code
search [7, 8, 9], no solutions exist today that can simultaneously determine
code relevance as well as compliance and non-compliance. In this paper, we set
up a learning framework and explore three research questions.

  * •

_Learning objective_ \- Compliance assessment is similar, but more fine-
grained than relevance assessment. Both the compliant and non-compliant
examples are relevant to a coding policy. _How can we frame the learning
problem?_ We propose learning different embeddings to represent compliant and
non-compliant facets of natural language policies. We formulate a
representation learning approach where the relationships between the compliant
and non-compliant policy facets and compliant, non-compliant, and irrelevant
code examples are preserved via the vector distances between their embeddings.
Representation learning can scale compliance assessment in a _zero-shot_
setting [10] as new policies and code examples can be mapped to their
embeddings, even if they are not part of the original training dataset.

  * •

_Training setup_ \- There are no task-specific labeled training datasets on
coding policies associated with their examples. Curating new training datasets
can be prohibitively expensive. _What training data can be used?_ We propose
repurposing general-purpose code and documentation datasets and propose
additional training and filtering tasks to reduce the semantic gap.

  * •

_Evaluation setup_ \- _How to quantify the performance of learning-based tools
for code compliance assessment?_ We contribute a new benchmark that
reinterprets and repurposes well known listings of coding issues curated by
program analysis experts: (a) Common Weaknesses Enumeration (CWE) [1] and (b)
Coding Best Practices (CBP) from Amazon CodeGuru test suite [11]. We consider
weakness or issue descriptions as policies, buggy examples as non-compliant
and correct code examples as compliant examples. For realistic search setting,
we further add 27K unlabeled code snippets from the CodeSearchNet test dataset
[12]. We use standard metrics to measure performance (_accuracy_ for
classification and _mean reciprocal rank (MRR)_ for search) to compare against
two strong baselines - CodeBERT and a multi-class model.

We develop a new tool _Policy2Code_ by exploring policy representations, loss
formulation, and training schemes. On CWE and CBP respectively, _Policy2Code_
achieves classification accuracies of (59%, 71%) and search MRR of (0.05,
0.21). In comparison, CodeBERT achieves classification accuracies of (37%,
54%) and MRR of (0.02, 0.02) respectively. The multi-class model achieves
classification accuracies of (54%, 60%) and MRR of (0.03, 0.13) respectively.
We set up Policy2Code search engine to on-demand retrieve compliant and non-
compliant examples of new policies. A user study with program analysis experts
showed that 24% detections from _Policy2Code_ were accepted compared to 7%
detections from CodeBERT and 15% by the multi-class model. _Policy2Code_ is
better than other ML alternatives, but still not practical as a stand alone
code compliance tool. We expect it to augment manual efforts in the near
future. We hope our work spawns future research in alternative and robust
formulations for automated code compliance assessment.

##  II Related Work

We discuss machine learning for code applications as well as key ideas in
information retrieval and representation learning.

###  II-A Machine Learning for Code

Machine learning and deep learning in particular, have advanced many software
engineering tasks such as variable naming [13], comment understanding [14],
bug detection [3], code review generation [15], and code and documentation
synthesis [16, 17]. Deep learning research has benefitted from efficient
transformer models [18] such as CodeBERT [19] \- a 124M parameter text-code
encoder with impressive performance on documentation generation and code
search tasks. Another useful concept is that of pre-training, allowing data-
hungry deep learning models to learn from large unlabeled or weakly labeled
datasets on related tasks, so that the models can be subsequently fine-tuned
using small task-specific training datasets [20, 21, 22]. CodeBERT is trained
with CodeSearchNet training dataset for two tasks: masked language modeling
and code-text relevance (method body to method descriptions).

_Neural Bug Finding_ is typically modeled in a supervised setting using
labeled training examples for a pre-determined set of issues. Training
examples may be obtained from existing static analyzer detections [3] or
synthesized with transformations [4]. Much of the research is devoted to
analyzing the effect of richer program representations. For example, Wang et
al. [5] show that combining different types of contexts (the Program
Dependence Graph (PDG), Data Flow Graph (DFG) and the method under
investigation) improves performance of bug detection in pre-determined
categories.

Earlier _Code Search_ systems predominantly relied on string matching, and in
some cases aiming to improve performance with code structures [23], API
matching [24], API usage patterns [25, 26], and query expansion [27]. More
recent neural code search systems perform search based on the vector distance
between their embeddings. This research focuses on three aspects (a) code
representations, for example using method names and API sequences [9], parse
trees [28], and control flow graphs [29, 30], (b) training strategy, for
example, unsupervised [7, 31] or supervised [28, 19, 32, 33], and (c) loss
formulations, for example, contrastive loss or [28, 19] or triplet loss [9,
29, 30, 33, 34].

###  II-B Representation learning

To the best of our knowledge, there are no settings for fine-grained
compliance assessment of code and policy pair. We review key research in
general information retrieval and representation (metric) learning [35, 36,
37, 38]. Representation learning has the attractive property that once the
embedding space is learned, novel input can be mapped without additional
training, essentially a _zero-shot_ setting [10].

A common ranking mechanism is _triplet loss_ that models relationships among
three data points - an anchor, a positive example, and a negative example. In
traditional code search, they map to natural language query (such as a method
description) to relevant example (such as the corresponding method body) and
an irrelevant example (such as the body of an unrelated method). [39, 40, 41].
The algorithm learns to minimize the distance between anchor-positive pairs
and maximize the distance between the anchor-negative pairs. Triplet mining is
an active area of research [39, 40, 41]. There are very few settings involving
higher order losses for fine-grained differentiation. For example, Chen et al.
[42] introduced a quadruplet loss network for person verification task. A
training example in this task consists of four data points
$x_{i},x_{j},x_{k},x_{l}$ where $x_{i},x_{j}$ belong to the same class (e.g.,
considered the same person) and $x_{k}$ and $x_{l}$ belong to two other
classes (faces from two other people).

###  II-C Other domains

Our objective may seem similar to the sentiment analysis task [43, 44] in
natural language understanding. The objective of this task is to classify a
natural language text into positive, negative, or neutral polarities. The
difference is that the polarities of natural language are not comparable with
the polarities of coding policies. The semantics of a polarity is globally
consistent for natural language. For example, both these sentences - ‘food is
exceptionally good’ or ‘the concert was exceptionally good’, convey similarly
positive emotions about two different entities. On the other hand, the
compliance or non-compliance of code is always conditioned on the specific
policy under consideration. The same snippet may be compliant with one policy
but non-compliant with another (for example, adheres to exception handling
requirements, but is not secure).

##  III Learning Problem Formulation

Let $r$ and $c$ denote a natural language policy and a code example. Let
$y\in\\{+,-\\}$ denote the compliant and non-compliant facets respectively,
and let irrelevance be denoted by $\sim$. Our objective is to learn a bimodal
embedding space for $r$ and $c$ such that the the compliant, non-compliant
(and irrelevance) relationships are represented via the vector distances
between their embeddings. In contrast to code search, where natural language
query has a single representation, we propose to learn separate $r^{+}$ and
$r^{-}$ embeddings for fine-grained differentiation. For compliant and non-
compliant example search, we then operate with two separate query inputs
$r^{+}$ and $r^{-}$. Embedding of $r^{+}$ should be closer to $c^{+}$,
followed by $c^{-}$, and dissimilar to that of $c^{\sim}$. For non-compliant
search, $r^{-}$ is closer to $c^{-}$, followed by $c^{+}$, and far from
$c^{\sim}$. Such space naturally supports search and classification based on
relative distances and learned thresholds. New policies and codes can be
mapped to their embeddings and evaluated in zero-shot setting. We now discuss
two aspects of representation learning - policy input representation and loss
functions.

###  III-A Facet Policy Input Representation

Policy representation is a complex problem. A policy contains a general
relevance context and a specific compliance context. For example, reconsider
(CWE-396): Catching overly broad exceptions promotes complex error handling
code that is more likely to contain security vulnerabilities. This policy is
relevant to all code snippets with exception handling. The compliance context
refers to using broad exception types (_Exception_) versus more specific
exceptions (_NoSuchAlgorithmException_ or _NullPointerException_). Policy
representations can incorporate domain knowledge and advanced parsing of
natural language text. In the context of this paper however, we explore two
purely learning based variations inspired from prior work in NLP and Computer
Vision.

####  III-A1 Facet-prefixed Policy

This idea is similar to T5 query format [45] with remarkable success in NLP
applications. Natural language policy description $r$ is prefixed with facet
token $y$. The concatenated representation $y:r$ is passed through an encoder
to generate faceted policy embedding. We use a transformer-based encoder with
self-attention, so that the facet token organically attends to the various
policy text components.

####  III-A2 Facet-masked Policy

This approach externally combines the facet token with the policy input using
a parametric model. We use an implementation of conditional masking [46],
which aims to learn as many mask vectors as the number of conditions (e.g.,
facets in our setting). One of the mask vector is selected based on the facet
$y$ and multiplied with embedding of policy $r$, essentially projecting the
policy into a different subspace conditioned on the facet label. See Appendix
A for details.

###  III-B Loss Functions

Figure 1: Illustration of compliance search task (with compliant facet): The
learning objective is to minimize the vector distance between the faceted
policy description and a compliant code embedding, and maximize the distance
with respect to non-compliant and irrelevant code embeddings.

Figure 1 illustrates the learning objective using an example of compliance
search task. Given a training quadruplet $(r^{+},c^{+},c^{-},c^{\sim})$, we
want to learn embeddings that minimize the relative distances between matched
pairs and maximize the distances between unmatched pairs. We test two
formulations.

####  III-B1 Quadruplet Loss (QL)

Quadruplet loss is an extension of triplet loss computed as follows:

$\displaystyle L^{+}_{quad}$
$\displaystyle=\sum_{r^{+},c^{+},c^{-}}^{N}[d(f_{r^{+}},f_{c^{+}})^{2}-d(f_{r^{+}},f_{c^{-}})^{2}+\alpha_{1}]_{+}$
(1)
$\displaystyle+\sum_{r^{+},c^{-},c^{\sim}}^{N}[d(f_{r^{+}},f_{c^{-}})^{2}-d(f_{r^{+}},f_{c^{\sim}})^{2}+\alpha_{2}]_{+}$

where $f_{x}$ denotes the embedding of sequence $x$, $d(.,.)$ denotes the
distance function between two embeddings, and $[]_{+}$ indicates that only
positive values contribute towards the loss calculation.
$\alpha_{2}>\alpha_{1}>0$ are tunable margin parameters for compliance and
relevance differentiation, respectively. The loss for the non-compliant facet
can be computed similarly over quadruplet $(r^{-},c^{-},c^{+},c^{\sim})$.
Finally, the total loss is computed as
$L_{quad}=\frac{1}{2}(L^{+}_{quad}+L^{-}_{quad})$.

The naive quadruplet loss formulation has two limitations. Eqn. 1 is anchored
around text only. It does not encode constraint for classification task
because there is no anchoring around code and no constraint to ensure that
embeddings of $r^{+}$ and $r^{-}$ are different given a code. Another
limitation is from practically available data. In general purpose datasets,
very few examples will contain both compliant and non-compliant codes.
However, there is an abundance of datasets with partial view (some contain
only relevant examples, some with only compliant or non-compliant examples,
etc.). To be able to use the latter datasets, we need an alternative
formulation as below.

####  III-B2 Bimodal Multi-task Triplet loss (BMT)

In this formulation, the quadruplets $(r^{+},c^{+},c^{-},c^{\sim})$ and
$(r^{-},c^{-},c^{+},c^{\sim})$ are un-pivoted into a batch $B$ of labeled
examples $(x,l)$, where $x$ can be either a text or code (e.g., $x$ is
bimodal) and label $l$ is generated on the fly such that only the $x$
originating from paired policy and facets share that label $l$. For example,
the quadruplets mentioned above can be un-pivoted into a list
$[(r^{+},l^{1}),(c^{+},l^{1}),(r^{-},l^{2}),(c^{-},l^{2}),(c^{\sim},l^{\sim}),\ldots]$.
Each irrelevant example $c^{\sim}$ is assigned a unique label $l^{\sim}$ to
prevent anchor-positive matching with another entity. We then calculate a
_bimodal multi-task triplet (BMT)_ loss over possible valid triplets $(a,p,n)$
where $a$, $p$, and $n$ are anchor, positive, and negative sequences in the
batch such that $a$ and $p$ share the same label and $n$ has a different
label. The term _multi-task_ denotes the simultaneous modeling of different
class labels.

$L_{B}=\sum_{(a,p,n);l(a)=l(p),l(a)\neq
l(n)}[d(f_{a},f_{p})^{2}-d(f_{a},f_{n})^{2}+\alpha]_{+}$ (2)

where $\alpha$ is a common margin which acts as a threshold to separate the
irrelevant examples from the relevant ones. The formulation naturally allows
both code and text entities to serve as anchors. Datasets with partial labels
(only relevant, compliant, and/or non-compliant) can be incorporated into
training. Compliance search task is served by selecting codes based on their
distances from faceted policy representations. For compliance classification,
we use the threshold $\alpha$ to determine if a code $c$ is relevant or
irrelevant with respect to the average policy embedding $(r^{+}+r^{-})/2$. If
it is deemed relevant, we use the distances with respect to $r^{+}$ and
$r^{-}$ to determine the closest facet as compliance label. The distances can
be mapped to probabilities using SoftMax function
$p(d)=\frac{e^{-d}}{\sum_{d^{\prime}}e^{-d^{\prime}}}$.

##  IV Training Setup

ML models are data hungry. For code compliance assessment however, there are
no large-scale datasets containing natural language policies paired with their
compliant and non-compliant examples. We re-interpret and repurpose general
datasets on code, documentation, code reviews, and bug-fixes with additional
training schemes to bridge this gap.

###  IV-A Multi-Task Pre-training

To capture patterns in higher level documentation guidelines, pre-training
with general documentation can be helpful. To capture coding issues and
aligned code examples, pre-training with code reviews can be beneficial. We
use below tasks.

  * •

Collocated documentation prediction (Doc): Paragraphs of software
documentation are segmented into non-overlapping fixed-length passages. The
passages belonging to the same paragraph are assigned a common label. The
objective of Collocated documentation prediction is to predict which candidate
segments (code snippet or text) in a batch come from the same paragraph.

  * •

Code-comment matching task (CC): Given multiple <code, review comment> pairs,
we attempt to determine (a) the correct review comment given a code (relevant
comment prediction) and (b) the correct code on which a review comment is made
(non-compliant code prediction).

###  IV-B Pre-fine-tuning

Examples containing bug-fixes are especially valuable to us, because the
example can be reinterpreted as a proxy to policy description with compliant
and non-compliant codes. Such a dataset can then be used to pre-finetune for
target task. See Figure 2 for a motivating example of a GitHub bug-fix review
comment. The comment provides a security best practice suggestion for which
the code-before and code-after versions can be used as non-compliant and
compliant examples, respectively. Each review comment is essentially treated
as a policy and the code-before and the code-after versions are taken to be
the non-compliant and the compliant code examples respectively. Irrelevant
examples $c^{\sim}$ are mined from codes of unrelated bug-fixes. This dataset
can be noisy, but we expect useful signals to emerge with large-scale
datasets.

Code review comment: Probably better to use ‘SecureRandom‘ for anything
security related..

Facet: non-compliant/ Code-before

⬇

private String generatePassword(){

String allowedChars="abcdefghijklmnopqrstuvwxyz0123456789";

int charRange=allowedChars.length();

Random random=new Random();

StringBuilder password=new StringBuilder();

int passwordLength=10;

for (int i=0; i < passwordLength; i++) {

password.append(allowedChars.charAt(random.nextInt(charRange)));

}

return password.toString();

}

Facet: compliant/ Code-after

⬇

private String generatePassword() throws NoSuchAlgorithmException {

String allowedChars="abcdefghijklmnopqrstuvwxyz0123456789";

int charRange=allowedChars.length();

SecureRandom random=SecureRandom.getInstanceStrong();

int passwordLength=20;

char[] password=new char[passwordLength];

for (int i=0; i < passwordLength; i++) {

password[i]=allowedChars.charAt(random.nextInt(charRange));

}

return new String(password);

}

Figure 2: Example entry in GitHub comment and bug-fix dataset

_Data Filtering_ : As an additional treatment, we apply _Doc2BP_ [47], a deep
learning classifier trained to detect coding policies and best practice
recommendations from natural language text. _Doc2BP_ analyzes keywords, parts
of speech patterns, and other linguistic properties to determine if a comment
resembles coding policies, and filters out non-policy-like comments to reduce
noise. Results with and without _Doc2BP_ indicate that the filtering step in
pre-finetuning is useful.

##  V Evaluation Setup

We construct a new benchmark by repurposing two listings of coding issues and
a public code search dataset.

  * •

Common Weaknesses Enumeration (CWE) View 702 [1] is a public domain list of
software weaknesses introduced during implementation. It contains 185 Java
weaknesses, each with a short natural language description and compliant and
non-compliant examples. We use the weakness description as the policy. The
number of policies with non-compliant, compliant, and both types of examples
are 182, 36, and 33, respectively.

  * •

Coding Best Practice (CBP) is a collection of Java best practice descriptions
and compliant and non-compliant examples obtained from Amazon CodeGuru [11]
static analyzer test-suite. Our dataset contains 46 Java best practices, each
with at least one compliant and one non-compliant example, and a total of 186
examples.

  * •

CodeSearchNet (CSN) is a public domain dataset [12]. We use the test split
with 27K unlabeled code examples.

The resulting benchmark contains 231 policies, 477 labeled codes, and 27K
unlabeled codes (total 28K code examples). Policies cover many topics like
input validation, cipher security, web security, command injection, preferred
data structures, etc. The dataset has no overlap with the training data, hence
this is a zero-shot benchmark. We investigate two tasks:

  * •

Compliance Classification: The objective is to predict whether a code is
compliant, non-compliant, or irrelevant given a policy. Performance is
measured in accuracy with respect to the known ground truth.

  * •

Compliance Search: The objective is to retrieve compliant and non-compliant
codes from a large code corpus for a user-specified natural language policy.
In a large corpus setting, we do not expect to know the ground truth labels
for all code examples. Hence we rely on another metric in information
retrieval research, the mean reciprocal rank metric ($MRR$) defined as
$\frac{1}{Q}\sum_{i=1}^{|Q|}\frac{1}{rank_{i}}$ where $Q$ denotes the number
of user queries and $rank_{i}$ denotes the rank position of the first example
from the known ground truth within each facet. We prefer MRR over other
information retrieval metrics such as precision@k for its ability to capture
the order of relevance. For example, a precision@5 metric would yield the same
score (0.2) whether a single relevant result appears at rank 1 or rank 5. MRR
on the other hand, would generate a score of $1$ for the first position and a
score of $0.2$ for the fifth position.

We propose two competitive baselines.

  * •

CodeBERT [19] is the off-the-shelf pre-trained HuggingFace implementation.
CodeBERT is a state-of-the-art deep learning approach for several code-related
tasks including code search (https://microsoft.github.io/CodeXGLUE/). Despite
using a token sequence representation of code, it is able to out-perform
complex models such as GraphCodeBERT that can represent programs as graphs.
This baseline is designed for relevance and not fine-grained differentiation.
It represents the current best tool available to developers to find relevant
examples, which can then be manually classified for compliance.

  * •

Multi-class classification is a three-class (compliant, non-compliant,
irrelevant) classifier. It is implemented by adding dropout and linear layers
on top of the CodeBERT’s [19] transformer embedding layer. This is also
trained for a zero-shot setting. A (policy, code) pair is represented using
CodeBERT’s input sequence convention
$[CLS]x_{1},x_{2},...x_{m}[SEP]y_{1},y_{2},...y_{n}[SEP]$ where $x$ and $y$
represent the tokens for policy and code respectively.

Our use of public domain datasets and easy to replicate baselines is expected
to fuel future research on this topic.

##  VI Results

This section presents a comprehensive evaluation. Bold-facing in tables is
used to indicate superior performance.

###  VI-A Experimental Setup

As the central bimodal code-text encoder for Policy2Code as well as the multi-
class classification baseline, we use the HuggingFace implementation of
CodeBERT [19]. The multi-class classification baseline and the Policy2Code
model are trained using the same pre-training and pre-fine-tuning data for
apples-to-apples comparison. For the collocated documentation pre-training
(DOC) task, we use Java 8, 11, and 16 documentation ($\sim$1 GB). For code
comment matching pre-training (CC), we used 150K GitHub code reviews. For bug-
fix-comment pre-fine-tuning, we leveraged 32K code reviews on GitHub that were
tagged as bug-fix, split into 80%-20% for training and validation. All these
datasets are in public domain. Further none of them intentionally overlap with
benchmark examples. Code is statically represented as token sequences with
sub-word tokenization and BytePair Encoding (BPE) [48].

###  VI-B Policy2Code Model Training

TABLE I: Effect of Loss Formulation and Policy Representation Loss | Policy Rep | Classification Accuracy | Search MRR
---|---|---|---
CWE (Conceptual) | CWE | CBP | CWE (Conceptual) | CWE | CBP
QL | Prefixed | 42.6 $\pm$ 3.6 | 48.13 $\pm$ 2.2 | 44.93 $\pm$ 0.98 | 0.021 $\pm$ 0.00 | 0.0321 $\pm$ 0.009 | 0.0915 $\pm$ 0.017
Masked | 33.41 $\pm$ 1.9 | 34.97 $\pm$ 2.2 | 39.3 $\pm$ 6.3 | 0.0298 $\pm$ 0.02 | 0.0318 $\pm$ 0.19 | 0.07641 $\pm$ 0.029
BMT | Prefixed |  45.3 $\pm$ 2.4 |  49.8 $\pm$ 5.1 |  48.0 $\pm$ 7.1 | 0.0264 $\pm$ 0.029 |  0.0363 $\pm$ 0.011 |  0.1496 $\pm$ 0.021
Masked | 41.6 $\pm$ 4.5 | 42.23 $\pm$ 3.7 | 46.21 $\pm$ 1.2 | 0.0242 $\pm$ 0.061 | 0.0213 $\pm$ 0.032 | 0.1127 $\pm$ 0.027
TABLE II: Effect of Pre-training and Filtering Schemes Pre-Training Scheme | Classification Accuracy | Search MRR
---|---|---
CWE (Conceptual) | CWE | CBP | CWE (Conceptual) | CWE | CBP
None | 45.3 $\pm$ 2.4 | 49.8 $\pm$ 5.1 | 48.0 $\pm$ 7.1 | 0.0264 $\pm$ 0.029 | 0.0363 $\pm$ 0.011 | 0.1496 $\pm$ 0.021
Doc | 48.3 $\pm$ 3.1 | 54.2 $\pm$ 4.6 | 56.2 $\pm$ 2.6 | 0.0319 $\pm$ 0.038 | 0.0313 $\pm$ 0.008 | 0.1773 $\pm$ 0.009
CC | 47.8 $\pm$ 6.1 | 43.77 $\pm$ 1.1 | 55.2 $\pm$ 2.9 | 0.0471 $\pm$ 0.129 | 0.0467 $\pm$ 0.008 | 0.1654 $\pm$ 0.017
Doc + CC | 53.2 $\pm$ 3.2 |  59.3 $\pm$ 7.4 | 67.6 $\pm$ 7.9 |  0.0492 $\pm$ 0.05 | 0.0482 $\pm$ 0.011 | 0.1982 $\pm$ 0.012
Doc + CC + _Doc2BP_ Filter [47] |  53.8 $\pm$ 6.1 |  59.3 $\pm$ 2.0 |  71.0 $\pm$ 2.8 | 0.0462 ± 0.013 |  0.0538 $\pm$ 0.01 |  0.2182 $\pm$ 0.012

We now discuss the effect of policy representations, losses, and training
schemes. Models were optimized using automatic hyper-parameter tuning. See
Appendix B for details.

Table I reports the performance of different combinations of loss functions
and policy representations on the CWE and CBP datasets for classification and
search tasks. _The BMT loss with facet-prefixed policy representation is the
most effective strategy across both datasets and tasks_.

Table II shows the effect of different pre-training and filtering schemes on
Policy2Code performance. All schemes use BMT loss with facet-prefixed policy
representation and subsequently followed by the same pre-finetuning step. The
filtering step reduces the dataset to 14K examples (out of 32K original bug-
fix dataset) which are predicted to be policy-like. _The pre-training schemes
as well as the filtering step improve model performance across tasks_. We
continue to see better performance for the CBP dataset than the CWE dataset.
We also observe that documentation and code-related pre-training impact CWE
performance differently. While code-related pre-training (CC) decreases the
CWE performance, documentation pre-training improves the performance. This
effect can be explained as CWE policies are more conceptual, hence more
similar to documentation patterns than specific code comments.

###  VI-C Baseline Comparison

Table III compares Policy2Code with baselines on compliance classification and
compliance search tasks. Policy2Code outperforms CodeBERT as well as the
multi-class classification baseline. Policy2Code achieves classification
accuracies of (59%, 71%) and search MRR of (0.05, 0.21) on CWE and CBP. On the
same datasets, a popular CodeBERT baseline achieves classification accuracies
of (37%, 54%) and MRR of (0.02, 0.02) respectively. Multi-class classification
achieves accuracies of (54%, 60%) and MRR of (0.03, 0.13) respectively.

We observe that the model performance is generally lower on the CWE dataset.
Our investigation shows that CWE contains multiple policies on related topics
and partially labeled ground truth. One code example may be applicable to
multiple policies, but not all such relationships are recorded. For example,
CWE-338333https://cwe.mitre.org/data/definitions/338.html suggests _using
cryptographically strong Pseudo-Random Number Generator (PRNG) in a security
context_. The below code is the only explicitly labeled example for this
policy.

⬇

Random random = new Random(System.currentTimeMillis());

int accountID = random.nextInt();

However, other code examples in the CWE dataset may also be related to
CWE-338. See the below code for example, labeled only for policy
CWE-336444https://cwe.mitre.org/data/definitions/336.html on _not using the
same seed across multiple PRNG initializations_. This example can also be
considered as a valid example for CWE-338.

⬇

private static final long SEED = 1234567890;

public int generateAccountID() {

Random random = new Random(SEED);

return random.nextInt();

}

This implies that the CWE ground truth is incomplete, leading to currently
computed metrics being a lower bound on the actual performance. CBP is
expected to have higher quality as it is manually curated for compliance
analysis task specifically.

TABLE III: Comparison of Policy2Code with Baselines on Compliance Assessment Tasks Method | Classification Accuracy | Search MRR
---|---|---
CWE (Conceptual) | CWE | CBP | CWE (Conceptual) | CWE | CBP
CodeBERT | 33.5 | 37.5 | 54.4 | 0.0223 | 0.0242 | 0.0217
Multi-class | 42.82 $\pm$ 3.1 | 54.36 $\pm$ 1.9 | 60.04 $\pm$ 2.7 | 0.0297 $\pm$ 0.03 | 0.0312 $\pm$ 0.02 | 0.1310 $\pm$ 0.03
Policy2Code |  53.84 $\pm$ 6.1 |  59.29 $\pm$ 2.0 |  71.04 $\pm$ 2.8 |  0.0462 $\pm$ 0.013 |  0.0538 $\pm$ 0.01 |  0.2182 $\pm$ 0.012

###  VI-D User Study

We conducted a user study to determine if Policy2Code can be used in a
practical setting for searching compliant and non-compliant code examples. One
application for such examples is to serve as test cases for creating new
static analysis rules.

We built a FAISS-GPU index [49] over all 28K code examples in the benchmark
dataset. On a p3.xlarge EC2 machine, index creation takes approximately 8
seconds. At inference time, top examples for any policy queries are retrieved
within milliseconds based on approximate and efficient nearest neighbor
algorithm. Exact similarity computation is avoided as it can take about 30
minutes per policy to compare all codes.

We recruited 16 experts with an average of over 10 years of software
engineering experience and over 5 years of programming language analysis
experience. For 25 policies in CWE dataset, we determined the top 5 compliant
and non-compliant examples detected by various approaches in the entire 28K
code example benchmark dataset. We masked the identity of the selection
algorithm and asked the assessor if they agree or disagree with the label
assignment. Table IV shows the acceptance rate of predictions made by
different techniques. _Policy2Code has the highest overall acceptance rate at
24%, followed by 15.2% for multi-class classification and 7.2% for CodeBERT,
averaged over all assessments._

TABLE IV: Acceptance Rate in User Study (%) Model | Compliant | Non-compliant | Overall
---|---|---|---
CodeBERT | 9.68 | 6.38 | 7.2
Multi-class | 27.01 | 11.57 | 15.2
Policy2Code | 41.93 | 17.64 | 24.13

###  VI-E Anecdotal Results

See Figures 3 and 4 for anecdotal examples. On the one hand, they may seem to
over-index on keyword similarity (Figure 3(c), (e)), and on the other hand, we
find surprisingly semantic associations (Figure 3(a)). Models trained on
textual features alone, without using any advanced program representations may
over-index on keyword similarity rather than higher order semantics. Here is
an incorrect finding, The software does not properly account for differences
in case sensitivity when accessing or determining the properties of a
resource, leading to inconsistent results. (CWE-178).

⬇

private boolean queryRelativeCatalogs () {

if (resources==null) readProperties();

if (resources==null) return defaultRelativeCatalogs;

try {

String allow = resources.getString(‘catalogs’);

␣␣␣␣␣␣␣␣return␣(allow.equalsIgnoreCase(‘true’) ||
allow.equalsIgnoreCase(‘y’)␣||␣allow.equalsIgnoreCase(‘1’));

} catch (MissingResourceException e) {

return defaultRelativeCatalogs;

}

}

The repeated usage of keywords resource, properties, and case in the result,
leads to the example being scored higher, even though the notion of case-
sensitivity is incorrectly modeled. Here, the case handling is a design
choice. Policy2Code makes a mistake on this very challenging example.

a) Truncation errors occur when a primitive is cast to a primitive of a
smaller size and data is lost in the conversion. (CWE-197)

⬇

public static byte[] toByteArray(InputStream input, long size) throws
IOException {

if (size > Integer.MAX_VALUE) {

throw new IllegalArgumentException(""Size greater than Integer max value: "" +
size);

}

return toByteArray(input, (int) size);

}

b) The software accepts XML from an untrusted source but does not validate the
XML against the proper schema. (CWE-112)

⬇

public static Document newDocument(String xmlString, boolean namespaceAware)
throws SAXException, IOException, ParserConfigurationException {

return XmlUtils.newDocument(new InputSource(new StringReader(xmlString)),
namespaceAware);

}

c) This program compares classes by name, which can cause it to use the wrong
class when multiple classes can have the same name. (CWE-486)

⬇

protected static String createTableRefName(final Object entity) {

Class type = entity.getClass();

type = (type == Class.class ? (Class) entity : type);

return (type.getSimpleName() + ’_’);

}

d) The product does not sufficiently enforce boundaries between the states of
different sessions, causing data to be provided to, or used by, the wrong
session. (CWE-488)

⬇

protected void update(CollectionUpdateType force) throws IOException {

State localState;

synchronized (lock) {

if (first) {

state = checkState();

state.last = System.currentTimeMillis();

return;

}

localState = state.copy();

}

updateCollection(localState, force);

localState.last = System.currentTimeMillis();

synchronized (lock) {

state = localState;

}

}

e) The product uses untrusted input when calculating or using an array index,
but the product does not validate or incorrectly validates the index to ensure
the index references a valid position within the array. (CWE-129)

⬇

private static int calculateEndIndex( double[] array, int originalIndex ) {

final int length = array.length;

Exceptions.requireNonNull( array, ""array cannot be null"" );

int index = originalIndex;

if ( index < 0 ) {

index = length + index;

}

if ( index < 0 ) {

index = 0;

}

if ( index > length ) {

index = length;

}

return index;

}

Figure 3: More Examples for Compliant Facet \- All examples are real
detections from Policy2Code on CodeSearchNet-extended corpus. CWE ids are
provided for reference only. a) Catching overly broad exceptions promotes
complex error handling code that is more likely to contain security
vulnerabilities. (CWE-396)

⬇

private static byte[] decrypt(byte[] src, byte[] key) throws RuntimeException
{

try {

SecureRandom sr = new SecureRandom();

DESKeySpec dks = new DESKeySpec(key);

SecretKeyFactory keyFactory = SecretKeyFactory.getInstance(DES);

SecretKey securekey = keyFactory.generateSecret(dks);

Cipher cipher = Cipher.getInstance(DES);

cipher.init(Cipher.DECRYPT_MODE, securekey, sr);

return cipher.doFinal(src);

} catch (Exception e) {

throw new RuntimeException(e);

}

}

b) Do not throw new checked exceptions (CheckedExceptions) forcing users to
handle them elsewhere. Code should either propagate existing checked
exceptions or handle them.

⬇

public static Cipher newCipher(String algorithm) {

try {

return Cipher.getInstance(algorithm);

}

catch (NoSuchAlgorithmException e) {

throw new IllegalArgumentException(""Not a valid encryption algorithm"", e);

}

catch (NoSuchPaddingException e) {

throw new IllegalStateException(""Should not happen"", e);

}

}

c) Instead of repeatedly creating a new Random object to obtain multiple
random numbers, create a single Random object and reuse it.

⬇

private static final long SEED = 1234567890;

public int generateAccountID() {

Random random = new Random(SEED);

return random.nextInt();

}

d) The product receives input that is expected to specify an index, position,
or offset into an indexable resource such as a buffer or file, but it does not
validate or incorrectly validates that the specified index/position/offset has
the required properties. (CWE-1285)

⬇

private static Object getCollectionProp(Object o, String propName, int index,
String[] path) {

o = _getFieldValuesFromCollectionOrArray(o, propName);

if ( index + 1 == path.length ) {

return o;

} else {

index++;

return getCollectionProp(o, path[index], index, path);

}

}

e) The software does not properly handle when the expected number of
parameters, fields,or arguments is not provided in input, or if those
parameters are undefined. (CWE-229)

⬇

public static <M extends Model> boolean isNew(M m, String pk_column) {

final Object val = m.get(pk_column);

return val == null || val instanceof Number && ((Number) val).intValue() <= 0;

}

Figure 4: More Examples for Non-compliant Facet \- All examples are real
detections from Policy2Code on CodeSearchNet-extended corpus. CWE ids are
provided for reference only.

##  VII Threats to Validity

As we formulate code compliance assessment as a machine learning problem, we
see certain threats to the validity.

  * •

ML models are known to carry biases from the training datasets. While more
training schemes (pre-training as well as fine-tuning) can help improve model
performance, some policies may still be under-represented in these training
datasets. Certain issues may rarely occur in code and even rarely detected by
human reviewers. The models trained on human code review comments and bug-
fixes may not learn to detect these issues well.

  * •

The benchmark dataset needs further curation. In using the CWE dataset, we see
that certain policy descriptions are conceptual and loosely scripted that do
not convey a strong recommendation or warning. Further, some ground truth
labels are missing. Future work should further enhance the benchmark in both
aspects to improve assessment.

  * •

Finally, ML predictions may never meet 100% accuracy unlike carefully crafted
static analysis rules. We expect automated tools to augment and not replace
manual code reviews and static analyzer creation. Some reduced degree of
manual vetting may still be required.

We need to understand how the performance of machine learning algorithms vary
by data characteristics (policy and code) and how it can be improved by
innovating on natural language (NL) and programming language (PL) research.

##  VIII Conclusion

Code compliance assessment is an important problem in software development and
an emerging area for machine learning. This paper explores novel research
questions related to learning framework, training data, and evaluation setup.
We proposed a representation learning approach that preserves the
relationships between policies and their compliant, non-compliant, and
irrelevant code examples via the vector distances between their embeddings. To
overcome the lack of task-specific training data, we proposed repurposing
general software datasets with pre-training, pre-fine-tuning, and filtering
steps. We evaluated the impact of policy representations, losses, training
schemes, and hyper-parameter optimization. Resulting Policy2Code model shows
promising results for compliance classification and search tasks. Policy2Code
achieves classification accuracies of (59%, 71%) and search MRR of (0.05,
0.21) on CWE and CBP, the two types of datasets in our benchmark. On the same
datasets, CodeBERT baseline achieves classification accuracies of (37%, 54%)
and MRR of (0.02, 0.02) respectively whereas multi-class baseline achieves
classification accuracies of (54%, 60%) and MRR of (0.03, 0.13) respectively.
In a user study of compliant and non-compliant findings, 24% detections from
Policy2Code were accepted compared to only 7% detections from CodeBERT and 15%
by the multi-class baseline.

We hope to encourage more theoretical and empirical research in automated code
compliance assessment. We expect ML solutions to significantly reduce future
manual efforts in code reviews and static analysis. Improvements can come from
innovating on input transformations, use of program analysis and domain
knowledge (API knowledge graph), and robust ML formulations.

##  Appendix A Facet-masked Policy Representation

The idea for facet-masked representation is based on the conditional masking
work, originally used in computer vision to compute image similarities across
aspects such as color and texture [46]. The embedding of policy $r$ is
factorized depending on facet value $y$. The factorization is implemented via
a conditional mask $\textbf{m}\in R^{d\times n_{k}}$, where $d$ is the
embedding vector length and $n_{k}$ is the number of facets (=2). The mask can
be paramterized as $\textbf{m}=\sigma(\beta)$, with $\sigma$ denoting a
rectified linear unit so that $\sigma(\beta)=max\\{0,\beta\\}$. The $k^{th}$
column $m_{k}$ plays the role of an element-wise gating function selecting the
relevant subspace of the embedding dimensions to attend to the $k^{th}$ facet.
Additional loss regularization terms are used to encourage embeddings to be
drawn from a unit ball ($L_{W}$) and for mask regularization ($L_{M}$).

##  Appendix B Model Tuning Experiments

Performance of neural models can be significantly improved with proper hyper-
parameter tuning (HPT). We used automatic selection of parameters (batch size,
learning rate, etc) based on a validation dataset and experimented with
triplet loss variations.

  * •

Batch All Triplet loss computes the loss for all possible, valid triplets in
the batch, i.e., anchor and positive must have the same label, anchor and
negative a different label.

  * •

Batch Hard Triplet loss computes the loss for all possible, valid triplets. It
then looks for the hardest positive (largest $d(f_{a},f_{p})$) and the hardest
negatives (smallest $d(f_{a},f_{n})$) per label class, and sums the loss only
over them.

  * •

Batch Semi-hard Triplet loss computes the loss for all possible, valid
triplets. It then looks for the semi hard positives and negatives and sums the
loss only over them.

  * •

Batch Hard Triplet Soft-Margin loss is a variation of the Batch Hard triplet
loss where the loss over the hardest positive and the hardest negative
examples are computed with a soft margin, e.g. $\log
1p(\exp(d(f_{a},f_{p})-d(f_{a},f_{n})))$.

The determination of hard, semi-hard, and easy triplets was made with a
tunable margin parameter.

  * •

Easy: $d(f_{a}-f_{p})+\alpha<d(f_{a}-f_{n})$. These are already well separated
and not useful for model training.

  * •

Medium: $d(f_{a}-f_{p})<d(f_{a}-f_{n})<d(f_{a}-f_{p})+\alpha$. These triplets
are typically more suited for optimisation.

  * •

Hard: $d(f_{a}-f_{n})<d(f_{a}-f_{p})$. Selecting the hardest negative examples
can sometimes lead to bad local minima and result in a collapsed model
training [39].

Figure 5 shows the distribution of distances between the embeddings of text
anchors and positive and negative code example for a sample of the dataset.
The distribution is partitioned in three sections based on the margin, used to
define easy, medium, and hard triplets. Figure 6 shows the effect of margin
with various triplet modelling strategies. Table V compares the performance of
loss variations on fine-tuning the CodeBERT model. The margin parameter is
tuned for each variation except the soft-margin formulation which does not use
a margin threshold. We observe that Batch All Triplet loss has the best
performance and use it for all subsequent experiments.

Figure 5: Distribution of triplet distances can be used to tune margins and
mine triplets.

Figure 6: Impact of margin on the pre-fine-tuning MRR of Batch All Triplet
loss.

TABLE V: Performance of triplet loss variations Batch Triplet Loss Type | Overall Accuracy | Overall MRR
---|---|---
All Triplets | 57.2 $\pm$ 6.2 | 0.080 $\pm$ 0.02
Hard Triplets | 48.1 $\pm$ 4.7 | 0.063 $\pm$ 0.01
Semi-Hard Triplets | 50.13 $\pm$ 2.8 | 0.068 $\pm$ 0.02
Hard Triplets Soft-Margin | 53.12 $\pm$ 5.1 | 0.065 $\pm$ 0.04

## References

  * [1] Mitre Corporation, “The Common Weakness Enumeration (CWE) Initiative,” http://cwe.mitre.org/, 2021, [Online; accessed 2021].
  * [2] Industry Research, “Global static code analysis software market report, history and forecast 2016-2027, breakdown data by companies, key regions, types and application,” https://www.industryresearch.biz/global-static-code-analysis-software-market-18726250, p. 105, 2021, published: 2021-07-12.
  * [3] A. Habib and M. Pradel, “Neural bug finding: A study of opportunities and challenges,” _CoRR_ , vol. abs/1906.00307, 2019. [Online]. Available: http://arxiv.org/abs/1906.00307
  * [4] M. Pradel and K. Sen, “Deepbugs: A learning approach to name-based bug detection,” _Proc. ACM Program. Lang._ , vol. 2, no. OOPSLA, Oct. 2018. [Online]. Available: https://doi.org/10.1145/3276517
  * [5] Y. Li, S. Wang, T. N. Nguyen, and S. Van Nguyen, “Improving bug detection via context-based code representation learning and attention-based neural networks,” _Proc. ACM Program. Lang._ , vol. 3, no. OOPSLA, Oct. 2019.
  * [6] N. Stulova, A. Blasi, A. Gorla, and O. Nierstrasz, “Towards detecting inconsistent comments in java source code automatically,” in _2020 IEEE 20th International Working Conference on Source Code Analysis and Manipulation (SCAM)_ , 2020, pp. 65–69.
  * [7] S. Sachdev, H. Li, S. Luan, S. Kim, K. Sen, and S. Chandra, “Retrieval on source code: A neural code search,” in _Proceedings of the 2nd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages_ , ser. MAPL 2018. New York, NY, USA: Association for Computing Machinery, 2018, p. 31–41. [Online]. Available: https://doi.org/10.1145/3211346.3211353
  * [8] J. Cambronero, H. Li, S. Kim, K. Sen, and S. Chandra, “When deep learning met code search,” 2019.
  * [9] X. Gu, H. Zhang, and S. Kim, “Deep code search,” in _Proceedings of the 40th International Conference on Software Engineering_ , ser. ICSE ’18. New York, NY, USA: Association for Computing Machinery, 2018, p. 933–944. [Online]. Available: https://doi.org/10.1145/3180155.3180167
  * [10] H. Larochelle, D. Erhan, and Y. Bengio, “Zero-data learning of new tasks,” in _Proceedings of the 23rd National Conference on Artificial Intelligence \- Volume 2_ , ser. AAAI’08. Chicago, Illinois: AAAI Press, 2008, p. 646–651.
  * [11] Amazon Web Services, _Amazon CodeGuru: Automate code reviews and optimize application performance with ML-powered recommendations_ , 2021, https://aws.amazon.com/codeguru/.
  * [12] H. Husain, H. Wu, T. Gazit, M. Allamanis, and M. Brockschmidt, “Codesearchnet challenge: Evaluating the state of semantic code search,” _CoRR_ , vol. abs/1909.09436, 2019. [Online]. Available: http://arxiv.org/abs/1909.09436
  * [13] M. Allamanis, M. Brockschmidt, and M. Khademi, “Learning to represent programs with graphs,” in _International Conference on Learning Representations_. Vancouver, BC, Canada: OpenReview.net, 2018. [Online]. Available: https://openreview.net/forum?id=BJOFETxR-
  * [14] P. Rani, M. Birrer, S. Panichella, M. Ghafari, and O. Nierstrasz, “What do developers discuss about code comments?” in _2021 IEEE 21st International Working Conference on Source Code Analysis and Manipulation (SCAM)_ , 2021, pp. 153–164.
  * [15] J. K. Siow, C. Gao, L. Fan, S. Chen, and Y. Liu, “CORE: automating review recommendation for code changes,” _CoRR_ , vol. abs/1912.09652, 2019. [Online]. Available: http://arxiv.org/abs/1912.09652
  * [16] T. H. M. Le, H. Chen, and M. A. Babar, “Deep learning for source code modeling and generation: Models, applications, and challenges,” _ACM Comput. Surv._ , vol. 53, no. 3, Jun. 2020. [Online]. Available: https://doi.org/10.1145/3383458
  * [17] A. Naghshzan, L. Guerrouj, and O. Baysal, “Leveraging unsupervised learning to summarize apis discussed in stack overflow,” in _2021 IEEE 21st International Working Conference on Source Code Analysis and Manipulation (SCAM)_ , 2021, pp. 142–152.
  * [18] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, u. Kaiser, and I. Polosukhin, “Attention is all you need,” in _Proceedings of the 31st International Conference on Neural Information Processing Systems_ , ser. NIPS’17. Red Hook, NY, USA: Curran Associates Inc., 2017, p. 6000–6010.
  * [19] Z. Feng, D. Guo, D. Tang, N. Duan, X. Feng, M. Gong, L. Shou, B. Qin, T. Liu, D. Jiang, and M. Zhou, “CodeBERT: A pre-trained model for programming and natural languages,” in _Findings of the Association for Computational Linguistics: EMNLP 2020_. Online: Association for Computational Linguistics, Nov. 2020, pp. 1536–1547. [Online]. Available: https://www.aclweb.org/anthology/2020.findings-emnlp.139
  * [20] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” in _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_. Minneapolis, Minnesota: Association for Computational Linguistics, Jun. 2019, pp. 4171–4186. [Online]. Available: https://www.aclweb.org/anthology/N19-1423
  * [21] P. Ramachandran, P. Liu, and Q. Le, “Unsupervised pretraining for sequence to sequence learning,” in _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_. Copenhagen, Denmark: Association for Computational Linguistics, Sep. 2017, pp. 383–391. [Online]. Available: https://www.aclweb.org/anthology/D17-1039
  * [22] A. Aghajanyan, A. Gupta, A. Shrivastava, X. Chen, L. Zettlemoyer, and S. Gupta, “Muppet: Massive multi-task representations with pre-finetuning,” 2021.
  * [23] E. Linstead, S. Bajracharya, T. Ngo, P. Rigor, C. Lopes, and P. Baldi, “Sourcerer: Mining and searching internet-scale software repositories,” _Data Min. Knowl. Discov._ , vol. 18, no. 2, p. 300–336, Apr. 2009. [Online]. Available: https://doi.org/10.1007/s10618-008-0118-x
  * [24] F. Lv, H. Zhang, J.-g. Lou, S. Wang, D. Zhang, and J. Zhao, “Codehow: Effective code search based on api understanding and extended boolean model,” in _Proceedings of the 30th IEEE/ACM International Conference on Automated Software Engineering_ , ser. ASE ’15. Lincoln, Nebraska: IEEE Press, 2015, p. 260–270. [Online]. Available: https://doi.org/10.1109/ASE.2015.42
  * [25] C. Mcmillan, D. Poshyvanyk, M. Grechanik, Q. Xie, and C. Fu, “Portfolio: Searching for relevant functions and their usages in millions of lines of code,” _ACM Trans. Softw. Eng. Methodol._ , vol. 22, no. 4, oct 2013. [Online]. Available: https://doi.org/10.1145/2522920.2522930
  * [26] M. Raghothaman, Y. Wei, and Y. Hamadi, “Swim: Synthesizing what i mean - code search and idiomatic snippet synthesis,” in _2016 IEEE/ACM 38th International Conference on Software Engineering (ICSE)_. Austin, TX, USA: IEEE, 2016, pp. 357–367.
  * [27] M. Lu, X. Sun, S. Wang, D. Lo, and Yucong Duan, “Query expansion via wordnet for effective code search,” in _2015 IEEE 22nd International Conference on Software Analysis, Evolution, and Reengineering (SANER)_ , no. 22. Montreal, CA: IEEE, 2015, pp. 545–549.
  * [28] M. Allamanis, D. Tarlow, A. D. Gordon, and Y. Wei, “Bimodal modelling of source code and natural language,” in _Proceedings of the 32nd International Conference on International Conference on Machine Learning - Volume 37_ , ser. ICML’15. Lille, France: JMLR.org, 2015, p. 2123–2132.
  * [29] Y. Wan, J. Shu, Y. Sui, G. Xu, Z. Zhao, J. Wu, and P. S. Yu, “Multi-modal attention network learning for semantic source code retrieval,” in _Proceedings of the 34th IEEE/ACM International Conference on Automated Software Engineering_ , ser. ASE ’19, no. 34. San Diego, California: IEEE Press, 2019, p. 13–25. [Online]. Available: https://doi.org/10.1109/ASE.2019.00012
  * [30] X. Ling, L. Wu, S. Wang, G. Pan, T. Ma, F. Xu, A. X. Liu, C. Wu, and S. Ji, “Deep graph matching and searching for semantic code retrieval,” _CoRR_ , vol. abs/2010.12908, 2020. [Online]. Available: https://arxiv.org/abs/2010.12908
  * [31] J. P. Diniz, D. Cruz, F. Ferreira, C. Tavares, and E. Figueiredo, “Github label embeddings,” in _2020 IEEE 20th International Working Conference on Source Code Analysis and Manipulation (SCAM)_ , 2020, pp. 249–253.
  * [32] J. Cambronero, H. Li, S. Kim, K. Sen, and S. Chandra, “When deep learning met code search,” in _Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering_ , ser. ESEC/FSE 2019. New York, NY, USA: Association for Computing Machinery, 2019, p. 964–974. [Online]. Available: https://doi.org/10.1145/3338906.3340458
  * [33] M. de Rezende Martins and M. A. Gerosa, “Concra: A convolutional neural networks code retrieval approach,” in _Proceedings of the 34th Brazilian Symposium on Software Engineering_ , ser. SBES ’20. New York, NY, USA: Association for Computing Machinery, 2020, p. 526–531. [Online]. Available: https://doi.org/10.1145/3422392.3422462
  * [34] Z. Yao, J. R. Peddamail, and H. Sun, “Coacor: Code annotation for code retrieval with reinforcement learning,” in _The World Wide Web Conference_ , ser. WWW ’19. New York, NY, USA: Association for Computing Machinery, 2019, p. 2203–2214. [Online]. Available: https://doi.org/10.1145/3308558.3313632
  * [35] M. Kaya and H. Bilge, “Deep metric learning: A survey,” _Symmetry_ , vol. 11, p. 1066, 2019.
  * [36] Y. Luan, J. Eisenstein, K. Toutanova, and M. Collins, “Sparse, Dense, and Attentional Representations for Text Retrieval,” _Transactions of the Association for Computational Linguistics_ , vol. 9, pp. 329–345, 04 2021. [Online]. Available: https://doi.org/10.1162/tacl_a_00369
  * [37] D. Tunkelang, “Faceted search,” in _Faceted Search_. NY, USA: Morgan and Claypool, 2009.
  * [38] O. Ben Yitzhak, N. Golbandi, N. Har’El, R. Lempel, A. Neumann, S. Ofek-Koifman, D. Sheinwald, E. Shekita, B. Sznajder, and S. Yogev, “Beyond basic faceted search,” _WSDM’08 - Proceedings of the 2008 International Conference on Web Search and Data Mining_ , vol. 8, pp. 33–44, 01 2008.
  * [39] R. Manmatha, C.-Y. Wu, A. Smola, and P. Krähenbühl, “Sampling matters in deep embedding learning,” _2017 IEEE International Conference on Computer Vision (ICCV)_ , vol. 1, no. 1, pp. 2859–2867, 2017.
  * [40] F. Schroff, D. Kalenichenko, and J. Philbin, “Facenet: A unified embedding for face recognition and clustering,” _CoRR_ , vol. abs/1503.03832, 2015\. [Online]. Available: http://arxiv.org/abs/1503.03832
  * [41] A. Hermans, L. Beyer, and B. Leibe, “In Defense of the Triplet Loss for Person Re-Identification,” _arXiv e-prints_ , vol. 1, no. 1, Mar. 2017.
  * [42] W. Chen, X. Chen, J. Zhang, and K. Huang, “Beyond triplet loss: a deep quadruplet network for person re-identification,” _CoRR_ , vol. abs/1704.01719, 2017. [Online]. Available: http://arxiv.org/abs/1704.01719
  * [43] F. Luo, P. Li, P. Yang, J. Zhou, Y. Tan, B. Chang, Z. Sui, and X. Sun, “Towards fine-grained text sentiment transfer,” in _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_. Florence, Italy: Association for Computational Linguistics, Jul. 2019, pp. 2013–2022. [Online]. Available: https://aclanthology.org/P19-1194
  * [44] J. Li, R. Jia, H. He, and P. Liang, “Delete, retrieve, generate: A simple approach to sentiment and style transfer,” _CoRR_ , vol. abs/1804.06437, 2018\. [Online]. Available: http://arxiv.org/abs/1804.06437
  * [45] C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu, “Exploring the limits of transfer learning with a unified text-to-text transformer,” _Journal of Machine Learning Research_ , vol. 21, no. 140, pp. 1–67, 2020. [Online]. Available: http://jmlr.org/papers/v21/20-074.html
  * [46] A. Veit, S. Belongie, and T. Karaletsos, “Conditional similarity networks,” in _2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , vol. 1. HI, USA: IEEE, 2017, pp. 1781–1789.
  * [47] N. Sawant and S. H. Sengamedu, “Learning-based identification of coding best practices from software documentation,” in _2022 IEEE International Conference on Software Maintenance and Evolution_. Limassol, Cyprus: IEEE Computer Society, oct 2022, pp. –.
  * [48] R. Sennrich, B. Haddow, and A. Birch, “Neural machine translation of rare words with subword units,” in _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. Berlin, Germany: Association for Computational Linguistics, Aug. 2016, pp. 1715–1725. [Online]. Available: https://www.aclweb.org/anthology/P16-1162
  * [49] J. Johnson, M. Douze, and H. Jégou, “Billion-scale similarity search with gpus,” _IEEE Transactions on Big Data_ , vol. -, no. -, pp. 1–1, 2019.