1Aryabhatta Research Institute of Observational Sciences (ARIES), Manora Peak,
Nainital, 263002, India.
2Physical Research Laboratory, Navrangpura, Ahmedabad 380009, India.
3Department of Astronomy and Astrophysics, Tata Institute of Fundamental
Research, Homi Bhabha Road, Mumbai 400005, India
4S.N. Bose National Centre for Basic Sciences, Block JD, Sector III, Salt
Lake, Kolkata 700106, West Bengal, India

# Investigating the morphology and CO gas kinematics of Sh2-112 region

Kshitiz K. Mallick1 Saurabh Sharma1 Lokesh K. Dewangan2 Devendra K. Ojha3
Neelam Panwar1 Tapas Baug4

###### Abstract

We present a study of the molecular cloud in Sh2-112 massive star forming
region using the 3–2 transition of CO isotopologues - CO, 13CO, and C18O;
supplemented in part by CGPS H I line emission and MSX data. Sh2-112 is an
optically visible region powered by an O8V type massive star BD +45 3216, and
hosts two Red MSX Survey sources – G083.7962+03.3058 and G083.7071+03.2817 –
classified as H II region and young stellar object, respectively. Reduced
spectral data products from the James Clerk Maxwell Telescope archive,
centered on the two RMS objects with $\sim$7’$\times$7’ field of view each,
were utilised for the purpose. The 13CO(3-2) channel map of the region shows
the molecular cloud to have filamentary extensions directed away from the
massive star, which also seems to be at the edge of a cavity like structure.
Multiple molecular cloud protrusions into this cavity structure host local
peaks of emission. The integrated emission map of the region constructed from
only those emission clumps detected above 5$\sigma$ level in the position-
position-velocity space affirms the same. MSX sources were found distributed
along the cavity boundary where the gas has the been compressed. Spectral
extraction at these positions yielded high Mach numbers and low ratios of
thermal to non-thermal pressure, suggesting a dominance of supersonic and non-
thermal motion in the cloud.

###### keywords:

Interstellar filaments—H II regions—Millimeter astronomy—Star
formation—Massive stars

<EMAIL_ADDRESS>1 January 20151 January 2015

12.3456/s78910-011-012-3 #### 000 0000 1– 1

##  1 Introduction

Formation and evolution of massive stars is a significant area of research
(Zinnecker & Yorke, 2007), as such stars can affect the evolution of the
Galaxy via their immense matter and radiation output. Though it has been
difficult to establish an evolutionary sequence for massive stars, due to
their faster evolution and relatively rarer occurrences, various high-mass
precursors have been sought to be established, such as massive cold molecular
cores, massive starless clumps, infrared dark clouds (IRDCs) (Motte et al.,
2018), massive young stellar objects (MYSOs) (Hoare et al., 2005), and so on.
The Lyman continuum radiation from high-mass stars ionizes the surrounding
medium, and the subsequent expanding H II regions have been subjects of
investigation for understanding the impact of radiation output in triggering
or quenching further star formation in the natal cloud (Elmegreen, 1998, 2011;
Ogura, 2010).

Sh2-112 is an optically visible H II region (Dickel et al., 1969) (Figure 1)
(_l_ $\sim$ 83.758°, _b_ $\sim$ +03.275°) at a distance of 2 kpc (Panwar et
al., 2020) in the northern Galactic plane. It is associated with the Cygnus
superbubble, and is one of the many Sharpless (Sharpless, 1959) H II regions
in Cygnus (Uyanıker et al., 2001). The region appears nearly circular at
optical wavelengths, and displays a blister morphology in radio emission
(Israel, 1978). Figure 1 displays a prominent dark lane against the optical
emission. The massive star BD +45 3216 – which has been estimated to be of
spectral type O8V (Lahulla, 1985; Panwar et al., 2020) – lies close to this
lane, offcenter from the optical nebulosity. The Red MSX Source (RMS) survey
by Lumsden et al. (2013) – which aims to catalog massive young stellar
population in the Galaxy – has identified two sources associated with the
Sh2-112 region, namely G083.7962+03.3058 and G083.7071+03.2817, classified as
H II region and YSO, respectively, in the catalog. Though the molecular cloud
in this region has been subject to investigations in literature in various
molecular transitions, such as HCN (J=1-0) (Burov et al., 1988), CO (Blitz et
al., 1982), 13CO (J=1-0) (Dobashi et al., 1994, 1996), and CO isotopologues
(Urquhart et al., 2008; Maud et al., 2015b; Panja et al., 2022), they have
either been at a low resolution, and/or as a part of a large statistical study
or a larger region encompassing Sh2-112. As such, a detailed examination of
the molecular cloud associated with Sh2-112 H II region is pending, a void we
try to fill in this paper.

The organisation of this paper is as follows. In section 2, we list the
datasets used and any processing steps. This is followed by section 3, where
we present the analysis results for the molecular gas kinematics. Finally we
discuss our results in section 4, followed by a summary and conclusions
section in section 5.

Figure 1: DSS2-Red image of the Sh2-112 region. Plus symbol marks the massive
star BD +45 3216. Circle and diamond denote the locations of the two RMS
sources, G083.7962+03.3058 and G083.7071+03.2817, respectively. Dashed boxes
show the field of view of the two JCMT fields.

##  2 Data Used

###  2.1 Archival Spectral Data Products

We obtained the archival reduced and calibrated JCMT (_James Clerk Maxwell
Telescope_) spectral cubes from the Canadian Astronomy Data
Centre111https://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/en/ (CADC), for CO(3-2)
(rest frequency = 345.79599 GHz), 13CO(3-2) (rest frequency = 330.587960
GHz),and C18O(3-2) (rest frequency = 329.330545 GHz). The _J=3–2_ transition
traces gas at a critical density of $\sim$ 104-5 cm-3 (Buckle et al., 2010).
In the direction of the Sh2-112 region, two fields have been observed by the
JCMT, namely G083.7962+03.3058 and G083.7071+03.2817 (see Figure 1), using the
HARP/ACSIS (Heterodyne Array Receiver Programme/Auto-Correlation Spectral
Imaging System; Buckle et al., 2009) spectral imaging system. Cubes for both
of these fields were retrieved. The temperature scale used for the pixel
brightness units is T${}_{A}^{*}$ (antenna temperature). A basic processing
using the starlink kappa (Currie et al., 2014) package was carried out,
wherein the downloaded reduced cubes for the two fields were mosaiced; the
spectral axis was converted to LSRK velocity scale; and the coordinate system
was transformed to Galactic from FK5 for ease of analysis. Thereafter the
cubes were rebinned along spectral axis to 0.5 km s-1 channel width. Table 1
provides the details of the JCMT data used. While the rebinned cubes were used
for the detection of spatial structures (sections 3.1 and 3.2) as they have
lower RMS noise; for the calculation of physical parameters (section 3.3) the
native channel width (13CO and/or C18O) cubes were used as high velocity
resolution is required for the same.

For the Sh2-112 region, we also procured archival spectral cube for 21 cm H I
line emission produced by the Canadian Galactic Plane Survey (CGPS) Consortium
(Taylor et al., 2003). We use the cube on an as is basis for examining the
morphology of the region. The 21 cm cube has an angular resolution of
$\sim$1’$\times$1’ cosec$\delta$, channel width of $\sim$ 0.8 km s-1, and a
velocity resolution of $\sim$ 1.3 km s-1.

Table 1: JCMT Data Used Line/Wavelength | Spatial | Native Channel | Rebinned Channel | Program
---|---|---|---|---
| Resolution | Width | Width (& Noise) | IDs Used
CO(3-2) |  $\sim$14” |  $\sim$0.42 km s-1 | 0.5 km s-1 | M08AU19
|  |  | ($\sim$0.37 K) |
13CO(3-2) |  $\sim$14” |  $\sim$0.05 km s-1 | 0.5 km s-1 | M08AU19,
|  |  | ($\sim$0.35 K) | M08BU18
C18O(3-2) |  $\sim$14” |  $\sim$0.05 km s-1 | 0.5 km s-1 | M08AU19,
|  |  | ($\sim$0.45 K) | M08BU18

###  2.2 Archival Imaging Data Products

We obtained the stellar sources present in MSX (Midcourse Space Experiment)
Catalog 6 (Egan et al., 2003) for our JCMT field of view, via the NASA/IPAC
Infrared Science Archive222https://irsa.ipac.caltech.edu/. Though observations
by the MSX satellite were carried out in 6 bands ranging from 4-21 $\mu$m, the
most sensitive observation has been in the mid-infrared (MIR) A band (8.28
$\mu$m) (Lumsden et al., 2002). The other three MIR bands used by MSX are C
(12.13 $\mu$m), D (14.65 $\mu$m), and E (21.3 $\mu$m). MSX has surveyed the
Galactic plane at MIR bands with a spatial resolution of $\sim$ 18.3” (Price
et al., 2001), and has helped in uncovering its MIR population, especially the
deeply embedded stellar objects (Lumsden et al., 2002).

Besides the above, H$\alpha$ image from the
IPHAS333https://www.iphas.org/images/ (Drew et al., 2005; Barentsen et al.,
2014) survey (Obs date:2003-08-14); the DSS2-Red
image444https://archive.eso.org/dss/dss; and the NVSS (NRAO VLA Sky Survey;
Condon et al., 1998) 1.4 GHz continuum image was downloaded for a visual
examination of the region.

##  3 Results

###  3.1 Channel Maps

Figure 2 shows the channel map for the Sh2-112 region in 13CO(3-2) emission.
The massive star BD +45 3216, and the two RMS sources (G083.7962+03.3058 and
G083.7071+03.2817) have been marked on the image. The filamentary nature of
the molecular emission is clearly brought out in the channel maps. While in
the eastern part, different filamentary structures seem to join at one end at
the RMS source G083.7962+03.3058 (circle); in the western portion such
structures seem to converge on G083.7071+03.2817 (diamond symbol). In the
velocity range -4.0 to -2.0 km s-1, the two parts seem to be connected by
molecular filaments. Another noticeable feature is that in the eastern part,
the molecular emission forms an arc like structure directed away from the
massive star BD +45 3216 (plus symbol), best delineated in the [-2.0,-1.0] km
s-1 panel. While in the velocity range [-2.0,0.0] km s-1, this arc appears to
be directed towards north; at the blueshifted velocities, for example
[-7.0,-5.0] km s-1 range, it appears curved towards the south. In the middle
velocity ranges, [-5.0,-2.0] km s-1, no proper pattern can be made out. It is
possible that two separate filaments, seen at blueshifted and redshifted
velocities, respectively, are merging at this juncture. Towards the north-west
of the massive star BD +45 3216, there appears to be a cavity where hardly any
molecular emission is seen. Furthermore, the western emission centered on the
G083.7071+03.2817 source is also directed away from the massive star.

Figure 2: Channel maps of 13CO(3-2) emission. Plus symbol marks the massive
star BD +45 3216. Circle and diamond denote the locations of the two RMS
sources, G083.7962+03.3058 and G083.7071+03.2817, respectively. An arc like
filamentary structure, a cavity like structure, and a connecting filament have
been indicated by arrows.

###  3.2 Moment Maps

In this section, we examine the m-0 (moment-0 or Integrated Intensity), m-1
(moment-1 or Intensity-weighted velocity), and linewidth (Intensity-weighted
dispersion) maps of the molecular emission in the Sh2-112 region. The
examination is confined to only those regions which are detected at $\geq$
5$\sigma$ detection level in the spectral cubes ($\sigma$ being the rms noise
level of the respective cubes). To achieve this, the clumpfind algorithm of
Williams et al. (1994) was utilised via starlink software’s (Currie et al.,
2014) cupid package (Berry et al., 2007). The 5$\sigma$ threshold was
conservatively chosen to eliminate false detections of clumps. As recommended
by Williams et al. (1994), the gap between contour levels was kept at
2$\sigma$. With these values, the clumpfind algorithm was implemented on the
mosaiced CO(3-2), 13CO(3-2), and C18O(3-2) position-position-velocity cubes.
After the detection of clumps, we mask those regions where no emission was
detected at 5$\sigma$ or above to create a masked cube. Thereafter, the
respective (masked) cubes were collapsed to create m-0, m-1, and linewidth
images.

The maps are shown in Figure 3 for CO(3-2), 13CO(3-2), and C18O(3-2) molecular
transitions. The massive star BD +45 3216, and the two RMS sources
(G083.7962+03.3058 and G083.7071+03.2817) have been marked on each of the
images. In the m-0 maps, CO(3-2) emission appears to be diffused with no
particular discernible structure, which is expected given the ubiquitous
nature of CO molecule and comparatively lower critical density than the other
two transitions. However, it is still worth noting that much of the CO(3-2)
emission still “faces away” from the massive star BD +45 3216. The 13CO(3-2)
m-0 map most clearly traces the molecular filamentary structures which were
examined in the channel maps (Figure 2). While the cavity to the west of the
massive star shows no 13CO(3-2) emission, the connecting filamentary structure
between the eastern and western regions is prominent. Given that all emission
here is at $\geq$ 5$\sigma$ level, this indicates that there is significant
molecular mass concentration in the connecting filament and it is not some
minor emission structure. The critical density is highest for C18O(3-2)
transition, and thus only the densest clumps are detected here.
Unsurprisingly, the bulk of C18O(3-2) emission is associated with the
positions of RMS sources. The prominent arc like filamentary structure marked
on the 13CO(3-2) channel map (Figure 2) is also partly seen in C18O(3-2) m-0
map, indicating the presence of dense gas along it. The linewidth maps for all
three transitions show a maxima towards the positions of the RMS sources as
compared to the outer parts, which could be due to outflow activity and/or
possibly suggest a convergence of different flows towards these sources. In
the 13CO(3-2) linewidth map, the connecting filament has a relatively lower
velocity dispersion, while the arc like filamentary structure (Figure 2)
displays a relatively larger dispersion along its length as compared to
others.

Figure 3: Row-wise : moment-0 (Integrated intensity), moment-1 (Intensity-
weighted velocity), and linewidth (Intensity-weighted dispersion) collapsed
images for three cubes – CO(3-2), 13CO(3-2), C18O(3-2) in first, second, and
third columns, respectively. The symbols are same as Figure 2.

###  3.3 Physical Parameters

Figure 4 shows the 13CO(3-2) m-0 map of the region overlaid with the massive
star BD +45 3216; the two RMS sources (G083.7962+03.3058 and
G083.7071+03.2817); and the MSX sources (m1 to m11) which lay in the regions
of emission $\geq$ 5$\sigma$ level (see section 3.2). The local peaks of m-0
emission, based on visual examination of the contours have also been marked on
the image (p1 to p10). To obtain the physical parameters for these sources
(except BD +45 3216 which did not have any molecular emission associated with
it), we extracted the spectra at their locations (from a 2$\times$2 pixel
area). As high-velocity resolution is required for the calculations, the non-
averaged cubes with the native channel width of $\sim$0.05 km s-1 were
utilised for spectrum extraction.

For the RMS sources, both 13CO(3-2) and C18O(3-2) spectra were available, and
are shown in Figure 5. However, for the MSX sources (m1-m11) and the local
13CO(3-2) integrated emission peaks (p1-p10), only 13CO(3-2) spectra could be
extracted, shown in Figures 6 and 7, respectively. The results from the
gaussian fitting are given in Table 2. The first thing to notice is that at
some of the locations, a simple gaussian is a poor fit to the spectra. For
example, in Figure 5, the spectra have broad wings on both sides, which is
usually an indication of some kind of likely outflow associated with the
sources. According to Maud et al. (2015b), while G083.7071+03.2817 is
associated with an outflow, G083.7962+03.3058 has a possibility of the same.
For some of the locations in Figures 6 and 7, there appears to be excess
emission on the red side. It had been discussed in section 3.1 that the arc
like structure in Figure 2 probably has a merger of blueshifted and redshifted
filaments. The excess redshifted emission seen at the locations which lie on
this arc, i.e. m1 (Figure 6), and p1, p2 (Figure 7) could be an indication of
the same. Emission peak positions p8 and p9 display significant self-
absorption in their spectra. The mean velocity for most of the fits lies in
the range -2.5 to -4.0 km s-1. It is notable that the locations towards the
eastern side (namely p1, p2, p3, and m1) have significantly more blueshifted
mean velocities as opposed to other locations. The RMS sources, though with
wide wings, have nearly same mean velocities ($\sim$ -4.0 km s-1), which is in
agreement with the radial velocity for the Sh2-112 region in Blitz et al.
(1982); Stark & Brand (1989); Brand & Blitz (1993). Though our locations
display a range of mean velocities, the overall distribution is consistent
with other values cited in literature within error limits (Dobashi et al.,
1994; Urquhart et al., 2008; Lumsden et al., 2013; Maud et al., 2015a, b;
Panja et al., 2022).

Using the FWHM (full width at half maxima) returned by the fit, we calculate
the standard deviation (or observed velocity dispersion), the non-thermal
velocity dispersion, and the total velocity dispersion using the following set
of equations (Fuller & Myers, 1992; Fiege & Pudritz, 2000) :

$\displaystyle\Delta V_{tot}^{2}$ $\displaystyle=$ $\displaystyle\Delta
V_{obs}^{2}+8~{}~{}ln\,2~{}~{}kT\left(\frac{1}{\bar{m}}-\frac{1}{m_{obs}}\right)$
(1) $\displaystyle\Rightarrow\frac{\Delta V_{tot}^{2}}{8~{}~{}ln\,2}$
$\displaystyle=$ $\displaystyle\frac{kT}{\bar{m}}+\left(\frac{\Delta
V_{obs}^{2}}{8~{}~{}ln\,2}-\frac{kT}{m_{obs}}\right)$
$\displaystyle\Rightarrow\sigma_{tot}^{2}$ $\displaystyle=$ $\displaystyle
c_{s}^{2}+\left(\sigma_{obs}^{2}-\sigma_{\textsc{t}}^{2}\right)$ (2)
$\displaystyle=$ $\displaystyle c_{s}^{2}+\sigma_{\textsc{nt}}^{2}~{}~{}.$ (3)

In the above equations, $\Delta V_{obs}$ is the FWHM of the fit;
$\sigma_{obs}$ ($=\Delta V_{obs}/\sqrt{8ln\,2}~{}$ for gaussian fits) is the
standard deviation (or dispersion); $\sigma_{\textsc{t}}(=\sqrt{kT/m_{obs}})$
is the thermal velocity dispersion; $m_{obs}$ is the mass of the relevant
molecule (29 amu and 30 amu for 13CO and C18O respectively);
$\sigma_{\textsc{nt}}$ is the non-thermal velocity dispersion;
$c_{s}$($=\sqrt{kT/\bar{m}}$) is the speed of sound; $\bar{m}$ is the average
molecular weight of the medium (2.37 amu); and T is the excitation or gas
kinetic temperature.

The FWHM shows a wide range at these locations, ranging from $\sim$ 1.5-3.0 km
s-1 for most of the sources, with some of the highest values associated with
locations in the vicinity of the two RMS sources, such as m2, m3, m4, m11, and
p10. This tallies with the linewidth map in Figure 3(h). For the above
calculations, we take the excitation temperature T as 20 K (see Figure 15 in
Panja et al., 2022). Furthermore, we also calculate Mach number
($=\sigma_{\textsc{nt}}/c_{s}$) and the ratio of thermal to non-thermal
pressure ($P_{\textsc{tnt}}=c_{s}^{2}/\sigma_{\textsc{nt}}^{2}$) (Lada et al.,
2003) for each of the locations. The results of these two calculations show
the presence of supersonic motion, and a dominance of non-thermal pressure in
the cloud. The mach number and P${}_{\textsc{tnt}}$ values are inversely
correlated, as expected from their dependence on $\sigma_{\textsc{nt}}$. Such
values would suggest that the emission mechanism is likely some supersonic
non-thermal phenomena, and could be via turbulence and magnetic fields (Myers
& Goodman, 1988; Crutcher, 1999), to suggest one such mechanism.

Figure 4:  13CO(3-2) integrated intensity map (contours at 2, 3.5, 6, 8, 10, 12, 14, 17, 21, 25, 30, 34, 40, and 45 K km s-1). Plus symbol marks the massive star BD +45 3216. The two RMS sources, G083.7962+03.3058 and G083.7071+03.2817, have been shown by circle and diamond symbols, respectively. The MSX sources have been marked in green boxes and labelled m1 to m11. Cyan boxes (labelled p1 to p10) are the locations of local contour peaks where also we extracted the spectra.  Figure 5:  13CO(3-2) (blue) and C18O(3-2) (red) spectra at positions of RMS sources marked in Figure 4. Green and yellow curves show the gaussian fits to the respective spectra, with blue dashed line marking the velocity of the peak of the 13CO(3-2) gaussian fit.  Figure 6:  13CO(3-2) spectra at MSX source positions marked in Figure 4 (green boxes labelled m1-m11). Green curve depicts the gaussian fit to the spectra, with blue dashed line marking the velocity of the peak of the gaussian fit.  Figure 7:  13CO(3-2) spectra at peak positions marked in Figure 4 (cyan boxes labelled p1-p10). Green curve depicts the gaussian fit to the spectra, with blue dashed line marking the velocity of the peak of the gaussian fit.  Table 2: Parameters derived from 13CO(3-2) spectra at the locations marked in Figure 4. For G083.7071+03.2817 and G083.7962+03.3058, C18O(3-2) spectrum was also available and thus used for calculation as well. clump | Mean | FWHM | Amplitude | $\sigma_{NT}$ | Mach | $P_{TNT}$ | _l_ | _b_
---|---|---|---|---|---|---|---|---
| ($km~{}s^{-1}$) | ($km~{}s^{-1}$) | (K) | ($km~{}s^{-1}$) | Number |  | (deg) | (deg)
G083.7071+03.2817
C18O | -3.88 | 1.79 | 3.18 | 0.76 | 2.86 | 0.12 | 83.7071 | 03.2817
13CO | -4.07 | 2.48 | 11.00 | 1.05 | 3.97 | 0.06 | - | -
G083.7962+03.3058
C18O | -4.11 | 2.69 | 2.62 | 1.14 | 4.30 | 0.05 | 83.7962 | 3.3058
13CO | -4.00 | 3.58 | 10.99 | 1.52 | 5.73 | 0.03 | - | -
MSX Sources
m1 | -4.41 | 1.92 | 4.78 | 0.81 | 3.06 | 0.11 | 83.8330 | 3.2881
m2 | -3.53 | 2.24 | 6.48 | 0.95 | 3.59 | 0.08 | 83.7978 | 3.3207
m3 | -1.91 | 2.81 | 4.52 | 1.19 | 4.49 | 0.05 | 83.7835 | 3.3079
m4 | -2.99 | 3.09 | 1.75 | 1.31 | 4.95 | 0.04 | 83.7803 | 3.3174
m5 | -3.35 | 1.47 | 6.74 | 0.62 | 2.34 | 0.18 | 83.7721 | 3.3279
m6 | -2.54 | 2.09 | 3.75 | 0.89 | 3.34 | 0.09 | 83.7607 | 3.3313
m7 | -3.62 | 1.25 | 4.38 | 0.53 | 1.99 | 0.25 | 83.7502 | 3.3258
m8 | -3.56 | 1.72 | 3.68 | 0.73 | 2.74 | 0.13 | 83.7359 | 3.3156
m9 | -3.84 | 1.97 | 4.88 | 0.83 | 3.15 | 0.10 | 83.7310 | 3.3121
m10 | -2.54 | 1.89 | 4.20 | 0.80 | 3.02 | 0.11 | 83.7112 | 3.3107
m11 | -2.74 | 2.87 | 4.45 | 1.22 | 4.60 | 0.05 | 83.7237 | 3.2822
m-0 contour peaks
p1 | -4.32 | 1.11 | 5.14 | 0.47 | 1.76 | 0.32 | 83.8544 | 3.2968
p2 | -4.87 | 2.28 | 6.74 | 0.96 | 3.64 | 0.08 | 83.8336 | 3.2942
p3 | -4.77 | 2.27 | 5.49 | 0.96 | 3.62 | 0.08 | 83.8170 | 3.3171
p4 | -3.22 | 1.97 | 10.01 | 0.83 | 3.15 | 0.10 | 83.7776 | 3.3288
p5 | -2.39 | 1.67 | 3.89 | 0.70 | 2.66 | 0.14 | 83.7634 | 3.3152
p6 | -3.63 | 1.91 | 6.85 | 0.81 | 3.05 | 0.11 | 83.7530 | 3.3291
p7 | -3.34 | 1.97 | 5.46 | 0.83 | 3.14 | 0.10 | 83.7405 | 3.3299
p8 | -3.87 | 2.66 | 4.38 | 1.12 | 4.25 | 0.05 | 83.7053 | 3.3157
p9 | -3.11 | 3.01 | 8.72 | 1.28 | 4.82 | 0.04 | 83.7194 | 3.3005
p10 | -2.80 | 3.30 | 10.26 | 1.40 | 5.29 | 0.04 | 83.7219 | 3.2869

##  4 Discussion

Figure 8:  Velocity channel maps for the CGPS H I 21 cm emission. Green
contour marks the 2 K km s-1 level of the 13CO(3-2) m-0 (Figure 4) image. In
-2.3 and -3.1 km s-1 channels, the red contours have been drawn at (outer to
inner) 65, 60, 55, and 50 K; while in -4.8 and -5.6 km s-1 channels, the red
contours are at 81 (outer) and 65 (inner) K. The dashed green circle in -3.1
km s-1 channel shows the location where spectrum was extracted (see Figure 9).
The rest of the symbols are same as Figure 4.  Figure 9:  13CO(3-2) spectrum
(in red) and CGPS H i spectrum (in blue) at the location marked in Figure 8
(green circle in -3.1 km s-1 channel) – demonstrating the H i self-absorption
feature centered at $\sim$ [-5,-4] km s-1 and its anti-correlation with the
molecular emission. The 13CO(3-2) spectrum has been scaled up by a factor of
30$\times$ for better visibility.  Figure 10:  _(a)_ 13CO(3-2) m-0 map. Cyan
contours are NVSS levels at 0.01, 0.02, 0.03, 0.04, 0.05, 0.06, 0.07, 0.08,
0.09, and 0.1 Jy/beam. _(b)_ H$\alpha$ image of the region. _(c)_ Continuum-
subtracted H2 emission (2.12 $\mu$m) map from Panwar et al. (2020). Green
contour marks the 2 K km s-1 level of the 13CO(3-2) m-0 (Figure 4) image. The
rest of the symbols are same as Figure 4.  Figure 11: Position-velocity
diagrams for the line segments marked L1-L8 (in red) on the central image
(13CO(3-2) m-0 from Figure 4). Green contour marks the 2 K km s-1 level of the
13CO(3-2) m-0 (Figure 4) image. The rest of the symbols are same as Figure 4.
On the L1-L8 p-v maps, the contour levels are at 1, 1.5, 2, 3, 4, 5, 6, 7, 8,
9, and 10 K.

Figure 8 shows the CGPS 21 cm velocity channel maps of this region. The
channel maps show a depression in H i emission (see Figure 9), which is
(anti-)correlated with strong (i.e. $\geq$ 5$\sigma$) molecular emission. Such
features have been referred to as H i self absorption (or HISA) features in
literature, and have been found to be an extensive presence in 21 cm surveys
(Kerton, 2005; Wang et al., 2020). HISA regions indicate the presence of cold
H I gas in foreground against warm H I emission from background. In our
channel maps, most intense depression in emission seems to be confined to the
north-eastern quadrant, in the vicinity of the source G083.7962+03.3058
(circle). Channels -2.3 km s-1 and -3.1 km s-1 display shell-like feature on a
smaller spatial scale, while the channels -4.8 km s-1 and -5.6 km s-1 show
large scale shell-like feature. The anti-correlation of molecular emission and
depression in H I emission can be most prominently seen in the -4.8 km s-1 and
-5.6 km s-1 channels. Such shell-like features have been found in other
regions as well, such as Sh2-237 (Dewangan et al., 2017), W4 and W5 (Hosokawa
& Inutsuka, 2007), and in Southern Galactic Plane Survey (SGPS) regions
(McClure-Griffiths et al., 2001).

Figure 10 shows the 13CO(3-2) m-0 image (overlaid with NVSS contours), the
H$\alpha$ image of the region, and the continuum-subtracted H2 emission map of
the region from Panwar et al. (2020). The radio emission seems to have
multiple peaks, and on a comparison of Figures 4 and 10(a), the radio peaks
appear to be associated with the locations of m3/m4, p5, and a location to the
east of p10. According to Panwar et al. (2020), the location to the east of
p10 is the ionized boundary layer. Based on the presence of this ionized
boundary layer and pressure calculations, they further conjecture the
possibility of triggered star formation towards the location of RMS source
G083.7071+03.2817 due to O8V type star BD +45 3216 (also see Morgan et al.,
2004, 2009; Urquhart et al., 2009). It is possible that the expansion of the
ionized gas could have had a role in the formation of the cavity towards the
west/northwest of the massive star BD +45 3216. There is significant H$\alpha$
emission in the cavity, while the H2 2.12$\mu$m emission seems to trace the
southern boundary of the molecular cloud. All along the cavity perimeter,
there are finger-like filamentary structures protruding into it, reminiscent
of bright-rimmed clouds which are seen in optical and infrared emission
(Chauhan et al., 2011; Sharma et al., 2016). Here these structures are seen in
absorption in optical (Figure 10(b)) and in emission in 13CO(3-2) integrated
intensity map (Figure 10(a)). Two out of three of these finger-like structures
seem to be hosting molecular peaks at their ends, i.e. p5 and p10. If there is
a possible case of triggering towards the source G083.7071+03.2817, then based
on size considerations of the cavity, there could be possible triggering all
along the cavity perimeter. The MSX sources m3, m4, m5, m7, m8, m9, and m11
seem to be distributed along the cavity boundary where there is seemingly
compression of molecular material, as evidenced by the sharp change in
13CO(3-2) contour levels (see Figure 4).

Lastly we present the position-velocity maps in Figure 11 along the different
line segments marked L1-L8 on the central image. While the lines L1-L5 trace
the filamentary structures which converge on the source G083.7962+03.3058
(circle), L6-L8 are cuts along the filamentary structures associated with
G083.7071+03.2817 (diamond symbol). The position-velocity maps show a complex
structure with clumping along their length. Significant velocity gradients can
also be seen in most of them, and especially in L1-L5 in the vicinity of the
source G083.7962+03.3058 (i.e. bright clump towards the right in L1-L5). Here
we note that such a gradient could indicate gas being channeled towards the
source, and such a flow of gas along the filamentary structures towards the
RMS sources – G083.7962+03.3058 and G083.7071+03.2817 – as also discussed in
section 3.2, is similar to the longitudinal flow in filaments towards hubs in
hub-filament systems (Dewangan et al., 2017, 2020; Williams et al., 2018).
Therefore, this region could present a good example of exploring various star
formation frameworks such as global hierarchical collapse (GHC Vázquez-
Semadeni et al., 2019), conveyor belt model (Longmore et al., 2014; Krumholz &
McKee, 2020), filaments to clusters model (Kumar et al., 2020), to name a few.

According to Panja et al. (2022), the column density of the molecular emission
region has been calculated to be $\sim$1022 cm-2. Such regions have been
designated as “hubs” in the context of hub-filament systems in literature
(Myers, 2009). The column density maps by Panja et al. (2022) also suggest
such a hub-filament configuration for the larger region of the order of a few
10s of parsec encompassing Sh2-112. Thus, what we have explored here seems to
be the detailed structure of a hub region where massive star formation is
going on. However, given the importance of filamentary structures in star
formation and expositions of various filament types in literature (see Hacar
et al., 2022, for a detailed review), it is essential to further study such
regions via high spectral and spatial resolution molecular transitions of
other species, magnetic fields, and so on so as to understand the evolution of
molecular clouds as they form stars.

##  5 Summary and Conclusions

We have carried out an analysis of the Sh2-112 region in CO(3-2), 13CO(3-2),
and C18O(3-2) molecular line transitions from the JCMT, supported by archival
data from CGPS H I line, MSX, NVSS, and IPHAS H$\alpha$ for visual
examination. Our main conclusions are as follows :

  1. 1.

The molecular emission appears filamentary in channel maps and seems to be
directed away from the massive star BD +45 3216, which is also located at the
edge of a cavity-like structure.

  2. 2.

Multiple local peaks were found associated with the molecular emission in the
13CO(3-2) integrated intensity emission map (m-0) which was generated using
clumps detected above 5$\sigma$ level in the position-position-velocity space.
The linewidth map shows high dispersion associated with the positions of the
RMS sources (G083.7962+03.3058 and G083.7071+03.2817).

  3. 3.

Analysis of CGPS 21 cm H i line emission reveals the presence of shell-like
HISA feature, where the molecular emission is nearly coincident with the
depression in H i emission.

  4. 4.

13CO(3-2) spectra was extracted at the locations of RMS sources, MSX sources,
and the local peaks of emission. For the RMS sources, C18O(3-2) spectra was
also extracted. Spectral profile fitting suggests significant deviation from a
gaussian profile for many sources. All the locations were found to have
significant non-thermal dispersions; large mach numbers ($\sim$ 2–6)
indicating dominance of supersonic motions within the clumps; and a small
thermal to non-thermal pressure ratio ($\sim$ 0.03–0.3).

## Acknowledgements

We thank the anonymous referee for a critical reading of the manuscript and
for the suggestions for the improvement of this paper. DKO acknowledges the
support of the Department of Atomic Energy, Government of India, under project
Identification No. RTI 4002. The James Clerk Maxwell Telescope has
historically been operated by the Joint Astronomy Centre on behalf of the
Science and Technology Facilities Council of the United Kingdom, the National
Research Council of Canada and the Netherlands Organisation for Scientific
Research. This research has made use of the NASA/IPAC Infrared Science
Archive, which is funded by the National Aeronautics and Space Administration
and operated by the California Institute of Technology. This research made use
of data products from the Midcourse Space Experiment. Processing of the data
was funded by the Ballistic Missile Defense Organization with additional
support from NASA Office of Space Science. This research has also made use of
the NASA/ IPAC Infrared Science Archive, which is operated by the Jet
Propulsion Laboratory, California Institute of Technology, under contract with
the National Aeronautics and Space Administration. This research has made use
of the services of the ESO Science Archive Facility.

* Barentsen et al. (2014) Barentsen, G., Farnhill, H. J., Drew, J. E., et al. 2014, MNRAS, 444, 3230. doi:10.1093/mnras/stu1651
* Berry et al. (2007) Berry, D. S., Reinhold, K., Jenness, T., et al. 2007, Astronomical Data Analysis Software and Systems XVI, 376, 425
* Blitz et al. (1982) Blitz, L., Fich, M., & Stark, A. A. 1982, ApJS, 49, 183. doi:10.1086/190795
* Brand & Blitz (1993) Brand, J. & Blitz, L. 1993, A&A, 275, 67
* Buckle et al. (2009) Buckle, J. V., Hills, R. E., Smith, H., et al. 2009, MNRAS, 399, 1026. doi:10.1111/j.1365-2966.2009.15347.x
* Buckle et al. (2010) Buckle, J. V., Curtis, E. I., Roberts, J. F., et al. 2010, MNRAS, 401, 204. doi:10.1111/j.1365-2966.2009.15619.x
* Burov et al. (1988) Burov, A. B., Vdovin, F. V., Zinchenko, I. I., et al. 1988, Pisma v Astronomicheskii Zhurnal, 14, 492
* Chauhan et al. (2011) Chauhan, N., Pandey, A. K., Ogura, K., et al. 2011, MNRAS, 415, 1202. doi:10.1111/j.1365-2966.2011.18742.x
* Condon et al. (1998) Condon, J. J., Cotton, W. D., Greisen, E. W., et al. 1998, AJ, 115, 1693. doi:10.1086/300337
* Crutcher (1999) Crutcher, R. M. 1999, ApJ, 520, 706. doi:10.1086/307483
* Currie et al. (2014) Currie, M. J., Berry, D. S., Jenness, T., et al. 2014, Astronomical Data Analysis Software and Systems XXIII, 485, 391
* Dewangan et al. (2017) Dewangan, L. K., Ojha, D. K., Zinchenko, I., et al. 2017, ApJ, 834, 22. doi:10.3847/1538-4357/834/1/22
* Dewangan et al. (2020) Dewangan, L. K., Ojha, D. K., Sharma, S., et al. 2020, ApJ, 903, 13. doi:10.3847/1538-4357/abb827
* Dickel et al. (1969) Dickel, H. R., Wendker, H., & Bieritz, J. H. 1969, A&A, 1, 270
* Dobashi et al. (1994) Dobashi, K., Bernard, J.-P., Yonekura, Y., et al. 1994, ApJS, 95, 419. doi:10.1086/192106
* Dobashi et al. (1996) Dobashi, K., Bernard, J.-P., & Fukui, Y. 1996, ApJ, 466, 282. doi:10.1086/177509
* Drew et al. (2005) Drew, J. E., Greimel, R., Irwin, M. J., et al. 2005, MNRAS, 362, 753. doi:10.1111/j.1365-2966.2005.09330.x
* Egan et al. (2003) Egan, M. P., Price, S. D., & Kraemer, K. E. 2003, AAS
* Elmegreen (1998) Elmegreen, B. G. 1998, Origins, 148, 150
* Elmegreen (2011) Elmegreen, B. G. 2011, EAS Publications Series, 51, 45. doi:10.1051/eas/1151004
* Fiege & Pudritz (2000) Fiege, J. D. & Pudritz, R. E. 2000, MNRAS, 311, 85. doi:10.1046/j.1365-8711.2000.03066.x
* Fuller & Myers (1992) Fuller, G. A. & Myers, P. C. 1992, ApJ, 384, 523. doi:10.1086/170894
* Hacar et al. (2022) Hacar, A., Clark, S., Heitsch, F., et al. 2022, arXiv:2203.09562
* Hoare et al. (2005) Hoare, M. G., Lumsden, S. L., Oudmaijer, R. D., et al. 2005, Massive Star Birth: A Crossroads of Astrophysics, 227, 370. doi:10.1017/S174392130500476X
* Hosokawa & Inutsuka (2007) Hosokawa, T. & Inutsuka, S.-. ichiro . 2007, ApJ, 664, 363. doi:10.1086/518396
* Israel (1978) Israel, F. P. 1978, A&A, 70, 769
* Kerton (2005) Kerton, C. R. 2005, ApJ, 623, 235. doi:10.1086/428490
* Kumar et al. (2020) Kumar, M. S. N., Palmeirim, P., Arzoumanian, D., et al. 2020, A&A, 642, A87. doi:10.1051/0004-6361/202038232
* Krumholz & McKee (2020) Krumholz, M. R. & McKee, C. F. 2020, MNRAS, 494, 624. doi:10.1093/mnras/staa659
* Lada et al. (2003) Lada, C. J., Bergin, E. A., Alves, J. F., et al. 2003, ApJ, 586, 286. doi:10.1086/367610
* Lahulla (1985) Lahulla, J. F. 1985, A&AS, 61, 537
* Longmore et al. (2014) Longmore, S. N., Kruijssen, J. M. D., Bastian, N., et al. 2014, Protostars and Planets VI, 291. doi:10.2458/azu_uapress_9780816531240-ch013
* Lumsden et al. (2002) Lumsden, S. L., Hoare, M. G., Oudmaijer, R. D., et al. 2002, MNRAS, 336, 621. doi:10.1046/j.1365-8711.2002.05785.x
* Lumsden et al. (2013) Lumsden, S. L., Hoare, M. G., Urquhart, J. S., et al. 2013, ApJS, 208, 11. doi:10.1088/0067-0049/208/1/11
* Maud et al. (2015a) Maud, L. T., Lumsden, S. L., Moore, T. J. T., et al. 2015, MNRAS, 452, 637. doi:10.1093/mnras/stv1334
* Maud et al. (2015b) Maud, L. T., Moore, T. J. T., Lumsden, S. L., et al. 2015, MNRAS, 453, 645. doi:10.1093/mnras/stv1635
* McClure-Griffiths et al. (2001) McClure-Griffiths, N. M., Dickey, J. M., Gaensler, B. M., et al. 2001, PASA, 18, 84. doi:10.1071/AS01010
* Morgan et al. (2004) Morgan, L. K., Thompson, M. A., Urquhart, J. S., et al. 2004, A&A, 426, 535. doi:10.1051/0004-6361:20040226
* Morgan et al. (2009) Morgan, L. K., Urquhart, J. S., & Thompson, M. A. 2009, MNRAS, 400, 1726. doi:10.1111/j.1365-2966.2009.15585.x
* Motte et al. (2018) Motte, F., Bontemps, S., & Louvet, F. 2018, ARA&A, 56, 41. doi:10.1146/annurev-astro-091916-055235
* Myers & Goodman (1988) Myers, P. C. & Goodman, A. A. 1988, ApJ, 329, 392. doi:10.1086/166385
* Myers (2009) Myers, P. C. 2009, ApJ, 700, 1609. doi:10.1088/0004-637X/700/2/1609
* Ogura (2010) Ogura, K. 2010, Astronomical Society of India Conference Series, 1, 19
* Panja et al. (2022) Panja, A., Sun, Y., Chen, W. P., et al. 2022, ApJ, 939, 46. doi:10.3847/1538-4357/ac940f
* Panwar et al. (2020) Panwar, N., Sharma, S., Ojha, D. K., et al. 2020, ApJ, 905, 61. doi:10.3847/1538-4357/abc42e
* Price et al. (2001) Price, S. D., Egan, M. P., Carey, S. J., et al. 2001, AJ, 121, 2819. doi:10.1086/320404
* Sharma et al. (2016) Sharma, S., Pandey, A. K., Borissova, J., et al. 2016, AJ, 151, 126. doi:10.3847/0004-6256/151/5/126
* Sharpless (1959) Sharpless, S. 1959, ApJS, 4, 257. doi:10.1086/190049
* Stark & Brand (1989) Stark, A. A. & Brand, J. 1989, ApJ, 339, 763. doi:10.1086/167334
* Taylor et al. (2003) Taylor, A. R., Gibson, S. J., Peracaula, M., et al. 2003, AJ, 125, 3145. doi:10.1086/375301
* Urquhart et al. (2008) Urquhart, J. S., Busfield, A. L., Hoare, M. G., et al. 2008, A&A, 487, 253. doi:10.1051/0004-6361:200809415
* Urquhart et al. (2009) Urquhart, J. S., Morgan, L. K., & Thompson, M. A. 2009, A&A, 497, 789. doi:10.1051/0004-6361/200811149
* Uyanıker et al. (2001) Uyanıker, B., Fürst, E., Reich, W., et al. 2001, A&A, 371, 675. doi:10.1051/0004-6361:20010387
* Vázquez-Semadeni et al. (2019) Vázquez-Semadeni, E., Palau, A., Ballesteros-Paredes, J., et al. 2019, MNRAS, 490, 3061. doi:10.1093/mnras/stz2736
* Wang et al. (2020) Wang, Y., Bihr, S., Beuther, H., et al. 2020, A&A, 634, A139. doi:10.1051/0004-6361/201935866
* Williams et al. (1994) Williams, J. P., de Geus, E. J., & Blitz, L. 1994, ApJ, 428, 693. doi:10.1086/174279
* Williams et al. (2018) Williams, G. M., Peretto, N., Avison, A., et al. 2018, A&A, 613, A11. doi:10.1051/0004-6361/201731587
* Zinnecker & Yorke (2007) Zinnecker, H. & Yorke, H. W. 2007, ARA&A, 45, 481. doi:10.1146/annurev.astro.44.051905.092549


# Evaluating Automatic Metrics
with Incremental Machine Translation Systems

Guojun Wu1 Shay B. Cohen2 Rico Sennrich1,2
1Department of Computational Linguistics, University of Zurich
2School of Informatics, University of Edinburgh
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

We introduce a dataset comprising commercial machine translations, gathered
weekly over six years across 12 translation directions. Since human A/B
testing is commonly used, we assume commercial systems improve over time,
which enables us to evaluate machine translation (MT) metrics based on their
preference for more recent translations. Our study confirms several previous
findings in MT metrics research and demonstrates the dataset’s value as a
testbed for metric evaluation. We release our
code.111https://github.com/gjwubyron/Evo

Evaluating Automatic Metrics
with Incremental Machine Translation Systems

Guojun Wu1 Shay B. Cohen2 Rico Sennrich1,2 1Department of Computational
Linguistics, University of Zurich 2School of Informatics, University of
Edinburgh<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>

##  1 Introduction

Automatic metrics for machine translation (MT) are typically assessed by
measuring their correlation with or accuracy with respect to human judgments
Macháček and Bojar (2013); Mathur et al. (2020b); Kocmi et al. (2021).
However, human evaluation is resource-intensive and time-consuming, and the
number of translation systems included in a meta-evaluation tends to be
relatively small. In this study, we explore the use of commercial machine
translations, collected weekly over a period of 6 years for 12 translation
directions, for the evaluation of MT metrics. Given the common use of human
A/B testing Tang et al. (2010); Caswell and Liang (2020), our base assumption
is that commercial systems show real improvements over time and that we can
assess metrics as to whether they prefer more recent MT outputs. Using our
dataset, we revisit a number of recent findings in MT metrics research, and
find that our dataset supports these.

Freitag et al. (2022, 2023) revealed that neural metrics exhibit significantly
higher correlation with human judgments compared to non-neural ones. In our
experiments, we analyze metric scores over time and evaluate metrics’ ability
to accurately rank MT systems. Our findings demonstrate that neural metrics
show a more consistent upward trend, and achieve higher accuracy than non-
neural metrics.

Ma et al. (2019) demonstrated that the correlation between metrics and human
judgments significantly decreased when considering only the top-performing
systems. However, the limited number of MT systems (typically 10–15 MT systems
per language pair) made it difficult to fully confirm this trend Mathur et al.
(2020a). We revisit this finding using a larger sample and observe that the
correlation tends to decrease for many language pairs as the quality of
evaluated systems improves.

High-quality synthetic references were found to produce a stronger correlation
between human evaluations and metrics compared to human-generated references
Freitag et al. (2023). We reexamine the effect of using synthetic references
with three language pairs and find that synthetic references can result in
comparable correlation.

##  2 Background and Related Work

Designed to directly learn human judgments, trained metrics Rei et al. (2020);
Sellam et al. (2020) have exhibited notable advancements in correlating with
human judgments compared to non-neural metrics like BLEU Papineni et al.
(2002); Freitag et al. (2021). Recent research Freitag et al. (2022, 2023)
reveals that these trained metrics can also generalize to new domains and
challenge sets.

Ma et al. (2019) assessed the stability of metrics across top-N MT systems,
and noticed that the correlation between metric and human scores diminished as
N decreased. A subsequent investigation Mathur et al. (2020a) suggested that
the decrease might be due to instability of small samples. They employed a
rolling window of N systems, moving from the worst to the best systems and
found that the correlation is unstable for small samples. Besides, due to the
limited number of MT systems, they could not determine if metric reliability
decreases as the quality of MT systems improves.

In WMT23 Metrics shared task Freitag et al. (2023), human translations
received unexpectedly low ratings, which prompted an investigation into using
synthetic references as a potential alternative. It was found that high-
quality synthetic references led to a stronger correlation between human and
metrics compared to humans references.

Instead of evaluating metrics through comparison with human judgement, Moghe
et al. (2023) explored a complementary approach by correlating metrics with
the outcome of downstream tasks. Similarly, our study does not use human
judgment directly; instead, we evaluate metrics based on their preference for
newer MT outputs.

##  3 Methods

###  3.1 Data

The original corpus contains sentences in English from Abstract Meaning
Representation (AMR) Annotation Release 2.0 Knight and et al. (2017), along
with their German, Italian, Spanish, and Chinese translations developed by
Damonte and Cohen (2020). This corpus contains 1371 sentences per language.
The source sentences were mainly drawn from content gathered in the news
domain.

Translations222Due to the origin of the translations, the data we use is to be
licensed by the Linguistic Data Consortium. If you would like to use this
dataset for your research, please contact the authors. The collection of the
data continues. are gathered weekly from May 2018 – March 2024 using Google
Translate from each of the five languages to the other four languages. Early
experiments revealed that for English$\rightarrow$Spanish, there was a
substantial similarity between professional translations and those generated
by the earliest systems (details in Appendix A). Consequently, Spanish was
removed from further investigation, reducing the number of language pairs to
12. As minimal variation was observed between consecutive weeks, we subsample
for the following analysis, with consecutive systems being approximately one
month apart. After removing duplicates (systems receiving identical scores
across all metrics), we retained 56–63 systems per language pair.

###  3.2 Metrics

####  3.2.1 Surface-level overlap

BLEU (Papineni et al., 2002) measures n-grams overlap between the translation
and its reference. We use _corpus_bleu_ in SacreBLEU Post (2018).

chrF (Popović, 2015) assesses the overlap between the characters of the
translation and the reference. We use _corpus_chrf_ in SacreBLEU.

####  3.2.2 Embedding based

BERTScore (Zhang* et al., 2020) derives contextual embeddings from BERT
(Devlin et al., 2019) models and computes cosine similarity between embeddings
of the translation and the reference. We use the F1 score without TF-IDF
weighting.

####  3.2.3 Trained with human judgements

COMET-20 Rei et al. (2020) is trained on top of XLM-R Conneau et al. (2020)
using Direct Assessments (DA) from WMT17 to WMT19. We utilize wmt20-comet-da.

UniTE (Wan et al., 2022a, b) is capable of evaluating translation outputs in
source-only, reference-only, and source-reference-combined assessment
scenarios. We use unite-mup.

COMET-22 (Rei et al., 2022a) is the current default model in COMET and trained
on DA from WMT17 to WMT20. We use wmt22-comet-da.

COMET-Kiwi (Rei et al., 2022b) is a reference-free metric trained using DA
from WMT17 to WMT20, and DA from the MLQE-PE corpus. We use wmt22-cometkiwi-
da.

MS-COMET-QE-22 (Kocmi et al., 2022) is a reference-free metric, extending
COMET by Microsoft Research with proprietary data.

Figure 1: The Spearman correlation measures the relationship between metric score rankings and time rankings for MT systems. A positive correlation indicates an upward trend, with a higher correlation indicating a stronger trend. A red star indicates lack of statistical significance (p-value $>$ 0.05). | All | Into EN | From EN | Into DE | Into IT | Into ZH
---|---|---|---|---|---|---
COMET-22 | 73.9 | 66.6 | 71.6 | 76.4 | 79.4 | 72.6
COMET-Kiwi | 73.4 | 72.1 | 73.9 | 74.8 | 75.3 | 71.4
UniTE | 73.2 | 66.5 | 73.7 | 77.1 | 75.0 | 73.9
COMET-20 | 72.5 | 66.1 | 74.6 | 74.3 | 74.0 | 74.9
chrF | 71.4 | 74.5 | 57.8 | 60.4 | 76.5 | 74.6
MS-COMET-22-QE | 69.9 | 57.4 | 68.1 | 68.8 | 73.9 | 78.6
BLEU | 68.2 | 71.7 | 57.3 | 56.3 | 68.9 | 76.4
BERTScore | 68.0 | 65.4 | 62.2 | 68.8 | 69.0 | 68.6
Table 1: Accuracy for ranking system pairs. Column “All” shows the results for
all system pairs. Each following column evaluates accuracy over a subset of
systems. Rows are sorted by the accuracy over all system pairs.

##  4 Results

###  4.1 How do metric scores change over time?

While it is reasonable to expect that systems improve over time, how metric
scores will reflect these improvements remains unclear. To investigate this,
we visualize how metric scores vary over time for individual language pairs in
Appendix B. In general, upward trends are evident for the metrics across the
language pairs.

We use Spearman correlation to measure whether the upward trends are
consistent. Metrics with higher correlation are deemed more reliable, as they
better reflect the overall ranking of the systems. As illustrated in Figure 1,
COMET-22, UniTE, COMET-20, and COMET-Kiwi consistently demonstrate high
correlation across the language pairs. Among the remaining four metrics, we
notice low correlations in specific language pairs, like BLEU and chrF in
English$\rightarrow$German or MS-COMET-22-QE in Italian$\rightarrow$English.

Figure 2: Accuracy over a rolling window of 36 systems. The x axis shows the
index of the starting system, and systems are sorted by time. Figure 3:
Accuracy across three language pairs using either human or synthetic
references. The two reference-free metrics are not included as they will not
be influenced by reference.

###  4.2 How good can the metrics rank incremental systems accurately?

In this section, we evaluate metrics in a common scenario (Mathur et al.,
2020a): ranking a pair of systems. As we assume newer systems are better than
old ones, accuracy Kocmi et al. (2021) is adopted as follows. For each system
pair, we calculate the difference of the metric scores (metric$\Delta$) and
the difference in time (time$\Delta$). Accuracy for a specific metric is
calculated as the ratio of rank agreements between metric and time deltas to
the total number of comparisons:

$\text{Accuracy}=\frac{|\text{sign(metric$\Delta$) = sign(time$\Delta$)
}|}{|\text{all system pairs}|}$

Since the systems span from 2018 to 2024, those separated by a substantial
time interval might exhibit considerable quality gaps, potentially resulting
in an overestimate of metric reliability Mathur et al. (2020a). Consequently,
we only pair systems with a gap of less than a year. Even within such a
timeframe, substantial improvements in quality are possible Caswell and Liang
(2020).

Table 1 shows that trained metrics generally outperform non-trained metrics.
For all system pairs, COMET-22 achieves the highest accuracy, followed by
COMET-Kiwi. In contrast, MS-COMET-QE-22 struggles to attain high accuracy
except for into Chinese. Among surface-level metrics, chrF outperforms BLEU,
reflecting results in previous studies Kocmi et al. (2021), and achieves the
highest accuracy for into English. We also examine performance for individual
language pairs. Trained metrics exhibit high accuracy, yet no single metric
excels across all pairs. More details in AppendixC.

###  4.3 Does the reliability of metrics depend on the quality of the systems
evaluated?

As mentioned in Section 2, metric reliability may decline as the quality of
evaluated systems improves Ma et al. (2019). However, the limited number of MT
systems made it difficult to fully confirm this trend Mathur et al. (2020a).
We revisit this issue using a larger sample of MT systems. Following the
approach of Mathur et al. (2020a), we implement a rolling window of N systems,
transitioning from the earliest to the most recent ones. Using accuracy as
explained in Section 4.2, we conduct tests with N varying from 24 to 40.
Figure 2 illustrates the results for N = 36, representing the identified
scenarios. Different metrics display varying trends. For instance, in
English$\rightarrow$German, trained metrics show an upward trend, while
surface-level metrics show a downward trend. A downward trend is most common,
with each metric showing a clear decline across 7 or more language pairs.
However, we also observe upward or relatively flat trends in the remaining
language pairs.

###  4.4 How will synthetic references impact the metrics’ judgement?

We generate synthetic references for three language pairs using another
commercial MT system, DeepL, and examine their impact on metric evaluation. As
depicted in Figure 3, we observe that for English$\rightarrow$German, all
metrics achieve a higher accuracy, while for the remaining language pairs,
there are some drops. Overall, synthetic references lead to a comparable
accuracy for the three language pairs we investigate.

##  5 Conclusion

We evaluated metrics based on their preference for newer translations,
confirming many prior findings on MT metrics. Our dataset, covering 12
language pairs with at least 56 systems each, surpasses previous datasets that
typically included only 3 pairs with around 15 systems each, providing larger-
scale evidence for debated questions such as the relationship between MT
quality and metric reliability. Additionally, the systems are incremental (a
baseline compared to improvements developed by the same group), reflecting the
most common use case of the metrics. We encourage the use of our dataset for
future investigations into MT metrics or the development of MT quality over
time.

## Limitations

Our study bases on the assumption that newer systems of Google Translate
outperform older ones due to the quality assurance measures, including human
testing, taken before deployment. Although this is a reasonable belief, it
might not always be true.

Recently, LLM-based evaluators have demonstrated great performance in
evaluating MT systems. However, we have not included any LLM-based evaluators
in this study because it would be costly to experiment with our extensive
dataset.

## References

  * Caswell and Liang (2020) Isaac Caswell and Bowen Liang. 2020.  Recent advances in google translate.  https://research.google/blog/recent-advances-in-google-translate/.  Google Research Blog.
  * Conneau et al. (2020) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020.  Unsupervised cross-lingual representation learning at scale.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 8440–8451, Online. Association for Computational Linguistics.
  * Damonte and Cohen (2020) Marco Damonte and Shay Cohen. 2020.  Abstract Meaning Representation 2.0 - Four Translations LDC2020T07.  Web Download. Philadelphia: Linguistic Data Consortium, 2020.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Freitag et al. (2023) Markus Freitag, Nitika Mathur, Chi-kiu Lo, Eleftherios Avramidis, Ricardo Rei, Brian Thompson, Tom Kocmi, Frederic Blain, Daniel Deutsch, Craig Stewart, Chrysoula Zerva, Sheila Castilho, Alon Lavie, and George Foster. 2023.  Results of WMT23 metrics shared task: Metrics might be guilty but references are not innocent.  In _Proceedings of the Eighth Conference on Machine Translation_ , pages 578–628, Singapore. Association for Computational Linguistics.
  * Freitag et al. (2022) Markus Freitag, Ricardo Rei, Nitika Mathur, Chi-kiu Lo, Craig Stewart, Eleftherios Avramidis, Tom Kocmi, George Foster, Alon Lavie, and André F. T. Martins. 2022.  Results of WMT22 metrics shared task: Stop using BLEU – neural metrics are better and more robust.  In _Proceedings of the Seventh Conference on Machine Translation (WMT)_ , pages 46–68, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
  * Freitag et al. (2021) Markus Freitag, Ricardo Rei, Nitika Mathur, Chi-kiu Lo, Craig Stewart, George Foster, Alon Lavie, and Ondřej Bojar. 2021.  Results of the WMT21 metrics shared task: Evaluating metrics with expert-based human evaluations on TED and news domain.  In _Proceedings of the Sixth Conference on Machine Translation_ , pages 733–774, Online. Association for Computational Linguistics.
  * Knight and et al. (2017) Kevin Knight and et al. 2017.  Abstract Meaning Representation (AMR) Annotation Release 2.0 LDC2017T10.  Web Download. Philadelphia: Linguistic Data Consortium, 2017.
  * Kocmi et al. (2021) Tom Kocmi, Christian Federmann, Roman Grundkiewicz, Marcin Junczys-Dowmunt, Hitokazu Matsushita, and Arul Menezes. 2021.  To ship or not to ship: An extensive evaluation of automatic metrics for machine translation.  In _Proceedings of the Sixth Conference on Machine Translation_ , pages 478–494, Online. Association for Computational Linguistics.
  * Kocmi et al. (2022) Tom Kocmi, Hitokazu Matsushita, and Christian Federmann. 2022.  MS-COMET: More and better human judgements improve metric performance.  In _Proceedings of the Seventh Conference on Machine Translation (WMT)_ , pages 541–548, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
  * Ma et al. (2019) Qingsong Ma, Johnny Wei, Ondřej Bojar, and Yvette Graham. 2019.  Results of the WMT19 metrics shared task: Segment-level and strong MT systems pose big challenges.  In _Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1)_ , pages 62–90, Florence, Italy. Association for Computational Linguistics.
  * Macháček and Bojar (2013) Matouš Macháček and Ondřej Bojar. 2013.  Results of the WMT13 metrics shared task.  In _Proceedings of the Eighth Workshop on Statistical Machine Translation_ , pages 45–51, Sofia, Bulgaria. Association for Computational Linguistics.
  * Mathur et al. (2020a) Nitika Mathur, Timothy Baldwin, and Trevor Cohn. 2020a.  Tangled up in BLEU: Reevaluating the evaluation of automatic machine translation evaluation metrics.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4984–4997, Online. Association for Computational Linguistics.
  * Mathur et al. (2020b) Nitika Mathur, Johnny Wei, Markus Freitag, Qingsong Ma, and Ondřej Bojar. 2020b.  Results of the WMT20 metrics shared task.  In _Proceedings of the Fifth Conference on Machine Translation_ , pages 688–725, Online. Association for Computational Linguistics.
  * Moghe et al. (2023) Nikita Moghe, Tom Sherborne, Mark Steedman, and Alexandra Birch. 2023.  Extrinsic evaluation of machine translation metrics.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 13060–13078, Toronto, Canada. Association for Computational Linguistics.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  Bleu: a method for automatic evaluation of machine translation.  In _Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics_ , pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
  * Popović (2015) Maja Popović. 2015.  chrF: character n-gram F-score for automatic MT evaluation.  In _Proceedings of the Tenth Workshop on Statistical Machine Translation_ , pages 392–395, Lisbon, Portugal. Association for Computational Linguistics.
  * Post (2018) Matt Post. 2018.  A call for clarity in reporting BLEU scores.  In _Proceedings of the Third Conference on Machine Translation: Research Papers_ , pages 186–191, Brussels, Belgium. Association for Computational Linguistics.
  * Rei et al. (2022a) Ricardo Rei, José G. C. de Souza, Duarte Alves, Chrysoula Zerva, Ana C Farinha, Taisiya Glushkova, Alon Lavie, Luisa Coheur, and André F. T. Martins. 2022a.  COMET-22: Unbabel-IST 2022 submission for the metrics shared task.  In _Proceedings of the Seventh Conference on Machine Translation (WMT)_ , pages 578–585, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
  * Rei et al. (2020) Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020.  COMET: A neural framework for MT evaluation.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 2685–2702, Online. Association for Computational Linguistics.
  * Rei et al. (2022b) Ricardo Rei, Marcos Treviso, Nuno M. Guerreiro, Chrysoula Zerva, Ana C Farinha, Christine Maroti, José G. C. de Souza, Taisiya Glushkova, Duarte Alves, Luisa Coheur, Alon Lavie, and André F. T. Martins. 2022b.  CometKiwi: IST-unbabel 2022 submission for the quality estimation shared task.  In _Proceedings of the Seventh Conference on Machine Translation (WMT)_ , pages 634–645, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
  * Sellam et al. (2020) Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020.  BLEURT: Learning robust metrics for text generation.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7881–7892, Online. Association for Computational Linguistics.
  * Tang et al. (2010) Diane Tang, Ashish Agarwal, Deirdre O’Brien, and Mike Meyer. 2010.  Overlapping experiment infrastructure: More, better, faster experimentation.  In _Proceedings 16th Conference on Knowledge Discovery and Data Mining_ , pages 17–26, Washington, DC.
  * Wan et al. (2022a) Yu Wan, Keqin Bao, Dayiheng Liu, Baosong Yang, Derek F. Wong, Lidia S. Chao, Wenqiang Lei, and Jun Xie. 2022a.  Alibaba-translate China’s submission for WMT2022 metrics shared task.  In _Proceedings of the Seventh Conference on Machine Translation (WMT)_ , pages 586–592, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
  * Wan et al. (2022b) Yu Wan, Dayiheng Liu, Baosong Yang, Haibo Zhang, Boxing Chen, Derek Wong, and Lidia Chao. 2022b.  UniTE: Unified translation evaluation.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 8117–8127, Dublin, Ireland. Association for Computational Linguistics.
  * Zhang* et al. (2020) Tianyi Zhang*, Varsha Kishore*, Felix Wu*, Kilian Q. Weinberger, and Yoav Artzi. 2020.  Bertscore: Evaluating text generation with bert.  In _International Conference on Learning Representations_.

##  Appendix A Metric scores for English $\rightarrow$ Spanish translations

Figure 4 displays the scores of four different metrics for
English$\rightarrow$Spanish translations in our early experiments. Early
systems achieved nearly perfect metric scores, whereas later systems displayed
markedly lower scores. Upon closer examination of the human translations, we
noticed roughly 25% of them are identical to that of the early systems. This
indicates the use of Google Translate in the professional translation.

(a) BLEU

(b) chrF

(c) BERTScore

(d) COMET-22

Figure 4: The metric scores for English$\rightarrow$Spanish translations.
While the earliest system achieved nearly perfect scores, subsequent systems
showed a notable decline.

##  Appendix B Metric scores over time

Figure 5 illustrates the findings regarding the change of metric scores over
time. Generally, upward trends are evident for the metrics across language
pairs. Furthermore, these trends sometimes appear as step-like progressions.
Based on a visual inspection of the results, we have some interesting findings
as follows:

  1. 1.

Although there have been concerns that MT systems were optimized for BLEU,
given its longstanding status as the primary evaluation metric, our findings
suggest that the upward trends of BLEU are less consistent compared to other
metrics. This observation might provide implicit evidence that BLEU is not
solely used during system development.

  2. 2.

The trajectories of BLEU and chrF exhibit a high degree of similarity, as do
the trajectories of COMET-20, COMET-22, COMET-Kiwi, and UniTE. In contrast,
BERTScore and MS-COMET-22-QE follow distinct trajectories of their own. These
similarities and discrepancies reflect the inherent properties of these
metrics. BLEU and chrF both rely on measuring surface-level overlap, while
BERTScore is unique in its reliance on contextual embeddings. As for the
trained metrics, although they are all trained in a similar manner, MS-
COMET-22-QE was trained using entirely different data.

  3. 3.

In certain language pairs, the trajectories of certain metrics may experience
a downturn. For instance, noticeable troughs are observed for BLEU and chrF in
English$\rightarrow$German, Italian$\rightarrow$German, and
English$\rightarrow$Italian; for BERTScore in English$\rightarrow$German,
German$\rightarrow$Italian, and English$\rightarrow$Italian; and for MS-
COMET-22-QE in Italian$\rightarrow$English, Italian$\rightarrow$German, and
Chinese$\rightarrow$English. On the other hand, the trajectories of the
remaining metrics may occasionally exhibit bumps but do not show clear
troughs.

(a)

(b)

Figure 5: Metric scores over time.

(a)

(b)

(c)

Figure 6: Metric scores over time.

(a)

(b)

(c)

Figure 7: Metric scores over time.

##  Appendix C Accuracy across the language pairs

Figure 8: Accuracy for ranking system pairs across individual language pairs.