11institutetext: Department of Space, Earth and Environment, Chalmers
University of Technology, Onsala Space Observatory, 439 92 Onsala, Sweden

# Polarisation of molecular lines in the circumstellar envelope of the post-
Asymptotic Giant Branch star OH 17.7-2.0

W. H. T. Vlemmings<EMAIL_ADDRESS>D. Tafoya 11

(accepted 31-Jan-2023)

###### Abstract

Context. The role of magnetic field in the shaping of Planetary Nebulae (PNe),
either directly or indirectly after being enhanced by binary interaction, has
long been a topic of debate. Large scale magnetic fields around pre-PNe have
been inferred from polarisation observations of masers. However, because
masers probe very specific regions, it is still unclear if the maser results
are representative of the intrinsic magnetic field in the circumstellar
envelope (CSE).

Aims. Molecular line polarisation of non-maser lines can provide important
information about the magnetic field. A comparison between the magnetic field
morphology determined from maser observations and that observed in the more
diffuse CO gas, can reveal if the two tracers probe the same magnetic field.

Methods. We compare observations taken with the Atacama Large
Millimeter/submillimeter Array (ALMA) of molecular line polarisation around
the post-Asymptotic Giant Branch (post-AGB)/pre-PNe star OH 17.7-2.0 with
previous observations of polarisation in the 1612 MHz OH maser region. Earlier
mid-infrared observations indicate that OH 17.7-2.0 is a young bipolar pre-
PNe, with both a torus and bipolar outflow cavities embedded in a remnant AGB
envelope.

Results. We detect CO $J=2-1$ molecular line polarisation at a level of $\sim
4\%$ that displays an ordered linear polarisation structure. We find that,
correcting for Faraday rotation of the OH maser linear polarisation vectors,
the OH and CO linearly polarised emission trace the same large scale magnetic
field. A structure function analysis of the CO linear polarisation reveals a
plane-of-the-sky magnetic field strength of $B_{\perp}\sim 1$ mG in the CO
region, consistent with previous OH Zeeman observations.

Conclusions. The consistency of the ALMA CO molecular line polarisation
observation with maser observations indicate that both can be used to
determine the magnetic field strength and morphology in CSEs. The new
observations indicate that the magnetic field has a strong toroidal field
component projected on the torus structure and a poloidal field component
along the outflow cavity. The existence of a strong, ordered, magnetic-field
around OH 17.7-2.0 indicates that magnetic fields are likely involved in the
formation of this bipolar pre-PNe.

###### Key Words.:

magnetic fields, circumstellar matter, stars: post-AGB, stars: individual: OH
17.7-2.0

##  1 Introduction

The processes involved in the formation of bipolar Planetary Nebulae (PNe) are
still under debate (e.g. Blackman, 2022). While there are strong indications
that binary interaction is required (e.g. Boffin & Jones, 2019, and references
therin), there are also observations that show that magnetic fields appear
responsible for launching collimated outflows that shape the nebula (e.g.
Vlemmings et al., 2006). It is possible that binarity and magnetic fields work
in tandem. In this case, the magnetic field is enhanced by the interaction
between the evolved star, typically during the red-giant branch (RGB) or
asymptotic-giant branch (AGB) phase of their evolution, and a companion (e.g.
Nordhaus & Blackman, 2006). This interaction could lead to a common-envelope
evolution (CEE), where the tenuous envelope of the RGB/AGB star is ejected
while accretion on the companion or evolved star core, assisted by enhanced
magnetic fields, launches a bipolar outflow (e.g. Ondratschek et al., 2022).
Observations of a large sample of evolved stars with fast bipolar outflows
indicate that these bipolar pre-PNe likely make up the majority of the recent
CEE ejection events (Khouri et al., 2021).

Figure 1: (left) The 222.5 GHz continuum emission from OH 17.7-2.0 in red
contours and greyscale. The contours are drawn at $0.625,1.25,2.5,5,10,20,40,$
and $80\%$ of the peak value of 71.8 mJy beam-1. The dashed line indicates the
direction of the outflow inferred from mid-infrared continuum and H2
observations (Gledhill et al., 2011). The four labelled solid circles are the
sources identified by Gaia within a radius of $2.5$″of OH 17.7-2.0 (see text).
(right) The polarised continuum emission (greyscale) and the 225.5 GHz
continuum contours (similarly spaced as in the left panel) of OH 17.7-2.0. The
blue line segments indicate the linear polarisation direction where polarised
emission is detected at $>3\sigma$, where the rms on the polarised emission
$\sigma=21\leavevmode\nobreak\ \mu$Jy beam-1. This means that the uncertainty
on the direction is $\lesssim 10^{\circ}$. The segments are scaled by the
linear polarisation fraction which peaks at $P_{\rm l,max}=0.47\%$. The dashed
line indicates the direction of the outflow from OH 17.7-2.0 as described for
the left panel. The filled ellipse indicates the beam size of the
observations.

Strong magnetic fields have also been observed around early AGB stars and
post-AGB/pre-PNe objects that have no confirmed companion or that have not yet
undergone a CEE event (e.g. Vlemmings, 2019, and references therein). The
origin of these fields is still unclear. The majority of the magnetic field
observations around evolved stars have relied on measurements of linear and
circular polarisation, due to the Zeeman-effect, of masers (e.g. Vlemmings et
al., 2002; Bains et al., 2003; Vlemmings et al., 2005, 2006; Herpin et al.,
2006; Leal-Ferreira et al., 2013; Gonidakis et al., 2014). Additionally, there
are observations of the surface magnetic field on one AGB star (Lèbre et al.,
2014) as well as on the surface of two post-AGB stars (Sabin et al., 2015).
Observations of polarised dust (e.g. Sabin et al., 2020) and radio synchrotron
emission (Perez-Sanchez et al., 2013) around post-AGB stars might also
indicate the involvement of magnetic fields in shaping PNe. Still, to date,
the strongest evidence of the role of magnetic fields specifically around
post-AGB objects comes from maser observations (e.g. Bains et al., 2003;
Vlemmings et al., 2006) that provide both a magnetic field morphology and
magnetic field strength. It has however been argued that the magnetic fields
determined from maser observations are biased due to the special nature of the
amplified stimulated emission that produces maser features in very specific
region in the CSE (e.g. Soker, 2002).

Molecular line polarisation from non-maser molecules, resulting from the
Goldreich-Kylafis (GK) effect (e.g. Goldreich & Kylafis, 1982; Lankhaar &
Vlemmings, 2020), can be used to determine if the maser magnetic field
observations probe the intrinsic large-scale magnetic field. The GK-effect of
CO and other molecules has been observed in star forming regions, proto-
planetary disks, and around a supergiant star (e.g. Cortés et al., 2005;
Beuther et al., 2010; Vlemmings et al., 2017; Stephens et al., 2020; Cortés et
al., 2021; Teague et al., 2021). It has also been detected, using the Sub-
Millimeter Array (SMA), around the AGB stars CW Leo and IK Tau (Girart et al.,
2012; Vlemmings et al., 2012) and with the Combined Array for Research in
Millimeter-wave Astronomy (CARMA) towards R Leo and R Crt (Huang et al.,
2020). While these observations indicated a possible large scale magnetic
field, the sensitivity of the SMA and CARMA was not sufficient for the
detection of polarised molecular line emission in more than a few compact
regions of the CSE. For most of the previously observed sources, no maser
polarisation observations probing the same region as probed by the GK-effect
exist. It has thus not been possible to compare the observations directly with
maser measurements.

One of the post-AGB/pre-PNe sources with extensive polarised OH maser
observations is the star OH 17.7-2.0. Polarisation observations of mainly the
1612 MHz OH masers around this source revealed an apparent coherent large
scale structure with a magnetic field strength in the OH maser region of
$B\approx 2.5-4.5$ mG (Bains et al., 2003, herafter B03). Here we present ALMA
observations of linear polarisation due to the GK-effect in the envelope of OH
17.7-2.0. In § 2, we present the observations and data reduction, and in § 3
we present the source, with a particular discussion regarding its distance. In
§ 4, we present the results of the ALMA observations and in § 5 we perform a
Structure Function Analysis (SFA) to estimate the magnetic field strength,
perform a detailed comparison with the OH maser observation of B03, and
present our results on the morphology of the magnetic field around OH
17.7-2.0. We end with our conclusions in § 6.

Table 1: Gaia results label | Gaia source_id | $\pi\pm\sigma_{\pi}$ | offset
---|---|---|---
|  | [mas] | [″]
1 | 4104128498748054272 | $0.39\pm 0.02$ | $2.35$
2 | 4104128503103791232 | $-0.04\pm 0.14$ | $2.39$
3 | 4104128503103804288 | - | $0.47$
4 | 4104128503164186496 | - | $0.32$

##  2 Observations and data reduction

The CSE of OH 17.7-2.0 was observed by ALMA in full polarisation mode on April
14 2018, spending a total time of 2.5 hours. The on-source time was $\sim 70$
min. The remaining time was spent observing the phase calibrator J1832+2039
and the amplitude and polarisation calibrator J1924+2914. Four spectral
windows of $1.875$ GHz and 960 channels each were centred on $214.6$, $216.5$,
$228.6$, and $230.5$ GHz resulting in a channel width of $\sim 2.6$ km s-1.
The observations were calibrated using the ALMA polarisation calibration
scripts (Nagai et al., 2016).

Figure 2: Channel maps of the polarised CO $J=2-1$ emission around the post-
AGB star OH 17.7-2.0. The Stokes I total intensity emission is indicated by
the solid red contours at $2.5,5,10,20,40,$ and $80\%$ of the peak emission
($I_{\rm CO,peak}$=1.04 Jy beam-1). The linearly polarised emission is shown
as a greyscale map, and the blue line segments denote the linear polarisation
direction when the polarised emission $>5\sigma_{P}$. The uncertainty on the
polarisation direction is thus $\lesssim 7^{\circ}$. The segments are scaled
to the level of fractional polarisation with the scale indicated in the bottom
left panel. The maximum polarisation $P_{\rm l,max}=4.1\%$. The panels are
labelled with the $V_{\rm lsr}$ velocity in km s-1, and the beam size is shown
in the bottom right panel. The stellar velocity $V_{\rm lsr,*}=62.0$ km s-1.
The dashed line indicates the direction of the outflow of OH 17.7-2.0 as in
Fig. 1

Subsequent calibration, self-calibration and imaging was done using CASA 5.7.2
(McMullin et al., 2007)111The data were checked against an error in visibility
amplitude calibration (https://almascience.eso.org/news/amplitude-calibration-
issue-affecting-some-alma-data) but ALMA staff from the ESO ALMA Regional
Centre determined that a correction was not necessary. We performed two rounds
of phase-only self-calibration on the compact continuum of OH 17.7-2.0, which
improved the dynamic range in the continuum from $\sim 530$ to $\sim 1010$.
The observations reach a continuum rms in Stokes I ($\sigma_{I}$) of
$71\leavevmode\nobreak\ \mu$Jy beam-1. In the Stokes Q and U continuum images,
the rms noise ($\sigma_{Q,U}$) are $\sim 13\leavevmode\nobreak\ \mu$Jy beam-1
and $\sim 18\leavevmode\nobreak\ \mu$Jy beam-1 respectively. We also produced
the Stokes V continuum image, for which we find an rms noise $\sigma_{V}\sim
14\leavevmode\nobreak\ mu$Jy beam-1. While the rms in the Stokes Q, U and V
images are close to the theoretical noise limit, that of Stokes I is almost
five time higher. The continuum Stokes I image is thus likely limited by
dynamic range. The continuum beam size, using Briggs weighing and a robust
parameter of $0.5$, is $0.78\times 0.65$″(PA $75.1^{\circ}$).

The continuum was subtracted using the CASA task uvcontsub after which the
strongest spectral lines were imaged using Briggs weighing and a robust
parameter of $0.5$. To improve polarisation sensitivity, we averaged two
channels, obtaining a velocity resolution of $\sim 5.2$ km s-1. The
$\sigma_{I}$, $\sigma_{Q}$, $\sigma_{U}$, and $\sigma_{V}$ rms noise level in
a line free channel are $0.64\leavevmode\nobreak\ $mJy beam-1,
$0.39\leavevmode\nobreak\ $mJy beam-1, $0.38\leavevmode\nobreak\ $mJy beam-1,
and $0.47\leavevmode\nobreak\ $mJy beam-1. The beam size, at 230 GHz is
$0.77\times 0.64$″(PA $77.3^{\circ}$). The maximum recoverable scale in our
observations is $\sim 8.1$″. From a comparison with the published CO $J=2-1$
spectrum (Heske et al., 1990) and a visual inspection of the images, there are
no indications of significant resolved out flux. Finally linear polarisation
maps were created from the Stokes Q and U imaged using
$P_{l}=\sqrt{Q^{2}+U^{2}-\sigma^{2}_{P}}$. The polarisation rms
$\sigma_{P}\approx 0.5$ mJy beam-1 is found from an analysis of the rms in a
line free spectral channel for each spectral line individually.

##  3 OH 17.7-2.0

The post-AGB/pre-PNe star OH 17.7-2.0, also known as IRAS 18276-1431, has been
studied across a wide range of wavelengths. It hosts strong OH masers (e.g.
Bowers, 1978), while its H2O masers disappeared after a rapid flux decline
between 1985 and 1990 (Engels, 2002) before reappearing more than 20 years
later (Wolak et al., 2013). The decrease of H2O maser emission was attributed
to a recent drop in mass-loss rate caused by the star having recently left the
AGB phase. Polarimetric measurements of OH masers appear to indicates that the
CSE of OH 17.7-2.0 has a strong large-scale magnetic field (B03). OH 17.7-2.0
was first suggested be a bipolar nebula based on an analysis of its infrared
(IR) spectral energy distribution (Le Bertre et al., 1984). Later imaging in
the optical and (near-)IR confirmed this view (e.g. Sánchez Contreras et al.,
2007; Gledhill et al., 2011; Lagadec et al., 2011; Murakawa et al., 2013).
Observations of H2 and CO emission in the near-IR suggest a bipolar outflow
velocity of $\sim 95$ km s-1. The outflow has created bipolar outflow cavities
that are $\gtrsim 125$ yr old and are inclined to the line-of-sight by $\sim
22^{\circ}$ (Gledhill et al., 2011). Observations of the 12CO and 13CO $J=1-0$
emission using the Owens Valley Radio Observatory (OVRO) revealed that the
source is embedded in a dense, mostly spherical, CO envelope which has an
outflow velocity of $\sim 17$ km s-1(Sánchez Contreras et al., 2007; Sánchez
Contreras & Sahai, 2012). No fast outflow is detected in the sub-millimetre CO
emission. In Murakawa et al. (2013), a model was created to match the near-IR
spectral energy distribution and polarised emission. They conclude, assuming a
distance of $3$ kpc, that most of the dust resides in a torus with inner and
outer radii of 30 and 1000 au. The torus has a total mass of $3.0$ M⊙ and,
assuming an expansion of the torus with a velocity similar to the velocities
measured in the CO $J=1-0$, an age of $\sim 300$ yr.

The distance to OH 17.7-2.0: Because foreground Faraday rotation of the
linearly polarised OH maser emission affects the comparison of the absolute
polarisation angle between the OH and sub-millimetre molecular line emission,
we carefully assess the distance to OH 17.7-2.0. In the literature, distances
are quoted between $\sim 2-5.4$ kpc. Some of the earliest distances were
derived using the phase-lag method, comparing the size of the OH maser shell
with the delay between the infrared and OH maser variability curve. This
yielded distances ranging from $3.4-5.4$ kpc using measurements from Bowers et
al. (1983) or $2-5.5$ kpc, using those from Herman & Habing (1985).
Alternatively, the (near) kinematic distance, for a source velocity $V_{\rm
LSR,*}=62$ km s-1, gives a distance of $4.1^{+0.2}_{-0.3}$ kpc (Reid et al.,
2014). Finally, the luminosity distance, assuming a source luminosity of
$6000$ L⊙, is $2.94\pm 0.38$ kpc (Vickers et al., 2015). In the Gaia DR3 data
release (Gaia Collaboration et al., 2021), four sources are identified within
$2.5$″of the position of the sub-millimetre continuum peak found in our ALMA
observations. The Gaia source identification number, offsets, and Gaia
parallax results are shown in Table 1 and the positions are noted in Fig. 1.
It is clear that the two first Gaia sources are not related to OH 17.7-2.0. In
fact, their positions coincide exactly with the field sources seen in the
near-infrared observations of Sánchez Contreras et al. (2007) and Gledhill et
al. (2011). The two remaining Gaia sources, correspond exactly with the
scattered light emission coming from the two outflow lobes of OH 17.7-2.0 seen
in the same observations. However, unsurprisingly, no reliable astrometry
could be performed on these two lobes. Based on these results, we adopt the
luminosity distance from Vickers et al. (2015) of $D=2.94\pm 0.38$ kpc as the
most reliable at the moment.

##  4 Observational results

We detected significant ($>3\sigma$) polarised emission towards the 222.5 GHz
continuum of OH 17.7-2.0. The maximum fraction of continuum polarisation is
$0.47\%$. This is a lower fraction than the continuum polarisation detected
with ALMA towards the post-AGB object OH 231.8+4.2 (Sabin et al., 2020) (at
$\sim 345$ GHz). We detect no circular polarisation signal in the Stokes V
continuum image, so can place a $3\sigma$ limit on the continuum circular
polarisation of $0.58\%$.

Molecular line linear polarisation is detected (at $>5\sigma$) for four of the
five strongest molecular transitions detected in our observations. The five
transitions, all detected with a peak emission $>0.1$ Jy beam-1 are presented
with their peak fluxes and maximum linear polarisation fraction (and $5\sigma$
polarisation upper limit, towards the peak of the line emission, in case of
the non-detection) in Table 2. Channel maps of the four lines for which
polarisation was detected, CO $J=2-1$, 29SiO $J=5-4$, SiO $J=5-4$, and SO
$J=5-4$ are shown in Figs. 2, 7, 8, and 9 respectively. In addition to the
linear polarisation, we also investigated if circular polarisation we detected
for any of the molecular lines. Compact, negative, Stokes V emission was
detected in four channels towards the peak of the Stokes I emission of CO
$J=2-1$. The (absolute) maximum fractional circular polarisation $P_{\rm
v,max}=0.41\%$. The other lines did not display significant Stokes V emission.
Currently, the minimum detectable degree of circular polarisation, defined as
three times the systematic calibration uncertainty, is $1.8\%$222ALMA Cycle 9
Technical handbook, compared to $0.1\%$ for linear polarisation. As the
measured level of the CO circular polarisation is below this, the circular
polarisation is likely a result of the systematic calibration uncertainties.
Still, we shortly discuss the measurement in § 5.2 in relation to possible
contributions from anisotropic resonance scattering (Houde et al., 2013,
2022).

Table 2: OH 17.7-2.0 line polarisation Molecular Line | $I_{\rm peak}$ | $P_{\rm l,max}$
---|---|---
| [Jy beam-1] | [$\%$]
12CO $J=2-1$ | 1.04 | 4.1
29SiO $v=0,J=5-4$ | 0.31 | 4.7
SiO $v=0,J=5-4$ | 0.48 | 2.9
p-H2S $(2_{2,0}-2_{1.1})$ | 0.29 | ¡0.5
SO $J=5-4$ | 0.12 | 2.2
Figure 3: A zoom-in on the continuum emission (grey scale) of OH 17.7-2.0 with
the blue line segments denoting the linear polarisation of CO $J=2-1$ for the
two channels around the systemic velocity (labelled in km s-1 in the top right
corner). The dashed red line segments indicate the linear polarisation
direction of the continuum polarisation. The black dashed line indicates the
direction of the bipolar outflow. Figure 4: A zoom-in to the central region of
the CO $J=2-1$ emission previously shown in Fig. 2. The greyscale indicates
the Stokes I emission and the blue line segments indicate the linearly
polarisation direction. The red dashed contours represent the (negative)
Stokes V emission at $-8,-6,-4\sigma$ where $\sigma=0.4$ mJy beam-1. The
outflow direction is indicated by the dashed line. The blue box indicates the
region of the 1612 MHz OH masers for which a linearly polarisation map is
presented in B03 (their figure 7). The majority of the OH masers is located
within $\sim 0.4$″of the centre of the map. The channel velocity in in km s-1
is given the top right corner of each panel.

Figure 5: The dispersion of the polarisation vectors (the square root of the
second order structure function), binned to the Nyquist sampled resolution,
for two velocity channels of the CO $J=2-1$ polarised emission around OH
17.7-2.0. The error bars indicate the variance in each bin. The vertical
dotted line indicates the size of the beam major axis. The solid line
indicates the fit of the Structure Function Analysis described in the text.

Only for the CO $J=2-1$ do we detect significant polarisation across several
consecutive velocity channels and in extended regions of more than $3$″in
size. The polarisation direction for the CO appears to curve from an angle of
$\sim 120^{\circ}$ towards the South along the outflow axis, to $\sim
35^{\circ}$ towards the North. This means it goes from being nearly
perpendicular to the outflow to being nearly parallel to the outflow. In
contrast, the 29SiO $v=0,J=5-4$ has, at blue-shifted velocities, polarisation
vectors that lie at and angle of approximately $45^{\circ}$ with respect to
the outflow axis. Around $V_{\rm lsr}=68.5$ km s-1, the detected polarisation
is perpendicular to the outflow. Polarisation of SiO $v=0,J=5-4$ is only
detected in one area, at $V_{\rm lsr}=68.3$ km s-1, towards the North on the
outflow axis, with the polarisation perpendicular to the outflow. Finally, the
SO $J=5-4$ also only displays one region of significant polarisation, at
$V_{\rm lsr}=56.1$ km s-1, with a direction similar to that of the CO at the
same velocity.

##  5 Discussion

###  5.1 Continuum polarisation

As seen in Fig. 1(right) the continuum polarisation is confined to a small
area slightly offset from the continuum peak. While the central star
contributes to the emission, the largest contribution comes from the
circumstellar dust. Considering our spatial resolution is not sufficient to
resolve details in the continuum emission, and there are no observations at
other frequencies that can be used to constrain the dust properties or the
alignment mechanism, we do not provide an in depth analysis of the dust
polarisation. As shown in e.g.Vlemmings et al. (2017), for the supergiant VY
CMa, the dust polarisation is likely due to alignment of the dust particles
with the magnetic field. In Fig. 3, we compare the continuum polarisation
direction with those of the CO $J=2-1$ emission in spectral channels close to
the systemic velocity. As seen in the comparison, the direction of the
polarisation is completely consistent within the polarisation direction
uncertainties, indicating that the origin of the polarisation in the molecular
lines and continuum is related.

###  5.2 Anisotropic resonant scattering

Anisotropic resonant scattering can affect the polarisation properties of
molecular line emission when it passes through a magnetised molecular region
in the foreground between the source and the telescope (Houde et al., 2013,
2022). It can also occur in the emitting molecular line region itself. This
process results in a transformation from linear to circular polarisation.
Consequently, the linear polarisation direction is potentially no longer
directly relatable to the magnetic field direction.

Since the emitting CO region around evolved stars is limited by CO photo-
dissociation in the outer CSE, there is unlikely to be a large column of CO
molecules in the foreground towards the source. Additionally, the large
velocity gradient in an expanding CSE, towards all but the extreme velocities,
further reduces the foreground column of gas at the emitting velocity towards
most lines of sight. Thus, anisotropic resonance scattering of the polarised
CO line emission from CSEs is likely limited to the scattering that occurs in
the emitting region itself.

If anisotropic resonance scattering affects the linearly polarised emission
detected in our observations, we expect a component of circular polarisation
in the affected regions. Although, as described in § 2, the level of circular
polarisation observed for the CO $J=2-1$ emission is below the level that is
considered significant in light of ALMA systematic circular polarisation
calibration errors, we still compare the detected circular polarisation of
$\sim 0.4\%$ to the observed linear polarisation. As can be seen in Fig. 4,
the circular polarisation is confined to the central part of the emission,
with a size of approximately one interferometric beam. The linear polarisation
vectors trace a smooth curved pattern through the region where there is a
potential detection of circular polarisation. In the same region where the
circular polarisation occurs, the direction of the continuum and CO linear
polarisation is consistent (Fig. 3), which implies that there is no extra
rotation of the polarisation vectors. We conclude that anisotropic resonant
scattering does not affect our measurements.

###  5.3 Magnetic field strength: Structure Function Analysis

Similar to what is often done in studies of magnetic fields during star
formation (e.g. Hildebrand et al., 2009; Koch et al., 2010; Houde et al.,
2016; Dall’Olio et al., 2019) we can use the CO polarisation observations to
provide an estimate of the magnetic field strength through a structure
function analysis (SFA). The SFA analysis relates the dispersion of
polarisation vectors at the smallest scales, to the turbulent motions in the
magnetised gas to provide the ratio between the turbulent and mean large scale
magnetic field strength. The SFA can thus be used to characterise turbulence.
Under the assumption that the magnetic field is frozen into the gas, that the
turbulent field arises from transverse Alfvén waves, and that the turbulence
is isotropic and incompressible, the ratio between the turbulent and mean
large scale magnetic field strength determined using a SFA analysis can also
be used to estimate the magnetic field strength.

In the following, we repeat the equations for the SFA from Koch et al. (2010).
In the SFA, the dispersion of polarisation vectors is determined using the
equation:

$\langle\Delta\chi^{2}(l_{k})\rangle^{1/2}\equiv\Biggl{\\{}\frac{1}{N(l_{k})}\sum^{N_{k}}_{l_{k}<r_{ij}<l_{k+1}}(\chi(r_{i})-\chi(r_{j}))^{2}\Biggr{\\}}^{1/2},$
(1)

where $\chi(r_{i})$ is the polarisation angle at position
$r_{i}=(x_{i},y_{i})$, $r_{ij}=\sqrt{(x_{i}-x_{j})^{2}+(y_{i}-y_{j})^{2}}$,
$l_{k}$ indicates the binning interval of the scale (taken in arcseconds), and
$N(l_{k})$ are the number of points averaged in each bin. It was shown in
Hildebrand et al. (2009) that, for a smooth magnetic field component $B_{0}$
and a small scale turbulent magnetic field component $B_{t}$, the structure
function, on scales between the smallest turbulent scale and the
characteristic length scale for variations in the large scale component, can
be described with:

$\langle\Delta\chi^{2}(l_{k})\rangle\simeq
b^{2}+m^{2}l_{k}^{2}+\sigma^{2}_{M}(l_{k}).$ (2)

Here $m$ is the slope of the linear dispersion term of the large scale field
$B_{0}$ and $\sigma_{M}(l_{k})$ propagates the observational uncertainties on
the polarisation vectors in the binning. The term $b$ is then related to the
ratio between the turbulent and large scale magnetic field strengths through:

$\frac{\langle B_{t}^{2}\rangle^{1/2}}{B_{0}}=\frac{b}{\sqrt{2-b^{2}}}.$ (3)

Under the assumptions introduced previously, the ratio between the turbulent
and large scale magnetic field component is equal to the ratio between the
turbulent line width $\sigma_{\nu}$ and the Alfvén velocity
$\sigma_{A}=\frac{B_{0}}{\sqrt{4\pi\rho}}$, with $\rho$ the density of the
gas. Using the turbulent velocity and average density in the CO emitting
region, allows us to determine the magnetic field strength in the plane of the
sky. This strength will be a lower limit in the case the magnetic field is not
fully frozen into the gas.

The result of the SFA for two spectral channels, with the CO gas at $V_{\rm
LSR}=57.3$ km s-1and $62.4$ km s-1around OH 17.7-2.0 are shown in Fig. 5. As
expected, the dispersion of the polarisation vectors increase from small to
larger scale. For the channel close to the stellar velocity, the dispersion
subsequently decreases steeply due to the aligned polarised emission at larger
radius towards the North-East. The structure function is fit using Eq. 3
within $\lesssim 2.3$″(three synthesised beams), which we take to correspond
to a characteristic length scale for variations in the large scale magnetic
field component. We note that for both velocity channels, the polarisation
vector dispersion at the smallest scales available in our observations is
$\sim 20^{\circ}$. The analysis provides a ratio between the turbulent and
large scale magnetic field component $\frac{\langle
B_{t}^{2}\rangle^{1/2}}{B_{0}}=0.10\pm 0.01$, and $0.14\pm 0.01$ for the two
channels respectively. This means that the large scale magnetic field strength
for both channels is:

$B_{0}=[2.3,1.6]\leavevmode\nobreak\ \Biggl{(}\frac{\langle
n_{H_{2}}\rangle}{10^{5}}\Biggr{)}^{1/2}\frac{v_{t}}{1.0}\leavevmode\nobreak\
{\rm mG}.$ (4)

Here, $v_{t}$ is the turbulent velocity in the CO gas, taken to be $1.0$ km
s-1derived from CO radiative transfer modelling of AGB envelopes (e.g.
Vlemmings et al., 2021). From similar models, the average H2 number density
$n_{H_{2}}$ in the CO region is assumed to be $10^{5}$ cm-3. These
assumptions, as well as the assumption that the magnetic field is frozen into
the gas, dominate the uncertainty in the magnetic field strength. Similar
results are obtained for the other channels for which a SFA could be
performed. The polarised emission of the other molecular lines is limited to
compact regions not much larger than our interferometric beam and hence a SFA
was not possible.

In Houde et al. (2016) it is shown that a combination of the resolution of the
observations and interferometric spatial filtering affect the results from the
SFA. In §2 we showed that the maximum recoverable scale of our observations
($\sim 8.1$″) is sufficiently large that spatial filtering does not affect our
analysis. In order to determine the effect of the beamsize, we estimate the
number of independent turbulent cells ($N$) probed by our observations using
the formula from Houde et al. (2016):

$N=\frac{(\delta^{2}+2W_{1}^{2})\Delta^{\prime}}{\sqrt{2\pi}\delta^{2}}.$ (5)

Here $\delta$ is the correlation length of the turbulent field, $W_{1}$ is
radius of the interferometric beam (we use $1/\sqrt{8\ln{2}}$ times an average
fwhm beam-size of $0.72$″) and $\Delta^{\prime}$ is the depth of the observed
molecular layer along the line of sight. For a source at $D=2.94$ kpc, our
beam radius $W_{1}\sim 900$ au. We estimate the depth $\Delta^{\prime}$ from
the total size of the CO envelope ($\sim 18000$ au) divided by the number of
channels, yielding $\Delta^{\prime}\sim 1800$ au (although considering the
sperically expanding CSE this value actually various across the envelope).
Finally, we estimate the correlation length $\delta$ to be similar to the size
of the molecular clumps that make up the envelope. Richards et al. (2012)
derived, from H2O maser measurements, that these clumps start out with a size
of the order of the stellar radius and expand for increasing distance to the
star. Combined with CO observations and models (e.g. Olofsson et al., 1996),
Richards et al. (2012) suggest that the clump size $r_{c}$ appears to scale
with distance from the star $r$ as $rc\propto r^{0}.8$. Applying these
estimates to the CO $J=2-1$ gas around OH 17.7-2.0 yields a size of $\sim 850$
au. Hence, taking $\delta\sim 850$ au, $W_{1}\sim 900$ au and
$\Delta^{\prime}\sim 1800$ au we find $N\sim 2.7$. As the ratio between the
turbulent and large scale magnetic field strengths scales approximately with
$\sqrt{N}$, our magnetic field estimate would be overestimated by a factor of
$\sim 1.7$. Considering the various uncertainties in all the different
assumptions, we conclude that the order of magnitude strength of the plane of
the sky component of the magnetic field that permeates the CO emitting region
is $B_{\perp}\sim 1$ mG.

We can compare these results with the OH maser Zeeman measurements from B03.
The Zeeman measurements of the paramagnetic OH molecule provide an estimate of
the total magnetic field strength of $B=2.5-4.6$ mG. Since our measurement
estimates the plane of the sky component of the magnetic field, the CO results
are fully consistent with the OH Zeeman measurements.

###  5.4 Comparison with OH linear polarisation results

As previously indicated, a large scale magnetic field was mapped around OH
17.7-2.0 using OH maser observations (B03). This allows for a direct
comparison between the magnetic field traced in individual 1612 MHz OH maser
clumps and that in the more diffuse CO gas. The area covered by the OH masers
is shown in a zoom-in of the three central velocity channels of our CO
observations in Fig. 4. First, the maser positions where aligned with the CO
observations by assuming that the centre of the maser shell fitted in B03
coincides with the peak of the continuum emission from our observations.
Subsequently, each maser spot in table 4 of B03 was identified with the
nearest $0.1$″pixel and nearest velocity channel in our observations. For
those pixels that contained multiple maser spots, a weighted average OH maser
polarisation angle was calculated. Subsequently, the mean angle was subtracted
from both the OH polarisation angle ($\langle\chi_{\rm
OH}\rangle=70.7^{\circ}$) and CO polarisation angle ($\langle\chi_{\rm
CO}\rangle=81.6^{\circ}$) distributions. This was done because the OH maser
polarisation angle is likely significantly affected by Faraday rotation (see
below). The absolute angle difference $|\Delta\chi|=|(\chi_{\rm
OH}-\langle\chi_{\rm OH}\rangle)-(\chi_{\rm CO}-\langle\chi_{\rm CO}\rangle)|$
is shown in Fig. 6.

Figure 6: The distribution of the absolute angle difference between the 1612
MHz OH masers (B03) and the CO polarisation direction. The mean difference,
which can be affected by Faraday rotation, has been removed (see text). The
solid black line indicates a normal distribution with $\sigma=22.5^{\circ}$,
which corresponds to the best fit normal distribution. The dashed line
represents a uniform distribution of angle differences.

We find that the angle difference distribution can be best described by a
normal distribution with a standard deviation $\sigma=22.5^{\circ}$. This is
similar to the dispersion angle of the CO at the smallest scales as found in
the SFA analysis in § 5.3. Hence, considering the similarity of the angle
difference distribution with the dispersion in the angle expected to occur in
both the OH masing and CO gas as a result of turbulence, we can confidently
state that the magnetic field traced by the more diffuse CO gas and the OH
maser clumps behaves similarly.

###  5.5 Magnetic field morphology

The OH maser observations of B03 indicated that a large scale magnetic field
is present around OH 17.7-2.0. The OH polarisation was attributed to emission
from elliptically polarised $\sigma$-components that are the result of Zeeman
splitting. This means that the polarisation vectors are perpendicular to the
magnetic field direction. However, because foreground Faraday rotation is
significant at OH maser frequencies, it was not possible to relate the
polarisation direction with for example the outflow direction or inferred
toroidal structure. Because the CO observations correspond to higher frequency
emission ($\sim 230$ GHz) compared to the OH maser observations at 1612 MHz,
foreground Faraday rotation will be significantly less for our CO
observations. We can estimate the Faraday rotation by using the best distance
estimate ($2.94$ kpc, see § 3), a typical value for the interstellar magnetic
field of $2\leavevmode\nobreak\ \mu$G (Sun et al., 2008) and an average
electron number density $n_{e}=0.01066$ cm-3 on the line of sight to OH
17.7-2.0 (Yao et al., 2017). For the 1612 MHz OH masers, this implies a
Faraday rotation of $\Delta\Phi=102^{\circ}$ while for CO the rotation would
be negligible ($<0.005^{\circ}$). Comparing this with the difference in mean
angle between the OH and CO polarisation vectors found in § 5.4 of
$\sim-11^{\circ}$ this would imply an intrinsic angle difference of $\sim
91^{\circ}$. This is remarkably close to perpendicular and strongly implies a
relation between the polarisation measured on the masers and in the CO
envelope. Considering the OH polarisation is perpendicular to the magnetic
field, the CO polarisation is parallel to the magnetic field. Thus, the
curvature in the CO polarisation from being perpendicular to the outflow to
being nearly parallel to the outflow implies that we are tracing a dominant
toroidal magnetic field component towards the South of the continuum peak and
a helical or nearly poloidal field component towards the North part of the
outflow. This is likely the result of the inclination of the outflow, which
was estimated to be $\sim 22^{\circ}$ with the blue-shifted outflow in the
North and the red-shifted outflow in the South (Gledhill et al., 2011).
Towards the South we thus mostly probe gas in front of the outflow cavity,
while in the North we probe part of the outflow cavity itself. Such a
morphology is similar to that of magnetically driven outflow models such as a
magnetic tower jet (e.g. Huarte-Espinosa et al., 2012), a configuration that
was also inferred for the post-AGB object OH231.8+4.2 (Sabin et al., 2020). As
already noted in B03, the magnetic energy dominates the mechanical energy in
the regions probed by the OH masers and considering the similar field strength
estimated from the CO observations, the same holds true for the gas probed by
the CO emission.

##  6 Conclusions

We have observed, using ALMA, molecular line polarisation in the CSE of the
post-AGB star OH 17.7-2.0. The observations of the polarisation arising from
the GK-effect, has allowed us to determine the magnetic field strength in the
CO emitting region using a Structure Function Analysis. The strength of the
magnetic field component in the plane of the sky was found to be $\sim 1$ mG,
which is consistent with previous Zeeman measurements using OH masers (B03). A
comparison between the OH maser and CO linear polarisation vectors indicates
that, although the OH maser polarisation direction is strongly affected by
foreground Faraday rotation, the magnetic field in the OH masers and in the CO
gas is likely the same. This confirms that the magnetic field properties,
except for the absolute magnetic field direction on the sky, derived from OH
masers are representative of the large scale circumstellar magnetic field. The
magnetic field structure derived from the ALMA CO observations is similar to
that expected for a magnetically driven outflow. As previously noted from the
OH Zeeman observations, the magnetic energy also dominated the energy budget
in the envelope. More detailed observations of the outflow launching region
around OH 17.7-2.0 are needed to firmly identify the mechanism responsible for
the structures seen around this post-AGB star. The observations presented here
show that molecular line polarisation observations, of both maser and non-
maser species, are invaluable to determine the role of magnetic fields during
the late-stages of stellar evolution.

###### Acknowledgements.

We thank the referee, Martin Houde, for comments that improved the paper. This
paper makes use of the following ALMA data: ADS/JAO.ALMA#2016.1.00251.S. ALMA
is a partnership of ESO (representing its member states), NSF (USA) and NINS
(Japan), together with NRC (Canada), NSC and ASIAA (Taiwan), and KASI
(Republic of Korea), in cooperation with the Republic of Chile. The Joint ALMA
Observatory is operated by ESO, AUI/NRAO and NAOJ. The project leading to this
publication has received support from ORP, that is funded by the European
Union’s Horizon 2020 research and innovation programme under grant agreement
No 101004719 [ORP]. We also acknowledge support from the Nordic ALMA Regional
Centre (ARC) node based at Onsala Space Observatory. The Nordic ARC node is
funded through Swedish Research Council grant No 2017-00648.

## References

  * Bains et al. (2003) Bains, I., Gledhill, T. M., Yates, J. A., et al. 2003, MNRAS, 338, 287 (B03)
  * Beuther et al. (2010) Beuther, H., Vlemmings, W. H. T., Rao, R., et al. 2010, ApJ, 724, L113
  * Blackman (2022) Blackman, E. G. 2022, arXiv:2202.07246
  * Boffin & Jones (2019) Boffin, H. M. J. & Jones, D. 2019, The Importance of Binaries in the Formation and Evolution of Planetary Nebulae, SpringerBriefs in Astronomy. ISBN 978-3-030-25058-4. The Author(s), under exclusive license to Springer Nature Switzerland AG, 2019
  * Bowers (1978) Bowers, P. F. 1978, A&AS, 31, 127
  * Bowers et al. (1983) Bowers, P. F., Johnston, K. J., & Spencer, J. H. 1983, ApJ, 274, 733
  * Cortés et al. (2005) Cortés, P. C., Crutcher, R. M., & Watson, W. D. 2005, ApJ, 628, 780
  * Cortés et al. (2021) Cortés, P. C., Sanhueza, P., Houde, M., et al. 2021, ApJ, 923, 204
  * Dall’Olio et al. (2019) Dall’Olio, D., Vlemmings, W. H. T., Persson, M. V., et al. 2019, A&A, 626, A36
  * Engels (2002) Engels, D. 2002, A&A, 388, 252
  * Gaia Collaboration et al. (2021) Gaia Collaboration, Brown, A. G. A., Vallenari, A., et al. 2021, A&A, 649, A1
  * Girart et al. (2012) Girart, J. M., Patel, N., Vlemmings, W. H. T., et al. 2012, ApJ, 751, L20
  * Gledhill et al. (2011) Gledhill, T. M., Forde, K. P., Lowe, K. T. E., et al. 2011, MNRAS, 411, 1453
  * Goldreich & Kylafis (1982) Goldreich, P. & Kylafis, N. D. 1982, ApJ, 253, 606
  * Gonidakis et al. (2014) Gonidakis, I., Chapman, J. M., Deacon, R. M., et al. 2014, MNRAS, 443, 3819
  * Herman & Habing (1985) Herman, J. & Habing, H. J. 1985, A&AS, 59, 523
  * Herpin et al. (2006) Herpin, F., Baudry, A., Thum, C., et al. 2006, A&A, 450, 667
  * Heske et al. (1990) Heske, A., Forveille, T., Omont, A., et al. 1990, A&A, 239, 173
  * Hildebrand et al. (2009) Hildebrand, R. H., Kirby, L., Dotson, J. L., et al. 2009, ApJ, 696, 567
  * Houde et al. (2013) Houde, M., Hezareh, T., Jones, S., et al. 2013, ApJ, 764, 24
  * Houde et al. (2016) Houde, M., Hull, C. L. H., Plambeck, R. L., et al. 2016, ApJ, 820, 38
  * Houde et al. (2022) Houde, M., Lankhaar, B., Rajabi, F., et al. 2022, MNRAS, 511, 295
  * Huang et al. (2020) Huang, K.-Y., Kemball, A. J., Vlemmings, W. H. T., et al. 2020, ApJ, 899, 152
  * Huarte-Espinosa et al. (2012) Huarte-Espinosa, M., Frank, A., Blackman, E. G., et al. 2012, ApJ, 757, 66
  * Koch et al. (2010) Koch, P. M., Tang, Y.-W., & Ho, P. T. P. 2010, ApJ, 721, 815
  * Khouri et al. (2021) Khouri, T., Vlemmings, W. H. T., Tafoya, D., et al. 2021, Nature Astronomy, 6, 275
  * Lagadec et al. (2011) Lagadec, E., Verhoelst, T., Mékarnia, D., et al. 2011, MNRAS, 417, 32
  * Lankhaar & Vlemmings (2020) Lankhaar, B. & Vlemmings, W. 2020, A&A, 636, A14
  * Le Bertre et al. (1984) Le Bertre, T., Epchtein, N., & Nguyen-Q-Rieu 1984, A&A, 138, 353
  * Leal-Ferreira et al. (2013) Leal-Ferreira, M. L., Vlemmings, W. H. T., Kemball, A., et al. 2013, A&A, 554, A134
  * Lèbre et al. (2014) Lèbre, A., Aurière, M., Fabas, N., et al. 2014, A&A, 561, A85
  * Li & Henning (2011) Li, H.-B. & Henning, T. 2011, Nature, 479, 499
  * McMullin et al. (2007) McMullin, J. P., Waters, B., Schiebel, D., Young, W., & Golap, K. 2007, in Astronomical Society of the Pacific Conference Series, Vol. 376, Astronomical Data Analysis Software and Systems XVI, ed. R. A. Shaw, F. Hill, & D. J. Bell, 127
  * Murakawa et al. (2013) Murakawa, K., Izumiura, H., Oudmaijer, R. D., et al. 2013, MNRAS, 430, 3112
  * Nagai et al. (2016) Nagai, H., Nakanishi, K., Paladino, R., et al. 2016, ApJ, 824, 132
  * Nordhaus & Blackman (2006) Nordhaus, J. & Blackman, E. G. 2006, MNRAS, 370, 2004
  * Olofsson et al. (1996) Olofsson, H., Bergman, P., Eriksson, K., et al. 1996, A&A, 311, 587
  * Ondratschek et al. (2022) Ondratschek, P. A., Röpke, F. K., Schneider, F. R. N., et al. 2022, A&A, 660, L8.
  * Perez-Sanchez et al. (2013) Perez-Sanchez, A. F., Vlemmings, W. H. T., Tafoya, D., et al. 2013, MNRAS, 436, L79
  * Reid et al. (2014) Reid, M. J., Menten, K. M., Brunthaler, A., et al. 2014, ApJ, 783, 130
  * Richards et al. (2012) Richards, A. M. S., Etoka, S., Gray, M. D., et al. 2012, A&A, 546, A16
  * Sabin et al. (2015) Sabin, L., Wade, G. A., & Lèbre, A. 2015, MNRAS, 446, 1988
  * Sabin et al. (2020) Sabin, L., Sahai, R., Vlemmings, W. H. T., et al. 2020, MNRAS, 495, 4297
  * Sánchez Contreras et al. (2007) Sánchez Contreras, C., Le Mignant, D., Sahai, R., et al. 2007, ApJ, 656, 1150
  * Sánchez Contreras & Sahai (2012) Sánchez Contreras, C. & Sahai, R. 2012, ApJS, 203, 16
  * Soker (2002) Soker, N. 2002, MNRAS, 336, 826
  * Stephens et al. (2020) Stephens, I. W., Fernández-López, M., Li, Z.-Y., et al. 2020, ApJ, 901, 71
  * Sun et al. (2008) Sun, X. H., Reich, W., Waelkens, A., et al. 2008, A&A, 477, 573
  * Teague et al. (2021) Teague, R., Hull, C. L. H., Guilloteau, S., et al. 2021, ApJ, 922, 139
  * Vickers et al. (2015) Vickers, S. B., Frew, D. J., Parker, Q. A., et al. 2015, MNRAS, 447, 1673
  * Vlemmings et al. (2002) Vlemmings, W. H. T., Diamond, P. J., & van Langevelde, H. J. 2002, A&A, 394, 589
  * Vlemmings et al. (2005) Vlemmings, W. H. T., van Langevelde, H. J., & Diamond, P. J. 2005, A&A, 434, 1029
  * Vlemmings et al. (2006) Vlemmings, W. H. T., Diamond, P. J., & Imai, H. 2006, Nature, 440, 58
  * Vlemmings et al. (2012) Vlemmings, W. H. T., Ramstedt, S., Rao, R., et al. 2012, A&A, 540, L3
  * Vlemmings (2019) Vlemmings, W. 2019, IAU Symposium, 343, 19
  * Vlemmings et al. (2017) Vlemmings, W. H. T., Khouri, T., Martí-Vidal, I., et al. 2017, A&A, 603, A92
  * Vlemmings et al. (2021) Vlemmings, W. H. T., Khouri, T., & Tafoya, D. 2021, A&A, 654, A18
  * Wolak et al. (2013) Wolak, P., Szymczak, M., Bartkiewicz, A., et al. 2013, The Astronomer’s Telegram, 5211
  * Yao et al. (2017) Yao, J. M., Manchester, R. N., & Wang, N. 2017, ApJ, 835, 29

##  Appendix A Polarisation maps for three further molecules

Figure 7: Same as Fig. 2 for the 29SiO $v=0,J=5-4$ emission around OH
17.7-2.0. The peak emission is $I_{\rm 29SiO,peak}$=0.31 Jy beam-1. The
maximum polarisation fraction $P_{\rm l,max}=4.7\%$. Figure 8: Same as Fig. 2
for the SiO $v=0,J=5-4$ emission around OH 17.7-2.0. The peak emission is
$I_{\rm SiO,peak}$=0.49 Jy beam-1 and the maximum polarisation fraction
$P_{\rm l,max}=2.9\%$. Figure 9: Same as Fig. 2 for the SO $J=5-4$ emission
around OH 17.7-2.0. The peak emission is $I_{\rm SO,peak}$=0.12 Jy beam-1 and
the maximum polarisation fraction $P_{\rm l,max}=2.2\%$.


Further author information: (Send correspondence to C.F.)
C.F.: E-mail<EMAIL_ADDRESS>
J.J.: E-mail<EMAIL_ADDRESS>
# Distributed Transmission Control for Wireless Networks using Multi-Agent
Reinforcement Learning

Collin Farquhar  Marconi-Rosenblatt AI/ML Innovation Lab, ANDRO Computational
Solutions,
7980 Turin Rd, Beeches Technical Campus Building 1, Rome, NY, USA  Prem Sagar
Pattanshetty Vasanth Kumar  Marconi-Rosenblatt AI/ML Innovation Lab, ANDRO
Computational Solutions,
7980 Turin Rd, Beeches Technical Campus Building 1, Rome, NY, USA  Anu
Jagannath  Marconi-Rosenblatt AI/ML Innovation Lab, ANDRO Computational
Solutions,
7980 Turin Rd, Beeches Technical Campus Building 1, Rome, NY, USA  Jithin
Jagannath  Marconi-Rosenblatt AI/ML Innovation Lab, ANDRO Computational
Solutions,
7980 Turin Rd, Beeches Technical Campus Building 1, Rome, NY, USA

###### Abstract

We examine the problem of transmission control, i.e., when to transmit, in
distributed wireless communications networks through the lens of multi-agent
reinforcement learning. Most other works using reinforcement learning to
control or schedule transmissions use some centralized control mechanism,
whereas our approach is fully distributed. Each transmitter node is an
independent reinforcement learning agent and does not have direct knowledge of
the actions taken by other agents. We consider the case where only a subset of
agents can successfully transmit at a time, so each agent must learn to act
cooperatively with other agents. An agent may decide to transmit a certain
number of steps into the future, but this decision is not communicated to the
other agents, so it the task of the individual agents to attempt to transmit
at appropriate times. We achieve this collaborative behavior through studying
the effects of different actions spaces. We are agnostic to the physical
layer, which makes our approach applicable to many types of networks. We
submit that approaches similar to ours may be useful in other domains that use
multi-agent reinforcement learning with independent agents.

###### keywords:

Reinforcement learning, machine learning, multi-agent, communications,
scheduling, MAC, protocol

##  1 INTRODUCTION

Modern communication environments can be large, complex, and dynamic, and thus
can pose a problem for traditional algorithms that are often designed for a
particular goal or environment, and may have difficulty scaling and adapting
to new circumstances. Additionally, many traditional approaches rely on a
centralized controller, and without such a controller may fail to find optimal
performance if a distributed approach is employed. Machine learning and
reinforcement learning approaches have been proposed as a promising
alternative as they may learn to find solutions even in for extremely
challenging complex and dynamic environments [1]. Herein, we present a Multi-
Agent Reinforcement Learning (MARL) approach applied to the problem of
distributed transmissions control.

Our distributed approach uses individual agents to control the transmission of
devices (if and when to transmit), providing a solution to transmission
control and medium access control tasks where a centralized control mechanism
is infeasible or disadvantageous. Distributed approaches have the advantage of
limiting communication overhead and may be more extensible in terms of the
number of devices and spacial range, as a centralized bottleneck is avoided
and devices do not need to be near a centralized controller. Additionally, in
some systems, especially large or sensitive systems, a centralized approach
may not be feasible in terms of security and robustness. For example, an
attack on a centralized mechanism may cause the entire network to fail.
Communication settings where this distributed approach may be advantageous
include tactical ad hoc networks, sensor networks, device-to-device
communications, and various other internet of things systems.

Moving away from centralized approaches poses some technical challenges. In
general, network-level planning is easier with a centralized mechanism. From a
reinforcement learning (RL) perspective, if centralized mechanism controls
each node’s actions and has complete access to all the nodes’ observations and
rewards, then the problem reduces to a classical Markov Decision Process (MDP)
and single-agent solutions can be used [2].

We consider the setting with $n$ transmitters represented as individual
reinforcement learning agents. The environment is such that only a limited
number of the agents are able to successfully transmit concurrently due to the
friendly interference caused by transmitting. It is then the task of the
agents to learn when to transmit. We devise a general form for the action
space and analyze the effects of different actions spaces sizes on the
performance in terms of network throughput and fairness. By tailoring the
action space to the environment we see that the agents are able to learn
cooperative transmission patterns that lead to efficient and fair performance.
Based on our analysis, we choose action spaces that perform well and compare
them against CSMA-style benchmarks. We find that our RL agents often
outperform the benchmarks in terms of throughput and fairness.

We test our approach in a custom simulation environment based on OpenAI Gym
[3]. Our environment is open source and is freely available for download on
GitHub with a link provided in Appendix A.

##  2 Related Work

A similar objective to ours is explored in Ref. 4 from the perspective of
domain-agnostic cooperative RL. The work proposes a method for optimizing the
trade-off between fairness and efficiency in MARL, which is analogous to our
desire to optimize optimize fairness and throughput. However, the agents are
allowed to share information with other local agents. In the context of
communications systems, however, this information sharing may induce unwanted
overhead, and in our approach we do not rely on information sharing.

An adjacent area or research involves learning to communicate for RL agents —
that is, learning how to share information so that agents can cooperatively
achieve some goal. [5, 6]. While this goal may not be related to
communications systems per se, there is an inherent scheduling component for
the information sharing which is similar to our task. In particular, Kim et
al. [6] considers an environment with bandwidth and medium access constraints.
While these works use distributed execution, they rely on a centralized
training mechanism. In contrast, in our work we focus on optimizing
communications systems using fully distributed MARL.

Ref. 7 investigates how a deep reinforcement learning agent can learn a Medium
Access Control (MAC) strategy to coexist with other networks that use
traditional (not machine learning–based) MAC algorithms. In some ways, it may
be easier to learn to coexist with a traditional MAC algorithms, as the
algorithms may have a certain structures, regularities, or periodicities that
can be learned by the agent. In this paper we focus purely on multi-agent
scheduling, where all agents have the same action spaces and it is difficult
to predict a priori what patterns may emerge.

Despite not using a machine learning approach, perhaps the closest to this
paper is the recent work of Ref. 8 which proposes an algorithm for
distributed, fair MAC scheduling and additionally provides theoretical
guarantees for service disparity.

##  3 Problem Formulation and Approach

We consider scenarios with $n$ agents acting as transmitting devices (nodes in
a wireless network). The transmission medium is limited such that only a
threshold number, $k$, of agents can successfully transmit at the same time.
The threshold is uniform for all agents: if more than $k$ agents transmit on
the same step, then all of the transmissions are unsuccessful at the
respective receivers. Given this threshold behavior, we are agnostic to the
PHY-layer. With $k>1$ we can model an abstracted CDMA PHY-layer where $k$ can
be interpreted as the spreading factor. For $k=1$ we can model many other
systems that use an OFDM-based PHY-layer.

The agents may only transmit once on a given step. We leave the definition of
a transmission abstracted so that a transmission could represent either the
transmission of a single packet, multiple packers, or some number of bits.

Figure 1:  Depiction of the flow of information between our software during
the reinforcement learning process. The threshold_env inherits from gym.Env,
the base class for OpenAI Gym, and serves as the environment in which the
agents act. In agent.py the agents are defined using a DQN model. State,
reward, and action information ($S_{t}$, $R_{t}$, and $A_{t}$) are passed
between the environment and the agents. Since this is a multi-agent setting,
$S_{t}$, $R_{t}$, and $A_{t}$ are lists containing the respective information
for each agent. E.g., $S_{t}=[s_{t}^{(1)},s_{t}^{(2)},s_{t}^{(3)},...]$ is a
list of the states for each agent at time $t$. Figure adapted from Ref. 9.

###  3.1 Simulation Software

We provide a software platform to model the interaction of agents in a
threshold-limited communication setting. A link to view or download the code
can be found in Appendix A. There are two primary components to the software,
the first is agent.py where the agents’ RL models are defined and used to
select actions, and the second is the simulation environment, called
threshold_env, which is built on top of OpenAI Gym and inherits from the
gym.Env base class. The primary functionality is implemented in the step
method, which advances the simulation. The step method takes a list of actions
(one action per agent) as input. The actions are performed in the environment
and the agents move into a new state and receive some reward. The information
regarding the new states and the reward is then returned by the step method to
be used by the agents’ RL models in the agent.py file. The information flow
between our software is depicted in Figure 1.

While we use our custom environment in this paper as it is suits our desired
level of abstraction, our approach can easily be tested using other dedicated
wireless communication simulators if one wanted to more finely model the
particularities a specific communication system. Internally, we have tested
our approach with ns3-gym [10] and found the results to be consistent with our
custom environment.

###  3.2 RL Background

A general reinforcement learning problem is formulated as a Markov Decision
Process (MDP), which may be represented by the tuple
$(\mathbb{S},\mathbb{A},P,R)$ where $\mathbb{S}$ is the set of states an agent
may encounter, $\mathbb{A}$ is the set of actions an agent can take,
$P(s_{t+1}|s_{t},a_{t})$ is the state transition function which follows the
Markov Property and gives the probability that the agent’s next state is
$s_{t+1}$ given that the agent is in state $s_{t}$ and takes action $a_{t}$,
and $R$ is the reward function which outputs a real number. It is then the
task of an agent to maximize it’s long-term reward over many steps.

Our environment can be classified as a multi-agent, Partially Observable
Markov Decision Process (POMDP). Each agent is taking actions regarding if and
when to transmit. The actions taken by an agent are not directly communicated
to the other agents in the environment. However, the other agents can
indirectly observe the effects of the these actions by measuring the overall
level of interference. In the following sections, we define the state space,
action space, reward function, and the RL model.

###  3.3 State Space

An agent’s observation of their state consists of four features (t, s, i, b)
which are described below.

  * •

Whether the agent transmitted, $t\in\\{0,1\\}$

  * •

Whether the transmission was successful, $s\in\\{0,1\\}$

  * •

Interference-sensed (normalized), $i\in[0,1]$

  * •

buffer size (normalized), $b\in[0,1]$

If an agent did not transmit then $s$ is automatically set to $0$. The
interference-sensed feature is normalized by dividing the number of
transmissions from other agents by the number of other agents. We constrain
the interference-sensed feature, $i$, such that an agent cannot transmit and
sense on the same step, as in a realistic system its transmission would
dominate any local sensing mechanism. If an agent transmits, then $i$ is set
to $0$. Buffer size can be thought of as representing how many packets are
waiting to be transmitted by an agent. The buffer size feature, $b$, is
normalized by setting a maximum buffer size.

###  3.4 Action Space

We provide a structure for the action space. In this work, we test actions
spaces of different sizes, but they all follow the same form. Effectively, the
agent has the choices: do not transmit on the next step, transmit on the next
step, or wait for certain number of steps and then transmit. The encoding for
a selected action $a_{i}$ is shown below.

  * •

$a_{0}$ $\rightarrow$ Do not transmit

  * •

$a_{1}$ $\rightarrow$ Transmit on the next step

  * •

$a_{2}$ $\rightarrow$ Transmit after one step

  * •

$a_{3}$ $\rightarrow$ Transmit after two steps

  * •

$a_{4}$ $\rightarrow$ Transmit after three steps

  * •

…

A transmission lasts for one simulation-step. If an agent’s buffer is zero,
then the RL model is not used and the agent takes no action as there are no
packets to be transmitted, and thus no decision to be made. Since we consider
multi-agent settings, the agents will be making decisions asynchronously as
some actions last over multiple steps. In section 4.1 we analyze the
performance of different action space sizes. An action space of size $m$,
($|\mathbb{A}|=m$), is defined as $\mathbb{A}=\\{a_{i}\\}_{i=0}^{m-1}$.

###  3.5 Reward Function

The reward for function for single-step actions is shown below.

$R=\begin{cases}1&\text{successful transmission}\\\ -1&\text{unsuccessful
transmission}\\\ 0&\text{no transmission}\\\ \end{cases}$ (1)

For multi-step actions, the single-step rewards are averaged over the number
of steps the action acts over.

###  3.6 RL Model

We use the vanilla Deep Q-Network (DQN) algorithm as the RL model for each
agent. At a high level, the algorithm consists of a neural network that takes
an agent’s state observation as input and outputs an estimate of the Q-value
for each possible action. The Q-value represents the long-term sum of rewards
an agent will receive if they take a particular action in a particular state
and then from that point on continue to act optimally with respect to the
Q-value estimates.

Unless the agent selects an action randomly according to it’s exploration
mechanism, the agent will select the action $a_{t}$ that corresponds to the
maximum Q-value. After taking the step in the environment where the agent
performs the action, the transition $(s_{t},a_{t},r_{t},s_{t+1})$ is saved to
the agent’s experience replay memory (assuming a single-step action). The
experience replay memory is sampled during training. If an agent selects an
action that extends over multiple steps, the state at the time of the
decision, $s_{t}$ is saved, and the per-step reward is averaged over the
duration of the action. For example, if an action taken at time $t$ extends
over $n$ steps, then let $\tilde{r}=\frac{1}{n}\sum_{i=t}^{t+n-1}r_{i}$, and
the transition that is saved is $(s_{t},a_{t},\tilde{r},s_{t+n})$.

If an agent’s buffer size is zero, then the RL model is not used since there
are no packets to send and no decision to make; i.e., the agent takes no
action and no transition is saved to the agent’s memory.

The DQN hyperparameters are described in Appendix B and the code is provided
in the repository linked to in Appendix A.

##  4 Experiments and Results

In this section we describe a number of experiments and analyze the agents’
performance. We start in Sec. 4.1 by investigating the effect of the action
space on throughput and fairness for different number of agents and different
threshold values for the maximum number of concurrent successful
transmissions, $k$. Based on these results, we provide an action space
recommendation that we use for the remaining experiments where we compare
agent performance versus CSMA-inspired benchmarks in Sec. 4.2. Lastly, in Sec.
4.3, we examine the how the advantage the RL agents provide relates to an
agent’s ability to manage its buffer size for different buffer rates (source
rates). All experiments are run 3 times for 10,000 steps and the results shown
are averages over the 3 runs.

Throughput is calculated as the number of successful transmissions per step.
Since an agent may only transmit one packet per step, the maximum possible
network throughput is $k$. For greater clarity, the throughput data are
smoothed over $100$ steps. The plotted value at time step $t$ is the average
over the $100$ values preceding time step $t$. The fairness at time $t$ is
calculated using Jain’s fairness index as

$F_{t}=\frac{(\sum_{i=1}^{n}x_{t}^{(i)})^{2}}{n\sum_{i=1}^{n}(x_{t}^{(i)})^{2}},$
(2)

where $n$ is the number of agents and $x_{t}^{(i)}$ is the smoothed throughput
for agent $i$ at time $t$. A value of 1 represents maximum fairness.

###  4.1 Action Space Analysis

We test a range of action space sizes for $2$, $4$, and $10$ agents. We also
vary the threshold, $k$, for additional tests with $4$ and $10$ agents. The
buffer rate is such that the buffer is incremented on every step and is
uniform for all agents, so the agents always have packets to transmit.

| Throughput, $k=2$ | Fairness, $k=2$
---|---|---
No. of Actions |  4 Agents |  10 Agents |  4 Agents |  10 Agents
2 Actions |  Mean = 1.23383 STDEV = 0.05380 |  Mean = 1.13013 STDEV = 0.09343 |  Mean = 0.79449 STDEV = 0.01814 |  Mean = 0.38081 STDEV = 0.04988
3 Actions |  Mean = 1.84297 STDEV = 0.00767 |  Mean = 1.05710 STDEV = 0.05481 |  Mean = 0.99914 STDEV = 0.00010 |  Mean = 0.53656 STDEV = 0.01878
4 Actions |  Mean = 1.29135 STDEV = 0.02454 |  Mean = 1.00863 STDEV = 0.06923 |  Mean = 0.99897 STDEV = 0.00073 |  Mean = 0.71488 STDEV = 0.02333
5 Actions |  Mean = 1.14423 STDEV = 0.26474 |  Mean = 1.04642 STDEV = 0.06590 |  Mean = 0.99289 STDEV = 0.00988 |  Mean = 0.70399 STDEV = 0.16237
6 Actions |  Mean = 0.90587 STDEV = 0.07312 |  Mean = 1.05044 STDEV = 0.19130 |  Mean = 0.98259 STDEV = 0.01012 |  Mean = 0.89317 STDEV = 0.04956
7 Actions |  Mean = 0.86968 STDEV = 0.05590 |  Mean = 1.21009 STDEV = 0.17026 |  Mean = 0.96761 STDEV = 0.02630 |  Mean = 0.91798 STDEV = 0.02299
8 Actions |  Mean = 0.86249 STDEV = 0.06013 |  Mean = 1.10272 STDEV = 0.23757 |  Mean = 0.92256 STDEV = 0.00266 |  Mean = 0.91769 STDEV = 0.06550
9 Actions |  – |  Mean = 1.00109 STDEV = 0.07765 |  – |  Mean = 0.73018 STDEV = 0.33294
10 Actions |  – |  Mean = 0.91435 STDEV = 0.12382 |  – |  Mean = 0.92223 STDEV = 0.05944
11 Actions |  – |  Mean = 0.96161 STDEV = 0.03941 |  – |  Mean = 0.94661 STDEV = 0.02341
Table 1: Average and standard deviation of network throughput and fairness for
4 and 10 agents with $k=2$. The values are computed from the last 1,000 steps
out of a 10,000 step training run, and are averaged over 3 runs.

Results for throughput and fairness for 4 and 10 agents with $k=2$ are show in
Table 1. For 4 agents throughout is maximized with an action space of size
$|\mathbb{A}|=3$; fairness is above $0.99$ for 3, 4, and 5 actions. For 10
agents the highest throughput value is achieved with $|\mathbb{A}|=7$. It is
interesting to note that the second best throughput is achieved with
$|\mathbb{A}|=2$, but the corresponding fairness is very poor; fairness is
near or above $0.9$ for $|\mathbb{A}|\geq 6$, with the exception of 9 actions,
but this may have been due to unlucky sampling as the standard deviation is
much larger than that for any other action space size. To manage the trade-off
between throughput and fairness for 10 agents, a larger actions space shows
better results, as this gives good results for fairness and does not
(significantly) diminish throughput performance.

The size of the action space clearly has an effect on throughput and fairness.
This fact is both obvious and interesting. An action space of size 2 amounts
to a decision of whether to transmit. Larger action spaces conceptually change
the decision to include: how long to wait before transmitting. It may not have
been predictable a priori that extending the action space in this way would
have such an effect as, fundamentally, on each step an agent is either
transmitting or not, and each action $a_{i}$ for $i>0$ only schedules one
transmission (with $a_{0}$ being a single-step decision not to transmit).

Viewed another way, extending the action space is a way of biasing the set of
actions so that the agents have a lower probability of miscoordination. Ref.
11 defines coordination in the context of MARL as “the ability of two or more
agents to jointly reach a consensus over which actions to perform in an
environment”, and miscoordination penalty is incurred when the joint actions
lead to a lower reward. In our setting, we can define miscoordination to mean
when more than $k$ agents transmit on the same step, giving a reward of $-1$
for all agents that transmitted.

For example, if $k=2$ and actions were selected uniformly at random for 10
agents with $|\mathbb{A}|=2$ ($a=0$ meaning “don’t transmit” and $a=1$ meaning
”transmit”), then probability of miscoordination (of more than 2 agents
selecting $a=1$ on a given step) is, by the binomial distribution,

$P(\text{miscoordination})=0.5^{10}\sum_{i=k+1=3}^{10}\binom{10}{i}\approx
95\%.$ (3)

With the assumption of randomly selected actions, as the action space
increases the probability of miscoordination decreases. More details and a
proof of this can be found in Appendix C. However, as the size of the action
space, $|\mathbb{A}|$, keeps increasing, eventually a lower probability of
miscoordination would also result in diminishing throughput. Of course, our
agents do not select actions uniformly at random, but still, this theoretical
analysis implies that there is some optimal middle-ground for $|\mathbb{A}|$.

Additional tabulated results are displayed in Appendix D, which together with
Table 1 help inform us as we formulate a heuristic for finding an optimal
action space. We make note of some trends emerging from the empirical data.
For example, the fairness data for $k=1$ in Appendix D shows a general trend
that fairness increases and then may plateau as the number of actions
increase. For throughput, we generally observe that as we increase the number
of actions throughput increases until a point, and then may decrease. This
trend can be seen clearly for for 4 agents with $k=2$ agents in Table 1; and
for 2 agents with $k=1$, and 10 agents with $k=5$ both in Appendix D;
incidentally, it is when $|\mathbb{A}|=3$ that throughput is maximized in all
the aforementioned examples, but we conjecture that the reason for this is not
caused by an inherent property of the 3-action space. Rather, for each
example, $3=\frac{n}{k}+1$ where $n$ is the number of agents.

Based our empirical results we recommend the following heuristic for choosing
the size of the action space which we choose to promote good performance for
both throughput and fairness; we create $\mathbb{A}$ such that

$|\mathbb{A}|=\frac{n}{k}+1.$ (4)

In some cases, like for 10 agents in Table 1, the throughput is high for the
smallest action space, but when this occurs the fairness is unacceptably low.
The same behavior can be observed for 10 agents with $k=1$ in the appendix. In
these cases as well, using our heuristic we find a much more equitable trade-
off between throughput and fairness.

###  4.2 Benchmarking

Using our heuristic in Equation (4) to set the size of action space, we
compare the performance of our RL agents against three distributed CSMA-
inspired benchmarks.

CSMA (Carrier Sense Multiple Access) is a MAC protocol used in Wi-Fi systems;
the premise is that a device will sense the spectrum to see if the
transmission medium (channel) is clear. If the channel is clear, then the
device may transmit. If the transmission is unsuccessful, because, for
example, another device also transmitted at the same time and caused
interference, then the device randomly chooses a number of steps, $x$, in some
predefined interval, $\\{z\in\mathbb{Z}:0\leq z<X\\}$, to wait before
attempting to transmit again. We’ll refer to $x$ as a backoff timer and $X$ as
the backoff–upper-bound.

We consider these algorithms reasonable to compare against for benchmarking as
our action space may be viewed as doing something similar: giving the agents
the ability to choose how long to wait before attempting a transmission.

We call the three algorithms we use for benchmarking exponential CSMA,
p-persistent CSMA, and p-persistent. For exponential CSMA, we initially set
$X=2$; if the a transmission is unsuccessful, a new $x$ is randomly selected
and the new backoff–upper-bound is set to $2X$. When the agent has a
successful transmission, the backoff–upper-bound is reset to $X=2$. In
p-persistent CSMA, $X$ is a constant and it set to the size of the action
space $|\mathbb{A}|$ that the RL agent is using in the benchmarking
experiment. The p-persistent algorithm works the same as p-persistent CSMA,
with the only difference being that a spectrum reading that shows no
transmissions is not required as a precondition to transmit, as it is for the
other two algorithms.

Figure 2 shows plots for throughput and fairness for an experiment with 10
agents, $k=5$, and as in the previous section, the agents’ buffers are
incremented on every step. Using Equation (4) we set the size of the action
space, $|\mathbb{A}|=3$. We see that our RL agents achieve nearly optimal
network performance, as a fairness of 1 is optimal, and a network throughput
of $k$ is optimal. The network throughput for the benchmarking algorithms are
comparatively low. We see that 3-persistent CSMA produces the lowest
throughput, while the 3-persistent algorithm does better, which is likely due
to the fact that it does not require sensing that the spectrum is clear before
transmitting. Exponential CSMA does better still in terms of throughput,
although still far off of the performance of the RL agents, but its
corresponding network fairness is poor. As noted in Ref. 8, the traditional
distributed heuristic algorithms based on CSMA may suffer from unfairness,
which we observe here. Five of the Exponential CSMA agents capture the
channel, as they can all successfully transmit with $k=5$, and this causes the
other five agents’ backoff–upper-bound to continually and exponentially
increase. The network fairness for the RL agents and 3-persistent, are both
quite good with the best fairness being achieved by the RL agents. The high
value for network fairness demonstrates that despite each RL agent being
controlled by a distinct model and trying to optimize its own reward, our
distributed approach can still lead to cooperative and fair network behavior.

Additional throughput and fairness benchmarks for 2, 4, and 10 agents with
varying values for $k$ can be found in Appendix E. We see that in almost all
cases the RL agents outperform the CSMA-inspired benchmarks in terms of
throughput. The only time this is not the case is for 10 agents, $k=1$,
$|\mathbb{A}|=11$, where exponential CSMA has higher throughput but much worse
fairness. Overall, we take this as confirmation that our heuristic for
choosing the size of the action space works well to optimize throughput and
fairness separately and to provide a reasonable trade-off when necessary.

(a) Throughput for 10 agents, $k=5$, $|\mathbb{A}|=3$.

(b) Fairness for 10 agents, $k=5$, $|\mathbb{A}|=3$.

Figure 2: Network throughput and fairness with the performance for the RL
agents and the benchmarking algorithms plotted against each other. The plotted
data is the averaged over three runs.

###  4.3 Buffer Experiments

In the previous experiments the agents’ buffers were incremented on every
step, so each agent would always have packets to transmit. While this was
useful to determine how the agents would act cooperatively even when all
agents had packets to transmit, in many real-world communications systems the
information demands are not as stringent or as constant. In the following
sections we test the RL agents in settings with less stringent and
heterogeneous buffer rates and analyze the performance in terms of throughput,
fairness, and how well the agents are able to manage their buffers. In real-
world systems, if a buffer reaches its maximum size then information loss or
bottlenecks may occur. We compare against the p-persistent benchmarking
algorithm as in the previous section it showed higher throughput than
p-persistent CSMA and better fairness than exponential CSMA. We test with 4
agents with a maximum buffer size of 100 units, $k=1$, and set
$|\mathbb{A}|=5$ according to Eqn. (4).

In the first experiment we use a homogeneous buffer rate and increment each
agent’s buffer every 8 steps. Plots for network throughput and the buffer size
for one of the agents are displayed in Fig. 3. The network throughput in Fig.
3a is clearly, though only slightly higher for the RL agents than for the
5-persistent algorithm. Despite there not being a large gap in throughput, the
RL agents do far better at managing their buffer, as can be seen in Fig. 3b:
both buffers rise initially, but as the training progress the RL agent learns
to transmit effectively with the other agents so that the buffer stays low,
while the buffer gets maxed-out with the 5-persistent algorithm.

Next, we devise an experiment where agents have their buffers incremented
heterogeneously. We increment the buffers every $(2,5,8,10)$ steps
respectively for each agent. (E.g., agent 1’s buffer is incremented every 2
steps, agents 2’s buffer is incremented every 5 steps, and so on.) In Fig. 10
in Appendix F, the normalized buffer sizes for each of the agents are plotted.
For both the RL and 5-persistent approaches, Agent 1 is not able to prevent
its buffer from being maxed-out: the buffer rate is too demanding. For Agent
2, both approaches lead to the buffer being maxed-out initially, but about
halfway through the simulation, the RL agent learns to transmit effectively
enough so that its buffer stays under maximum buffer size. The buffer sizes
for Agent 3 are similar to those in the homogeneous buffer experiments, which
makes sense as the buffer rate (incremented every 8 steps) is the same: the RL
agent’s buffer rises initially, but it quickly learns to act such that the
buffer is kept low while for the 5-persistent algorithm the buffer quickly
maxes out. For agent 4, the RL agent handles this buffer rate with ease, and
while the 5-persistent approach manages to slow the increase of the buffer
size, the buffer size is still maxed-out at the end (or is at least very close
to being maxed-out).

These experiments show our RL approach and heuristic for selecting an action
space continues works well even when applied to settings with different, and
potentially heterogeneous, information demands.

Additional plots for throughput and fairness for both buffer experiments are
shown in Appendix F.

(a) Network throughput

(b) Normalized buffer size for Agent 1

Figure 3: Network throughput and normalized buffer size of Agent 1 for a
setting where the agents’ buffers are homogeneously incremented every 8 steps.
There are 4 agents with $k=1$ and $|\mathbb{A}|=5$. Network throughput is
averaged over 3 runs.

##  5 Conclusions and Future Directions

###  5.1 Summary

In this work we presented a distributed MARL approach for transmission control
(an MARL-based MAC multiple access method): we formulated the RL agents,
theoretically and empirically evaluated the action space to come up with a
heuristic for selecting the size of the action space, benchmarked against
CSMA-style algorithms, and tested performance with different buffer rates.
Overall, the RL agents outperform the benchmarks in terms of throughput,
fairness, and the ability to manage their buffer. We share the open source
code for the custom Gym environment, RL agents, and the algorithms used for
benchmarking through the link in Appendix A.

###  5.2 Discussion

The results of the action space analysis in Sec. 4.1 show that sometimes the
same reinforcement learning problem can be formulated with different action
spaces that are functionally equivalent (meaning that any trajectory of
actions that is possible using one action space is also possible in the
other), and yet some actions spaces will be better at maximizing an objective.
Equation (4) provides a heuristic for choosing the size of the action space
given the structure for the action space formulated in Sec. 3.4. Choosing the
size of the action space in this way can be viewed as tailoring the action
space to the problem within a general structure. Selecting a good action space
creates a bias towards selecting actions that are likely to create successful
patterns of transmission at a network-level. Our method of theoretically
analyzing the action space with the assumption of randomly selected actions
combined with empirical testing could be used in other cases where there is
freedom in choosing an action space.

The benchmarking in Sec. 4.2 showed that the RL agents outperform the CSMA-
style algorithms, and that this performance gap is consistent for varying
number of agents and different threshold values for $k$. The fact that the
benchmarking algorithms did not perform well indicates some of the
shortcomings of traditional distributed algorithms for transmission control,
namely, mediocre throughput performance, or good throughput performance but at
the expense of fairness. Meanwhile, the RL agents performed well not only in
terms of network throughput, but also in fairness demonstrates the power of a
distributed MARL approach, where even though the agents act individually to
maximize their reward, cooperative behavior can emerge that maximizes network
metrics as well if the MARL approach is constructed properly.

The advantage provided by the RL agents was demonstrated further in the buffer
experiments in Sec. 4.3. In these settings the RL agents were often able to
keep there buffer sizes low and prevent the their buffers from reaching the
maximum size. In real-world systems if an agents buffer were to be maxed-out
that could represent information loss or a major bottleneck in the network, so
the ability to keep buffer sizes low confers significant value. These
experiments also provide evidence that our action space with the heuristic
defined in Eqn. (4) to set the size of the action space works well in
different settings. The initial action space analysis in Sec. 4.1 was for a
setting where each agent’s buffer was incremented on every step. This
represents an impossibly high information demand if $k$ is smaller than the
number of agents, but in basing our action space analysis off of this setting
we find the best solution for the most demanding scenario, that is, when all
agents always have a constant stream of new packets to transmit. The tests in
Sec. 4.3 confirm that the RL approach combined and heuristic continues to
perform well even if the information demands are reduced or are heterogeneous.

###  5.3 Future Directions

In this work we dealt with the problem of transmission control/medium access
at a highly abstracted level. The potential advantage of this is that our
approach may be widely applicably to a variety of communications systems, or
potentially of independent interest to those interested in MARL. That being
said, if one wanted to deploy our approach on physical communications systems,
then one may want a more particularized environment to test the approach under
the confines of a specific real-world system. Fortunately, there are well-
developed network simulators available. Internally we have tested our approach
with ns3-gym [10] and observed similar performance.

While our approach seems to work well, there may also be room for improvement
in the formulation of the RL problem in terms of defining an optimal feature
space, action space, and reward function. The buffer experiments in Sec. 4.3
showed good results for the RL agents, however, the reward function used, Eqn.
(1), is quite simplistic and does not directly use any buffer terms. If one
wanted to incentivize certain buffer management (e.g., penalize when the
buffer is maxed-out or have the reward be proportional to buffer size), then
buffer terms could be added to the reward function accordingly.

We tested with 2, 4, and 10 agents, but it would be interesting to see how the
approach scales to larger number of agents and with different threshold values
for $k$. Additionally, there may be other algorithms to benchmark against that
could perform better than the CSMA-style algorithms we employed. Lastly,
further study could be directed towards the exploration of different RL
algorithms instead of DQN and/or different hyperparameters for the neural
networks.

To make it easier for those interested in further investigating or expanding
on this work, we provide the open source software for the Gym environment and
RL agents in Appendix A.

##  Appendix A Link to Software

The custom Gym environment as well as the RL model and benchmarking algorithms
can be found at https://github.com/Farquhar13/RL_Transmission_Control.

##  Appendix B DQN Hyperparameters

Each agents is equipped with a separate DQN model, but the hyperparameters are
the same for all the models. The DQN model uses a small feed-forward neural
network for Q-value estimation. We use an $\epsilon$-greedy exploration
strategy where a random action is taken with probability $\epsilon$, and on
each step $\epsilon$ is decreased by a factor of $\epsilon_{\text{decay}}$
until $\epsilon\leq\epsilon_{\text{min}}$. The agent trains on every step. The
training batch is constructed by randomly sampling past transitions.

DQN Hyperparameters
---
Hyperparameter |  Value
Input dimension |  $|s_{t}|=4$
Hidden layer 1 dimension |  128
Hidden layer 2 dimension |  256
Activation function to hidden layers |  ReLu
Activation function to output layer |  Linear
Starting Epsilon |  $\epsilon=1$
Epsilon decay |  $\epsilon_{\text{decay}}=0.996$
Minimum epsilon |  $\epsilon_{\text{min}}=0.05$
Gamma |  $\gamma=0.99$
Learning rate |  $\alpha=10^{-4}$
Batch size |  64
Loss function |  Mean Square Error
Optimizer |  Adam

##  Appendix C Miscoordination Penalty Probabilities

A miscoordination will occur when more than $k$ agents transmit at the same
time. The probability of miscoordination for 10 agents selecting actions
uniformly at random with, $k=2$, $|\mathbb{A}|=2$, is calculated in Eqn. (3)
using the probability mass function for the binomial distribution:

$P(X=x)=\binom{n}{x}p^{x}(1-p)^{n-x}.$ (5)

We can calculate the miscoordination penalty when $|\mathbb{A}|=2$ by letting
$n$ be number of agents and $p=0.5$ be the probability of choosing to
transmit. Then,

$P(\text{miscoordination})=\sum_{i=k+1}^{n}\binom{n}{i}p^{i}(1-p)^{n-i}=\sum_{i=k+1}^{n}\binom{n}{i}0.5^{n}.$
(6)

Calculating the probability of miscoordination for larger action spaces is
more complicated, as for $|\mathbb{A}|>2$ the action space will have some
multi-step actions, which means that agents will be taking actions
asynchronously.

We can calculate the probability of miscoordination in different way by taking
taking a certain view of the actions. If we map the actions such a $1$ denotes
a transmission and a $0$ denotes no transmission, then we can view the actions
as lists. The action space for $|\mathbb{A}_{m}|=m$ can be written:

$\displaystyle a_{0}=(0)$ $\displaystyle a_{1}=(1)$ $\displaystyle
a_{2}=(0,1)$ $\displaystyle a_{3}=(0,0,1)$ $\displaystyle\ldots$
$\displaystyle a_{m-1}=(\underbrace{0,0,\ldots,0}_{\text{m - 2 times}},1).$

So $\mathbb{A}_{m}=\\{a_{i}\\}_{i=0}^{m-1}$. If we are at least $m$ steps into
the simulation and assume, as before, that actions are taken uniformly at
random, then we can calculate the probability of transmitting on an arbitrary
step. Let $c_{0}$ and $c_{1}$ respectively be the count of the number of zeros
and ones in the lists in $\mathbb{A}$. Then the transmission probability,
$p_{m}$ for $|\mathbb{A}_{m}|=m$ is,

$\text{transmission probability for
}\mathbb{A}_{m}=\frac{c_{1}}{c_{0}}=p_{m},$ (7)

and we find for $m\geq 2$,
$c_{0}=1+\sum_{j=2}^{m-1}(j-1)=1+\sum_{j=1}^{m-2}j$, and $c_{1}=m-1$ since
there is one transmission for every $a_{i}$ except for $a_{0}$. The
transmission probability will decrease as the size of the action space $m$
gets larger. Formally, this can be seen by noting that $\sum_{j=1}^{m-2}j$ is
an arithmetic series and it can be shown that $c_{0}=1+\frac{m^{2}-3m+2}{2}$,
and thus scales quadratically with the size of the action space $m$, whereas
$c_{1}$ only scales linearly with $m$. Using the fact that actions are taken
independently, we now can calculate the probability of miscoordination on an
arbitrary step for $n$ agents with threshold $k$ and action size
$|\mathbb{A}|=m$ as,

$P(\text{miscoordination for }\mathbb{A}_{m})=\sum_{i=k+1}^{n}p_{m}^{i}.$ (8)

Assuming that actions are selected uniformly at random, we can now prove that
a larger action space leads to a lower miscoordination probability for $n$
agents with threshold $k$. Let $|\mathbb{A}_{x}|=x$, $|\mathbb{A}_{x}|=y$
where $x<y$. We have shown that a larger action space leads to a smaller
transmission probability, so $p_{x}>p_{y}$. Suppose $P(\text{miscoordination
for }\mathbb{A}_{x})>P(\text{miscoordination for }\mathbb{A}_{y})$. So,

$\displaystyle P(\text{miscoordination for }\mathbb{A}_{x})$
$\displaystyle>P(\text{miscoordination for }\mathbb{A}_{y})$ (9)
$\displaystyle\sum_{i=k+1}^{n}p_{x}^{i}$
$\displaystyle>\sum_{i=k+1}^{n}p_{y}^{i}$
$\displaystyle\sum_{i=k+1}^{n}p_{x}^{i}-\sum_{i=k+1}^{n}p_{y}^{i}$
$\displaystyle>0$ $\displaystyle\sum_{i=k+1}^{n}(p_{x}^{i}-p_{y}^{i})$
$\displaystyle>0.$

The last line is true since, for any $a,b,c\in\mathbb{R}^{+}$ with $a>b$, then
$a^{c}>b^{c}$, so $a^{c}-b^{c}>0$, thus every $(p_{x}^{i}-p_{y}^{i})$ is
positive and a sum of positive terms is greater than zero. Therefore,
$P(\text{miscoordination for }\mathbb{A}_{x})>P(\text{miscoordination for
}\mathbb{A}_{y})$. $\blacksquare$

Put simply, a larger action space will result in a lower transmission
probability, which implies a lower probability for miscoordination.

##  Appendix D Action Space Analysis Additional Data

Throughput, $k=1$
---
No. of Agents / No. of Actions |  2 Agents |  4 Agents |  10 Agents
2 Actions |  Mean = 0.78006 STDEV = 0.17884 |  Mean = 0.60261 STDEV = 0.03866 |  Mean = 0.57710 STDEV = 0.03215
3 Actions |  Mean = 0.95078 STDEV = 0.02402 |  Mean = 0.61586 STDEV = 0.02624 |  Mean = 0.46621 STDEV = 0.06823
4 Actions |  Mean = 0.64971 STDEV = 0.01450 |  Mean = 0.63784 STDEV = 0.03712 |  Mean = 0.48662 STDEV = 0.06416
5 Actions |  Mean = 0.73348 STDEV = 0.06607 |  Mean = 0.69083 STDEV = 0.17607 |  Mean = 0.44202 STDEV = 0.03817
6 Actions |  Mean = 0.81372 STDEV = 0.07375 |  Mean = 0.66441 STDEV = 0.01132 |  Mean = 0.40869 STDEV = 0.05880
7 Actions |  – |  Mean = 0.67896 STDEV = 0.12725 |  Mean = 0.43054 STDEV = 0.05287
8 Actions |  – |  Mean = 0.69733 STDEV = 0.07090 |  Mean = 0.40404 STDEV = 0.05060
9 Actions |  – |  – |  Mean = 0.37100 STDEV = 0.02976
10 Actions |  – |  – |  Mean = 0.41004 STDEV = 0.07357
11 Actions |  – |  – |  Mean = 0.43033 STDEV = 0.05623
Table 2: Average & standard deviation of throughput for threshold 1. Data is
from the last 1,000 steps of a 10,000 step training period and averaged over
three runs. Fairness, $k=1$
---
No. of Agents / No. of Actions |  2 Agents |  4 Agents |  10 Agents
2 Actions |  Mean = 0.87845 STDEV = 0.20949 |  Mean = 0.52672 STDEV = 0.03866 |  Mean = 0.29744 STDEV = 0.04632
3 Actions |  Mean = 0.99983 STDEV = 0.00010 |  Mean = 0.67614 STDEV = 0.13102 |  Mean = 0.42953 STDEV = 0.03979
4 Actions |  Mean = 0.99976 STDEV = 0.00001 |  Mean = 0.83247 STDEV = 0.00112 |  Mean = 0.49201 STDEV = 0.02939
5 Actions |  Mean = 0.99159 STDEV = 0.00820 |  Mean = 0.93710 STDEV = 0.06327 |  Mean = 0.55159 STDEV = 0.07052
6 Actions |  Mean = 0.99634 STDEV = 0.00291 |  Mean = 0.88923 STDEV = 0.06939 |  Mean = 0.63227 STDEV = 0.02151
7 Actions |  – |  Mean = 0.90435 STDEV = 0.10433 |  Mean = 0.65110 STDEV = 0.05479
8 Actions |  – |  Mean = 0.97265 STDEV = 0.00634 |  Mean = 0.67006 STDEV = 0.02951
9 Actions |  – |  – |  Mean = 0.67199 STDEV = 0.04562
10 Actions |  – |  – |  Mean = 0.69593 STDEV = 0.02341
11 Actions |  – |  – |  Mean = 0.67655 STDEV = 0.07446
Table 3: Average & standard deviation of fairness for threshold 1. Data is
from the last 1,000 steps of a 10,000 step training period and averaged over
three runs. 4 Agents, Throughput & Fairness, $k=3$
---
No. of Action |  Throughput |  Fairness
2 Actions |  Mean = 1.71945 STDEV = 0.15431 |  Mean = 0.89216 STDEV = 0.03130
3 Actions |  Mean = 1.99699 STDEV = 0.02159 |  Mean = 0.99979 STDEV = 0.00009
4 Actions |  Mean = 1.61192 STDEV = 0.04602 |  Mean = 0.99045 STDEV = 0.00238
5 Actions |  Mean = 1.55774 STDEV = 0.06220 |  Mean = 0.98197 STDEV = 0.00516
6 Actions |  Mean = 1.57845 STDEV = 0.08176 |  Mean = 0.94473 STDEV = 0.02596
7 Actions |  Mean = 1.45513 STDEV = 0.08087 |  Mean = 0.94238 STDEV = 0.00399
8 Actions |  Mean = 1.56845 STDEV = 0.08251 |  Mean = 0.90685 STDEV = 0.02018
Table 4: Average & standard deviation of throughput & fairness for 4 agents
threshold 3. Data is from the last 1,000 steps of a 10,000 step training
period and averaged over three runs. 10 Agents, Throughput & Fairness, $k=5$
---
No. of Action |  Throughput |  Fairness
2 Actions |  Mean = 3.19046 STDEV = 0.58891 |  Mean = 0.85694 STDEV = 0.14962
3 Actions |  Mean = 3.91997 STDEV = 0.03990 |  Mean = 0.98970 STDEV = 0.00431
4 Actions |  Mean = 3.31367 STDEV = 0.00995 |  Mean = 0.99861 STDEV = 0.00126
5 Actions |  Mean = 2.59365 STDEV = 0.17936 |  Mean = 0.98708 STDEV = 0.00375
6 Actions |  Mean = 2.43987 STDEV = 0.08961 |  Mean = 0.96973 STDEV = 0.01043
7 Actions |  Mean = 2.41671 STDEV = 0.15506 |  Mean = 0.96493 STDEV = 0.00990
8 Actions |  Mean = 2.44283 STDEV = 0.16497 |  Mean = 0.92002 STDEV = 0.00580
9 Actions |  Mean = 1.85070 STDEV = 0.05589 |  Mean = 0.94923 STDEV = 0.01256
10 Actions |  Mean = 1.85649 STDEV = 0.07320 |  Mean = 0.90869 STDEV = 0.02141
11 Actions |  Mean = 1.70585 STDEV = 0.08802 |  Mean = 0.92496 STDEV = 0.01213
Table 5: Average & standard deviation of throughput & fairness for 10 agents
threshold 5. Data is from the last 1,000 steps of a 10,000 step training
period and averaged over three runs.

##  Appendix E Benchmarking Additional Data

Figure 4: Network throughput and fairness for 2 agents, $k=1$,
$|\mathbb{A}|=3$. (Data averaged over three runs.)

Figure 5: Network throughput and fairness for 4 agents, $k=1$,
$|\mathbb{A}|=5$. (Data averaged over three runs.)

Figure 6: Network throughput and fairness for 10 agents, $k=1$,
$|\mathbb{A}|=11$. (Data averaged over three runs.)

Figure 7: Network throughput and fairness for 4 agents, $k=2$,
$|\mathbb{A}|=3$. (Data averaged over three runs.)

Figure 8: Network throughput and fairness for 10 agents, $k=2$,
$|\mathbb{A}|=6$. (Data averaged over three runs.)

##  Appendix F Buffer Experiments Additional Data

Figure 9: Network fairness for 4 agents, $k=1$, $|\mathbb{A}|=5$, where each
agent’s buffer is incremented every 8 steps. (Fairness averaged over three
runs.)

Figure 10: Buffer sizes for 4 agents, k = 1, $|A|=5$, where buffers are
incremented heterogeneously every (2, 5, 8, 10) steps respectively for each
agent. (Buffer sizes averaged over three runs.)

Figure 11: Network throughput and fairness for 4 agents, $k=1$,
$|\mathbb{A}|=5$, where buffers are incremented heterogeneously every (2, 5,
8, 10) steps respectively for each agent. (Throughput and fairness averaged
over three runs.)

## References

  * [1] Jagannath, J., Polosky, N., Jagannath, A., Restuccia, F., and Melodia, T., “Machine learning for wireless communications in the internet of things: A comprehensive survey,” Ad Hoc Networks 93, 101913 (oct 2019).
  * [2] Zhang, K., Yang, Z., Liu, H., Zhang, T., and Başar, T., “Fully decentralized multi-agent reinforcement learning with networked agents,” (2018).
  * [3] Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., and Zaremba, W., “Openai gym,” (2016).
  * [4] Jiang, J. and Lu, Z., “Learning fairness in multi-agent systems,” (2019).
  * [5] Foerster, J. N., Assael, Y. M., de Freitas, N., and Whiteson, S., “Learning to communicate with deep multi-agent reinforcement learning,” (2016).
  * [6] Kim, D., Moon, S., Hostallero, D., Kang, W. J., Lee, T., Son, K., and Yi, Y., “Learning to schedule communication in multi-agent reinforcement learning,” (2019).
  * [7] Yu, Y., Wang, T., and Liew, S. C., “Deep-reinforcement learning multiple access for heterogeneous wireless networks,” in [2018 IEEE International Conference on Communications (ICC) ], 1–7 (2018).
  * [8] Raeis, M. and Golestani, S. J., “Distributed fair scheduling for information exchange in multi-agent systems,” CoRR abs/2102.08814 (2021).
  * [9] Sutton, R. S. and Barto, A. G., [Reinforcement learning: An introduction ], MIT press (2018).
  * [10] Gawłowicz, P. and Zubow, A., “ns-3 meets OpenAI Gym: The Playground for Machine Learning in Networking Research,” in [ACM Intl. Conf. on Modeling, Analysis and Sim. of Wireless and Mobile Systems ], (2019).
  * [11] Kapetanakis, S. and Kudenko, D., “Reinforcement learning of coordination in heterogeneous cooperative multi-agent systems,” in [Adaptive Agents and Multi-Agent Systems II ], 119–131, Springer (2004).